Skip to content

Helm: Remove metadata Secret references from Kubernetes tasks - #74912

Open
anzinmhd wants to merge 9 commits into
apache:mainfrom
anzinmhd:fix/74850-worker-metadata-secret
Open

anzinmhd wants to merge 9 commits into
apache:mainfrom
anzinmhd:fix/74850-worker-metadata-secret

Conversation

@anzinmhd

@anzinmhd anzinmhd commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Remove the chart-generated AIRFLOW__DATABASE__SQL_ALCHEMY_CONN and AIRFLOW_CONN_AIRFLOW_DB variables from KubernetesExecutor task containers so tasks can start without a metadata database Secret in their namespace.

The shared environment helper accepts an internal ExcludeDatabase flag. The task base container always enables it. The worker-kerberos native sidecar enables it when airflowVersion >= 3.4.0, relying on the Kerberos CLI startup fix in #75000. Older Airflow versions retain the database references because their CLI still performs mandatory database checks. Other helper callers retain their existing behavior.

Closes #74850.

Reproduction

  1. Deploy Airflow 3.2.2 with KubernetesExecutor and multiNamespaceMode: true, with Kerberos disabled.
  2. Run a task in a separate namespace using pod_override.metadata.namespace, omitting the metadata database Secret from that namespace.
  3. The task container fails to start with CreateContainerConfigError reporting the missing Secret.
  4. Create the Secret with fake database credentials and trigger a fresh Dag run; the task succeeds.

After the fix, a fresh task can start without the metadata Secret in its namespace.

Verification

  • Airflow 3.2.2 reproduction: confirmed that the task pod fails when the metadata Secret is absent and a fresh task run succeeds when a placeholder Secret is present.
  • Regression coverage: added rendering checks that verify the task base container omits both database variables and that the worker-kerberos sidecar retains them on Airflow 3.2.2 and 3.3.2, but omits them on 3.4.0.
  • Earlier test runs: 125 pod-template tests and 39 common chart tests passed before the latest main template update. The revised targeted Helm test has not yet been rerun locally; Breeze could not resolve flit-core because PyPI was unreachable.
  • Formatting and linting: Ruff formatting and linting passed for the updated test file.

A previous Kubernetes smoke test used a 3.4.0 development image with a stub kinit and confirmed that the Kerberos CLI reached ticket initialization without database environment variables. It did not test real KDC authentication or an Airflow task using Kerberos.

Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-6)

@aaron-y-chen aaron-y-chen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks 😄

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:helm-chart Airflow Helm Chart

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metadata secret required for workers

3 participants