Repository navigation
Read a team's running task logs through the team's executor - #75255
Conversation
|
Tagging @o-nikolas @vincbeck as this is a general multi-team fix |
In multi-team mode, the API server reads a running task's logs through the global default executor, or looks up the task's named executor in the global scope only. A team's executor has its own config, such as the cluster its KubernetesExecutor pods run on, so a team task's live logs came from the wrong executor: nothing from a global LocalExecutor, or "Cannot find pod" from a global KubernetesExecutor on another cluster. Resolve the executor the same way the scheduler does: the team's executor, and a global one if the team doesn't have it.
d8951f4 to
7594e76
Compare
Move the log reader's lookup to ExecutorLoader.resolve_executor_name, so the scheduler can call the same function instead of its own loop. A match from the task's team, by alias, module path or class name, wins over a global one, the first match wins among equals, and another team's executor never matches.
50ab02c to
3f1b418
Compare
|
Hi maintainer, this PR was merged without a milestone set.
|
1 similar comment
|
Hi maintainer, this PR was merged without a milestone set.
|
Backport successfully created: v3-4-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
…or (#75255) In multi-team mode, the API server reads a running task's logs through the global default executor, or looks up the task's named executor in the global scope only. A team's executor has its own config, such as the cluster its KubernetesExecutor pods run on, so a team task's live logs came from the wrong executor: nothing from a global LocalExecutor, or "Cannot find pod" from a global KubernetesExecutor on another cluster. Resolve the executor the same way the scheduler does: the team's executor, and a global one if the team doesn't have it. (cherry picked from commit 1e3a232) Co-authored-by: Sean Ghaeli <58916776+seanghaeli@users.noreply.github.com>
Backport failed to create: v3-4-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker 1e3a232 v3-4-testThis should apply the commit to the v3-4-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
…or (apache#75255) In multi-team mode, the API server reads a running task's logs through the global default executor, or looks up the task's named executor in the global scope only. A team's executor has its own config, such as the cluster its KubernetesExecutor pods run on, so a team task's live logs came from the wrong executor: nothing from a global LocalExecutor, or "Cannot find pod" from a global KubernetesExecutor on another cluster. Resolve the executor the same way the scheduler does: the team's executor, and a global one if the team doesn't have it. (cherry picked from commit 1e3a232) Co-authored-by: Sean Ghaeli <58916776+seanghaeli@users.noreply.github.com>
…or (#75255) (#75408) In multi-team mode, the API server reads a running task's logs through the global default executor, or looks up the task's named executor in the global scope only. A team's executor has its own config, such as the cluster its KubernetesExecutor pods run on, so a team task's live logs came from the wrong executor: nothing from a global LocalExecutor, or "Cannot find pod" from a global KubernetesExecutor on another cluster. Resolve the executor the same way the scheduler does: the team's executor, and a global one if the team doesn't have it. (cherry picked from commit 1e3a232) Co-authored-by: Sean Ghaeli <58916776+seanghaeli@users.noreply.github.com>
In multi-team mode, the API server reads a running task's live log through the global executor, never the task's team executor. Only the Kubernetes executors (and so
AwsEksExecutor) fetch live logs through the executor, from the pod, so a team running one with its own namespace or cluster gets "Cannot find pod", or nothing at all when the global executor is aLocalExecutor. This resolves the executor the way the scheduler does: the team's own executor, else a global one, cached per team. Finished tasks and other executors aren't affected. Found while testing per-team clusters (#75251) on two real EKS clusters, where each team's live log now comes from its own cluster.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines