[rjgoyln]

Summary

Airflow derives Dag run and task run span ids from the metadata database through a custom OpenTelemetry IdGenerator, so spans from different components join into one trace. That generator was only ever installed on a TracerProvider Airflow built itself — under opentelemetry-instrument or a vendor distro the provider already exists, OpenTelemetry refuses to replace it, and the linkage is lost without a word.

Change

• Register OverrideableRandomIdGenerator under the opentelemetry_id_generator entry point group, selectable as OTEL_PYTHON_ID_GENERATOR=airflow.
• Patch the generator onto a provider Airflow did not build, and warn, instead of discarding it.
• Document the setup, the fallback's limits, and that [traces] exporter settings do not reach a borrowed provider.

The fallback is best-effort by construction — a Tracer keeps the generator it was built with, so only later tracers are linked — which is why the environment variable is the documented path.

The entry point is declared in airflow-core because shared/observability is vendored into consumers rather than published, so it has no installed metadata for OpenTelemetry to discover. Workers ride propagated traceparent values and need no override.

Scope

Issue 1 of #70388 only; issues 2 and 3 remain open.

• related: Support OpenTelemetry Auto-Instrumentation, Distros & Configurators #70388

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[tailg8nj]

This is a quick turn around! Overall this now allows a distro to set this environment variable (removing the need for custom code in local settings) if it detects it is operating on an airflow env. 👍

[potiuk]

Nice piece of work. Registering the generator as a standard entry point is the right shape for this, and the caveat you documented is accurate — I checked the SDK rather than taking it on trust: TracerProvider.get_tracer() passes self.id_generator into Tracer.__init__ (sdk/trace/__init__.py:1454 → :1119-1130) and the tracer then uses its own captured reference, so patching after the fact really does affect only tracers built later. Precisely what traces.rst says.

Two design decisions I'd like written down before this lands. Neither is a defect.

The adoption runs before otel_on is read

def configure_otel(conf: ConfigParser):
    if _adopt_existing_tracer_provider():
        return
    otel_on = conf.getboolean("traces", "otel_on", fallback=False)

So a deployment running under opentelemetry-instrument with otel_on=False — tracing deliberately switched off in Airflow — gets its TracerProvider.id_generator swapped anyway, plus a warning on every component start advising it to set OTEL_PYTHON_ID_GENERATOR=airflow. That's guidance about a feature the operator has turned off, repeated at every startup. Before this change, configure_otel with otel_on=False did nothing at all.

Your test parametrizes otel_on=[True, False] and asserts the patch in both, so this looks deliberate — could you say why in a comment? My instinct is that the adoption should sit behind the otel_on check.

The early return silently retires the [traces] exporter settings

When a provider already exists and otel_on=True, the return skips all exporter and span-processor setup. I think that's correct — with a vendor distro you don't want Airflow bolting a second OTLP exporter onto a provider the distro already configured — but nothing tells the operator. The warning mentions only span ids, and the new docs section covers only the generator caveat.

Someone who has otel_host / otel_port configured and then adopts auto-instrumentation gets no signal that those settings stopped applying. One sentence in the warning and one in the docs would cover it. If you considered provider.add_span_processor(...) on the borrowed provider and rejected it, that reasoning is worth a line in the docs too.

Smaller things

The entry point hard-codes airflow._shared.observability.traces, but that source is importable by three paths — airflow._shared…, airflow.sdk._shared… (task-sdk symlinks the same directory) and airflow_shared… — each a distinct module object with a distinct class object. So the isinstance check in _adopt_existing_tracer_provider only matches when both sides arrived via the same path. It holds today, since settings.py:42 is the sole caller and uses airflow._shared…. But if task-sdk ever calls configure_otel it would silently fail the check and double-patch. A duck-typed check, or a comment recording the coupling, would make that safe.

The description says the newsfragment is 70388.improvement.rst; commit 2062b425 renamed it to 70758.improvement.rst but the body still names the old one.

Finally, the Gen-AI disclosure box is unchecked here while your other two open PRs declare Claude Code — just confirming that's deliberate.

Worth a second look from

This touches the span-id propagation design, so I'd like input from the people who built it:

• @xBis7 — authored the trace-linkage work in shared/observability/.../traces/ and part of traces.rst; best placed to say whether adopting a foreign TracerProvider preserves the Dag-run/task-run linkage in practice.
• @ferruzzi — deepest recent history on Airflow's OTel metrics and telemetry config; the otel_on question above is really about how Airflow's telemetry switches should behave when someone else owns the provider.

No rush on either — this has been open since July and the branch is 1271 commits behind main, so it needs a rebase before it can merge regardless.

---

This review was drafted by an AI-assisted tool and confirmed by an Airflow maintainer. The findings below are observations, not blockers; an Airflow maintainer — a real person — will take the next look at the PR. If you think a finding is mis-applied, please reply on the PR and a maintainer will weigh in.

More on how Airflow handles maintainer review: contributing-docs/05_pull_requests.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[rjgoyln]

Thanks for the review. Rebased onto main and pushed as f55fedb.

On otel_on

This is deliberate. otel_on only controls whether Airflow creates its own provider; spans can still reach a distro-owned provider when it is false. Gating this on otel_on would leave those spans with random IDs. Added a comment and parametrised test to cover this.

On exporter settings

Agreed — documented in the warning, traces.rst, and newsfragment. The settings were already ineffective with a foreign provider, so the early return avoids constructing a provider that cannot be installed.

On import paths

Confirmed. The task SDK already imports this module, so the check stays at the configure_otel entry point, with the coupling documented there.

Happy to switch to duck typing if preferred.

[xBis7]

There is already the option to integrate a Dag run span into an external trace. See

#69633

I looked at the linked issue. Can you explain the motivation behind using an external tracer_provider? Maybe an example use-case?

I get it for Dag runs but what about spans for Airflow internal operations? We don't have any right now because they have been cleaned up but if we add new ones in the future, they will all use the external provider.

[potiuk]

Hello @rjgoyln - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 24 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• Support OpenTelemetry auto-instrumentation and vendor distros #70758 - Support OpenTelemetry auto-instrumentation and vendor distros
• Honor OTEL_PROPAGATORS when propagating trace context across Airflow #70840 - Honor OTEL_PROPAGATORS when propagating trace context across Airflow
• Fix API client errors losing detail behind a reverse proxy #70936 - Fix API client errors losing detail behind a reverse proxy
• Fix airflowctl config lint filters matching partial config names #70940 - Fix airflowctl config lint filters matching partial config names
• Stop failing reschedule-mode sensors on a stale executor success #71194 - Stop failing reschedule-mode sensors on a stale executor success
• Drop the cncf.kubernetes provider dependency from airflow-core tests #71868 - Drop the cncf.kubernetes provider dependency from airflow-core tests
• Drop the git provider dependency from airflow-core tests #72041 - Drop the git provider dependency from airflow-core tests
• Fail deferred sensors on non-timeout errors when soft_fail is set #72053 - Fail deferred sensors on non-timeout errors when soft_fail is set
• Drop the celery provider dependency from airflow-core tests #72249 - Drop the celery provider dependency from airflow-core tests
• Honor config_file in deferrable Kubernetes tasks #72301 - Honor config_file in deferrable Kubernetes tasks
• Fix Dataproc job not cancelled when killing a deferred task #72591 - Fix Dataproc job not cancelled when killing a deferred task
• Stop the scheduler deadlocking with the triggerer on deferred-task timeouts #72595 - Stop the scheduler deadlocking with the triggerer on deferred-task timeouts
• Make the scheduler's HITL and orphan sweeps retry transient DB errors #72596 - Make the scheduler's HITL and orphan sweeps retry transient DB errors
• Keep the triggerer running when a trigger constructor raises #72758 - Keep the triggerer running when a trigger constructor raises
• Stop failing scheduled tasks when a serialized Dag is briefly missing #72814 - Stop failing scheduled tasks when a serialized Dag is briefly missing
• Stop fetching the run conf once per task instance in the scheduler #73022 - Stop fetching the run conf once per task instance in the scheduler
• Catch format-string arguments passed to exception constructors #73076 - Catch format-string arguments passed to exception constructors
• Fail deferrable KPO when a GC'd pod takes its XCom with it #73131 - Fail deferrable KPO when a GC'd pod takes its XCom with it
• Block new EM102 f-string exception messages in prek #73166 - Block new EM102 f-string exception messages in prek
• Fail the local yml mounts prek hook when synchronization fails #73170 - Fail the local yml mounts prek hook when synchronization fails
• Fix provider.yaml doc URL check with several suspended providers #73204 - Fix provider.yaml doc URL check with several suspended providers

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Batch upstream map-length lookups when resolving mapped kwargs #70620 - Batch upstream map-length lookups when resolving mapped kwargs
• Re-anchor custom queued deadline references when a Dag run is cleared #71850 - Re-anchor custom queued deadline references when a Dag run is cleared
• Make the default task failure email say what failed #73221 - Make the default task failure email say what failed

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

[tailg8nj]

I looked at the linked issue. Can you explain the motivation behind using an external tracer_provider? Maybe an example use-case?

I believe this to be fairly well documented in the linked issue. Open telemetry offers a way for vendors to ship distributions that preconfigure their preferred settings of the SDK and other auto-instrumentation features. These are not limited to: collector endpoint, format and compression, additional span processors, propagation strategies (like setting trace state), framework and library instrumentation preferences (like urllib, requests, httpx, celery, etc). It is a well supported valid way of setting up your Python asset w/o code.

Whether or not internal spans eventually get added can be address by the vendors themselves.

How else do you suggest we address issues 1 and 2 from the linked issue?

[xBis7]

@tailg8nj I just remembered this older comment of yours

#69633 (comment)

Same question as here.

I believe this to be fairly well documented in the linked issue.

The issue documents what distros and auto-instrumentation are, how they work and what is the standard process of using them.

I've been asking you what Airflow gains by supporting this, and not how OTel intends them to be used.

If Airflow documents that it supports an external provider and also supports Dag run traces, people are going to use both of them and if there are bugs they will then expect the Airflow community to fix them. So accepting an external provider means to also make sure that the existing functionality doesn't break.

How else do you suggest we address issues 1 and 2 from the linked issue?

I'm not arguing against the changes, we can discuss them later. I'm asking what are the gains and if they are worth the long-term commitment.