Skip to content

Shared-stream triggers: add producer-side ack channel for non-idempotent upstreams #67179

Description

@Lee-W

Body

Context

PR #66584 added SharedStreamManager and the opt-in BaseEventTrigger.{shared_stream_key,open_shared_stream,filter_shared_stream} hooks. Scope is idempotent / read-only / subscriber-side-effect upstreams (directory scans, polling APIs, Kafka with enable.auto.commit=true).

filter_shared_stream is one-way producer → subscriber.

What's missing

Upstreams whose consumption needs a producer-held handle to advance can't use shared-stream today:

  • Kafka manual commit, SQS delete-on-process / visibility-extend, Pub/Sub ack_id, Service Bus peek-lock.

Each needs the subscriber's accept/reject decision to flow back to the producer's handle.

Design candidates (from Jason)

  1. (event, ack) tuple — filter calls ack closure on accept. Simple, but silently lossy: first-ack moves the offset past events siblings never
    saw.
  2. Producer waits for N acks before advancing — closes the at-least-once gap, but slow filters block the group; interacts with the shared_stream_subscriber_queue_size bound.
  3. Watermark / commit-up-to-offset — matches real Kafka consumer-group semantics; meaningfully larger API change.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Activity

  1. added
    kind:metaHigh-level information important to the community
    on May 19, 2026
  2. self-assigned this
    on May 19, 2026
  3. added theissue type on May 19, 2026
  4. added 10 commits that reference this issue on May 26, 2026
    f1e1ea0
    939c9da
    ea2c1d9
    c3b856b
    c66055e
    5b341ff
    8d174b9
    691be0a
    c652c5c
    4c32f7a
  5. 6 remaining items

  6. added 14 commits that reference this issue on Jun 10, 2026
    59b4df7
    8ee0022
    bac24f9
    4673baa
    3169469
    c6209dd
    36c48df
    f4769a2
    7c06a8f
    29a3da2
    3a991fd
    e2d7076
    f7ecace
    9830478
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:corekind:featureFeature Requestskind:metaHigh-level information important to the community

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions