Status of testing of Apache Airflow 3.4.0rc3

@ashb

We are kindly requesting that contributors to Apache Airflow RC 3.4.0rc3 help test the RC.

Please let us know by checking the box / commenting if the change is addressed in the latest RC.

Significant Changes

Full details (migration notes, changed endpoints, behavioural changes) for each item are in the 3.4.0 release notes. The entries below are for test-tracking only.

New Features

Bug Fixes

Miscellaneous

Thanks to all who contributed to the release — the full contributor list is in the first comment below. 🙏

@14041999ai

Thanks to all who contributed to the release:

@14041999ai @1fanwang @202201142 @23tae @aaiss0927 @Aakcht @aaron-y-chen @aaronsteers @Aaryan123456679 @Abdel39 @abhijeets25012-tech @abhinav-phi @abhisek343 @abhishekmauryaKsolves @ABiscuitttt @Adityakk9031 @aeroyorch @ahilashsasidharan @alant94 @AlejandroMorgante @AlexanderBLR @allenc97 @Alwaysgaurav1 @Amitkumar293 @AmoghAtreya @amoghrajesh @Andrej730 @Andrushika @anisengupta @anishgirianish @anmolxlight @AntonioBergonzi @antruigon @Anuragp22 @anxkhn @anzinmhd @aoelvp94 @arijitroy003 @arkadiuszbach @arnavsingh0418 @arnoldmr01 @Arose26 @arunppsg @ashb @AshleyAHuang @ashokraminedi @ashutosh264 @atakanyenel @atamagrawal @atharvaajmera @auroraxo @AutomationDev85 @auyua9 @Avcu @Bafff @baha-bouali @baraka-akeyless @basantjamwal @BasPH @bbovenzi @Beat-Nick @bentorb @bhavya2109sharma @bingqin2 @bmanan7 @bnspopi @br413 @bramhanandlingala @bugraoz93 @bujjibabukatta @ccoliu @CharoenwitKunna @ChickenBenny @choo121600 @chrispatterson @cmettler @codewithdaniel1 @codewithsruthi @Codingaditya17 @coleheflin @ColtenOuO @cool9850311 @Crowiant @cruseakshay @CyberNativeAI @dabla @daesung9511 @daniyarndh @darshil929 @davidnzhang @deepujain @Dev-iL @dev-xong @developer-rpai @dgvj-work @DhanushAnegondi @DHBhensdadia @dheerajturaga @dheerenmohta @dhkim1920 @dilnazanlid @dingo4dev @divyanshus2404 @DjVinnii @dkranchii @dlactin @DonHaul @dosedaf @Dotify71 @drewrukin @dstandish @dsuhinin @durgaprasadml @dybyte @Eason09053360 @ecodina @ed-kyu @Ed3110 @eddiesr93 @edsu @ehab20011 @Ei-Sandi @eitan-shalev @eladkal @ellisms @EncrypterParv @ephraimbuddy @fallintoplace @fat-catTW @feberbo @ferruzzi @Feudjo @filipeaaoliveira @firasbouzazi @fpiped @FrankYang0529 @gang-zh @GayathriSrividya @glostis @gocoolp @goingforstudying-ctrl @gopidesupavan @gps23 @gtxu @guan404ming @hanu-14 @hanxdatadog @Har1sh-k @harishkesavarao @harjothkhara @harshhh817 @haseebmalik18 @HasnainRaz @henry3260 @HeroCC @himanshuchawla0807-gif @hkc-8010 @HsuehYi @hunnguye @hussein-awala @huyua9 @hypnguyen1209 @IcebreakerSA @ignacioparicio @ihorlukianov @imrichardwu @itayweb @itej13 @ivonaest @jacobcbeaudin @jakezwang @jakubmatyszewski @jason810496 @jasperjonkhans @jayachandrakasarla @jayamanikharyono @jedcunningham @jeff3071 @JelyFishhhhhh @JH-A-Kim @Jie211 @jingi723 @jlaneve @jonathanleek @Jonghyun-An @JoonKyoLee @joseph-bergin @josix @jroachgolf84 @jscheffl @juan-pablo-guereca @Junwon @justinpakzad @jx2lee @k-popov @Kaap10 @kacpermuda @kada2004 @kadubhumika @kakatur @kalluripradeep @karenhsieh75 @karthedew @KarthikMohankumar @kartik00052 @kaxil @keemgdeok @keith991001 @KenanAdel @KenanAdel-M @kevinhongzl @KidAmnesiac1 @KingLizard1020 @koleharv @korex-f @krystianpl26 @KSchmidAmilar @Kunal8954 @KushagraB424 @kutsibalci @LE0-Lin @Lee-W @leeyspaul @li-xiang-db @liwenjie200543 @lohitkolluri @Lothnic @lubimow-xwf @luc-pimentel @LuchoFormisano @LuisFelipeUrena @lyric2249 @MahathirMohammadShuvo @MaksYermak @MalteNiederstadt @manish1337 @MannXo @MarthalaJagruthiReddy @martin-ubm @matthieuauger @mattwang44 @maupatel @MeghanaGangarapu @MhussainD4772 @michaelneely @MichalJaroslawKrzywanski-TomTom @mihirduvedi @minyeamer @Miretpl @mitre88 @mobuchowski @MOJAN3543 @Mola-maker @molcay @moomindani @MukundKulkarni @myps6415 @nagasrisai @namanjain24-sudo @nandeshkanagaraju @nanheekim-ovo @Nataneljpwd @neel-astro @nickstenning @NihalRadhakrishna @nikoki @nikolauspschuetz @noah-gil @noamst-monday @noron12234 @nyang404 @o-nikolas @olayiwola-gbenga @olegkachur-e @olmaks @onlyarnav @osintalex @ozemin @Pandry @pankajastro @pankajkoti @parkhojeong @Pebble32 @Pedrinhonitz @peloyeje @phanikumv @Phinhas214 @pickfire @pierrejeambrun @PolitePp @potiuk @Prab-27 @Prajwal7842 @pranampagi @pranayyelugam @PreethamSanji @Priyaj11 @purva-code24 @Pushkal-Gupta @puzzle-rpa-team @r3wretrhy @RabahAmrouche05 @radhwene @ramitkataria @rammmaa @raphaelauv @RaphCodec @rapsealk @rawsun007 @renat-sagut @rino0601 @rjgoyln @RNHTTR @robertpofuk @rohan9446 @Rohitkanithi @roshanprabu @roykoand @RoyLee1224 @ryanahamilton @RyukR1 @saadbelgi @safaehar @saitejabandaru @SakshamKapoor2911 @SameerMesiah97 @Samin061 @samraj2k @SamWheating @schizophrenicmaniac @Sdnsoumy @sduvvuri26 @seanghaeli @seanmuth @sehajsandhu123 @sejal-gupta-ksolves @SEPURI-SAI-KRISHNA @sgoel2be24-cyber @shaealh @shahar1 @shashank-iitbhu @shivaam @ShivanshhhG @shoemoney @ShousenZHANG @shreyamalviya @ShubhamKapoor992 @shubhamraj-git @shubhgurav0590 @SIBTAIN-ASAD @simar-rekhi @simon-hoon @singhAbhishek16 @singlaamitesh @sivavarma-x @sjyangkevin @sln-jack @sohamwaghe @sokojh @somaz94 @sortega @souradeep07 @spider-man-tm @sreecharan-desu @SreeramaYeshwanthGowd @stegololz @stephen-bracken @stephenyang0215 @stephsi @steveahnahn @Subham-KRLX @subhramit @SulimanAbdulrazzaq @szzhoujiarui @takayoshi-makabe @te-horie @Tegh25 @thc1006 @theJohnnyBrown @TheoLauw @TimurRakhmatullin86 @tirkarthi @TJaniF @tomron @topherinternational @ucaeon @udsy19 @uplsh580 @uranusjr @vaishsriram @Vamsi-klu @vatsrahul1001 @Vedant-Agarwal @vgkowski @victorymakes @viiccwen @vikramkoka @vincbeck @VinolFernandes @Vishwaspatel2401 @vitorantoniazzi @vividbaek @VladaZakharova @vsima @w-shahid @warreee @weiyu1029 @wiasliaw @Will-thom @wolvery @WonYong-Jang @X33834 @xBis7 @xlsongc @yamakazushi @YAshhh29 @YashrajG @ydsakshi @ykuc @ylv01 @yuseok89 @yuyua9 @zach-overflow @zachliu @Zeeshan-Chaudhry @zhangxinyao88 @ZhaoMJ @zohaibfast99 @zozo123

No issues found from my testing of #73116 and #74244 against Airflow core (3.4.0rc1) / task-sdk (1.4.0rc1) with Python 3.11 in Breeze, both work as expected.

Artifacts: Verified that both the core and Task SDK wheels have no remaining httpx imports and use httpx2 in every migrated module. Task SDK metadata requires httpx2 + truststore and no longer certifi (expected, as per the httpx2 docs here).
Task SDK client works for both HTTP and HTTPS: ran a test dag using Variable/Connection/XCom API calls and a deferrable task succeeded E2E for both protocols (the latter was run with the API server on a self-signed [api] ssl_cert).
airflow-core in-process transports: tested the following without error:
1. task execution in airflow dags test
2. dag-processor parse-time Variable.get API calls
3. a trigger reading a API-based Variable GET.
4. local CLI commands along with the JWKS fetch (local HTTP and public HTTPS).
truststore TLS matrix (private CA, Linux): [api] ssl_ca_file is trusted. An unknown CA is rejected. A CA added to the system store is trusted via truststore, both by the SDK client and by bare httpx2. client_use_public_certs=False with ssl_ca_file still works.
httpx2 request lines do not leak into task logs.

Did not test: macOS/Windows truststore behaviour described in the significant newsfragment, since this was Linux only.

I've tested my changes in #58543 and #74377, and they're working well :)

I found a scheduler crash on Postgres that looks related to #70972.

Since #70972, asset_dag_run_queue keeps one row for each asset event. When the scheduler creates an asset-triggered run, it puts all queued rows of the Dag into one IN (...). With many rows, Postgres rejects the statement. The scheduler exits, and it fails again on each restart. Then no Dag gets scheduled.

What I saw on apache/airflow:3.4.0rc1 with Postgres 16. I sent the events with POST /api/v2/assets/events.

A Dag with schedule=[a, b] and 9,000 queued events fails with stack depth limit exceeded.
A single-asset Dag with [scheduler] batch_asset_events = True waits at max_active_runs while events arrive. With 9,000 queued events, it fails with the same error.
A single-asset Dag with default settings and 66,199 queued events fails with number of parameters must be between 0 and 65535.

On 3.3.2, the same [a, b] Dag with 33,500 events works fine.

Drafted-by: Claude Opus 5.5 (reviewed by me)

All tested for #71192, #73580, #75255 and #68917 on 3.4.0rc1

I've tested my changes in [#70714](#70714), and everything is working

Tested #72425 on 3.4.0rc1, and it works as expected

@vatsrahul1001

Please let us know by checking the box / commenting if the change is addressed in the latest RC.

@vatsrahul1001 Both of my changes work as expected in 3.4.0rc1, compared against 3.3.2.

#71529: with OTEL_CONFIG_FILE set, a real scheduler now exports through the file's exporter and view. 3.3.2 discards them.

#71251: with only PyMySQL installed, db migrate and a Dag run succeed on MySQL 8.4. 3.3.2 raises the mysqlclient RuntimeError.

Evidence

apache-airflow 3.4.0rc1 and task-sdk 1.4.0rc1 from PyPI with the 3.4.0rc1 constraints file, Python 3.12, macOS arm64. The control is apache-airflow 3.3.2, the latest 3.3.x release.

71529: opentelemetry-instrument airflow scheduler for 35 s with otel_on=True and an OTEL_CONFIG_FILE that declares a console exporter and a *_duration view with explicit bounds.

3.3.2     console exports: 0
          [error] Failed to export metrics batch code: None, reason: Failed to parse: http://None:None/v1/metrics
3.4.0rc1  console exports: 9
          [info] OTEL_CONFIG_FILE is set; using the MeterProvider it built.
          airflow.scheduler.scheduler_loop_duration  explicit_bounds=[0.5, 1, 2, 4, 8]  bucket_counts=[0, 0, 0, 0, 0, 5]
          same bounds on critical_section_duration, critical_section_query_duration, executor_heartbeat_duration

The repro script from the PR (bootstrap, then get_otel_logger()):

3.3.2     declarative views: ['None']        same provider object? False   RESULT: declarative view was DISCARDED
3.4.0rc1  declarative views: ['*_duration']  same provider object? True    RESULT: declarative view survived

Without OTEL_CONFIG_FILE the re-init path is unchanged on both versions: the provider is replaced and a post-fork counter still exports.

71251: MySQL 8.4.13 on localhost, PyMySQL 1.1.3 and aiomysql installed, mysqlclient not installed.

3.3.2     $ airflow db migrate
          RuntimeError: You do not have `mysqlclient` package installed. Please install it with `pip install mysqlclient` ...
3.4.0rc1  $ airflow db migrate
          [info] Running stamp_revision  -> b4f1a7d2c903
          [info] Database migration done!
          $ airflow db check
          [info] Connection successful.
          $ airflow standalone, then airflow dags trigger mysql_smoke (two tasks passing an XCom)
          DagRun Finished: dag_id=mysql_smoke, ..., run_duration=2.945011, state=success

Neither driver installed, 3.4.0rc1:

RuntimeError: You have a MySQL connection string but neither `mysqlclient` nor `PyMySQL` is installed. Install one of them, ...

Both drivers installed, 3.4.0rc1 with mysqlclient 2.3.0: airflow db check prints "Connection successful." with both mysql+mysqldb and mysql+pymysql URLs.

Drafted-by: GitHub Copilot CLI (Claude Sonnet 5.5); reviewed by @1fanwang before posting

My changes work as expected.

#73433 was not automatically backported for some reason - here's a manual PR to do it: #75939

From my side, the following changes were tested:

Asset-triggered Dags get one run per event by default (Make asset event batching opt-in #68517). Also indirectly covers the asset-queue rework (Reference asset events from asset_dag_run_queue #70972).
Mapped tasks never running on scheduled runs (Fix dynamically mapped tasks never running on scheduled Dag runs #74861)
Short-circuit inside a mapped task group not skipping downstream (Fix short-circuit inside a mapped task group not skipping downstream #73959, [v3-4-test] Show secondary states of mapped tasks in Grid view (#74883) #75426)
Python 3.10 dropped (Drop Python 3.10 from CI, Breeze and the container images #74151)
The 3.4 schema migrations on SQLite (Postgres migrations run in airflow-init)

We also performed static-scan and deprecation-warning tests to look for patterns affected by these changes:

removal of the Dataset alias (Remove deprecated dataset aliases #70154)
deprecation of airflow.utils.email (Deprecate airflow.utils.email in favor of the SMTP provider #75414)
validate_key now raising ValueError (Refactor validate_key to raise ValueError instead of AirflowException #68890)
XCom backends needing store_value (Let XCom backends tell loop iterations apart through store_value #74858)
heartbeats no longer using execution_api_retries (Don't retry task heartbeats according to [workers] execution_api_retries #73244)
removal of airflow info --file-io (Remove airflow info --file-io as file.io dropped anonymous uploads #73054)

All looks good.

Tested on Airflow 3.4.0rc1 with local airflow standalone.

Fix dynamically mapped tasks never running on scheduled Dag runs #74861: Scheduled dynamic task mapping completed successfully. All mapped tasks ran and passed.
Add draining state for graceful Dag pausing #72407: “Pause after completion” let the active run finish, created no new scheduled runs, then paused the DAG.

They work as expected.

Tested on Airflow 3.4.0rc1 using Breeze with PostgreSQL:

Fix Dag list run badges and counters not refreshing after completion #74734: Dag run badges and counters updated without refreshing.
Show errors when the task clear confirmation check fails #75269: Blocking the clear-task confirmation check displayed an error; Cancel worked and the page remained usable.

Both behaved as expected.

Tested: #73717 on local Airflow.
Working as expected!

Tested #74548 on 3.4.0rc2 (breeze start-airflow --use-airflow-version 3.4.0rc2) using the scenarios documented in the PR, and it works as expected.

Tested on Apache Airflow 3.4.0rc2 (Python 3.12, SQLite):

Fix Dag run filtering for null start dates #71654: verified start_date_gte and start_date_gt exclude completed Dag runs with start_date=NULL; the Dag runs API returned the expected results.
Fix XCom slice endpoint for crossed bounds #72584: verified the XCom slice endpoint across 20 slice cases, including all crossed-bound cases. All passed.
both changes work as expected.

Seems like I found another regression in 3.4.0rc2... Might be worth another look before release

Issue: the API server stalls when it gets many write requests at the same time. 3.3.2 does not have this problem.

Setup: airflow standalone, Postgres 16, default DB pool (5 + 10 overflow), [api] workers = 1. I sent 600 POST /api/v2/assets/events, with 24 requests at the same time.

version	finished	time	500 errors	max latency
3.3.2	600 / 600	4.9 s	0	0.54 s
3.4.0b2	600 / 600	15 s	0	1.3 s
3.4.0rc1	146 / 600 (stopped at 180 s)	-	19	90 s
3.4.0rc2	600 / 600	136 s	10	60.7 s

POST /api/v2/variables gives the same result on rc2. During the stall, one running task also failed because its heartbeat timed out.

It looks like the audit log commit in log_action blocks the event loop when the DB pool is busy. Moving it to a thread (await run_in_threadpool(session.commit)) fixed it in my test.

I think it's worth taking a look before the release, but it needs many write requests at the same time, so I am not quite sure if this is enough for -1.

Issue: the API server stalls when it gets many write requests at the same time. 3.3.2 does not have this problem.

Thanks for reporting the regression, I will try reproduce the issue first.
Update: I confirmed that it's a real regression, working on it (but not in await run_in_threadpool way).

Tested #73555 on apache/airflow:3.4.0rc2 (Docker Compose, Postgres + LocalExecutor) with 3.3.2 as the control. Works as expected.

Two observations, not bugs:

With a mapped task in the run, the time axis does not shrink after filtering, since the aggregate bar is unaffected by the window (as stated in the PR).
The date pills have minute granularity, so tries less than a minute apart cannot be separated from the UI. The API works fine with seconds.

Found another one while testing rc2: #76190

Reproduced on 3.4.0rc2 (Breeze, Postgres, LocalExecutor) with the Dag from the issue.

Setup: a fail_fast=True Dag where a task gets removed or renamed while a run is still going (e.g. a deploy lands in the middle of a long run). When the next task fails:

The API server returns 500 on the state update (KeyError: 'victim' in _stop_remaining_tasks).
The scheduler hits the same KeyError in handle_failure and exits.
After a restart it re-queues the orphaned TI, the task fails again, and the scheduler dies again. In my run the scheduler exited 6 times in about 5 minutes, and boom (retries=0) got to try 7.

Not a regression though: _stop_remaining_tasks is the same in 3.3.2, so I don't think it should block 3.4.0. Fix is in #76191, would be nice to get it into 3.4.1.

#74022 breaks the sync of Dag-level permissions in 3.4.0rc2 when apache-airflow-providers-fab is 3.10.1 or older. With these versions, core no longer writes the permissions from a Dag's access_control into the FAB permission tables.

#74022 is not on the list, but its fix needs to raise the minimum FAB version to 3.10.2. Not sure whether we should include it in 3.4.0. I create #76509 to fix it.

Tested Airflow 3.4.0rc3 on a clean Linux environment (Python 3.12, SQLite and Postgres backends):

DB migrations initialized and stamped cleanly from scratch (airflow db migrate / db check).
Core CLI commands (version, info, dags list, tasks test) function as expected without unhandled warnings.
Baseline asset-triggered and scheduled DAGs execute normally.

While testing deeper on RC3, I also spotted a few related edge cases:

Connection.get_connection_from_secrets() in airflow/models/connection.py still lacks session forwarding to MetastoreBackend.get_connection(), forcing an extra connection checkout from the pool on every lookup.
In the Task SDK supervisor (supervisor.py:1180), invalid IPC frames skip sending a response frame, which deadlocks the child task runner process while heartbeating continues.
Silent discard of network errors during bundle.refresh() with 0 import errors recorded in the UI.

I am currently working on the solutions for these and will be opening the tracking issues with reproductions and raising PRs for them in a few moments.

PR's for the above listed issues:

Found a thin edge case with #71192 if the Execution API takes more than 5 seconds to respond (which I've never seen happen but is potentially possible). #76571 handles this case, will leave it up to maintainer discretion if we should backport this into 3.4

I believe 76550 76551 and 76552 are all carry-overs from previous versions (the first in particular probably has existed since 3.0). 76553 is real, but not a blocking bug (further description in the issue).

Yeah agreed, none of these should block 3.4.0. Raised them against main for the next release.

🎉 Airflow 3.4.0 is now released! https://apache-airflow.slack.com/archives/CCQ18L37F/p1793969658540499

Huge thanks to everyone who contributed and helped test this release 🙌

Navigation Menu

Repository navigation

Status of testing of Apache Airflow 3.4.0rc3 #75865

Significant Changes

New Features

Bug Fixes

Miscellaneous

25 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Status of testing of Apache Airflow 3.4.0rc3 #75865

Description

Significant Changes

New Features

Bug Fixes

Miscellaneous

Activity

vatsrahul1001 commented on Oct 28, 2026

zach-overflow commented on Oct 28, 2026

aaron-y-chen commented on Oct 28, 2026

Andrushika commented on Oct 28, 2026

seanghaeli commented on Oct 29, 2026

imrichardwu commented on Oct 29, 2026

ccoliu commented on Oct 29, 2026

1fanwang commented on Oct 29, 2026

shahar1 commented on Oct 29, 2026

anisengupta commented on Oct 29, 2026

fat-catTW commented on Oct 29, 2026

anzinmhd commented on Oct 29, 2026

samraj2k commented on Oct 29, 2026

25 remaining items

sjyangkevin commented on Nov 4, 2026

fat-catTW commented on Nov 4, 2026

Andrushika commented on Nov 4, 2026

jason810496 commented on Nov 4, 2026

wiasliaw commented on Nov 4, 2026

ChickenBenny commented on Nov 4, 2026

FrankYang0529 commented on Nov 5, 2026

meel-ayush commented on Nov 5, 2026

meel-ayush commented on Nov 5, 2026

seanghaeli commented on Nov 6, 2026

uranusjr commented on Nov 6, 2026

meel-ayush commented on Nov 6, 2026

vatsrahul1001 commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions