Skip to content

Update execute_tasks_new_python_interpreter docs for posix_spawn - #76095

Open
gang-zh wants to merge 1 commit into
apache:mainfrom
gang-zh:docs-task-interpreter-posix-spawn
Open

gang-zh wants to merge 1 commit into
apache:mainfrom
gang-zh:docs-task-interpreter-posix-spawn

Conversation

@gang-zh

@gang-zh gang-zh commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

Since #73503, when [core] execute_tasks_new_python_interpreter is on (and always on macOS), the task supervisor starts a fresh interpreter with os.posix_spawn instead of fork() followed by execv(). The option's description, its 3.4.0 newsfragment and the two security pages still say the task process is forked and then execs. That describes a mechanism the code no longer uses, and it leaves out the property that motivated #73503: at-fork handlers registered in the supervisor no longer run in the task.

This is a docs-only change with no change in behaviour.

On v3-4-test the newsfragment has already been compiled into RELEASE_NOTES.rst, so this will not cherry-pick cleanly. A manual backport that edits RELEASE_NOTES.rst instead will follow.

Checks run:

  • prek run --files on the four files: every hook passed except check-default-configuration, which needs a Breeze CI image that I couldn't build locally. I ran its two commands, airflow config list --default and then airflow config lint, in apache/airflow:3.4.0rc1-python3.12 with this branch's config.yml mounted. The new description renders, and lint reports no issues.
  • In the same image, os.posix_spawn did not run os.register_at_fork handlers, while a bare os.fork() ran both before and after_in_child.
  • On 3.4.0rc1 with the option on, task processes ran in the spawned bootstrap interpreter (sys.argv == ['-c']) with PR_GET_DUMPABLE at 0, which matches the updated security docs.

related: #72164, #73503


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 5.5)

Generated-by: Claude Code (Opus 5.5) following the guidelines

🤖 Generated with Claude Code

Since apache#73503 the supervisor starts the task in a fresh interpreter with
os.posix_spawn instead of forking and then exec-ing. The option's
description, its 3.4.0 newsfragment and the security docs still told
operators that the task process is forked first, which no longer matches
what they get when they turn the option on, and they left out that
at-fork handlers registered in the supervisor (the reason for apache#73503) no
longer run in the task.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant