Skip to content

[GSOC 2026] Airflow Contribution & Verification Agent Skills #62500

Open

Description

@jason810496

Important

  1. We had already decided to use contributor-docs as source of truth here instead of solely based on Breeze CLI: [GSOC 2026] Airflow Contribution & Verification Agent Skills #62500 (comment)
  2. Please add the link to your PoC (draft PR, feature-branch link on your fork, etc.) if you have at the very beginning of your proposal so that we don’t overlook it.

Body

Background

Apache Airflow’s Breeze environment is the de facto way to reproduce CI, run tests, and verify changes locally. It encapsulates complex tooling (Docker, integrations, static checks, tests, system verification) behind a single, consistent developer interface.

However, modern AI coding tools (e.g. Claude Code, Gemini CLI, GitHub Copilot–style agents) currently treat Airflow’s repo like any generic Python project. They rarely:

  • Understand whether they are running inside or outside Breeze.
  • Choose the correct commands for host vs. container.
  • Follow the same workflows that Airflow contributors actually use (e.g. prek, breeze shell, breeze start-airflow).

We already expose some information through docs (e.g. AGENTS.md), but this mostly inflates the context window rather than giving agents a structured, machine-usable interface to Breeze.

This project aims to bridge that gap by creating an “Airflow Breeze Contribution / Contribution Verification” AI skill (final name TBD) that systematically encodes common contribution workflows and makes them reliably executable and testable by AI agents.

Goal

The overarching goal is to make AI tools:

Breeze-aware: able to detect whether they are running inside or outside Breeze and act accordingly.
In practice, this means that for a typical contributor PR, an AI agent can:

  • Run the right static checks.
  • Run the right subset of tests in Breeze.
  • Spin up Airflow and verify system behavior for a Dag representing the change (nice-to-have).
  • Do all of the above while respecting host/container boundaries.

Additionally, the solution should be consistency-focused, meaning that we want to keep Breeze CLI as the single source of truth for agent skills. This can be achieved by auto-syncing CLI docstrings and behaviors into the AI skill using existing tooling (e.g. prek), ensuring that the skill definitions always reflect the current state of the Breeze CLI.

Core Tasks

1. Environment Awareness & Detection
  • Design and implement a simple, robust mechanism for the agent skills to detect:
    • “Host” vs “inside Breeze container”.
    • Relevant environment variables, markers, or file paths that indicate context.
  • Encode decision logic for when to run:
    • Host-only commands (e.g. breeze shell, breeze start-airflow, git operations).
    • Container-only commands (e.g. pytest, airflow ...).
  • Provide a clear API/contract that AI tools can call to query current context and get recommended commands.

Note: Maybe we need to add some explicit markers, files in the repo, or write a small helper script that can be called to determine context in a reliable way. Or maybe we can rely on existing environment variables or filesystem cues. This is an open design question to explore.

2. Modeling Core Contributor Workflows as Skills

Based on the three scenarios described, define and implement skills that represent common contribution flows:

Scenario 1: Static checks pass

  • Stage changes (git add ...).
  • Run prek.
  • Collect and surface failures in a structured way so that an agent can fix them.

Scenario 2: Unit tests in Breeze

  • Start or attach to a Breeze container with breeze shell or breeze exec.
  • Run pytest with a targeted module/test path (not the whole suite).
  • Then the agent can inspect results and decide on next steps (e.g. fix code, exit Breeze).
3. Syncing with Breeze CLI as Source of Truth (via prek)
  • Investigate existing Breeze CLI docstrings and structure.
  • Define a mapping from Breeze commands (and their docstrings) to skill definitions, paths, and parameters.
  • Implement a prek hook that:
    • Generates or updates the agent skills definition files from Breeze CLI docstrings.
    • Fails when drift is detected (e.g. a command changed but the skill spec was not updated).
  • Integrate these checks into existing static check pipelines so the skills stay in sync automatically.
4. Evaluation & Test Harness
  • Design a testable user scenario or “exam” that simulates a typical contribution workflow (e.g. fixing a simple bug, adding a small feature) to verify that the added skills work as intended.
  • Add unit tests for any additional scripts or helper functions created.
5. Documentation & Developer Guide
  • Add or extend documentation (e.g. AGENTS.md, Breeze docs) to:
    • Describe the new Breeze-aware skills.
    • Show example workflows for human contributors and AI tools.
    • Document how other tools can integrate with the skills (e.g. path to spec file, key commands).

Advanced Tasks (Optional / Stretch Goals)

Scenario: System behavior verification

  • Write a Dag representing the feature/bugfix being contributed (or use an existing one).
  • Run breeze start-airflow (with --integration when needed).
  • Trigger the Dag via CLI (instead of UI) and wait for completion.
  • Inspect logs/status to determine success/failure from the TaskInstance logs.
  • Inspect logs/status from all the component services (scheduler, api-server, triggerer, etc) to determine if there are any underlying issues.
  • The agent can then decide to fix code, fix the Dag, or exit Breeze based on the results.

Expected Outcome

By the end of the project, we expect:

  • A Breeze-aware AI skill that can:
    • Detect host vs. container context.
    • Choose appropriate commands and environment transitions.
  • The AI toolings will be "smart-enough" to handle the core workflows for contributions, including:
    • Static checks with prek.
    • Targeted unit tests in Breeze.
    • Continue iterating based on results (e.g. fix code, fix tests, exit).
  • A sync mechanism (likely using prek) that:
    • Keeps Breeze CLI and the skill definitions in sync.
    • Fails CI when they diverge, ensuring Breeze remains the single source of truth.
  • Initial evaluation “exam(s)” and test harnesses that:
    • Verify that an implementation of the skill behaves correctly on at least the core scenarios.
  • Updated documentation explaining how contributors and AI tools can make use of the new capability.

A successful project will make it much easier for future AI tooling (IDEs, CLIs, bots) to interact with Breeze in a reliable and Airflow-native way, increasing contributor productivity and lowering the barrier to entry.

Recommended Skills

  • Programming & Tooling
    • Solid Python skills (CLI tools, packaging, basic testing).
    • Familiarity with Docker and containerized development environments.
    • Experience with writing or using CLIs and handling subprocesses.
  • Dev Workflow & CI
    • Understanding of typical open source contribution workflows (git, PRs, static checks, unit tests, pre-commit).
    • Exposure to CI systems and concepts of reproducible environments.
  • AI/Agents
    • Interest in or experience with AI coding assistants, Agent Skills, tool-calling, or agent frameworks.
    • Comfort reasoning about what “smart enough” means in terms of concrete, testable behaviors.
  • Airflow/Breeze (Nice to Have)
    • Basic knowledge of Apache Airflow concepts (Dags, tasks, operators).
    • Prior use of Breeze for development or testing is a plus, but not strictly required.

Motivation to work at the intersection of developer experience, tooling, and AI is more important than prior deep expertise in all of these areas.

Mentors

Learning Materials

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Activity

  1. added
    area:dev-envCI, pre-commit, pylint and other changes that do not change the behavior of the final code
    on Feb 26, 2026
  2. added
    kind:metaHigh-level information important to the community
    on Feb 26, 2026
  3. removed
    kind:metaHigh-level information important to the community
    on Feb 26, 2026
  4. Pyasma commented on Feb 26, 2026

    @Pyasma
    Contributor

    hi @jason810496,

    I’m interested in the “Airflow Breeze Contribution / Contribution Verification” AI skill project and wanted to share my current understanding before going deeper.

    My reading is that the core goal is to encode existing Airflow contributor workflows (static checks, targeted tests, environment transitions) into Breeze-aware, machine-readable skills that AI tools (Gemini, Claude Code, GPT, etc.) can reliably execute.

    The main challenge seems to be making the agent aware of the host vs Breeze-container environment and enforcing correct command boundaries, while keeping the Breeze CLI as the single source of truth via auto-syncing (for example, through prek).

    System-level DAG execution for end-to-end verification, where an agent can trigger and validate real Airflow behavior in the same way a contributor would, using these skills.

    I want to confirm that this framing matches the intent of the project before drafting a more detailed proposal.

  5. jason810496 commented on Feb 27, 2026

    @jason810496
    MemberAuthor

    Hi @Pyasma , yes, your understanding is correct.

  6. Pyasma commented on Feb 27, 2026

    @Pyasma
    Contributor

    @jason810496 thanks for confirming
    I will work on the proposal

    Where i can find template for the proposal ? ( is there a specific template for it )

    https://community.apache.org/gsoc/ -> there is template in this page can i use that?

  7. jason810496 commented on Feb 27, 2026

    @jason810496
    MemberAuthor

    -> there is template in this page can i use that?

    Sure, feel free to use any template as long as your proposal is clear. The format doesn’t really matter, but the thinking behind the solution is important. Thanks.

  8. KatalKavya96 commented on Feb 27, 2026

    @KatalKavya96
    Contributor

    Hello @jason810496,

    I have carefully gone through the full description of the Airflow Contribution & Verification Agent Skills project and the related discussions. From my understanding, the core challenge is not just exposing Breeze commands to AI tools, but designing a structured, Breeze-aware abstraction layer that:

    • Reliably detects host vs. container context
    • Encodes common contributor workflows (prek, targeted pytest runs, breeze shell/start-airflow)
    • Enforces correct command boundaries
    • Keeps Breeze CLI as the single source of truth via automated synchronisation

    Having worked on Apache Airflow previously (including Breeze-based development, local testing, and debugging inside containers) I can directly relate to the host/container boundary issues and the importance of reproducible contributor workflows.

    I am currently working on a structured proposal that focuses on:

    • A robust environment detection mechanism
    • A sync mechanism tied to Breeze CLI (likely via prek)
    • A small evaluation to verify correctness of agent behaviour

    I would really appreciate staying in close touch while refining the proposal to ensure it aligns well with the project’s expectations.

    Looking forward to contributing meaningfully to this effort.
    Thank You

  9. Abhishekmishra2808 commented on Mar 8, 2026

    @Abhishekmishra2808
    Contributor

    @jason810496 @potiuk, I have few questions from the community before making a proposal,

    1. How strongly do we want the Breeze CLI to remain the single source of truth for agent skills? Should the skills be entirely generated from CLI metadata, or should there be an intermediate schema describing workflows that Breeze commands map into?
    2. If Breeze CLI evolves, do maintainers expect the AI skill layer to adapt automatically through the prek sync mechanism, or should we maintain some backward compatibility layer for agents?
  10. jason810496 commented on Mar 9, 2026

    @jason810496
    MemberAuthor

    @jason810496 @potiuk, I have few questions from the community before making a proposal,

    1. How strongly do we want the Breeze CLI to remain the single source of truth for agent skills? Should the skills be entirely generated from CLI metadata, or should there be an intermediate schema describing workflows that Breeze commands map into?
    2. If Breeze CLI evolves, do maintainers expect the AI skill layer to adapt automatically through the prek sync mechanism, or should we maintain some backward compatibility layer for agents?

    The main question here is the long-term maintainability of the Airflow project itself. That’s why it would be better to have a single source of truth for agent skills, to avoid agent skills becoming outdated over time.

    It’s totally fine to have an intermediate schema for Breeze and agent skills, as long as we consider the long-term maintainability effort and make sure the mechanism itself is easy enough for maintainers to keep the agent skills synchronized with the Breeze CLI.

    For the second question, in most cases, Airflow contributors/maintainers will use the Breeze CLI directly installed from the ‎main branch. The Breeze CLI supports the ‎--use-airflow-version flag for some common commands.

    To sum up, the latest Breeze CLI itself should always be able to work with any Airflow version. We just need to make sure the mechanism is able to keep agent skills up to date with changes to the Breeze CLI -- or, more accurately, that changes to the Breeze CLI and changes to the agent skills are part of the same atomic Git commit history from my perspective.

  11. sonianuj287 commented on Mar 9, 2026

    @sonianuj287

    Hi, I’m exploring this issue and would like to work on it.
    I’ve started looking into the relevant module and will
    share an implementation soon. Any pointers are welcome.

  12. Abhishekmishra2808 commented on Mar 9, 2026

    @Abhishekmishra2808
    Contributor

    @jason810496 Thanks
    Looking at the Breeze structure, it seems feasible to derive the skill layer directly from the CLI definitions rather than maintaining a separate manually curated spec. A prek hook could generate the skill definitions from the Breeze command metadata and fail when the generated output diverges from the committed files. That would effectively enforce synchronization and ensure that any CLI change requires updating the corresponding agent skill in the same commit.
    This also keeps the Breeze CLI as the authoritative interface while allowing the skill layer to remain a thin, generated abstraction over it.
    One point I’m curious about: should the skill layer stay close to individual Breeze commands, or should it represent higher-level contributor workflows (e.g., static checks, targeted tests, environment transitions) that internally orchestrate multiple Breeze commands depending on context?

  13. potiuk commented on Mar 9, 2026

    @potiuk
    Member

    One point I’m curious about: should the skill layer stay close to individual Breeze commands, or should it represent higher-level contributor workflows (e.g., static checks, targeted tests, environment transitions) that internally orchestrate multiple Breeze commands depending on context?

    I think we should aim for something that will be both Human and Agents usable.

    I would even go as far as to say that we should use the contributing guides as a single source of truth rather than breeze CLI. If anything I would focus on making sure that the workflows described for humans are always updated when we add new breeze command.

    One thing that is important in those workflows. NOT EVERYTHING should be done with breeze. While breeze is good for repeatable environment and making sure that "works for me" syndrom does not kick in, a lot of the contributor's workflows - such us running tests or pre-commit hooks can be done without breeze.

    In vast majority of cases this is enough to run tests for example:

    uv run --project distribution_folder pytest

    Only when this is not working because some system dependency is missing (mysql librarires for example) we should fall back to breeze way of running tests. Also in case we see discrepancy between the CI and local tests.

    And I think skills should reflect that.

    And it's faster and easier to debug as well - as you can run it from the IDE (and agent can also debug it better I guess).

    So I am wondering if the intermediate layer that should be the "source of truth" should be contributing docs.

  14. Pyasma commented on Mar 9, 2026

    @Pyasma
    Contributor

    I think we should aim for something that will be both Human and Agents usable.

    I would even go as far as to say that we should use the contributing guides as a single source of truth rather than breeze CLI. If anything I would focus on making sure that the workflows described for humans are always updated when we add new breeze command.

    @potiuk oh this makes more sense. AGENTS.md has something similar mentioned as well.

    i was thinking of something similar too when i was reading AGENTS.md (you have this mentioned there):

    If uv tests fail with missing system dependencies, run the tests with breeze: breeze run pytest <tests> -xvs

    this might also be easier to maintain long term since the agent skills would reflect the contributing docs, just in a more structured way for agents

  15. Abhishekmishra2808 commented on Mar 9, 2026

    @Abhishekmishra2808
    Contributor

    From the discussion, it seems the important distinction is that the skill layer should represent contributor workflows, not just mirror the Breeze CLI commands. In that sense, Breeze becomes an execution mechanism for reproducibility, rather than the primary abstraction exposed to agents.So instead of a CLI-driven model, the system would likely look like a workflow-driven execution model, where the agent decides whether to run locally or fall back to Breeze depending on context.

    For example, a typical “run tests” workflow could follow something like:

    flowchart TD
        A[Code Change or PR Update] --> B[Run Static Checks]
        B --> C{Static Checks Pass?}
    
        C -- No --> D[Fix Lint or Formatting Issues]
        D --> B
    
        C -- Yes --> E[Run Tests Locally]
    
        E --> F{Dependencies Available?}
    
        F -- Yes --> G[Run Local Tests<br/>uv run --project distribution_folder pytest]
        F -- No --> H[Fallback to Breeze]
    
        H --> I[Start Breeze Environment<br/>breeze shell or breeze exec]
        I --> J[Run pytest inside container]
    
        G --> K[Analyze Test Results]
        J --> K
    
        K --> L{Tests Pass?}
    
        L -- No --> M[Fix Code or Tests]
        M --> E
    
        L -- Yes --> N[Ready for PR / CI Verification]
    
    Loading

    Conceptually the architecture could then look like:

    flowchart LR
        A[Contributing Guides] --> B[Workflow Definitions]
        B --> C[Agent Skill Layer]
    
        C --> D{Execution Strategy}
    
        D --> E[Local Execution]
        D --> F[Breeze Execution]
    
        E --> G[Local Commands<br/>uv run pytest, prek, etc.]
        F --> H[Breeze Commands<br/>breeze shell, breeze exec, airflow services]
    
    Loading

    In this model:

    • Contributing guides act as the source of truth for workflows.
    • Agent skills encode those workflows in a structured format.
    • The execution strategy layer decides whether commands should run locally or via Breeze.
    • Breeze remains the reproducibility fallback when local execution is insufficient (missing dependencies, CI mismatch, etc.).

    One possible implication is that Breeze CLI metadata could still be used to derive or validate the Breeze execution layer, while the workflow abstraction remains aligned with the contributing documentation. right?

  16. 72 remaining items

  17. Pyasma commented on Apr 21, 2026

    @Pyasma
    Contributor

    @gsemet I agree with your point about keeping things simple and using AGENTS.md as an entrypoint, that makes sense from a usability perspective.

    But the main problem here is long term maintainability and whether contributors can actually trust the agent’s behavior. It is not just about exposing information, it is about making sure the agent consistently gets correct and minimal context.

    We need to make sure:

    • the information given to the agent stays accurate over time
    • the mapping between docs and skills does not drift from real workflows
    • the agent is guided to the right commands without extra context that could lead to wrong decisions

    So the challenge is finding a balance between keeping things simple and having a mechanism that keeps everything reliable and maintainable as the project evolves.

  18. added 5 commits that reference this issue on Apr 21, 2026
    05d8d04
    b6c0eff
    134e056
    a20e39e
    96205c2
  19. jason810496 commented on Apr 23, 2026

    @jason810496
    MemberAuthor

    Just noted down related discussion in dev list that focus on user-facing docs and the actual code implementation. It's out of scope for this project, but it could be considered as post-gsoc project or follow-up project.

    https://lists.apache.org/thread/zswfsdgg2qw6bnk8yhxbvpzzookr8pnx

    It's also yet another kind of reflect changes from the source of truth, but the output will be user-facing docs or ADR.

  20. Subham-KRLX commented on Apr 24, 2026

    @Subham-KRLX
    Contributor

    Just noted down related discussion in dev list that focus on user-facing docs and the actual code implementation. It's out of scope for this project, but it could be considered as post-gsoc project or follow-up project.

    https://lists.apache.org/thread/zswfsdgg2qw6bnk8yhxbvpzzookr8pnx

    It's also yet another kind of reflect changes from the source of truth, but the output will be user-facing docs or ADR.

    Got it @jason810496 User docs/ADRs are explicitly marked as post GSoC in the proposal Keeping this PoC focused on agent skills only.

  21. gsemet commented on Apr 24, 2026

    @gsemet

    That is the new science of harness engineering !

    It is pretty simple to have a mental representation of what is going on: each time you start a session, you bring a contractor on your project that knows nothing of your internal working (actually for an opensource project it does a bit), and you want to give him the minimal information map (the "starter kit" doc) and it will jump to the right section.

    And you have a "budget" of ~100 rules you can give to your contractor on top of its internal model to adapt to your project, at any given point. That is why splitting task by tech or module is key.

    That is half the job. The other half is to actually verify the job done (usually you have a coding agent that take a task and code it, and a second subagent that verify it)

    Yesterday I did a simple change, I changed the place in these task files where the guidelines/rules are place (was after the acceptance criteria, now it is on top), and I get a huge bump in accuracy. That means the tasks needs to be organized correctly for agent to understand.

    Everybody learns at the same time, that is a bit odd. But it does the job.

    You can use superpowers or some other SDD framework, even plan mode works fine, but the agents.md priming is key for accuracy (respect of your rules).

  22. Subham-KRLX commented on Apr 24, 2026

    @Subham-KRLX
    Contributor

    That is the new science of harness engineering !

    It is pretty simple to have a mental representation of what is going on: each time you start a session, you bring a contractor on your project that knows nothing of your internal working (actually for an opensource project it does a bit), and you want to give him the minimal information map (the "starter kit" doc) and it will jump to the right section.

    And you have a "budget" of ~100 rules you can give to your contractor on top of its internal model to adapt to your project, at any given point. That is why splitting task by tech or module is key.

    That is half the job. The other half is to actually verify the job done (usually you have a coding agent that take a task and code it, and a second subagent that verify it)

    Yesterday I did a simple change, I changed the place in these task files where the guidelines/rules are place (was after the acceptance criteria, now it is on top), and I get a huge bump in accuracy. That means the tasks needs to be organized correctly for agent to understand.

    Everybody learns at the same time, that is a bit odd. But it does the job.

    You can use superpowers or some other SDD framework, even plan mode works fine, but the agents.md priming is key for accuracy (respect of your rules).

    That makes sense We will keep this PoC focused on the agent skills workflow for now and treat the docs/ADR reflection idea as a follow up after GSoC.

  23. alok-kumar0421 commented on Apr 24, 2026

    @alok-kumar0421

    Hi, I’ve added a PoC in my proposal and shared the repo as well.
    Right now it shows how workflows can run differently based on host vs Breeze.
    I’m improving it step by step and planning to make the workflows more structured.
    Would really appreciate your feedback!

  24. added 3 commits that reference this issue on Apr 24, 2026
    1978089
    d92c828
    5a3e0e3
  25. cschanhniem commented on Jul 8, 2026

    @cschanhniem

    Great project on structured agent interfaces for Breeze. Since this defines what agents can do in the Airflow repo, AGENTOWNERS could complement this as the policy layer — defining which agents may act on which paths, what requires human approval, and what's blocked entirely. AGENTS.md + AGENTOWNERS.yml together cover both the how (skills) and the what's-allowed (policy).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:dev-envCI, pre-commit, pylint and other changes that do not change the behavior of the final code

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions