Commit d217a9b
Enable parallel backfill by eliminating shared state between providers (apache#63288)
Add --provider and --providers-json flags to extract_parameters.py and
extract_connections.py so each backfill run uses an isolated temp
providers.json and only scans the target provider. In --provider mode,
modules.json is not written (it would be incomplete), so concurrent
runs don't clobber each other.
The backfill command now creates a TemporaryDirectory with per-version
providers.json files instead of patching a shared file.1 parent de3333d commit d217a9b
9 files changed
Lines changed: 288 additions & 191 deletions
File tree
- dev
- breeze
- doc
- images
- src/airflow_breeze/commands
- tests
- registry
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
82 | 93 | | |
83 | 94 | | |
84 | 95 | | |
| |||
Loading
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
0 commit comments