Skip to content

Commit d217a9b

Browse files
kaxilPyasma
authored andcommitted
Enable parallel backfill by eliminating shared state between providers (apache#63288)
Add --provider and --providers-json flags to extract_parameters.py and extract_connections.py so each backfill run uses an isolated temp providers.json and only scans the target provider. In --provider mode, modules.json is not written (it would be incomplete), so concurrent runs don't clobber each other. The backfill command now creates a TemporaryDirectory with per-version providers.json files instead of patching a shared file.
1 parent de3333d commit d217a9b

9 files changed

Lines changed: 288 additions & 191 deletions

File tree

dev/breeze/doc/11_registry_tasks.rst

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,17 @@ Example usage:
7979
# Backfill a hyphenated provider
8080
breeze registry backfill --provider microsoft-azure --version 11.0.0
8181
82+
Each run uses an isolated temporary ``providers.json``, so different providers
83+
can be backfilled in parallel from separate terminal sessions:
84+
85+
.. code-block:: bash
86+
87+
# Terminal 1
88+
breeze registry backfill --provider amazon --version 9.15.0 --version 9.14.0
89+
90+
# Terminal 2 (safe to run simultaneously)
91+
breeze registry backfill --provider google --version 14.0.0 --version 13.0.0
92+
8293
Output is written to ``registry/src/_data/versions/{provider}/{version}/``:
8394

8495
- ``parameters.json`` — operator/sensor/hook parameters

dev/breeze/doc/images/output_registry.svg

Lines changed: 11 additions & 7 deletions
Loading
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
297843509448a55e7941eed3c0485df8
1+
8c9be6264d33af7facd1fbdf435697b7

0 commit comments

Comments
 (0)