Skip to content

Fix race condition in control loop that could cause pipeline starts to be ignored#892

Merged
mwylde merged 1 commit into
masterfrom
controller_race
May 27, 2025
Merged

Fix race condition in control loop that could cause pipeline starts to be ignored#892
mwylde merged 1 commit into
masterfrom
controller_race

Conversation

@mwylde

@mwylde mwylde commented May 27, 2025

Copy link
Copy Markdown
Member

This PR fix a race condition in the controller that could cause configuration updates to be missed.

The potential race worked like this: the run_to_completion loop would read the config at the end of an iteration. If a JobConfig update arrived (via StateMachine::update) after this read but before the execute_state call in the next iteration, the JobContext for that next iteration would be created with the stale config. The StateMachine::update method would send a JobMessage::ConfigUpdate, but the state machine might have already proceeded with an older config.

This is addressed by adding a new Applied or NotApplied status to each configuration read from the database, which allows us to track whether we've started a controller loop with a particular configuration or not, preventing us from missing a configuration update.

@mwylde mwylde requested review from cmackenzie1 and hhoughgg May 27, 2025 17:07
@mwylde mwylde merged commit 2fe1e93 into master May 27, 2025
6 checks passed
@mwylde mwylde deleted the controller_race branch May 27, 2025 19:22
mwylde added a commit that referenced this pull request Jun 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants