Fix flaky Hypothesis stateful test for unique name generation#10946
Merged
max-sixty merged 1 commit intopydata:mainfrom Nov 24, 2025
Merged
Fix flaky Hypothesis stateful test for unique name generation#10946max-sixty merged 1 commit intopydata:mainfrom
max-sixty merged 1 commit intopydata:mainfrom
Conversation
The original approach used st.shared(st.builds(set), ...) with mutation to track unique names. While this pattern works for regular @given tests, it doesn't work correctly in RuleBasedStateMachine because the shared set's contents depend on which rules ran before, causing inconsistent replay during shrinking. Per Hypothesis maintainer guidance, moved uniqueness tracking to instance state (self.used_names) which is properly isolated per test case. Co-authored-by: Claude <noreply@anthropic.com>
dcherian
approved these changes
Nov 24, 2025
Contributor
dcherian
left a comment
There was a problem hiding this comment.
Thank you! I looked at it a couple of months ago and couldn't figure out what was going on
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude tried to fix the flakiness issue in CI. I probed its logic a bit, it looks reasonable but I don't fully understand. Our summary below...
Summary
DatasetStateMachinetest that was causing CI failures withFlakyStrategyDefinition: Inconsistent data generation!st.shared()strategy to state machine instance stateProblem
The nightly Slow Hypothesis CI has been failing:
https://github.com/pydata/xarray/actions/runs/19603404994
The test
properties/test_index_manipulation.py::DatasetTest::runTestfails intermittently with:hypothesis.errors.FlakyStrategyDefinition: Inconsistent data generation!
Data generation behaved differently between different runs.
Is your data generation depending on external state?
Root Cause
The original code used a pattern from [Stack Overflow](https://stackoverflow.com/questions/73737073/c
reate-hypothesis-strategy-that-returns-unique-values):
While this pattern works for regular
@giventests, it does not work for RuleBasedStateMachinebecause:
Per HypothesisWorks/hypothesis#2338:
"I think you'll need to set these up as instance attributes, rather than class attributes."
Solution
Move uniqueness tracking to state machine instance state:
This follows the Hypothesis-recommended pattern for stateful tests where cross-rule state belongs on
the instance.