fix: swap operand order so _metaflow parameter is respected in Metafl…#3022
fix: swap operand order so _metaflow parameter is respected in Metafl…#3022khushthecoder wants to merge 2 commits intoNetflix:masterfrom
Conversation
…owObject.__init__ Fixes Netflix#3019 ## Root Cause In MetaflowObject.__init__ (line 288), the expression: self._metaflow = Metaflow(_current_metadata) or _metaflow always evaluated to the freshly constructed Metaflow() because Metaflow.__init__ always sets self.metadata, making the object truthy. The 'or _metaflow' fallback was therefore dead code. This caused: - Unnecessary Metaflow object allocation for every child - Cross-environment unpickling broken (children ignore the restored metadata provider and fall back to global) ## Fix Swap the operands: self._metaflow = _metaflow or Metaflow(_current_metadata) This correctly uses the passed-in _metaflow when available, and only constructs a new Metaflow() as a fallback. ## Tests Added 5 regression tests in test/unit/test_metaflow_object_init.py: - test_passed_metaflow_is_used - test_no_metaflow_falls_back_to_new_instance - test_child_inherits_parent_metaflow - test_passed_metaflow_takes_precedence_over_current_metadata - test_getitem_passes_parent_metaflow
Greptile SummaryThis PR fixes a silent bug in Key changes:
The fix is minimal, backward-compatible (the Confidence Score: 5/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["MetaflowObject.__init__(pathspec, _metaflow, _current_metadata)"] --> B{"_metaflow is truthy?"}
B -- "Yes (caller supplied instance)" --> C["self._metaflow = _metaflow\n(reuse parent's provider)"]
B -- "No (_metaflow is None / default)" --> D["self._metaflow = Metaflow(_current_metadata)\n(construct new provider)"]
C --> E["Continue __init__\n(namespace, tags, id resolution)"]
D --> E
E --> F["__iter__ / __getitem__\npasses self._metaflow\nto child objects"]
F --> G["Child MetaflowObject\nreuses same metadata backend"]
style C fill:#d4edda,stroke:#28a745
style D fill:#fff3cd,stroke:#ffc107
Last reviewed commit: 2e17951 |
Rename test_child_inherits_parent_metaflow -> test_run_stores_passed_metaflow with an accurate docstring. The test validates that Run stores a directly passed _metaflow, not parent->child propagation via iteration (which is covered by test_getitem_passes_parent_metaflow).
| def test_getitem_passes_parent_metaflow(self): | ||
| """Flow.__getitem__ should pass self._metaflow to the child Run.""" | ||
| parent_mf = _DummyMetaflow() | ||
|
|
||
| from metaflow.client.core import Flow | ||
|
|
||
| child_obj = { | ||
| "run_number": "99", | ||
| "ts_epoch": 1700000000000, | ||
| "tags": [], | ||
| "system_tags": [], | ||
| } | ||
|
|
||
| with patch.object(Flow, "_get_object", return_value={ | ||
| "flow_id": "TestFlow", | ||
| "ts_epoch": 1700000000000, | ||
| "tags": [], | ||
| "system_tags": [], | ||
| }): | ||
| flow = Flow( | ||
| pathspec="TestFlow", | ||
| _metaflow=parent_mf, | ||
| _namespace_check=False, | ||
| ) | ||
|
|
||
| # Patch _get_child so __getitem__ finds something | ||
| with patch.object(Flow, "_get_child", return_value=child_obj): | ||
| run = flow["99"] | ||
|
|
||
| assert run._metaflow is parent_mf, ( | ||
| "Flow.__getitem__ did not propagate _metaflow to child Run." | ||
| ) |
There was a problem hiding this comment.
__iter__ propagation not tested
test_getitem_passes_parent_metaflow verifies the __getitem__ path, but there is no corresponding test for the __iter__ path (lines 394–416 of core.py), which is the other major call site that passes _metaflow=self._metaflow to child objects. The two paths are symmetric, so a bug regressing one without the other could go undetected.
Consider adding a test that patches Flow.__iter__ (or metadata.get_object used inside __iter__) and verifies that each yielded child carries the parent's _metaflow instance:
def test_iter_passes_parent_metaflow(self):
"""Flow.__iter__ should propagate _metaflow to every yielded child."""
parent_mf = _DummyMetaflow()
child_obj = {
"run_number": "7",
"ts_epoch": 1700000000000,
"tags": [],
"system_tags": [],
}
from metaflow.client.core import Flow
with patch.object(Flow, "_get_object", return_value={
"flow_id": "TestFlow",
"ts_epoch": 1700000000000,
"tags": [],
"system_tags": [],
}):
flow = Flow(pathspec="TestFlow", _metaflow=parent_mf, _namespace_check=False)
# Patch metadata.get_object so __iter__ yields one child
parent_mf.metadata.get_object.return_value = [child_obj]
children = list(flow)
assert len(children) == 1
assert children[0]._metaflow is parent_mf|
hi @savingoyal @saikonen I have submitted a PR addressing #3019 issue. Could you please review it? If any changes are required, I am available to make them. |
fix: swap operand order so
_metaflowparameter is respected inMetaflowObject.__init__Fixes #3019
PR Type
Summary
_metaflowparameter passed toMetaflowObject.__init__was silently discarded due to wrongoroperand order, causing every child object to allocate a newMetaflow()from the global provider instead of reusing the parent's. This broke cross-environment unpickling where children queried a different metadata backend than the parent.Issue
Fixes #3019
Reproduction
Runtime: local
Commands to run:
Where evidence shows up: parent console
Before (error / log snippet)
The passed
_metaflowis discarded; a freshMetaflow()is constructed instead.After (evidence that fix works)
The passed
_metaflowis now correctly used.Root Cause
In
metaflow/client/core.py, line 288 insideMetaflowObject.__init__:Python's
oroperator returns the first truthy value.Metaflow(_current_metadata)always returns a truthy object because its__init__(lines 2619–2628) unconditionally assignsself.metadatato a real provider — either via_metadata()or fromcurrent_metadata. Since the left operand is always truthy,or _metaflowis never evaluated.Cascading consequences:
__iter__(line 409),__getitem__(line 525), andMetaflow.__iter__(line 665) passes_metaflow=self._metaflow, but it's silently ignored._unpickle_21227restores the parent's metadata via_current_metadata, but when the parent iterates children, those children pass_metaflow=self._metaflowwhich is discarded — children fall back to the global provider, potentially querying a different backend.Why This Fix Is Correct
Swapping the operands to
_metaflow or Metaflow(_current_metadata)restores the intended invariant: use the caller-supplied_metaflowwhen available, only construct a new one as fallback. The fix is minimal — a single operand swap — and preserves all existing behavior for the_metaflow=None(default) case sinceNone or Metaflow(...)still constructs a new instance.Failure Modes Considered
Backward compatibility for
_metaflow=None(default path): When no_metaflowis passed (the common case for top-levelFlow('name')orRun('name/id')calls),None or Metaflow(_current_metadata)still constructs a newMetaflow()— identical behavior to before.Truthy
_metaflowskipping_current_metadata: When_metaflowis passed (internal code paths:__iter__,__getitem__, unpickling),_current_metadatais not used. This is correct because the parent already has the right metadata provider — constructing a new one would be wasteful and potentially inconsistent.Falsy
_metaflowedge case: If someone were to pass a falsy non-None value for_metaflow, it would fall through to constructMetaflow(_current_metadata). All existing call sites pass eitherNone(default) orself._metaflow(always truthy), so this path is never hit in practice but fails safe.Tests
5 tests added in
test/unit/test_metaflow_object_init.py:test_passed_metaflow_is_used— verifiesself._metaflow is passed_mf(the core bug)test_no_metaflow_falls_back_to_new_instance— verifies default path still workstest_child_inherits_parent_metaflow— verifiesRun('X/42', _metaflow=mf)holds the parent's instancetest_passed_metaflow_takes_precedence_over_current_metadata— verifies_metaflowwins when both are giventest_getitem_passes_parent_metaflow— verifiesflow["99"]propagates_metaflowto childNon-Goals
Metaflow.__init__,_unpickle_*methods, or__getstate__/__setstate__— those are correct as-is.AI Tool Usage
AI tools were used (describe below)
Tool: Antigravity (AI coding assistant)
Used for: Identifying the fix location, generating the test scaffolding, and drafting the PR description
All generated code was reviewed, understood, and tested manually. I can explain every line of the fix and tests independently.