Skip to content

Fix unit-switch revert corrupting tables with genuine partial 100x blocks - #2927

Open
lenamonj wants to merge 4 commits into
ranaroussi:devfrom
lenamonj:fix/unit-switch-partial-block
Open

lenamonj wants to merge 4 commits into
ranaroussi:devfrom
lenamonj:fix/unit-switch-partial-block

Conversation

@lenamonj

@lenamonj lenamonj commented Jul 28, 2026 •

Copy link
Copy Markdown

Fixes the main finding in #2924, plus the capital-gains ZeroDivisionError confirmed still present on dev.

The defect. The revert in _fix_prices_sudden_change (added in d3704c8 to stop the unit-switch repair applying twice) fires whenever any row was repaired downward and _standardise_currency() had run. When the table instead contains a genuine interior 100x block, the block repair is correct, and the revert then rescales the whole table 100x and inverts Repaired?: 90 of 90 rows wrong, the 20 broken rows flagged clean, the 70 clean rows flagged repaired.

The fix. No external anchor, and no revert. A switch that passed the no-volume-jump gate is a unit switch, so the bigger numbers are the minor currency and the smaller the major. When a lone range would divide everything from some date back to the oldest row, in a table _standardise_currency() actually divided, the bigger older side is by that rule already at major scale and the smaller newest segment is what standardisation wrongly divided: repair that segment up directly. The whole-table revert and its flag inversion are deleted; interior blocks repair locally without consulting metadata. _standardise_currency() now records whether it divided (currencyScaled), because it sets currencyRepaired even when it only relabels GBp to GBP, and the two states need opposite repair directions. Net -19 lines in history.py.

Tests. All offline, none carries regularMarketPrice. The unit-switch cases drive _fix_unit_switch on CSV fixtures readable as a spreadsheet, in the exact state _standardise_currency() leaves behind:

case state golden outcome
partial-100x-block genuine interior 100x block in a divided table block divided down, only block flagged - fails on dev: whole table rescaled, flags inverted
wrongly-divided-block interior GBP block the division left 100x too small block multiplied up, only block flagged
reapplied-conversion GBp->GBP quotation switch whose GBP segment the division wrongly divided (the d3704c8 scenario, which shipped without a regression test) newest segment up, only it flagged
switch-not-divided same switch when standardisation only relabelled (prices_in_subunits False) older pence segment down, only it flagged - fails on dev: the revert un-repairs a correct repair
mixed-units-no-standardise the block fixture read as a GBP-labelled mixed table nothing has converted, no currencyRepaired same as partial-100x-block - interior repair independent of metadata
test_standardise_currency_records_scaling both branches of _standardise_currency currencyScaled True after a real division, False after a relabel - fails on dev
capital gains gain with no dividend the same day; gain on the first row no ZeroDivisionError - fails on dev

tests.test_price_repair: every pre-existing test has the same outcome on this branch as on dev - the failures on both sides are the live-fetch tests hitting rate limits in my environment, everything else passes on both.

@ValueRaider

ValueRaider commented Aug 1, 2026 •

Copy link
Copy Markdown
Collaborator

The problem with relying on self._history_metadata['regularMarketPrice'] is you're assuming the fetch is to today. But end date might be 2 years ago, so you get false positives.

Also you have given the repair logic a lie:

hist._history_metadata = {"currency": "GBP", "currencyRepaired": True,

The prices are still in GBp not GBP, currencyRepaired is False.


If a switch is detected without volume jump, then it's a unit-switch and the only truth is:

  • the bigger numbers are the minor currency
  • the smaller numbers are the major currency

This should be sufficient.

@lenamonj

lenamonj commented Aug 2, 2026

Copy link
Copy Markdown
Author

Both points taken, and fixed by deleting the anchor rather than defending it.

On the false positives: agreed, regularMarketPrice is today's quote and the window can end anywhere, so any comparison against it misreads old windows. It is worse than that, though: Yahoo delivers rmp in the quote currency, pence for a GBp ticker, and the test fed it 50.1993. With a truthful pence value the anchor would have reverted and failed its own test. The anchor is gone and no test carries regularMarketPrice any more.

On the fixture: the 50s-with-a-4800-block table was meant as the state after _standardise_currency() divides a pence table carrying a doubly-scaled block, which is why the metadata said currencyRepaired: True. Your reading of it - a GBP-labelled mixed table nothing has converted, currencyRepaired rightly absent - is at least as real, so it is now its own test case with exactly that metadata, and the repair produces the same golden outcome in both readings, because the interior repair no longer consults that metadata at all.

The direction now comes from your rule. A switch that passed the no-volume-jump gate is a unit switch, so the bigger numbers are the minor currency and the smaller the major. When a lone range would divide everything from some date back to the oldest row in a table standardisation actually divided, the bigger older side is by that rule already at major scale, and the smaller newest segment is what standardisation wrongly divided: the repair multiplies that segment up directly. The whole-table revert and its flag inversion are deleted, net -19 lines in history.py.

Your rule needed one recorded bit to apply after the pipeline has run: _standardise_currency() sets currencyRepaired even when it only relabels GBp to GBP without dividing, and the two states need opposite directions - after a real division the smaller side is the wrongly-divided one, after a relabel the bigger side genuinely is still pence and must come down. It now records currencyScaled, and there is a network-free test driving both of its branches.

Six offline cases in total. Three fail on current dev: the interior block (the whole table rescaled, flags inverted), the relabel-only switch (the revert un-repairs a correct repair), and the currencyScaled recording. Three pin behaviour that must not change: the wrongly-divided interior block, the d3704c8 switch scenario, and your mixed-table reading of the block fixture. Every pre-existing test has the same outcome on this branch as on dev - the same seven live-fetch failures in my offline environment on both sides.

Pushed as 1ac1130, PR description updated to match.

Comment thread yfinance/scrapers/history.py Outdated
del ranges[i]
if unit_switch and len(ranges) == 1 and ranges[0][1] == n \
and self._history_metadata.get('currencyRepaired', False) \
and self._history_metadata.get('currencyScaled', True):

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, that state can't exist. _standardise_currency() is the only writer and always sets the two keys together, the post-repair revert deletes them together, and every fetch rebuilds the metadata from Yahoo's raw response, so the pair can never disagree. The compound check was guarding key absence, not that combination: currencyRepaired (default False) kept the gate closed for currencies standardisation never touches, against the True default on currencyScaled. That is the wrong shape for a question one flag answers, so the gate is now the single check self._history_metadata.get('currencyScaled', False), defaulting closed. Equivalent in every reachable state, and the offline cases pin all three: divided, relabel-only, and never-standardised. Pushed as 0fe6ad3.

@ValueRaider

Copy link
Copy Markdown
Collaborator

Ignore that review, I was being dumb.

This seems fine. But before merging I'm going to stress-test it against my cached yf data.

@ValueRaider

Copy link
Copy Markdown
Collaborator

Rebase to latest dev, just in case this commit affects e285bd1

@lenamonj
lenamonj force-pushed the fix/unit-switch-partial-block branch from 0fe6ad3 to ebf713e Compare August 8, 2026 16:12
@lenamonj

lenamonj commented Aug 8, 2026

Copy link
Copy Markdown
Author

Rebased onto f854309. The three commits are unchanged (range-diff clean); one new commit adapts the offline metadata test to _standardise_currency()'s three-tuple return. Re-ran the new tests against unpatched f854309: the three red cases still fail there for their original reasons, so the fix composes cleanly with your change. Offline suite otherwise matches clean dev in my environment.

@ValueRaider

Copy link
Copy Markdown
Collaborator

I saw you reply to my review with:

You're right, that state can't exist

But my comment was nonsense. Are you AI?

@lenamonj

lenamonj commented Aug 8, 2026

Copy link
Copy Markdown
Author

I'm a person, 100%. I run a Claude Code loop that analyzes repos and drafts the fixes. I review what it produces and I only open a PR when I think the finding is genuinely useful. That reply agreeing with your review is one I didn't read closely enough before it went out, and that's on me.

@lenamonj

lenamonj commented Sep 4, 2026

Copy link
Copy Markdown
Author

@ValueRaider no rush, just checking whether the stress test turned anything up, and whether a rebase onto dev would help.

@ValueRaider

ValueRaider commented Sep 8, 2026 •

Copy link
Copy Markdown
Collaborator

I'm not seeing this have any effect in my own pipeline - it runs on yfinance-cache, so I would expect to see 100x/drops jumps as I add to cache with your change.

I want to see a real-world change from this.

…ocks

The revert added in d3704c8 fires whenever any row was repaired downward after _standardise_currency() ran. A genuine partial 100x block inside an already-converted table then gets the whole table rescaled 100x and the Repaired? flags inverted. Distinguish the two cases with the same anchor _standardise_currency() already uses: after this pass's repairs, a re-applied conversion leaves the table ~change-x below regularMarketPrice, a repaired partial block leaves it at the market level. Revert only in the first case. Adds fixture-based tests for both scenarios, and a guard for _repair_capital_gains raising ZeroDivisionError when a fund distributes capital gains with no dividend that day (or only on the first row).
…arMarketPrice

Review feedback on the previous commit was right twice over: anchoring on
regularMarketPrice assumes the fetch window ends near today, and the test fed
the anchor a value in the wrong unit. Both problems go away by removing the
anchor entirely.

In a genuine unit switch the bigger numbers are the minor currency and the
smaller the major. So when a lone repair range divides everything from some
date back to the oldest row in a table _standardise_currency() actually
divided, the switch is genuine and standardisation wrongly divided the newest
segment: repair that segment up directly, instead of dragging the correct
majority down and reverting the whole table afterwards. The whole-table revert
and its flag inversion are deleted; interior 100x blocks repair locally as
before. _standardise_currency() now records whether it divided
(currencyScaled), because it sets currencyRepaired even when it only relabels,
and the two states need opposite repair directions.

Tests cover all four states, no regularMarketPrice anywhere: a genuine
interior 100x block, an interior block standardisation wrongly divided, a
quotation switch standardisation wrongly divided (bigger side already major),
and the same switch when standardisation only relabelled (older pence side
divided down).
currencyScaled is only ever written alongside currencyRepaired, so
gating on both keys with opposite defaults was redundant and read as
if the pair could disagree. One check, defaulting closed, is
equivalent in every reachable state.
f854309 dropped div_scaled from the return tuple. The test unpacked
four values with the third discarded, so it raised ValueError on the
rebase. Unpack three.
@lenamonj
lenamonj force-pushed the fix/unit-switch-partial-block branch from ec5fc19 to f6ae542 Compare September 8, 2026 10:24
@lenamonj

lenamonj commented Sep 8, 2026

Copy link
Copy Markdown
Author

CNE.L (Capricorn Energy, quoted in GBp) shows it on Ticker("CNE.L").history(period="max", repair=True, auto_adjust=False, end="2026-09-08"), comparing dev at 686a031 with this branch rebased onto it.

Raw closes (repair=False): 47.111 on 1988-12-21, 249.742 on 2024-05-23, 383.0 on 2026-09-07. dev returns 4711.1, 24974.19 and 38300.0: every one of the 9668 rows comes back 100x larger than Yahoo serves it, 86 of them 10000x, and every row is flagged repaired. This branch returns 47.111, 249.742 and 383.0. The interior blocks the 100x repair corrects (1377 closes between 1992-09-01 and 2020-12-22, and the 2020 consolidation block) are identical on both branches, so the only difference is the whole-table revert firing on dev and rescaling everything.

period="2y" shows no difference on either branch; the revert only fires once the fetch reaches the 1988 rows, which is why a cache built from recent windows would not see it. I have rebased the branch onto current dev; the repair tests pass.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants