[dwaddle]

Fixes #3004. Supersedes #3018.

Taking over #3018 as requested by @ValueRaider:

• Switches get_news() to Yahoo's GraphQL endpoint (financeTickerList -> mlRecsStream) with recency ranking (aboutnessWeight: 0), mapping assets back to the existing article schema.
• Follows yf.config.locale for the news list slug and clientContext, with no per-call method argument.
• Caches requests per (count, tab, lang, region).

[arhancanli]

Read the diff (not run against Yahoo), two small things in the get_news() loop in base.py:

1. The pagination loop stops on not edges or not hasNextPage or not after, but never checks that endCursor moved. If Yahoo ever returns the same cursor with hasNextPage: true and pages that are all AssetError (so news does not grow), the loop re-posts the same request forever. Remembering the previous cursor and breaking when after == previous is a one-line guard, and a test with two identical pages would pin it.
2. if self._news.get(cache_key) treats an empty list as a miss, so for a locale Yahoo has no list for (the ja-JP case in the docs) every .news access sends a fresh POST. cache_key in self._news would cache the empty result too; the Ticker.news property is the path that would hit this repeatedly.

Does the NOT_FOUND error code come from a real ja-JP response, or from the mocked payload in the tests?

[dwaddle]

Good catches! Pushed a commit with both fixes and tests:

1. Stalled cursor: added after == prev_after break check in the pagination loop, plus a unit test simulating stalled endCursor with hasNextPage=True.
2. Empty cache: switched to cache_key in self._news so empty list responses are cached on subsequent accesses without re-querying.
3. NOT_FOUND error code: yes, that comes directly from Yahoo's live GraphQL gateway. For locales without news streams (like ja-JP or nl-NL), Yahoo returns errors: [{"extensions": {"code": "NOT_FOUND", "service": "networkapi_subgraph"}}] with HTTP 200 and lightyearList: null.

[arhancanli]

Read 3afb664 (not run). Both fixes look right: cache_key in self._news caches the empty result, and prev_after starts as the initial after, so the first page cannot trip the guard.

One gap in the stalled-cursor test: its second page is self._response([], ...), and the loop already breaks on not edges, so that test would pass without the new after == prev_after check. To exercise the guard the second page needs at least one edge, for example another _story("b") that is filtered out as an AssetError or returns the same cursor with has_next=True, and then post.call_count should still be 2 rather than 3. Deleting the guard should turn that test red; right now it would not.

[dwaddle]

Spot on, thanks! Updated the test in a266a5b to include an AssetError edge on page 2 and a third page. Verified locally that deleting the after == prev_after guard turns the test red (it requests page 3 and returns ['a', 'b']), whereas with the guard it breaks cleanly at call_count 2 with ['a'].

[arhancanli]

Thanks for the a266a5b update; the page-3 case is what makes the guard observable. One more edge to consider: if page 2 returns a new cursor but every edge is an AssetError, news does not grow, so the only stop is hasNextPage going false. Is there a page cap in the loop, or does it rely on Yahoo ending the stream?

[dwaddle]

Currently Yahoo's cursor (endCursor) is an encoded session offset ({"sessionId": "...", "index": N}) that terminates when reaching the backend recommendation limit (hasNextPage: false), but you're completely right that relying solely on remote API termination is risky if a feed ever yielded endless AssetErrors with advancing indices.

Pushed 5039c7b adding an explicit ceiling max_pages = max(5, (count + 9) // 10 * 2) (5 pages for default count=10), along with a unit test.

Also, Mehdy has pulled the stalled-cursor guard and empty cache fixes into #3018 (in 11ebed1). Whichever PR @ValueRaider prefers to merge is great by me!

[arhancanli]

Read 5039c7b (diff only, tests not run). The cap closes the case I asked about: pages_fetched is incremented before the request, so the all-AssetError test makes exactly 5 calls for count=5, and max(5, ...) keeps small counts from getting a cap of 2. One thing I would check: when the loop exits on max_pages with len(news) < count, the caller gets a short list with no sign of why. Is a utils.get_yf_logger().debug(...) line at that exit worth adding, so a truncated feed is distinguishable from a ticker that simply has few articles?

[dwaddle]

Added that debug line in 447a70a when len(news) < count and pages_fetched >= max_pages, along with an assertion in the unit test. Good call!

[dwaddle]

Closing as superseded and merged via #3018.