Inconsistent results when streaming to unidiff (does not happen with urlopen or requests)

[AdamWill]

In an LLM code review project we've been looking at different ways of parsing diffs. Today I was looking at different ways of reading diffs from a forge (GitHub, GitLab etc.) into unidiff, a diff parser. I encountered some weird behavior which I first reported to unidiff but now think is probably an httpx issue.

Here's the reproducer:

#!/bin/python

import hashlib
import urllib.request

import httpx
import requests
import unidiff

diffurl = "https://forge.fedoraproject.org/releng/tickets/pulls/12599.diff"

def check(patches):
    md5 = hashlib.md5(str(patches).encode("utf-8")).hexdigest()
    if md5 != "c28ad8dd59833ff6d8f2f7d680cf49ca":
        with open(f"/var/home/adamw/diff-{md5}.diff", "w", encoding="utf-8") as fh:
            fh.write(str(patches))
            print("DUMPED " + md5)
    else:
        print("OK")

def naive():
    diff = httpx.get(diffurl).text
    patches = unidiff.PatchSet(diff)
    check(patches)

def sophisticated_httpx():
    with httpx.stream("GET", diffurl) as r:
        patches = unidiff.PatchSet(r.iter_bytes(), encoding="utf-8")
    check(patches)

def sophisticated_requests():
    with requests.get(diffurl, stream=True) as r:
        patches = unidiff.PatchSet(r.raw, encoding="utf-8")
    check(patches)

def sophisticated_urlopen():
    req = urllib.request.Request(diffurl)
    req.add_header("Accept", "text/plain")
    with urllib.request.urlopen(req) as r:
        patches = unidiff.PatchSet(r, encoding="utf-8")
    check(patches)

for i in range(50):
#    sophisticated_requests()
    sophisticated_httpx()

as you can see, that implements various ways of getting the same diff file, feeding it to unidiff, and checking that it generates the expected result.

If you run the 50-iteration loop for naive, sophisticated_requests, or sophisticated_urlopen, they pass: they print OK 50 times. If you run it for sophisticated_httpx, it will produce some OKs but also fail in various interesting ways, documented in detail in the issue I reported to unidiff - it either misses some content, or crashes.

All three 'sophisticated' methods use the same amount of memory, indicating they all are actually streaming the content to unidiff. But urlopen and requests work reliably, while httpx doesn't. I'm not sure why this is.

[AdamWill]

This is with python3-httpx-0.28.1-9.fc43.noarch on Fedora.