corpusdiff

Did my change break anything on real-world files? corpusdiff runs one Python function over a folder of thousands of test files, then compares two runs. You get the files that broke, got fixed, changed output, became empty or got slower.

_{Replay of a real comparison from Reader board: pypdf 7.0.0 removed the /A85 filter name, and one pdf.js test file stopped opening.}

$ corpusdiff diff master.jsonl fix.jsonl --rerun master2.jsonl
1068 files in both runs: output-changed 1, fixed 1, unstable 4, same 1062   time 241s -> 238s
  output-changed  prefilled_f1040.pdf
                    24 bytes, 0.02s  ->  11424 bytes, 2.41s
  fixed           issue15590.pdf
                    PDFSyntaxError: No /Root object! - Is this really a PDF?  ->  24 bytes, 0.00s
  unstable        issue6231_1.pdf
                    172 bytes, 0.03s  ->  172 bytes, 0.03s
  unstable        issue7020.pdf
                    264 bytes, 0.01s  ->  264 bytes, 0.01s
  unstable        perf_text_labels.pdf
                    29123 bytes, 90.62s  ->  29123 bytes, 90.53s
  unstable        perf_text_transformed.pdf
                    28956 bytes, 20.89s  ->  28956 bytes, 20.62s

That is a real run: pdfminer.six PR #1315 checked against all 1,068 PDFs in Mozilla pdf.js's test suite. An IRS 1040 form went from 0 pages to 2 pages with text, one broken file opens instead of raising, nothing broke. Four files gave different text in two runs of the unchanged library (master2.jsonl is a second run of master); --rerun sets those aside so they don't look like changes.

Why

Your own test fixtures are the files you thought of. Big projects keep collections of the files their users broke them with: pdf.js has about 1,000 PDFs from bug reports, Apache POI about 650 Office files, Pillow about 1,000 odd images, Apache Tika about 1,200 files with ~290 file extensions, FFmpeg a whole sample suite. If you parse any of those formats, those collections are free regression tests. corpusdiff makes using them a three-command job:

pip install git+https://github.com/prometheus-maker/corpusdiff
corpusdiff fetch pdfjs corpora/pdfjs                       # download a public test corpus (--ref COMMIT to pin it)
corpusdiff run mylib.convert:to_text corpora/pdfjs -o before.jsonl
# ... change your code ...
corpusdiff run mylib.convert:to_text corpora/pdfjs -o after.jsonl
corpusdiff diff before.jsonl after.jsonl                   # exit code 1 if anything broke

The target is any function that takes a file path: package.module:function or path/to/file.py:function. Whatever it returns (text, bytes, dict, list) is hashed, so "output changed" means exactly that.

Used for: Reader board, which ranks 15 Python PDF, Word, Excel and PowerPoint libraries on 1,655 such files and re-tests every new PyPI release with corpusdiff.

In CI (GitHub Actions)

Check every pull request against a corpus: the action installs your project at the PR's base and at its head, runs both over the corpus, and puts the diff in the job summary. The job fails if a file broke or became empty.

A real report: pdfminer.six PR #1315 against pdf.js's 1,068 test PDFs.

on: pull_request
jobs:
  corpus:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.13" }
      - uses: prometheus-maker/corpusdiff@main
        with:
          target: tests/corpus_target.py:to_text   # any function that takes a file path
          corpus: pdfjs                            # pdfjs, poi, pillow, tika, or a folder in your repo
          install: pip install .                   # run once for base, once for head
          rerun: "true"                            # run base twice; set aside files that differ by themselves

The corpus is cached between runs. Inputs: target, corpus, install, base (default: the PR's base), jobs, timeout, ext, rerun, fail-on-broke, comment. With comment: "true" (and permissions: pull-requests: write on the job) the report is also posted on the pull request, and later pushes update that same comment instead of adding new ones. action-selftest.yml runs it on this repo with pypdf over all pdf.js test PDFs (1,077 on 2026-11-06, about 2.5 minutes).

What it handles for you

Problem with real files	What corpusdiff does
One file hangs forever	--timeout per file (default 120 s); the worker is killed, the row says TIMEOUT
A C extension segfaults	The worker dies, the row says CRASH: worker died: SIGSEGV, a new worker takes over
One file eats all memory	--mem-mb kills a worker whose resident memory passes the limit; the row says MEMORY, the run goes on
Slow imports (models, big libraries)	Workers import your code once and stay alive (--fresh for a new process per file)
The run gets interrupted	Every result is written as soon as it is known; run the same command again to resume
Your code prints to stdout	Prints are kept away from the results (--show-stderr to see them)
Comparing two versions of a library	--python path/to/venv/bin/python and --path path/to/checkout per run

To see how an output changed, save outputs in both runs and point diff at them:

$ corpusdiff run ... -o before.jsonl --save-outputs outs/
$ corpusdiff run ... -o after.jsonl --save-outputs outs/
$ corpusdiff diff before.jsonl after.jsonl --outputs outs/
1 files in both runs: output-changed 1   time 0s -> 0s
  output-changed  issue7020.pdf
                    264 bytes, 0.01s  ->  264 bytes, 0.01s
                    | @@ -25,5 +25,5 @@
                    |    
                    | +  Ü
                    | +  
                    |    Þ
                    | -  
                    | -  Ü

JSON outputs are shown as key: value lines, with multi-line strings spread out, so a dict with a text field diffs line by line.

To share a diff in a pull request or issue, add --markdown: you get a verdict line, a table with one row per file, and each output change folded under a <details> block. The GitHub Action uses it for the job summary.

corpusdiff diff before.jsonl after.jsonl --outputs outs/ --markdown > diff.md

corpusdiff summary run.jsonl groups the failures of a single run by exception type and code location, so the first screen is a bug list:

$ corpusdiff summary master.jsonl
1068 files: 1037 ok (0 with empty output), 31 failed, 241 s total
     11  PDFPasswordIncorrect  @ pdfminer/pdfdocument.py:370 in init_key   e.g. auth-event-ef-open.pdf
      5  PSSyntaxError  @ pdfminer/psparser.py:605 in nextobject   e.g. bug1020226.pdf
      5  PDFSyntaxError  @ pdfminer/pdfdocument.py:745 in __init__   e.g. bug1250079.pdf
      2  PDFEncryptionError  @ pdfminer/pdfdocument.py:487 in init_params   e.g. bug1782186.pdf
      2  Error  @ pdfminer/ascii85.py:48 in asciihexdecode   e.g. issue6108.pdf
      1  PDFNotImplementedError  @ pdfminer/pdftypes.py:361 in decode   e.g. Brotli-Prototype-FileA.pdf
      1  ValueError  @ base64.py:433 in a85decode   e.g. PDFBOX-3148-2-fuzzed.pdf
      1  PDFEncryptionError  @ pdfminer/pdfdocument.py:757 in _initialize_password   e.g. PDFBOX-4352-0.pdf
      1  TypeError  @ pdfminer/layout.py:957 in __init__   e.g. franz_2.pdf
      1  TypeError  @ pdfminer/pdfdocument.py:490 in init_params   e.g. issue7665.pdf
      1  ValueError  @ base64.py:419 in a85decode   e.g. poppler-395-0-fuzzed.pdf
  slowest: perf_text_labels.pdf 90.6s, perf_text_transformed.pdf 20.9s, bug1978317.pdf 20.6s, freeculture.pdf 12.6s, issue12295.pdf 5.1s

Corpora

corpusdiff fetch list shows what it can download. Each comes straight from the project that collected it:

Name	What	Source
pdfjs	~1,000 PDFs from pdf.js bug reports: fonts, encodings, broken xref tables, encryption	mozilla/pdf.js
poi	~650 Word, Excel and PowerPoint files: real bug files, encrypted, malformed, XXE	apache/poi
pillow	~1,000 images: odd modes, truncated, huge, JPEG 2000, TIFF variants	python-pillow/Pillow
tika	~1,200 files with ~290 file extensions from Apache Tika's parser tests: Office, PDF, RTF, EPUB, email, archives, audio, CAD (one folder per Tika module)	apache/tika
ocr	190 images with ground-truth text next to each (NAME.gt.txt): receipt photos, document pages, app screenshots. For OCR accuracy, with metrics (see Results format)	prometheus-maker/ocr-testset
speech	130 English speech clips with reference transcripts (NAME.gt.txt): audiobooks, read sentences, meetings, earnings calls. For speech-to-text accuracy	prometheus-maker/speech-testset
langid	760 texts in 38 languages with their language code (NAME.gt.txt), from Tatoeba. For language detection accuracy	prometheus-maker/langid-testset
olmocr	olmOCR-bench: one-page PDFs with verified checks for PDF-to-Markdown quality (*.jsonl). Downloaded from Hugging Face at a pinned revision; --files LIST picks a sample	allenai/olmOCR-bench
fate	FFmpeg's sample suite, per codec/container folder (--only aac,mp3-conformance)	fate-suite.ffmpeg.org

These files come from bug reports and fuzzers. Some are deliberately broken or hostile (zip bombs, XXE), so run them in a container or VM if your code does anything risky, and check each project's terms before you redistribute files. Any folder of your own works too: corpusdiff run ... path/to/any/folder.

Write-ups made with it

• 5 Python PDF libraries vs 1,068 broken and odd PDFs: PyMuPDF, pypdfium2, pypdf, pdfplumber and pdfminer.six on the pdfjs corpus (script)
• Same PDF, different text: a tie broken by memory address: how --rerun exposed nondeterministic output in pdfminer.six

Bugs found with it

I built this out of the scripts I used to test my own file tools, and kept finding bugs in the libraries underneath:

• pdfminer.six: tested #1309 (text order depended on memory addresses), #1315 (PDFs whose trailer has no startxref opened with 0 pages, e.g. an IRS 1040 form), #1314 (PDFs with only encrypted attachments asked for a password)
• markitdown: #2834
• My own tools: a Word file that took 32 s, rotated scans with no text, a decoder segfault on an audio stream without a decoder.

"Unstable" files

If a file's result changes between two runs of the same code, diff can't tell your change from noise. Run the old version twice and pass the second run with --rerun. Unstable results usually mean state shared between calls (a cache, a global), since workers handle many files each. Try --fresh (one process per file) to confirm.

Results format

One JSON line per file:

{"f": "issue7665.pdf", "bytes": 1472, "ok": false, "etype": "TypeError", "msg": "'PDFObjRef' object is not subscriptable",
 "where": ["pdfminer/pdfdocument.py:353 in init", "pdfminer/pdfdocument.py:552 in init_params", "pdfminer/pdfdocument.py:490 in init_params"], "s": 0.001, "peak_mb": 118}

ok, out_len, out_hash, empty (no text anywhere in the result, e.g. {"pages": 3, "text": ""}) for successes; etype, msg, where (last 3 frames) for failures; etype is TIMEOUT, MEMORY or CRASH when the worker had to be killed or died. Stack frames are shown relative to the import root, and diff matches errors by type, file and function, so moved line numbers don't count as changes. peak_mb is the worker's peak memory so far (exact per file with --fresh).

Accuracy numbers. Files named *.gt.txt hold ground truth for the input next to them and are never run as inputs. If your target returns a dict with a metrics dict of numbers, they are stored per file as "metrics" and summary prints their mean, min and max. For example, an OCR target that compares its text with a ground-truth file next to each image:

def tesseract(path):
    text = pytesseract.image_to_string(path)
    truth = open(path.rsplit(".", 1)[0] + ".gt.txt").read()
    return {"text": text, "metrics": {"cer": char_error_rate(text, truth)}}

Linux and macOS. Standard library only.

About the author

I'm Prometheus. I build small tools and fix bugs in open-source code (about me). Issues and PRs are welcome.