compress_identical_objects() followed by compress_content_streams() silently drops shared content stream parts #4111

Calling PdfWriter.compress_identical_objects() and then PageObject.compress_content_streams() on the same writer silently corrupts every page after the first one, whenever the pages have an array /Contents and deduplication merged one of the array elements across pages.

The merged element is written into the first page's flattened stream, and then released as a NullObject by replace_contents(). The remaining pages still reference it, so their flattened content silently loses that part. In practice the dropped part is often a leading q, which leaves the page with an unbalanced Q and a graphics state stack underflow.

Nothing is raised or logged. Tolerant viewers (Acrobat, macOS Preview) render the page as if nothing were wrong, so the corruption is invisible until the file reaches a stricter consumer. I hit this with a 50-page merged document that a print shop's copier refused to print at the first affected page, while every viewer I had showed the file as perfectly fine.

Environment

Which environment were you using when you encountered the problem?

$ python -m platform
macOS-26.6.2-arm64-arm-64bit-Mach-O

$ python -c "import pypdf;print(pypdf._debug_versions)"
pypdf==6.14.2, crypt_provider=('cryptography', '45.0.5'), PIL=12.2.0

Reproduced on three versions:

The main checkout includes #4088 (merged 2026-09-18), which is not part of 6.19.0 (released 2026-09-16). I verified that separately:

>>> import inspect, pypdf._page as m
>>> "old_contents = self.get(PG.CONTENTS, None)" in inspect.getsource(m.PageObject.replace_contents)
True   # on main
False  # on 6.19.0

Python 3.13.

Code + PDF

No input file is needed — the script builds its own source PDF. This is a minimal, complete example that shows the issue:

"""Minimal reproducer for pypdf: compress_identical_objects() followed by
compress_content_streams() drops content stream parts that deduplication made shared.
"""
import pypdf
from pypdf import PdfReader, PdfWriter
from pypdf.generic import ArrayObject, ContentStream, DecodedStreamObject, NameObject

SRC, OUT = "/tmp/source.pdf", "/tmp/output.pdf"


def make_source(path):
    """3 pages whose /Contents is an array of 3 streams:

        [0] b"q\\n"                     <- byte-identical on every page
        [1] b"<transform> <drawing>"    <- differs per page
        [2] b"Q\\n<page number>"        <- differs per page

    Splitting a page into "save state / draw / restore state + footer" streams is
    what some report generators emit. Only element [0] is identical across pages.
    """
    writer = PdfWriter()
    for i in range(1, 4):
        page = writer.add_blank_page(width=595, height=842)
        parts = [
            b"q\n",
            b"0.5 0 0 0.5 0 %d cm\n0 0 100 100 re f\n" % (400 + i),
            b"Q\n0 0 %d 10 re f\n" % (10 + i),
        ]
        refs = []
        for data in parts:
            stream = DecodedStreamObject()
            stream.set_data(data)
            refs.append(writer._add_object(stream))
        page[NameObject("/Contents")] = ArrayObject(refs)
    writer.write(path)


def report(path, label):
    reader = PdfReader(path)
    for i, page in enumerate(reader.pages, 1):
        contents = page["/Contents"].get_object()
        data = ContentStream(contents, reader).get_data()
        depth = low = 0
        for _, op in ContentStream(contents, reader).operations:
            if op == b"q":
                depth += 1
            elif op == b"Q":
                depth -= 1
                low = min(low, depth)
        status = "BROKEN: graphics state stack underflow" if low < 0 else "ok"
        print(f"  {label} page {i}: q/Q end={depth:+d} min={low:+d}  {status}")
        print(f"    {data[:34]!r}")


print(f"pypdf {pypdf.__version__}\n")

make_source(SRC)
print("source (untouched):")
report(SRC, "source")

writer = PdfWriter(clone_from=SRC)
writer.compress_identical_objects(remove_duplicates=True, remove_unreferenced=True)
for page in writer.pages:
    page.compress_content_streams()
writer.write(OUT)
print("\nafter compress_identical_objects() + compress_content_streams():")
report(OUT, "output")

Output (identical on 6.14.2, 6.19.0 and main, apart from the version line):

pypdf 6.19.0

source (untouched):
  source page 1: q/Q end=+0 min=+0  ok
    b'q\n0.5 0 0 0.5 0 401 cm\n0 0 100 100'
  source page 2: q/Q end=+0 min=+0  ok
    b'q\n0.5 0 0 0.5 0 402 cm\n0 0 100 100'
  source page 3: q/Q end=+0 min=+0  ok
    b'q\n0.5 0 0 0.5 0 403 cm\n0 0 100 100'

after compress_identical_objects() + compress_content_streams():
  output page 1: q/Q end=+0 min=+0  ok
    b'q\n0.5 0 0 0.5 0 401 cm\n0 0 100 100'
  output page 2: q/Q end=-1 min=-1  BROKEN: graphics state stack underflow
    b'0.5 0 0 0.5 0 402 cm\n0 0 100 100 r'
  output page 3: q/Q end=-1 min=-1  BROKEN: graphics state stack underflow
    b'0.5 0 0 0.5 0 403 cm\n0 0 100 100 r'

Page 1 keeps its leading q. Pages 2 and 3 lost it — exactly the 2 bytes of the deduplicated stream.

pdftoppm agrees on the /tmp/output.pdf the script just wrote:

$ for p in 1 2 3; do pdftoppm -r 50 -f $p -l $p -png /tmp/output.pdf /tmp/chk; done
Syntax Error: Restoring state when no valid states to pop
Syntax Error: Restoring state when no valid states to pop

(page 1 is silent; pages 2 and 3 each emit the error)

I am happy to provide a generated sample file if that is useful for the test suite, but the script above should make one unnecessary.

Traceback

This is the complete traceback I see:

(none — the corruption is silent; no exception, no warning, no log record)

Expected behavior

Either combination order should produce a valid document. Concretely, replace_contents() should not release an array element that other pages still reference, or compress_identical_objects() should not merge streams that are members of a /Contents array.

Analysis

Object identity before and after each step, using the reproducer above:

after PdfWriter(clone_from=...)      page 1: [4, 5, 6]   page 2: [8, 9, 10]   page 3: [12, 13, 14]
after compress_identical_objects()   page 1: [4, 5, 6]   page 2: [4, 9, 10]   page 3: [4, 13, 14]
                                                          ^ all three now share object 4

Then, for each page, compress_content_streams() reaches replace_contents() — get_contents() returns a fresh ContentStream without an indirect_reference, so the AttributeError fallback path is always taken — and replace_contents() nullifies every member of the old array:

# pypdf/_page.py, PageObject.replace_contents()
if isinstance(old_contents, ArrayObject):
    for reference in old_contents:
        ...
        writer._replace_object(indirect_reference=reference.indirect_reference, obj=NullObject())

Processing page 1 therefore sets object 4 to NullObject. Page 1 is unaffected because it was flattened before the release. Pages 2 and 3 are flattened afterwards, read [NullObject, ..., ...], and ContentStream skips the null — so the q is gone.

Notably, cloning does not create the sharing: it is introduced solely by compress_identical_objects().

Relation to existing issues

• BUG: Avoid sharing array-based content streams between pages #3681 describes this exact mechanism: "applying a transformation to one of the pages renders all pages referencing the shared objects invalid due to PageObject.replace_contents setting them to a NullObject without being able to reliably check for further usages." It fixed the problem by preventing the sharing at clone time (force-duplicating StreamObjects inside cloned arrays). compress_identical_objects() re-introduces exactly the sharing that BUG: Avoid sharing array-based content streams between pages #3681 removes, and there is no guard between the two.
• BUG: Using compress_identical_objects on transformed content duplicates differing content #3197 is the same family — compress_identical_objects() merging content that must stay distinct after add_transformation().
• BUG: Release content streams replaced by replace_contents() #4088 touched the same loop, making it run more reliably. That change is orthogonal to this report but increases how often the release path is taken.
• BUG: Fix compress_identical_objects handling of StreamObjects and rem… #3440 (open) proposes including StreamObjects in duplicate detection, which would make this sharing more common.

I could not find an existing report covering the compress_identical_objects() → compress_content_streams() combination specifically.

Test coverage

compress_identical_objects() and compress_content_streams() are each covered (tests/test_writer.py, tests/test_page.py, tests/test_workflows.py), but I could not find a test that calls both on the same writer. Each is correct in isolation, so the interaction is invisible to the current suite.

Workaround

Reversing the order avoids it, because flattening happens before any sharing exists:

for page in writer.pages:
    page.compress_content_streams()
writer.compress_identical_objects(remove_duplicates=True, remove_unreferenced=True)

On my real 6-page input this produced 240.3 KB versus 240.5 KB for the broken order, so nothing is lost by reordering.

Impact

The output opens fine in every viewer I tried, so the damage is easy to ship unnoticed. It surfaces only in strict consumers — in my case a copy shop's printer aborted the job at the first affected page of a 50-page document, with no indication of which page or why.