Skip to content

V2 chat: file comparisons end partial because Analyze rejects correctly quoted evidence over formatting #1540

Open

Description

@paullizer

Issue

In the V2 (React) chat, an orchestrated request such as "compare these two files" over a table-heavy PDF and a CSV ends Partially completed. The last step ("Prepare the file comparison") is a compose step. It writes the answer as text and doesn't create a file. It fails with:

The operation did not produce the complete named results declared by the plan.

The retry card re-runs only that step, and it fails the same way.

Compose isn't the cause. The PDF preparation step (document_analyze) returns a partial result because its evidence check rejects correctly quoted passages over formatting differences. The compose inputs were planned with allow_partial: false, so compose refuses to run. The error message then blames compose.

This affects anyone who compares or analyzes documents whose extracted text contains tables or rich formatting. Document Intelligence Layout output stores tables as HTML and keeps line breaks.

Steps to Reproduce

  1. In V2 chat, select a narrative PDF that contains several tables, plus a CSV (for example, a 200-row dataset).
  2. Ask "compare these two files".
  3. Approve the plan: Analyze the PDF, run tabular analysis on the CSV, then compose the comparison.
  4. Step 1 shows Partially Completed, step 2 Completed and step 3 Failed. Retrying step 3 fails again.

Expected Behavior

  • A finding whose quoted evidence is in the source is accepted when the quote differs from the stored chunk text only in formatting:
    • whitespace and line breaks
    • HTML table markup and entities
    • markdown pipes and emphasis
    • typographic quotes and dashes
    • letter case
  • Ordinary caveats and observations stay with the finding or slice and appear in the report, but don't block the result. Examples: "the excerpt doesn't state the currency", "the document says the data is synthetic".
  • A step that prepares one source doesn't report the other source as missing. The comparison happens in compose.
  • When a step can't run because an earlier step was partial, the message says so. The logs name the producing step and why it was partial.

Actual Behavior

This comes from one failing run, using the application logs plus the saved Analyze artifacts. The original run and its retry failed identically.

  • Analyze read all 4 windows (20 chunks) and produced 35 candidate findings: 1 accepted, 34 unresolved.
  • Recorded issues:
    • unmatched_evidence ×203
    • invalid_evidence_reference ×20 (findings left with no located evidence)
    • unresolved_finding ×18
    • window_requirement_unresolved ×5
  • All 226 quoted passages were re-checked against the indexed chunks:
What the quote needed in order to match the chunk text Passages
Nothing: exact substring (the only form accepted today) 23
Whitespace and line breaks collapsed +6
HTML table tags removed +23
Table pipes treated as cell separators +174
Not in the source at all 0
Ambiguous (matches more than one chunk) 0

Every quote was genuinely in the source. The checker (collect_analysis_window_candidates) uses a literal str.find against the raw chunk text. Any table or line-break difference counts as unmatched, and one unmatched passage makes the whole finding unresolved.

  • All 35 findings had status: "supported". 18 of them also carried caveat-style text in issues, for example "the excerpt does not state the year, currency or units". Any issue text is treated as blocking.
  • There were 5 window-level issues. Any window issue marks the whole result partial.
    • 2 said the CSV wasn't supplied, so the comparison couldn't be done. The comparison goal had leaked into the per-source Analyze step.
    • 1 noted that the slice ended partway through a table.
    • 2 were informational.
  • Compose then raised ResultNotReadyError on its strict inputs before any model call. The executor reported it as result_invalid.
  • No log records which producer was partial, the issue-code counts or the unresolved count. The window page_range debug log also always prints None:None, because it reads the wrong keys.

Impact

  • Comparisons and analyses over table-heavy or richly formatted documents end partial or fail, even when the model's work is correct and fully grounded.
  • Users see a misleading error on the compose step. Retrying doesn't help, because the retry reuses the same partial Analyze result.
  • Admins can't diagnose the cause from the logs.

Notes

Proposed fix. The fix is general: nothing is keyed to specific files, names or values.

  1. Formatting-insensitive evidence matching in functions_document_analysis_results.py.
    • Try an exact match first, so today's offsets don't change.
    • Then try a normalized match that ignores whitespace, HTML tags and comments, entities, markdown table, emphasis, heading and list markup, and invisible characters. It also normalizes Unicode (NFKC) and typographic quote and dash variants.
    • Then try a case-insensitive match.
    • Keep an index map back to the original offsets, and require the match to start and end on word boundaries.
    • Keep exactly-one-chunk uniqueness for each tier.
    • When the model's chunk or page selector is wrong, fall back to searching the whole assigned window.
    • Store the source's own text span (never the model's wording), the match tier and a matcher version.
    • Keep all-or-nothing per finding: a passage that truly isn't in the source still leaves the finding unresolved.
  2. Non-blocking caveats and notes. Add optional finding-level caveats and window-level notes to the analyze-final-v1 window response. Caveats stay with accepted records, so compose sees them, and both show in the Analyze report. issues is kept for requested values that the slice can't conclude.
  3. Per-source scoping. Add planner mixed-source guidance and window-prompt guidance so that each preparation step extracts what its own sources contribute and doesn't report other named sources as missing. Compose does the comparison.
  4. Diagnostics only. Nothing changes in compose, allow_partial or retry behavior. Add:
    • an accurate failure code and message when a strict input meets a partial producer
    • a warning log naming the consumer and producer capabilities and the partial counts
    • an Analyze validation summary log with finalized and unresolved counts, per-issue-code counts and match-tier counts
    • the page_range log fix
  5. Add functional tests and fix documentation, and bump the version.

Runs that already failed will still fail on retry, because the retry reuses the saved partial Analyze result. Ask again once the fix is deployed.

Out of scope: native mixed narrative/tabular Compare (#1059), changing compose's partial-input policy, and changing retry behavior.

Related: #1055, #1059, #1233, #1509.

Suspected areas:

  • functions_document_analysis_results.py
  • functions_document_analysis.py: _build_window_analysis_prompt and validation logging
  • functions_orchestration_planner.py: mixed-source guidance
  • functions_orchestration_result_runtime.py and functions_orchestration_executor.py: partial-input failure mapping
  • functions_appinsights.py: log allowlist keys

Activity

  1. added
    bugSomething isn't working
    P1Priority 1: urgent, address before P2 work
    on Sep 28, 2026
  2. paullizer commented on Sep 28, 2026

    @paullizer
    CollaboratorAuthor

    The fix is in review in #1558, into paullizer-react-v2-ui, as v0.261.191.

    What changed

    • Analyze locates quoted evidence through presentation differences: line breaks and repeated spaces, HTML and Markdown table markup, inline tags, entities, emphasis marks, invisible characters, typographic quotes and dashes, Unicode compatibility forms, and case. Words, numbers, signs and citations stay strict. A match must start and end on whole words and numbers and never end before an apostrophe suffix, saved evidence is the verbatim source span, and each unlocated passage records a diagnostic reason.
    • Findings can carry non-blocking caveats, and windows non-blocking notes. Both appear in the report, and issues still block.
    • Planner and window-prompt guidance scope each preparation step to its own source's contribution.
    • A strict input that meets a partial producer now fails as input_partial_not_accepted, and the log names the producer. The new [DOCUMENT_ANALYSIS] Final findings validated event gives the counts behind a partial result, and the page_range debug log is fixed. Compose, allow_partial and retry behavior are unchanged.

    Differences from the proposed fix

    • List, heading and bullet markers are compared as written instead of being ignored. Deleting them let changed symbols match, for example >50% and <50%.
    • There's no whole-window fallback for a wrong citation. Citations stay strict, and such a miss is counted as not_in_cited_location. Every citation in the failing run named the right chunk.
    • The exact tier keeps its substring behavior. Applying the boundary rules to it is a follow-up.

    Validation

    • Replaying the failing run's saved window responses locates 226 of 226 passages (23 exact, 203 normalized), up from 23. Unresolved candidates fell from 34 to 18 of 35. Those 18 carry blocking issues the old prompt produced, and the new prompt addresses them in new runs.
    • New test_document_analysis_evidence_matching.py: 94 passed, also under python -O. Related Analyze and orchestration suites: 291 passed.
    • Three independent reviews found false acceptances in earlier drafts, such as 1,200 inside 1,200,000 and They can inside They can't. All are fixed, and each is a rejection test.

    Remaining

    • A live comparison on a deployed environment. Runs that already failed must be asked again, because a retry reuses the saved partial result. The PR has a query for checking the result.
    • Follow-ups: exact-tier boundary rules, readable table evidence in the evidence panel, and notes in exports from a saved result.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P1Priority 1: urgent, address before P2 workbugSomething isn't working

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions