Repository navigation
Description
Issue
In the V2 (React) chat, an orchestrated request such as "compare these two files" over a table-heavy PDF and a CSV ends Partially completed. The last step ("Prepare the file comparison") is a compose step. It writes the answer as text and doesn't create a file. It fails with:
The operation did not produce the complete named results declared by the plan.
The retry card re-runs only that step, and it fails the same way.
Compose isn't the cause. The PDF preparation step (document_analyze) returns a partial result because its evidence check rejects correctly quoted passages over formatting differences. The compose inputs were planned with allow_partial: false, so compose refuses to run. The error message then blames compose.
This affects anyone who compares or analyzes documents whose extracted text contains tables or rich formatting. Document Intelligence Layout output stores tables as HTML and keeps line breaks.
Steps to Reproduce
- In V2 chat, select a narrative PDF that contains several tables, plus a CSV (for example, a 200-row dataset).
- Ask "compare these two files".
- Approve the plan: Analyze the PDF, run tabular analysis on the CSV, then compose the comparison.
- Step 1 shows Partially Completed, step 2 Completed and step 3 Failed. Retrying step 3 fails again.
Expected Behavior
- A finding whose quoted evidence is in the source is accepted when the quote differs from the stored chunk text only in formatting:
- whitespace and line breaks
- HTML table markup and entities
- markdown pipes and emphasis
- typographic quotes and dashes
- letter case
- Ordinary caveats and observations stay with the finding or slice and appear in the report, but don't block the result. Examples: "the excerpt doesn't state the currency", "the document says the data is synthetic".
- A step that prepares one source doesn't report the other source as missing. The comparison happens in compose.
- When a step can't run because an earlier step was partial, the message says so. The logs name the producing step and why it was partial.
Actual Behavior
This comes from one failing run, using the application logs plus the saved Analyze artifacts. The original run and its retry failed identically.
- Analyze read all 4 windows (20 chunks) and produced 35 candidate findings: 1 accepted, 34 unresolved.
- Recorded issues:
unmatched_evidence×203invalid_evidence_reference×20 (findings left with no located evidence)unresolved_finding×18window_requirement_unresolved×5
- All 226 quoted passages were re-checked against the indexed chunks:
| What the quote needed in order to match the chunk text | Passages |
|---|---|
| Nothing: exact substring (the only form accepted today) | 23 |
| Whitespace and line breaks collapsed | +6 |
| HTML table tags removed | +23 |
| Table pipes treated as cell separators | +174 |
| Not in the source at all | 0 |
| Ambiguous (matches more than one chunk) | 0 |
Every quote was genuinely in the source. The checker (collect_analysis_window_candidates) uses a literal str.find against the raw chunk text. Any table or line-break difference counts as unmatched, and one unmatched passage makes the whole finding unresolved.
- All 35 findings had
status: "supported". 18 of them also carried caveat-style text inissues, for example "the excerpt does not state the year, currency or units". Any issue text is treated as blocking. - There were 5 window-level issues. Any window issue marks the whole result partial.
- 2 said the CSV wasn't supplied, so the comparison couldn't be done. The comparison goal had leaked into the per-source Analyze step.
- 1 noted that the slice ended partway through a table.
- 2 were informational.
- Compose then raised
ResultNotReadyErroron its strict inputs before any model call. The executor reported it asresult_invalid. - No log records which producer was partial, the issue-code counts or the unresolved count. The window
page_rangedebug log also always printsNone:None, because it reads the wrong keys.
Impact
- Comparisons and analyses over table-heavy or richly formatted documents end partial or fail, even when the model's work is correct and fully grounded.
- Users see a misleading error on the compose step. Retrying doesn't help, because the retry reuses the same partial Analyze result.
- Admins can't diagnose the cause from the logs.
Notes
Proposed fix. The fix is general: nothing is keyed to specific files, names or values.
- Formatting-insensitive evidence matching in
functions_document_analysis_results.py.- Try an exact match first, so today's offsets don't change.
- Then try a normalized match that ignores whitespace, HTML tags and comments, entities, markdown table, emphasis, heading and list markup, and invisible characters. It also normalizes Unicode (NFKC) and typographic quote and dash variants.
- Then try a case-insensitive match.
- Keep an index map back to the original offsets, and require the match to start and end on word boundaries.
- Keep exactly-one-chunk uniqueness for each tier.
- When the model's chunk or page selector is wrong, fall back to searching the whole assigned window.
- Store the source's own text span (never the model's wording), the match tier and a matcher version.
- Keep all-or-nothing per finding: a passage that truly isn't in the source still leaves the finding unresolved.
- Non-blocking caveats and notes. Add optional finding-level
caveatsand window-levelnotesto theanalyze-final-v1window response. Caveats stay with accepted records, so compose sees them, and both show in the Analyze report.issuesis kept for requested values that the slice can't conclude. - Per-source scoping. Add planner mixed-source guidance and window-prompt guidance so that each preparation step extracts what its own sources contribute and doesn't report other named sources as missing. Compose does the comparison.
- Diagnostics only. Nothing changes in compose,
allow_partialor retry behavior. Add:- an accurate failure code and message when a strict input meets a partial producer
- a warning log naming the consumer and producer capabilities and the partial counts
- an Analyze validation summary log with finalized and unresolved counts, per-issue-code counts and match-tier counts
- the
page_rangelog fix
- Add functional tests and fix documentation, and bump the version.
Runs that already failed will still fail on retry, because the retry reuses the saved partial Analyze result. Ask again once the fix is deployed.
Out of scope: native mixed narrative/tabular Compare (#1059), changing compose's partial-input policy, and changing retry behavior.
Related: #1055, #1059, #1233, #1509.
Suspected areas:
functions_document_analysis_results.pyfunctions_document_analysis.py:_build_window_analysis_promptand validation loggingfunctions_orchestration_planner.py: mixed-source guidancefunctions_orchestration_result_runtime.pyandfunctions_orchestration_executor.py: partial-input failure mappingfunctions_appinsights.py: log allowlist keys
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsPending Evaluation
Activity
paullizer commentedon Sep 28, 2026
The fix is in review in #1558, into
paullizer-react-v2-ui, as v0.261.191.What changed
reason.caveats, and windows non-blockingnotes. Both appear in the report, andissuesstill block.input_partial_not_accepted, and the log names the producer. The new[DOCUMENT_ANALYSIS] Final findings validatedevent gives the counts behind a partial result, and thepage_rangedebug log is fixed. Compose,allow_partialand retry behavior are unchanged.Differences from the proposed fix
>50%and<50%.not_in_cited_location. Every citation in the failing run named the right chunk.Validation
issuesthe old prompt produced, and the new prompt addresses them in new runs.test_document_analysis_evidence_matching.py: 94 passed, also underpython -O. Related Analyze and orchestration suites: 291 passed.1,200inside1,200,000andThey caninsideThey can't. All are fixed, and each is a rejection test.Remaining