Skip to content

Review agent should re-map inline comments on 422 stale-diff errors instead of falling back to review body #760

Open

Description

@fullsend-ai-retro

What happened

On PR fullsend-ai/fullsend#5996 (Aug 10, 19:05 UTC), fullsend-ai-review[bot] completed a review but received HTTP 422 from GitHub when posting inline comments because new commits had been pushed during the review window. The review was DISMISSED and 5 findings (including 1 HIGH: link-mark leak at *ast.Link) were dumped into the review body text, losing their inline file:line context. The HIGH finding was independently surfaced with correct inline positions by waynesun09's review sweep 47 minutes later at 19:52, but in the interim the finding lacked the actionable inline context that helps the PR author locate and fix the issue.

What could go better

The current fallback (posting findings in the review body) preserves information but loses the inline context that makes findings actionable — reviewers and authors must manually locate the relevant code. This is likely systemic: any PR with active development during the review window (typically 10–20 minutes for complex reviews) will encounter stale diff positions, especially during intense review-fix cycles. On this PR, the review agent ran 8 reviews over the course of Aug 10–11, during which 15+ commits were pushed; only one review hit the 422, but the pattern is structurally inevitable on active PRs. Confidence: medium — I can confirm the 422 and DISMISSED state from the API data, but I cannot inspect the post-review script internals to verify the exact failure mechanism or how much re-mapping effort would be required.

Proposed change

In the review agent's comment-posting mechanism (likely scripts/post-review.sh or equivalent in fullsend-ai/agents), when a 422 error occurs on inline comment submission: (1) re-fetch the current PR diff at the new head, (2) attempt to re-map original file:line positions using the file path and surrounding code context (a simple heuristic: search for the same code snippet within a few lines of the original position), (3) retry posting inline comments at the updated positions. If re-mapping fails for a specific comment, post it as a standalone PR comment with an explicit file:line reference (e.g., "adf.go:245 — [HIGH] link-mark leak...") rather than burying it in the review body. This preserves discoverability even when exact inline placement is impossible.

Validation criteria

On the next 5 PRs where the review agent encounters 422 errors during inline comment posting, at least 80% of findings should be posted either as inline comments at re-mapped positions or as standalone comments with explicit file:line references, rather than falling back to undifferentiated review body text.


Generated by retro agent from fullsend-ai/fullsend#5996

Activity

  1. fullsend-ai-prioritize commented on Aug 25, 2026

    @fullsend-ai-prioritize

    RICE Priority Score: 0.93

    Score breakdown
    Dimension Score Reasoning
    Reach 2 The review agent is a platform-wide tool used across all PRs. The issue is structurally inevitable on any PR with active development during the review window — the issue itself documents 8 reviews and 15+ commits on a single PR, with at least one 422. All users relying on inline review findings on active PRs are affected, scoring 2 (most active users across orgs).
    Impact 1 When this occurs, findings (including HIGH-severity ones) lose their inline file:line context and are buried in the review body text. Information is preserved but actionability is significantly degraded — authors must manually locate relevant code. A workaround exists (reading the body text) but it is inconvenient, making this a noticeable workflow improvement rather than a blocker. Score: 1 (medium).
    Confidence 0.7 The issue is well-described with a specific incident (PR #5996), concrete validation criteria (80% re-mapped on next 5 occurrences), and a clear proposed solution. The retro bot itself notes medium confidence on the exact failure mechanism and re-mapping effort required, and we lack visibility into the post-review script internals. Score: 0.7 (high but with some implementation uncertainty).
    Effort 1.5 Implementation requires: catching 422 errors in the comment-posting flow, re-fetching the current diff, building a heuristic to re-map file:line positions using code context/snippets, retrying inline comment posting, and implementing a better fallback (standalone comments with explicit file:line). The diff re-mapping heuristic is the main complexity — it touches multiple components and requires design work and testing across varied scenarios. Score: 1.5 (moderate).

    Formula: (2 x 1 x 0.7) / 1.5 = 0.93

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions