Skip to content
This repository was archived by the owner on Oct 16, 2026. It is now read-only.
This repository was archived by the owner on Oct 16, 2026. It is now read-only.

bug: LLM grader reports 'no response provided' when system prompt is present in input #982

Closed
#983
Closed
bug: LLM grader reports 'no response provided' when system prompt is present in input#982
#983
Labels
bugSomething isn't workingcoreAnything pertaining to core functionality of AgentV

Description

@christso
christso
opened on Apr 8, 2026

Bug Description

When running evals with a system prompt in the input messages, the LLM grader frequently reports "No response content was provided to evaluate" even when the model actually generated a correct response. This was discovered during benchmarking with with-superpowers vs without-superpowers experiments.

Reproduction

  1. Create an eval with a system message followed by a user message:
tests:
  - id: test-1
    input:
      - role: system
        content: "You are a helpful assistant that thinks step by step..."
      - role: user
        content: "Solve this logic puzzle..."
    assertions:
      - type: llm-grader
        prompt: "Check if the response correctly solves the puzzle"
  1. Run against multiple targets:
agentv eval test.EVAL.yaml --target azure --experiment with-system-prompt
agentv eval test.EVAL.yaml --target gemini --experiment with-system-prompt
  1. Observe that the LLM grader returns low scores with reasoning like "No response content was provided to evaluate" even when the model's actual response is correct.

Observed behavior

Experiment Target Actual Response Grader Score Grader Reasoning
without-superpowers gemini Correct (A=Knave, B=Knave) 1.0 Correct analysis
with-superpowers gemini Correct (A=Knave, B=Knave) 0.5 "No response text was provided"
without-superpowers azure Correct 0.97 Good solution
with-superpowers azure Correct 0.0 "No solution steps were included"

The contains evaluator confirms the correct content IS present in the response. The issue is specific to the LLM grader's prompt construction when system messages are in the input.

Expected behavior

The LLM grader should correctly receive and evaluate the model's response regardless of whether the input includes system messages.

Environment

Likely cause

The LLM grader prompt construction in packages/core/src/evaluation/evaluators/llm-grader.ts may not correctly handle multi-turn inputs with system messages when building the evaluation context for the judge model.

Activity

  1. christso
    added
    bugSomething isn't working
    coreAnything pertaining to core functionality of AgentV
    on Apr 8, 2026

  2. christso commented on Apr 8, 2026

    @christso
    christso
    on Apr 8, 2026
    CollaboratorAuthor

    Root Cause Analysis

    Investigated the codebase — found two related bugs in the system message handling pipeline:

    Bug 1: System message not extracted in buildPromptInputs

    File: packages/core/src/evaluation/formatting/prompt-builder.ts (lines 80-90)

    The buildPromptInputs function correctly includes system messages in the chatPrompt object, but never extracts them separately into the systemMessage field of the returned PromptInputs. The return statement is:

    return { question, chatPrompt }; // Missing: systemMessage

    Bug 2: System prompt passed in wrong location

    File: packages/core/src/evaluation/orchestrator.ts (lines ~1220 and ~2673)

    Even if systemMessage were populated, it's being placed in metadata instead of as a direct systemPrompt field:

    metadata: {
  systemPrompt: promptInputs.systemMessage ?? '', // always '' due to Bug 1
},

    This should be:

    systemPrompt: promptInputs.systemMessage,

    Effect Chain

    1. Eval YAML includes role: system message in input
    2. buildPromptInputs returns systemMessage: undefined
    3. Orchestrator passes metadata.systemPrompt: ''
    4. Provider invocation doesn't receive the system prompt
    5. Something in the response handling gets confused with multi-turn inputs
    6. LLM grader can't find the response content → "No response provided"

    Evidence

    From the benchmark results:

    • without-superpowers (no system message): Grader scores correctly (gemini 100%, azure 73.5%)
    • with-superpowers (has system message): Grader fails with "no response provided" even though the contains evaluator confirms correct content IS present

    Proposed Fix

    See the detailed code changes in the investigation notes. Two files need changes:

    1. prompt-builder.ts — extract system messages and return them in PromptInputs
    2. orchestrator.ts — pass systemPrompt as a direct field, not in metadata
  3. christso
    added
    in-progressClaimed by an agent — do not duplicate work
    on Apr 8, 2026
  4. christso
    added this to AgentV OSS Boardon Apr 8, 2026
  5. christso
    moved this to In progress in AgentV OSS Boardon Apr 8, 2026
  6. christso
    added a commit that references this issue on Apr 8, 2026

    fix(core): extract system messages in prompt builder for LLM grader

    4a5394d

  7. christso commented on Apr 8, 2026

    @christso
    christso
    on Apr 8, 2026
    CollaboratorAuthor

    Updated Root Cause (from E2E investigation)

    The initial analysis was incorrect. The bug is NOT caused by system message extraction. The real root cause is in resolveCustomPrompt() in packages/core/src/evaluation/evaluators/prompt-resolution.ts (lines 61-63):

    const promptValue = promptConfig.prompt;
if (typeof promptValue === 'string') {
  return promptValue;
}

    When an assertion has prompt: "Check if the response...", this returns the bare prompt text as evaluatorTemplateOverride, which completely replaces the DEFAULT_EVALUATOR_TEMPLATE. Since the bare prompt has no {{output}}, {{input}}, or {{criteria}} template variables, the grader never sees the question or response.

    Evidence

    • criteria: at test level → grader gets full template (662 input tokens) → score 1.0
    • prompt: in assertion → grader gets bare text only (124 input tokens) → score 0.0 ("no response provided")
    • This occurs with AND without system messages — system messages are irrelevant

    The with-superpowers correlation was a red herring

    The with-superpowers eval file happened to use prompt: in assertions while without-superpowers used criteria: at the test level. The failure was caused by the template replacement, not by system messages.

    What PR #983 actually does

    PR #983's changes (extracting systemMessage and passing systemPrompt on ProviderRequest) are a valid improvement but do NOT fix this bug. The PR needs to be revised to fix resolveCustomPrompt() instead.

  8. christso
    added a commit that references this issue on Apr 8, 2026

    fix(core): treat bare assertion prompt as criteria, not template over…

    056f8f4
  9. christso
    closed this as completedin #983on Apr 8, 2026
  10. christso
    removed
    in-progressClaimed by an agent — do not duplicate work
    on Apr 8, 2026
  11. christso
    moved this from In progress to In review in AgentV OSS Boardon Apr 8, 2026
  12. christso
    moved this from In review to Done in AgentV OSS Boardon Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcoreAnything pertaining to core functionality of AgentV

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions