Skip to content

feat(api): add opt-in graph trace to the /rag response - #411

Open
bosprimigenious wants to merge 3 commits into
apache:mainfrom
bosprimigenious:feat/rag-include-trace
Open

bosprimigenious wants to merge 3 commits into
apache:mainfrom
bosprimigenious:feat/rag-include-trace

Conversation

@bosprimigenious

Copy link
Copy Markdown

Closes #347.

What

/rag returns only the requested answer fields, so retrieval debugging needs log parsing even though the graph flow already computes the intermediates that /rag/graph exposes. This adds an opt-in trace to /rag and leaves the default response unchanged.

How

  • RAGRequest.include_trace: bool = False (opt-in, default false).
  • rag_answer() gained include_trace: bool = False. When true it returns the trace dict as a fifth element; when false it returns the same four-tuple as before, so the Gradio callers and existing callers are untouched.
  • The trace reuses the /rag/graph field names: keywords, match_vids, graph_result_flag, gremlin, graph_result, vertex_degree_list. Only fields present in the flow state are emitted.
  • /rag attaches it as "trace" and keeps the output-flag filtering for the answer fields.
{
  "query": "...",
  "graph_only": "...",
  "trace": {"keywords": [...], "match_vids": [...], "gremlin": "...", "graph_result": [...], ...}
}

Scope

Vector recall and rerank traces are intentionally left out, per the issue's suggested first version. This does not change /rag/graph, the graph flow contract, or default responses.

Tests

src/tests/api/test_rag_api.py:

  • test_rag_default_response_has_no_trace_field — default response keeps its old shape.
  • test_rag_include_trace_returns_graph_recall_fields — include_trace=true returns the trace under "trace", keeps the answer-field filtering, and passes the flag through to the callback.
$ pytest src/tests/api/test_rag_api.py -q
21 passed

(Environment note: run with proxy env vars unset; the sandbox's NO_PROXY entry [::1] makes httpx raise InvalidURL at import time.)

/rag only returned the final answer fields, so debugging retrieval meant
parsing logs. The graph flow already computes keywords, match_vids,
graph_result_flag, gremlin, graph_result and vertex_degree_list for
/rag/graph; expose the same fields on /rag behind include_trace=false by
default.

- RAGRequest gains include_trace (opt-in, default false).
- rag_answer() returns the trace dict as a fifth element only when asked, so
  the Gradio callers and the existing four-answer contract keep working.
- /rag attaches the trace under "trace" and reuses the /rag/graph names.
- Contract tests cover the default shape and the traced shape.

Signed-off-by: bosprimigenious <bosprimigenious@foxmail.com>
@github-actions github-actions Bot added the llm label Nov 8, 2026

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: include_trace=true always returns "trace": {}, because every RAG flow's post_deal strips the state down to the four answer fields before rag_answer builds the trace.

if not include_trace:
return answers
# Same field names as the `/rag/graph` response so callers can compare.
trace = {key: res[key] for key in TRACE_FIELDS if key in res}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: res never holds a TRACE_FIELDS key, because post_deal in all four /rag flows returns only the four answer fields unless is_graph_rag_recall is set, so the trace is always {}. Have the flows return the trace fields when tracing is requested, and test through the real rag_answer instead of a mocked rag_answer_func.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. At 99cdef7, the two graph flows return their full state when include_trace is set, and rag_answer() selects trace fields from that state. The new tests cover preparation/state propagation and post_deal; the trace API test still mocks rag_answer_func, so the real-rag_answer() regression you requested is not covered yet.

include_trace was always empty: the flows' post_deal strips the pipeline
state down to the four answer fields, so rag_answer never saw keywords,
match_vids, gremlin, graph_result or vertex_degree_list.

- RAGGraphVectorFlow gains is_graph_rag_recall and, like RAGGraphOnlyFlow
  already did, returns the full state when it is set.
- rag_answer passes include_trace through as is_graph_rag_recall, so the
  graph flows hand the trace back without changing the default contract.
- Flow tests cover both flows with and without the flag.

Signed-off-by: bosprimigenious <bosprimigenious@foxmail.com>
@bosprimigenious

Copy link
Copy Markdown
Author

Good catch — you are right, the trace was always {}. Fixed in c270a36.

What changed

  • RAGGraphVectorFlow now accepts is_graph_rag_recall and, like RAGGraphOnlyFlow already did, returns the full wkflow_state from post_deal when it is set; the default path still returns exactly the four answer fields.
  • rag_answer() passes include_trace through as is_graph_rag_recall, so both graph flows hand the trace back and /rag can pick keywords / match_vids / graph_result_flag / gremlin / graph_result / vertex_degree_list from the real state.
  • Vector-only and raw flows keep their answer-only contract (they have no graph state to return); the trace is {} there, which is consistent with asking for a graph trace when no graph recall ran.

Tests — src/tests/flows/test_rag_flow_trace.py covers both flows with and without the flag:

flow.post_deal(_Pipeline(payload))                 # -> four answer fields
flow.post_deal(_Pipeline(payload | {is_graph_rag_recall: True}))  # -> full state
$ pytest src/tests/flows/test_rag_flow_trace.py src/tests/api/test_rag_api.py -q
25 passed

The /rag response shape itself is unchanged, so existing callers see no difference unless they ask for include_trace.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: Requesting a trace disables graph-only answer synthesis. Evidence: Traced GraphRecallCondition and the graph-only pipeline at this head.

topk_per_keyword=topk_per_keyword,
# Graph-only and graph+vector flows return their full state when this
# is set, which is where the trace fields live.
is_graph_rag_recall=include_trace,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ When include_trace is true, this also enables recall-only mode in RAGGraphOnlyFlow: GraphRecallCondition.choose() selects the empty region instead of AnswerSynthesizeNode, so a normal graph-only /rag request returns an empty answer. Please use a separate flag for returning trace state and add a test that exercises the real graph-only flow with tracing enabled.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At 99cdef7, tracing uses a separate include_trace flag. rag_answer() passes that flag without enabling recall-only mode, and GraphRecallCondition still checks only is_graph_rag_recall. The current tests do not execute the real graph-only pipeline with tracing, so that requested regression is still missing.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: With include_trace=true, the graph+vector flow still returns "trace": {} because the flag never reaches the workflow state. The graph-only answer suppression raised at this head on rag_block.py line 112 also still applies.

return {"error": "No pipeline provided"}
res = pipeline.getGParamWithNoEmpty("wkflow_state").to_json()
log.info("RAGGraphVectorFlow post processing success")
if res.get("is_graph_rag_recall", False):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important: prepare sets is_graph_rag_recall on prepared_input but leaves it out of data_json, so WkFlowState.is_graph_rag_recall stays None, this branch never runs, and graph_vector_answer=true with include_trace=true still returns "trace": {}. Add "is_graph_rag_recall": is_graph_rag_recall to prepared_input.data_json, as RAGGraphOnlyFlow.prepare does. test_graph_vector_post_deal_returns_trace_when_requested misses this because it injects the flag into the fake state.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At 99cdef7, RAGGraphVectorFlow.prepare() writes include_trace into prepared_input.data_json, and post_deal() checks that separate flag. The new preparation test transfers this JSON into a real WkFlowState instead of injecting the flag directly into the fake post-processing state. Full pipeline execution is not covered by these tests.

Both review findings came from reusing is_graph_rag_recall:

- it also switches RAGGraphOnlyFlow into recall-only mode, so a normal
  graph-only answer came back empty;
- setting it on prepared_input never reached WkFlowState, because prepare()
  did not put it into data_json, so post_deal saw None and the trace stayed
  empty.

Introduce a dedicated include_trace flag on WkFlowInput and WkFlowState, put it
into data_json in both graph flows, and have post_deal return the full state
when it is set. is_graph_rag_recall keeps its recall-only semantics untouched.

Tests now prepare the real flows and assert that data_json -> WkFlowState
carries the flag, instead of injecting it into a fake state.

Signed-off-by: bosprimigenious <bosprimigenious@foxmail.com>
@bosprimigenious

Copy link
Copy Markdown
Author

Both review findings are fixed in 99cdef7 — thanks, they were exactly right.

1. Separate flag instead of is_graph_rag_recall (@imbajin)

is_graph_rag_recall is untouched and keeps its recall-only semantics (GraphRecallCondition still switches on it). The trace now has its own include_trace flag:

  • WkFlowInput.include_trace / WkFlowState.include_trace
  • rag_answer() passes include_trace=include_trace to both graph flows
  • RAGGraphOnlyFlow.post_deal returns the full state when either is_graph_rag_recall or include_trace is set; a normal graph-only answer still goes through AnswerSynthesizeNode

2. The flag now reaches WkFlowState (@bitflicker64)

prepare() in RAGGraphOnlyFlow and RAGGraphVectorFlow writes "include_trace": include_trace into prepared_input.data_json, which WkFlowState.assign_from_json copies into the state. post_deal reads res.get("include_trace"), so graph_vector_answer=true + include_trace=true actually returns the trace.

3. Test gap you called out

test_rag_flow_trace.py no longer injects the flag into a fake state. It now:

  • runs the real prepare() on both flows and asserts prepared_input.data_json["include_trace"] is set, and that WkFlowState.assign_from_json() carries it into to_json() — this is the assertion that would have failed on the previous head;
  • keeps the post_deal contract tests (answers only by default, full state when tracing);
  • keeps a guard that is_graph_rag_recall still returns the full state for the recall path.
$ pytest src/tests/flows/test_rag_flow_trace.py src/tests/api/test_rag_api.py -q
28 passed

Still not a live-cluster run — it is a unit-level check of the plumbing; the flow still needs a real HugeGraph instance to produce trace values end to end.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: yes. Summary: Generated Gremlin is dropped before it can appear in the requested trace. Evidence: Traced GraphQueryNode through BaseNode and WkFlowState at this head.

"keywords",
"match_vids",
"graph_result_flag",
"gremlin",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ When Gremlin generation is enabled, GraphQueryNode puts the generated query in context["gremlin"], but BaseNode.run() copies results through WkFlowState.assign_from_json(), which ignores keys without a declared attribute; the state has no gremlin field. Consequently this trace field is always omitted in the real flow. Please retain and reset the query in workflow state and cover the real node-to-state transition in a test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Add optional trace output to the RAG answer API for retrieval debugging

3 participants