Repository navigation
feat(api): add opt-in graph trace to the /rag response - #411
bosprimigenious wants to merge 3 commits into
Conversation
/rag only returned the final answer fields, so debugging retrieval meant parsing logs. The graph flow already computes keywords, match_vids, graph_result_flag, gremlin, graph_result and vertex_degree_list for /rag/graph; expose the same fields on /rag behind include_trace=false by default. - RAGRequest gains include_trace (opt-in, default false). - rag_answer() returns the trace dict as a fifth element only when asked, so the Gradio callers and the existing four-answer contract keep working. - /rag attaches the trace under "trace" and reuses the /rag/graph names. - Contract tests cover the default shape and the traced shape. Signed-off-by: bosprimigenious <bosprimigenious@foxmail.com>
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: include_trace=true always returns "trace": {}, because every RAG flow's post_deal strips the state down to the four answer fields before rag_answer builds the trace.
| if not include_trace: | ||
| return answers | ||
| # Same field names as the `/rag/graph` response so callers can compare. | ||
| trace = {key: res[key] for key in TRACE_FIELDS if key in res} |
There was a problem hiding this comment.
Important: res never holds a TRACE_FIELDS key, because post_deal in all four /rag flows returns only the four answer fields unless is_graph_rag_recall is set, so the trace is always {}. Have the flows return the trace fields when tracing is requested, and test through the real rag_answer instead of a mocked rag_answer_func.
There was a problem hiding this comment.
Thanks for catching this. At 99cdef7, the two graph flows return their full state when include_trace is set, and rag_answer() selects trace fields from that state. The new tests cover preparation/state propagation and post_deal; the trace API test still mocks rag_answer_func, so the real-rag_answer() regression you requested is not covered yet.
include_trace was always empty: the flows' post_deal strips the pipeline state down to the four answer fields, so rag_answer never saw keywords, match_vids, gremlin, graph_result or vertex_degree_list. - RAGGraphVectorFlow gains is_graph_rag_recall and, like RAGGraphOnlyFlow already did, returns the full state when it is set. - rag_answer passes include_trace through as is_graph_rag_recall, so the graph flows hand the trace back without changing the default contract. - Flow tests cover both flows with and without the flag. Signed-off-by: bosprimigenious <bosprimigenious@foxmail.com>
|
Good catch — you are right, the trace was always What changed
Tests — flow.post_deal(_Pipeline(payload)) # -> four answer fields
flow.post_deal(_Pipeline(payload | {is_graph_rag_recall: True})) # -> full stateThe |
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: Requesting a trace disables graph-only answer synthesis. Evidence: Traced GraphRecallCondition and the graph-only pipeline at this head.
| topk_per_keyword=topk_per_keyword, | ||
| # Graph-only and graph+vector flows return their full state when this | ||
| # is set, which is where the trace fields live. | ||
| is_graph_rag_recall=include_trace, |
There was a problem hiding this comment.
include_trace is true, this also enables recall-only mode in RAGGraphOnlyFlow: GraphRecallCondition.choose() selects the empty region instead of AnswerSynthesizeNode, so a normal graph-only /rag request returns an empty answer. Please use a separate flag for returning trace state and add a test that exercises the real graph-only flow with tracing enabled.
There was a problem hiding this comment.
At 99cdef7, tracing uses a separate include_trace flag. rag_answer() passes that flag without enabling recall-only mode, and GraphRecallCondition still checks only is_graph_rag_recall. The current tests do not execute the real graph-only pipeline with tracing, so that requested regression is still missing.
bitflicker64
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: With include_trace=true, the graph+vector flow still returns "trace": {} because the flag never reaches the workflow state. The graph-only answer suppression raised at this head on rag_block.py line 112 also still applies.
| return {"error": "No pipeline provided"} | ||
| res = pipeline.getGParamWithNoEmpty("wkflow_state").to_json() | ||
| log.info("RAGGraphVectorFlow post processing success") | ||
| if res.get("is_graph_rag_recall", False): |
There was a problem hiding this comment.
Important: prepare sets is_graph_rag_recall on prepared_input but leaves it out of data_json, so WkFlowState.is_graph_rag_recall stays None, this branch never runs, and graph_vector_answer=true with include_trace=true still returns "trace": {}. Add "is_graph_rag_recall": is_graph_rag_recall to prepared_input.data_json, as RAGGraphOnlyFlow.prepare does. test_graph_vector_post_deal_returns_trace_when_requested misses this because it injects the flag into the fake state.
There was a problem hiding this comment.
At 99cdef7, RAGGraphVectorFlow.prepare() writes include_trace into prepared_input.data_json, and post_deal() checks that separate flag. The new preparation test transfers this JSON into a real WkFlowState instead of injecting the flag directly into the fake post-processing state. Full pipeline execution is not covered by these tests.
Both review findings came from reusing is_graph_rag_recall: - it also switches RAGGraphOnlyFlow into recall-only mode, so a normal graph-only answer came back empty; - setting it on prepared_input never reached WkFlowState, because prepare() did not put it into data_json, so post_deal saw None and the trace stayed empty. Introduce a dedicated include_trace flag on WkFlowInput and WkFlowState, put it into data_json in both graph flows, and have post_deal return the full state when it is set. is_graph_rag_recall keeps its recall-only semantics untouched. Tests now prepare the real flows and assert that data_json -> WkFlowState carries the flag, instead of injecting it into a fake state. Signed-off-by: bosprimigenious <bosprimigenious@foxmail.com>
|
Both review findings are fixed in 99cdef7 — thanks, they were exactly right. 1. Separate flag instead of
2. The flag now reaches
3. Test gap you called out
Still not a live-cluster run — it is a unit-level check of the plumbing; the flow still needs a real HugeGraph instance to produce trace values end to end. |
imbajin
left a comment
There was a problem hiding this comment.
Blocking: yes. Summary: Generated Gremlin is dropped before it can appear in the requested trace. Evidence: Traced GraphQueryNode through BaseNode and WkFlowState at this head.
| "keywords", | ||
| "match_vids", | ||
| "graph_result_flag", | ||
| "gremlin", |
There was a problem hiding this comment.
GraphQueryNode puts the generated query in context["gremlin"], but BaseNode.run() copies results through WkFlowState.assign_from_json(), which ignores keys without a declared attribute; the state has no gremlin field. Consequently this trace field is always omitted in the real flow. Please retain and reset the query in workflow state and cover the real node-to-state transition in a test.
Closes #347.
What
/ragreturns only the requested answer fields, so retrieval debugging needs log parsing even though the graph flow already computes the intermediates that/rag/graphexposes. This adds an opt-intraceto/ragand leaves the default response unchanged.How
RAGRequest.include_trace: bool = False(opt-in, default false).rag_answer()gainedinclude_trace: bool = False. When true it returns the trace dict as a fifth element; when false it returns the same four-tuple as before, so the Gradio callers and existing callers are untouched./rag/graphfield names:keywords,match_vids,graph_result_flag,gremlin,graph_result,vertex_degree_list. Only fields present in the flow state are emitted./ragattaches it as"trace"and keeps the output-flag filtering for the answer fields.{ "query": "...", "graph_only": "...", "trace": {"keywords": [...], "match_vids": [...], "gremlin": "...", "graph_result": [...], ...} }Scope
Vector recall and rerank traces are intentionally left out, per the issue's suggested first version. This does not change
/rag/graph, the graph flow contract, or default responses.Tests
src/tests/api/test_rag_api.py:test_rag_default_response_has_no_trace_field— default response keeps its old shape.test_rag_include_trace_returns_graph_recall_fields—include_trace=truereturns the trace under"trace", keeps the answer-field filtering, and passes the flag through to the callback.(Environment note: run with proxy env vars unset; the sandbox's
NO_PROXYentry[::1]makes httpx raiseInvalidURLat import time.)