Repository navigation
feat(query): rank source above tests and docs, and answer file-first - #4982
rohit-jsfreaky wants to merge 1 commit into
Conversation
|
Thanks for the pull request, @rohit-jsfreaky. A maintainer will review it soon. Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions. A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic. |
|
Thanks @rohit-jsfreaky! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example: Prefixes we use: |
query scores node names only, so a test named after a bug (UsernameValidatorsTests) wins the exact-name tier, and the answer is ordered by hop and degree, so hub files fill the budget. On 177 SWE-bench Lite questions 43% of the start nodes were tests. Inside query only, test, doc, config and sourceless nodes now count x0.3 unless the question asks about them (reusing paths._is_test_path), and the answer is listed file by file, best file first. path, explain, traversal, the budget and the per-term seed guarantee are unchanged. Top-5 on 293 SWE-bench Lite and Verified questions: 49% -> 56% (26 gained, 4 lost). Co-Authored-By: Claude <noreply@anthropic.com>
86a5c7e to
9b2e076
Compare
There was a problem hiding this comment.
Graphify reviewed this change.
No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.
Not checked: tests were not run; no formal proof of the changed code.
No blocking issues. I didn't find anything I'd hold the merge for.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 0 functions depend on the 0 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).
Diff could not be reliably attributed to this PR: the base/head merge-base was unreachable in the shallow clone, so detailed findings were skipped (rather than emit a two-dot diff, which would misattribute base-branch changes).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Test selection
Test selection
0 of 502 test file(s) selected (0%) via static blast radius.
Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.
|
Landed in 0.9.102 (published to PyPI), cherry-picked with your authorship preserved. Fixes #4983. Thanks @rohit-jsfreaky. |
Closes #4983
What does this PR do?
I noticed
graphify querykept starting from tests and docs. A test named after a bug repeats the bug's words in its own name, so it wins the exact-name match, and the real source file ends up 6th or later or gets cut by the budget. On 177 SWE-bench Lite questions, 43% of the start nodes were tests.This PR changes
queryonly, in two ways:paths._is_test_path, plus atesting/folder andconftest.py.path,explain, traversal, depth and the token budget do not change. The new arguments default toNone, so every other caller runs the old code. The per-term seed guarantee still holds, because the weight is applied to the per-term score too. Nothing new is stored, sograph.jsonand the cache do not change.Diff:
graphify/serve.py+83 / -10,tests/test_query_source_first.py(9 new tests).Numbers
I measured with
graphify benchmark --questions(#4342) at the MCP defaults (BFS, depth 3, 2,000-token budget) on 293 SWE-bench Lite + Verified questions from 9 Python repos, each against the graph of its own base commit. "Top-5" means a file the real fix changed is in the first 5 files of the answer.Top-5 paired: 26 questions gained, 4 lost (McNemar p = 6e-5). I picked the 0.3 weight and the file-first order on the Django questions only; on the other 8 repos, which I did not tune on, it still goes up (Lite 49% to 56%, Verified 59% to 65%).
The cost: "anywhere" drops from 67% to 62%. In 28 of the 30 questions that lost it, v8 had shown the right file only 6th to 32nd, so an agent would rarely have read it anyway.
Data, scripts and the per-repo table: research/query/.
Overlap
Other open PRs try similar things: #2403 (exclude tests from seeds), #3417 (demote tests and vendor, sort by relevance), #4601 (file-type weights), #4481 (lead with the best file), and #3284 / #1782 / #1658 (relevance order). On the same 177 Lite questions the best of them reached 51% top-5. This PR also makes a small
serve.pyconflict with #3284, #3636, #3866, #4478, #4576 and #4601.Type of change
Verification & Invariants
Kept:
path/explainscoring is unchanged (test 6), the per-term seed guarantee still seeds every matched term (test 7), and seeds are still the first lines of the answer, so truncation keeps them. No persisted state changes.How was this tested?
Graphify-specific checklist
uv run python -m tools.skillgen --bless) when changing their source fragments. (Not needed: no skill fragments changed.)