Skip to content

feat(query): rank source above tests and docs, and answer file-first - #4982

Closed
rohit-jsfreaky wants to merge 1 commit into
Graphify-Labs:v8from
rohit-jsfreaky:query-source-first
Closed

rohit-jsfreaky wants to merge 1 commit into
Graphify-Labs:v8from
rohit-jsfreaky:query-source-first

Conversation

@rohit-jsfreaky

@rohit-jsfreaky rohit-jsfreaky commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Closes #4983

What does this PR do?

I noticed graphify query kept starting from tests and docs. A test named after a bug repeats the bug's words in its own name, so it wins the exact-name match, and the real source file ends up 6th or later or gets cut by the budget. On 177 SWE-bench Lite questions, 43% of the start nodes were tests.

This PR changes query only, in two ways:

  1. Tests, docs, config and nodes with no source file count x0.3 when picking seeds. If the question mentions tests ("test", "tests", "testing", "testcase", "unittest") or docs ("doc", "docs", "documentation", "readme"), those keep full weight. Tests are detected with the existing paths._is_test_path, plus a testing/ folder and conftest.py.
  2. The answer is listed file by file, best-scoring file first. Nodes the question did not match keep the old order after them.

path, explain, traversal, depth and the token budget do not change. The new arguments default to None, so every other caller runs the old code. The per-term seed guarantee still holds, because the weight is applied to the per-term score too. Nothing new is stored, so graph.json and the cache do not change.

Diff: graphify/serve.py +83 / -10, tests/test_query_source_first.py (9 new tests).

Numbers

I measured with graphify benchmark --questions (#4342) at the MCP defaults (BFS, depth 3, 2,000-token budget) on 293 SWE-bench Lite + Verified questions from 9 Python repos, each against the graph of its own base commit. "Top-5" means a file the real fix changed is in the first 5 files of the answer.

top-1 top-3 top-5 anywhere median tokens
v8 27% 40% 49% 67% 2,193
this PR 30% 48% 56% 62% 2,197

Top-5 paired: 26 questions gained, 4 lost (McNemar p = 6e-5). I picked the 0.3 weight and the file-first order on the Django questions only; on the other 8 repos, which I did not tune on, it still goes up (Lite 49% to 56%, Verified 59% to 65%).

The cost: "anywhere" drops from 67% to 62%. In 28 of the 30 questions that lost it, v8 had shown the right file only 6th to 32nd, so an agent would rarely have read it anyway.

Data, scripts and the per-repo table: research/query/.

Overlap

Other open PRs try similar things: #2403 (exclude tests from seeds), #3417 (demote tests and vendor, sort by relevance), #4601 (file-type weights), #4481 (lead with the best file), and #3284 / #1782 / #1658 (relevance order). On the same 177 Lite questions the best of them reached 51% top-5. This PR also makes a small serve.py conflict with #3284, #3636, #3866, #4478, #4576 and #4601.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Tests or CI
  • Refactor
  • Security fix

Verification & Invariants

Kept: path / explain scoring is unchanged (test 6), the per-term seed guarantee still seeds every matched term (test 7), and seeds are still the first lines of the answer, so truncation keeps them. No persisted state changes.

  • Read the CONTRIBUTING.md guide.
  • Reproduced the issue and identified the invariant.
  • Made the smallest fix necessary.
  • Added a regression test (if bug fix) or isolated boundary test.
  • Kept the PR description synchronized with the final implementation.
  • Documented any limitations / unsupported cases explicitly.

How was this tested?

uv run --frozen pytest -q tests/test_query_source_first.py      # 9 passed
uv run --frozen pytest -q                                       # 9,551 passed; 11 fail/error in test_language_override_lifecycle.py
                                                                # and test_graph_invariance.py (watch/rebuild timing). Run alone 3 times:
                                                                # 3/3 pass on this branch, plain v8 failed 1 of 3, so flaky on v8 too
uv run --frozen ruff check graphify/serve.py tests/test_query_source_first.py   # clean
uv run --frozen pyright graphify/serve.py tests/test_query_source_first.py      # 67 errors, the same 67 as v8
uv run --frozen bandit -q -ll graphify/serve.py                 # same 3 B104 findings as v8
graphify benchmark --questions <SWE-bench questions>            # numbers above

Graphify-specific checklist

  • I updated generated skill artifacts (uv run python -m tools.skillgen --bless) when changing their source fragments. (Not needed: no skill fragments changed.)
  • I confirmed that AST/structural extraction remains deterministic (no ambient state dependencies like ENV variables).
  • I reviewed changes for security implications (no unsafe interpolation into shell/Python).
  • I confirmed no API keys or local-only graph data are included.
  • (If applicable) I disclosed AI authorship in my commit messages.

@github-actions github-actions Bot added the contributor Pull request from a returning community contributor label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Thanks for the pull request, @rohit-jsfreaky. A maintainer will review it soon.

Want to talk it through while it is in review? Come join us on our Discord server. For longer-form discussion there is also GitHub Discussions.

A couple of things that speed up review: make sure the test suite passes on Python 3.10 and 3.13, and that the change keeps extraction deterministic.

@github-actions github-actions Bot added the needs:title PR title should follow the conventional-commit format label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown

Thanks @rohit-jsfreaky! One small thing: our changelog and release notes are built from PR titles, so we use the conventional-commits format, for example:

fix(python): resolve a bare stdlib import correctly

Prefixes we use: feat, fix, docs, chore, refactor, test, perf, build, ci. Editing the PR title clears this automatically. This is just a nudge and does not block review.

query scores node names only, so a test named after a bug (UsernameValidatorsTests) wins the
exact-name tier, and the answer is ordered by hop and degree, so hub files fill the budget.
On 177 SWE-bench Lite questions 43% of the start nodes were tests.

Inside query only, test, doc, config and sourceless nodes now count x0.3 unless the question
asks about them (reusing paths._is_test_path), and the answer is listed file by file, best
file first. path, explain, traversal, the budget and the per-term seed guarantee are
unchanged. Top-5 on 293 SWE-bench Lite and Verified questions: 49% -> 56% (26 gained, 4 lost).

Co-Authored-By: Claude <noreply@anthropic.com>
@rohit-jsfreaky rohit-jsfreaky changed the title query: rank source above tests and docs, and answer file-first feat(query): rank source above tests and docs, and answer file-first Nov 3, 2026
@github-actions github-actions Bot removed the needs:title PR title should follow the conventional-commit format label Nov 3, 2026

@graphify-labs graphify-labs Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Graphify reviewed this change.

No issues found by static checks — no coupling regressions or blocking issues in the code graph. That is not the same as safe to merge: see what was not checked.

Not checked: tests were not run; no formal proof of the changed code.


No blocking issues. I didn't find anything I'd hold the merge for.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 0 functions depend on the 0 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 0 functions in the blast radius were not formally verified this run (proofs are advisory here).

Diff could not be reliably attributed to this PR: the base/head merge-base was unreachable in the shallow clone, so detailed findings were skipped (rather than emit a two-dot diff, which would misattribute base-branch changes).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Test selection

Test selection

0 of 502 test file(s) selected (0%) via static blast radius.

Selection is safe under the controlled-regression assumption; always-run tests + a periodic full run are the backstops. Advisory — it never changes the check verdict.

@safishamsi

Copy link
Copy Markdown
Member

Landed in 0.9.102 (published to PyPI), cherry-picked with your authorship preserved. Fixes #4983. Thanks @rohit-jsfreaky.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor Pull request from a returning community contributor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: query starts from tests and docs instead of the code the question is about

2 participants