Skip to content

[Bug]: query prints per-term guaranteed seeds in alphabetical order, burying the answer file on multi-term queries #4357

Closed

Description

@devkancheti4-design

Problem

_pick_seeds guarantees one seed per query term (#1445) and appends those seeds in sorted(best_seed_by_term),
which is alphabetical term order. _subgraph_to_text prints seeds first. So on a multi-term query, a weak match on an
early-alphabet term is printed ahead of a specific term's exact match, and the answer file can fall below what an
agent reads first.

A minimal case: the query alpha bravo charlie apple zebra on a graph where all five are exact labels and eight
more labels contain apple. The start line is ['alpha', 'bravo', 'zebra', 'apple', 'charlie'], so the common term's
seed is printed before the rare exact match charlie.

Measured

I measured this on AST-only graphs (graphify update --no-cluster) with graphify's own _score_query, _pick_seeds
and _query_graph_text, on graphify 0.9.86 (v8 a5ba60d). hit@5 = the answer file is among the first 5 files
graphify query prints:

  • 114 Django issues from SWE-bench Lite; the answer is the file the official fix changed. Queried by the title plus
    the first 400 characters: hit@5 53. Yet a start node lies in the answer file on 83 of them.
  • In those misses the answer's seed usually sits at positions 6-10, after per-term seeds ordered alphabetically.

Proposal

Order the guaranteed seeds by the strength of each term's best match:

  • _score_query already ranks each term's winner with (-singleton score, -degree, label length, node id), and the
    singleton score carries the term's IDF weight.
  • _score_query would return that order (term_order), and _pick_seeds would iterate it, keeping the sorted
    order when term_order is not given.
  • The same seeds are picked, the traversal reaches the same nodes, and the order stays deterministic.
set hit@3 v8 / proposed hit@5 v8 / proposed
Django issues, title + 400 chars (114) 53 / 58 53 / 63
Django issues, title only (114) 46 / 48 54 / 55
full-stack-fastapi-template, 100 questions 29 / 33 36 / 37

Per question, hit@5 moved +10/-0, +4/-3 and +1/-0 on those three sets. The
evaluation script and data
reproduce every number above.

I have a PR ready, with tests that fail on v8 and pass with the change.

Activity

  1. github-actions commented on Oct 12, 2026

    @github-actions

    Thanks for opening this issue, @devkancheti4-design. A maintainer will take a look soon.

    If you would like to discuss it in real time, come say hi on our Discord server. For longer-form questions and ideas there is also GitHub Discussions.

    To help us triage, please make sure the report includes what you expected, what actually happened, and the steps (and a small sample) to reproduce it.

  2. safishamsi commented on Oct 12, 2026

    @safishamsi
    Member

    Fixed in v0.9.87 (graphifyy 0.9.87 on PyPI) via #4358.

  3. devkancheti4-design commented on Oct 12, 2026

    @devkancheti4-design
    ContributorAuthor

    Thank you @safishamsi for the quick fix and release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions