Similarity Search/Semantic Search for GraphQL Schema exploration and discovery

[Keweiqu]

To reduce the size of the schema into LLM's context window, similarity search has been mentioned a couple times as a potential solution. Let's follow up on this in the next meeting

• assignee(s): @ThoreKoritzius @michael-watson @PascalSenn
• source:

---

Note: Action Item issues are reviewed and closed during Working Group
meetings.

[ThoreKoritzius]

My thought on an agentic tool for general GraphQL type discovery:

Tool with normal regex search might not be flexible enough when LLM has no prior knowledge about the schema -> BM25 maybe a little better but lacks semantic flexibility (e.g., fails on synonyms, multilingual inputs, or loose phrasing).

Approach:
Combine sparse (BM25) and dense (vector) retrieval over pre-computed type summaries.
Pre-index the schema into a fast-lookup structure that also captures graph connectivity (e.g., related types, root fields) to supply the LLM with a compact, context-fit subset.

Thoughts
For best approach vector search might be needed but would require external embedding models, complex and needs compute. We could propose and train a tiny vector embedding model domain specific for GraphQL types to run on CPU with light requirements so that we could score important types for the query and return parts of the graph related to it, but might need further evaluation which we could test against the new benchmark dataset.

General Simpler Thoughts:
Standardize schema naming (English, CamelCase, consistent verbs/nouns) so that lightweight lemmatization + sparse search performs better, leveraging the LLM’s built-in semantic bias toward common English terms, but less flexible to existent schemas

[Keweiqu]

Sounds like a great plan! We need the similarity search for both selection (including types), and for filtering (removing fields). e.g. the root type Query/Mutation will have a lot of fields, similarly any super connected types like 'Viewer' will have the same problem. So after we discovered relevant types, we need to trim it. I'm wondering if we need two separate representations?
One is for type: with summary like the fields, args, and return types.
One is for field: with summary like which type it belongs to, args, return type?

[ThoreKoritzius]

One could combine discovery with filtering. If discovery simply applies a scoring function on a flattened layer, we can select based on top_k or relevance thresholds. A quick coding example (using OAI embeddings) to test this with the benchmark dataset. Each flattened type->field combo (644 instances) is embedded and scored by cosine similarity.

query="I need to cancel part of my reservation and get a refund, can you help me with that?", top_k=5

CancellationPolicy
  refundableUntilHours [score=0.352, signature=Int!]
RoomType
  cancellationPolicy [score=0.348, signature=CancellationPolicy]
Hotel
  cancellationPolicy [score=0.347, signature=CancellationPolicy]
Refund
  payment [score=0.337, signature=Payment!]
Mutation
  cancelBooking [score=0.334, signature=Booking!]

query="What kinds of rooms does the Hilton Downtown have?", top_k=7

Hotel
  rooms [score=0.366, signature=RoomTypeConnection!]
  conferenceRooms [score=0.349, signature=[ConferenceRoom!]]
HotelFeatures
  CONFERENCE_HALL [score=0.347, signature=Boolean!]
  STEAM_ROOM [score=0.346, signature=Boolean!]
  MEETING_ROOMS [score=0.346, signature=Boolean!]
  GAME_ROOM [score=0.342, signature=Boolean!]
  BUSINESS_CENTER [score=0.341, signature=Boolean!]

However, this does not yet include relevant graph connections constrains in the lookup, just matching fields based on the query. For example, "Hotel->rooms" is of type RoomTypeConnection, but LLM doesn't know that type. If we constrain the rest of the results to the previous signature and match recursively with breadth-first expansion we would constrain based on similarity and type dependencies together until we finish the top_k window

Hotel
  rooms [score=0.366, signature=RoomTypeConnection!] //constrain on RoomTypeConnection next
RoomTypeConnection
  totalCount [score=0.236, signature=Int!]
  edges [score=0.235, signature=[RoomTypeEdge!]!]  //constrain on RoomTypeEdge next
RoomTypeEdge
  node [score=0.252, signature=RoomType!]  //constrain on RoomType next
RoomType
  id [score=0.301, signature=ID!]
  name [score=0.300, signature=String!]
  description [score=0.292, signature=String]

So, we could recursively constrain the results to improve relevant field results, to expand based on the relevant fields. The early testing script for this can be found here Schema Discovery Experiments