Templatize concrete path keys from OpenAPI discovery#877
Open
fmhall wants to merge 1 commit into
Open
Conversation
Some providers (e.g. war-tracker.com) deliberately use concrete path
keys like `/api/v1/events/397003` instead of `/api/v1/events/{event_id}`
so their discovery probes hit a real URL. The canonical templates live
in the operation summary/description or in `info.x-guidance`.
We previously stored those concrete keys verbatim, producing resource
rows with real IDs in them.
Add a `templatizePath` utility that:
1. Returns already-templated URLs unchanged.
2. Mines the operation summary and doc-level guidance for path
patterns containing `{param}` placeholders, then matches them
against the concrete URL prefix to derive provider-preferred
parameter names (e.g. `{event_id}`, `{iso2}`).
3. Falls back to safe heuristics for unmatched concrete segments
(numeric / UUID / long hex → `{id}`; slug after `{id}` → `{slug}`).
Plumb a `canonicalUrl` field through `DiscoveredResource` and
`registerResource`. Use it for both DB upsert and stale-resource
deprecation comparison.
Request `GuidanceMode.Always` from the discovery package so the mining
step has access to the full guidance text regardless of length.
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Some OpenAPI providers (notably war-tracker.com) deliberately use concrete path keys like
/api/v1/events/397003instead of/api/v1/events/{event_id}so their discovery probes hit a real URL. Their canonical templates live in the operation'sdescriptionor ininfo.x-guidance. We previously stored those concrete keys verbatim, producing ugly resource rows with real IDs baked into them (see image in the kickoff message).What changed
templatizePath(url, hints)utility (apps/scan/src/lib/discovery/path-template.ts) with three-tier resolution:{...}segments are left alone.summary/descriptionand the doc-levelinfo.x-guidancefor templated path patterns. When one matches our concrete URL's prefix, we adopt the provider's own parameter names (e.g.{event_id},{iso2},{slug}).{id}; a segment immediately following an{id}→{slug}. Ambiguous segments (e.g.events,accounts) are left alone to avoid false positives.canonicalUrlthroughDiscoveredResource→registerResourcesFromDiscovery→registerResource. The DB storescanonicalUrlwhen present, falls back to the concrete probe URL otherwise.deprecateStaleResourcesnow compares against the same form that gets stored.fetchDiscoveryDocumentrequestsGuidanceMode.Alwaysso the mining step has the full guidance text regardless of length.Why this design
{event_id}and{iso2}— by mining their spec we adopt those names verbatim, rather than rewriting everything to our generic{id}/{code}.{param}templates pass through untouched./api/v1/widgets/123works out of the box; a provider with/api/v1/widgets/bluewon't getbluefalsely turned into{slug}.Examples (war-tracker)
/api/v1/events/397003/api/v1/events/{event_id}/share/397003/share/{event_id}/share/397003/strike/share/{event_id}/{slug}/media/397003/media/{event_id}/region/middle-east/region/{slug}/country/UA/country/{iso2}/event-type/drone-strike/event-type/{slug}Test plan
pnpm test:run— 115 tests pass (26 new fortemplatizePath)pnpm types:check— cleanpnpm lint— clean🤖 Generated with Claude Code