[Models] Pluggable model routing + multi-backend addressing (resolves #510 fallback decision)

[heskew]

Context

Once custom backends can be registered (see the backend-registration issue, #1325), an app can have any number of backends live at once — the registry is Map<logicalName, backend> per kind, not one-remote-one-local. What's missing is (a) a clean addressing scheme for picking among many, and (b) a way to make the routing decision pluggable so a separate plugin can do cost-, latency-, capability-, or tenant-aware selection and fallback — without baking policy into core.

This also resolves #510 open decision #2 (fallback chains: "declarative-in-config vs imperative"). The answer is neither baked into core: fallback, A/B, canary, and tenant routing all become one extension point.

Addressing — how apps refer to many backends

Today opts.model: string is a flat logical-name lookup (default 'default'), and embedding / generative are already separate namespaces. Keep that app-facing surface; layer a two-level scheme under it:

• Physical backends get provider-namespaced ids — openai:gpt-4o, ollama:llama3.2:3b, local:bge-small. The id is the registry key; namespacing avoids collisions when multiple plugins register.
• Logical roles are operator-defined aliases — default, fast, accurate, cheap — mapping to one or an ordered set of backend ids.
• opts.model accepts either a role or a concrete id. Apps speak intent ({ model: 'fast' }); ops/router own the physical mapping.

The shift that makes mixing first-class: an alias resolves to a group (ordered list), not necessarily a single backend.

Routing — the pluggable seam

Resolution today is a dumb Map.get(opts.model) at one call site each in Models.embed/generate/generateStream. That single seam is where routing plugs in:

interface ModelRouter {
  route(req: {
    kind: 'embedding' | 'generative';
    logicalName?: string;            // opts.model — a role or a concrete id
    requires?: Capability[];         // 'embed' | 'generate' | 'stream' | 'tools' | 'adapters'
    hints?: { tenant?: string; app?: string; promptTokens?: number; [k: string]: unknown };
  }): ModelBackend[];                // ordered: [primary, ...fallbacks]
}

function registerRouter(router: ModelRouter): void;  // replaces the default map-lookup router

Models loops the returned candidates, recording one hdb_model_calls row per attempt (the success / error_code columns already exist), so a fallback reads as "attempt 1 failed on A, attempt 2 succeeded on B". The router receives the call context Models already assembles (tenant, handlerPath, requested capabilities, signal), so it can route on cost, latency, tenant, prompt size, or required capabilities.

Two implementation paths

• Interim — router-as-backend. A routing plugin registers a normal ModelBackend (via the registration issue) that internally picks and delegates. Needs no new core surface; composable. Caveat: avoid double-counting analytics — either delegate through models.generate({ model: realId }) with a delegating: true capability bit so core skips the outer row, or delegate straight to the registry. (Same "outer call abstains, inner calls record" pattern already used by the toolMode: 'auto' loop in Models.ts.)
• Target — first-class router seam. The ModelRouter interface above, with the default router preserving exact current behavior. This is where ordered fallback, capability negotiation, and per-attempt billing live cleanly in core.

Config (additive superset)

Single-backend entries are unchanged; a fallback: list opts into ordered fallback the default router walks:

models:
  generative:
    default:
      backend: openai
      model: gpt-4o
      apiKey: ${OPENAI_API_KEY}
      fallback: [anthropic:claude, local:llama]   # NEW

New optional opts field (default router ignores it):

type GenerateOpts = { model?: string; requires?: Capability[]; /* …all existing… */ };

App call sites never change: models.generate(input, { model: 'fast' }).

Backward compatibility

Additive only — new symbols (registerRouter, ModelRouter) and optional fields (opts.requires, config fallback:, capabilities().delegating?). Namespaced ids are new valid values for opts.model; flat names keep resolving. Alias→group is a superset of single-backend config.

Regression guard (must hold): today Models.embed/generate throw ModelBackendNotFoundError synchronously, and generateStream runs its capability check synchronously before returning the iterable. The default router must preserve that exact behavior — same error class, same sync timing — or callers relying on the throw point break. The default router stays synchronous (a thin wrapper over the current resolve*() calls).

Acceptance

• opts.model resolves both role aliases and provider-namespaced backend ids; flat names still work.
• A role alias can map to an ordered group of backend ids.
• registerRouter(router) overrides selection; the default router reproduces current 5.1.0 behavior exactly (incl. synchronous ModelBackendNotFoundError and generateStream sync capability check — explicit regression test).
• Ordered fallback works and records one hdb_model_calls row per attempt.
• A routing plugin can route on requested capabilities / hints (cost, tenant, prompt size).
• Add unified model-access API (scope.models) #510 open decision [WIP] Enable Node.js Type Stripping #2 (fallback) is resolved by this mechanism; the decision is closed out on Add unified model-access API (scope.models) #510.
• No change to existing public surface.

Relationship

• Part of Add unified model-access API (scope.models) #510; resolves its open decision [WIP] Enable Node.js Type Stripping #2.
• Builds on the backend-registration issue ([Models] Public backend registration API — registerBackend + defineBackend (custom/in-process backends) #1325) — the router-as-backend interim path needs only that.

---

🤖 Generated with Claude Code