Add embedding engine #3280

ptelang · 2026-01-13T23:15:10Z

Add MCPEmbedding CRD for embedding model deployment in operator

Introduces a new MCPEmbedding custom resource to deploy HuggingFace
embedding models as MCP servers in Kubernetes. This enables semantic
search and similarity features for MCP tools and resources.

Key additions:

MCPEmbedding CRD with model caching, resource management, and grouping
Controller managing deployment lifecycle and model initialization
Helm chart templates and CRD definitions
Example configurations for basic and advanced use cases
CRD API documentation updates

The embedding server uses huggingface-embedding-inference with persistent
volume caching for downloaded models and integrates with MCPGroups for
organizing backend workloads.

Large PR Justification

Multiple related changes that would break if separated

github-actions

Large PR Detected

This PR exceeds 1000 lines of changes and requires justification before it can be reviewed.

How to unblock this PR:

Add a section to your PR description with the following format:

## Large PR Justification

[Explain why this PR must be large, such as:]
- Generated code that cannot be split
- Large refactoring that must be atomic
- Multiple related changes that would break if separated
- Migration or data transformation

Alternative:

Consider splitting this PR into smaller, focused changes (< 1000 lines each) for easier review and reduced risk.

See our Contributing Guidelines for more details.

This review will be automatically dismissed once you add the justification section.

codecov · 2026-01-13T23:19:51Z

Codecov Report

❌ Patch coverage is 56.66328% with 426 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.91%. Comparing base (546f5cb) to head (79ae443).
⚠️ Report is 48 commits behind head on jerm/2026-01-13-optimizer-in-vmcp.

Files with missing lines	Patch %	Lines
...operator/controllers/embeddingserver_controller.go	45.68%	272 Missing and 43 partials ⚠️
...d/thv-operator/controllers/mcpserver_controller.go	13.79%	49 Missing and 1 partial ⚠️
pkg/authserver/server/handlers/callback.go	73.57%	28 Missing and 9 partials ⚠️
pkg/authserver/server/handlers/authorize.go	76.78%	11 Missing and 2 partials ⚠️
cmd/thv-operator/main.go	0.00%	8 Missing ⚠️
cmd/thv-operator/pkg/oidc/resolver.go	88.88%	1 Missing and 1 partial ⚠️
pkg/transport/http.go	83.33%	1 Missing ⚠️

Additional details and impacted files

@@                          Coverage Diff                          @@
##           jerm/2026-01-13-optimizer-in-vmcp    #3280      +/-   ##
=====================================================================
- Coverage                              64.82%   61.91%   -2.92%     
=====================================================================
  Files                                    375      401      +26     
  Lines                                  36626    39935    +3309     
=====================================================================
+ Hits                                   23744    24725     +981     
- Misses                                 11011    13250    +2239     
- Partials                                1871     1960      +89

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

therealnb

A few comments, but mostly looks good.

cmd/thv-operator/api/v1alpha1/embeddingserver_types.go

cmd/thv-operator/controllers/embeddingserver_controller_test.go

jerm-dro

This seems like it's on the right track, but I think we want an e2e test demonstrating:

defining embedding servers with the CRD results in the creation of a deployment, service, etc
we can call that service and get the expected results
when we delete the CRD, the other objects are also deleted.

Also, please modify the pull request so it's against main. With an e2e test, I'd prefer to merge this right into main. It'll be one fully functional and validated change, so no need to wait on merging into main.

examples/operator/embedding-servers/basic-embedding.yaml

examples/operator/embeddings/embedding-with-group.yaml

cmd/thv-operator/api/v1alpha1/mcpembedding_types.go

cmd/thv-operator/api/v1alpha1/embeddingserver_types.go

cmd/thv-operator/controllers/embeddingserver_controller.go

cmd/thv-operator/controllers/mcpembedding_controller.go

jerm-dro · 2026-01-14T23:17:02Z

cmd/thv-operator/controllers/embeddingserver_controller.go

+	if controllerutil.ContainsFinalizer(embedding, embeddingFinalizerName) {
+		return ctrl.Result{}, false, nil
+	}
+
+	controllerutil.AddFinalizer(embedding, embeddingFinalizerName)
+	err := r.Update(ctx, embedding)
+	if err != nil {
+		return ctrl.Result{}, true, err
+	}
+	return ctrl.Result{}, false, nil


nit: this only returns true if there's also an err and ctrl.Result is always ctrl.Result{}. I'd recommend removing the two and updating the caller:

err := r.ensureFinalizer(...) if err != nil { return ctrl.Result{} }

cmd/thv-operator/controllers/embeddingserver_controller.go

jerm-dro · 2026-01-14T23:58:43Z

This seems like it's on the right track, but I think we want an e2e test demonstrating:
...

You can crib off existing tests here: https://github.com/stacklok/toolhive/tree/main/test/e2e/thv-operator/virtualmcp

Putting these changes in the virtualmcp directory is fine with me, since we'll eventually be testing this when it's wired up to the gateway.

github-actions · 2026-01-16T14:44:06Z

✅ Large PR justification has been provided. The size review has been dismissed and this PR can now proceed with normal review.

Large PR justification has been provided. Thank you!

…erver

…containers

…ller

…unction

* Add header forward middleware for remote MCP servers Implement middleware that injects configured headers into requests before they are forwarded to remote MCP servers. This enables operators to configure headers server-side, removing the burden from clients. This commit adds the core middleware; subsequent commits add RunConfig types, CLI flags, runner wiring, and CRD support. Key design decisions: - RestrictedHeaders blocklist prevents misconfiguration of hop-by-hop, request smuggling, and identity spoofing headers - Authorization is allowed with a warning (valid for static tokens) - Header names are pre-canonicalized at creation time - Supports both factory pattern (thv run) and direct creation (thv proxy) - Header values are never logged, only names at DEBUG level Related: #3316 * Add Forwarded and Http2-Settings to restricted headers Add two headers to the RestrictedHeaders blocklist per review feedback: - Forwarded (RFC 7239): The standardized equivalent of X-Forwarded-* headers, which are already blocked. Omitting it left an identity spoofing gap. - Http2-Settings (RFC 7540 Section 3.2.1): A hop-by-hop header used in HTTP/1.1 to HTTP/2 upgrades. Forwarding it can cause protocol confusion and request smuggling. It is the companion to the already blocked Upgrade header.

Also add scenario for stop/restart/status endpoints for workloads.

#3425) authserver DCR hardening: Add grant_types and response_types allowlist validation Reject unsupported grant types and response types in DCR validation to prevent clients from requesting capabilities the server cannot fulfill (e.g. client_credentials, implicit token).

* refactor rbac Signed-off-by: Chris Burns <29541485+ChrisJBurns@users.noreply.github.com> * Refactor RBAC management to eliminate code duplication Introduce a common EnsureRBACResources method in pkg/kubernetes/rbac that consolidates the pattern of creating ServiceAccount, Role, and RoleBinding resources. This eliminates ~170 lines of duplicated code across controllers. Co-authored-by: Yolanda Robla <yolanda@stacklok.com> * add license headers * increase test coverage * changes from review * fix ci * changes from review * bump chart version --------- Signed-off-by: Chris Burns <29541485+ChrisJBurns@users.noreply.github.com> Co-authored-by: Chris Burns <29541485+ChrisJBurns@users.noreply.github.com> Co-authored-by: taskbot <taskbot@users.noreply.github.com>

* Add token endpoint handler Implement POST /oauth/token handler that exchanges authorization codes for access tokens using fosite's access request/response flow. The handler validates the incoming token request, retrieves the stored authorization session, generates JWT access tokens, and supports RFC 8707 resource parameter for audience-restricted tokens targeting specific MCP servers. This handler completes the OAuth 2.0 authorization code flow started by the authorize and callback handlers. When a client presents an authorization code, fosite retrieves the session that was stored during the callback phase - this session contains the user's subject, the upstream token session ID (tsid), and the client ID binding. The token endpoint uses these stored claims to generate the access token, maintaining the link between issued tokens and upstream IDP tokens for later token injection by the proxy middleware. The test infrastructure is extended to properly track authorization code and PKCE sessions across the full authorize→callback→token flow. * Add RFC 8707 audience validation for token endpoint Implement proper validation of the resource parameter in the token endpoint per RFC 8707. Previously, any client-provided resource was blindly granted as the token audience, which was a security risk. Changes: - Add ErrInvalidTarget error for RFC 8707 invalid_target responses - Add ValidateAudienceURI to validate URI format (absolute, no fragment, http/https only) - Add ValidateAudienceAllowed to check resources against an allowlist - Add AllowedAudiences config field to AuthorizationServerParams and AuthorizationServerConfig - Update TokenHandler to validate before granting audience - Secure default: empty AllowedAudiences rejects all resource requests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

github-actions bot added the size/XL Extra large PR: 1000+ lines changed label Jan 13, 2026

github-actions bot previously requested changes Jan 13, 2026

View reviewed changes

ptelang requested review from jerm-dro and therealnb January 13, 2026 23:15

ptelang marked this pull request as draft January 13, 2026 23:17

ptelang force-pushed the add-embedding-engine branch from 111676e to 3ba5cc0 Compare January 14, 2026 15:00

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 14, 2026

ptelang force-pushed the add-embedding-engine branch from 3ba5cc0 to bca112f Compare January 14, 2026 15:29

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 14, 2026

therealnb approved these changes Jan 14, 2026

View reviewed changes

cmd/thv-operator/api/v1alpha1/embeddingserver_types.go Show resolved Hide resolved

cmd/thv-operator/controllers/embeddingserver_controller_test.go Show resolved Hide resolved

ptelang force-pushed the add-embedding-engine branch from bca112f to e290799 Compare January 14, 2026 17:57

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 14, 2026

ptelang force-pushed the add-embedding-engine branch from e290799 to 2165ce8 Compare January 14, 2026 18:26

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 14, 2026

ptelang marked this pull request as ready for review January 14, 2026 19:46

jerm-dro requested changes Jan 14, 2026

View reviewed changes

ptelang force-pushed the add-embedding-engine branch from c88d4e5 to 4dc64c0 Compare January 16, 2026 14:43

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 16, 2026

ptelang force-pushed the add-embedding-engine branch from 2377edf to 9f438b8 Compare January 16, 2026 15:19

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 21, 2026

Merge branch 'main' into add-embedding-engine

60f052e

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 22, 2026

ptelang force-pushed the add-embedding-engine branch from 8982c8d to a465f3d Compare January 22, 2026 15:48

Bump toolhive-operator-crds chart version

47f3623

ptelang force-pushed the add-embedding-engine branch from a465f3d to 47f3623 Compare January 22, 2026 15:54

Update all places from deployment to statefulset in ref to embeddings…

5a8e464

…erver

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 23, 2026

ptelang added 2 commits January 22, 2026 21:04

Remove the unnecessary updateStatefulSetWithRetry function

de85d9d

Fix embedding server statefulset update detection to support sidecar …

56d4f9b

…containers

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 23, 2026

Refactored statefulSetNeedsUpdate function in embedding server contro…

9a5d19d

…ller

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 23, 2026

ptelang added 3 commits January 22, 2026 21:23

Removed left-over TODO comment

e558afd

Replaced conditional branches with an immediately-invoked anonymous f…

941537f

…unction

Removed unnecessary README.md files from test scenarios

79ae443

github-actions bot added size/XL Extra large PR: 1000+ lines changed and removed size/XL Extra large PR: 1000+ lines changed labels Jan 23, 2026

jhrozek and others added 6 commits January 23, 2026 15:23

Add E2E tests for group endpoints (#3402)

2d8da5d

Also add scenario for stop/restart/status endpoints for workloads.

Merge branch 'main' into add-embedding-engine

ff03438

ptelang force-pushed the add-embedding-engine branch from c2f99b1 to ff03438 Compare January 23, 2026 18:08

Add embedding engine #3280

Are you sure you want to change the base?

Add embedding engine #3280

Uh oh!

Conversation

ptelang commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Large PR Justification

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Large PR Detected

How to unblock this PR:

Alternative:

Uh oh!

codecov bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

therealnb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jerm-dro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jerm-dro Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jerm-dro commented Jan 14, 2026

Uh oh!

github-actions bot commented Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ptelang commented Jan 13, 2026 •

edited

Loading

codecov bot commented Jan 13, 2026 •

edited

Loading