Skip to content

fix: serialize per-server tool-approval writes so baselines and approvals are complete (UX-02) - #1555

Closed
Dumbris wants to merge 21 commits into
mainfrom
fix/qa070-ux02-approval-serialization
Closed

Dumbris wants to merge 21 commits into
mainfrom
fix/qa070-ux02-approval-serialization

Conversation

@Dumbris

@Dumbris Dumbris commented Oct 10, 2026

Copy link
Copy Markdown
Member

Summary

Fixes UX-02 from the 0.70.0 UX QA audit. A stable toolset sometimes stayed partly pending after a trusted-server baseline or after an approval, while the approval reported success.

Root cause (shown by deterministic tests and live debug logs): several writers changed one server's tool-approval records at the same time, with no mutual exclusion. Each did read-modify-write on whole records. The writers were the discovery passes (reactive connect, boot sweep, post-reload sweep, HandleUpstreamServerChange, approval reindex, quarantined capture), operator approve/block/toggle, the server-approval baseline promotion, and the scanner's baseline+blocks commit. When two overlapped, a stale pass overwrote a fresher record:

  • A non-baseline pass filed tools as pending over the approvals a baseline pass had just written.
  • A pass's "stays pending" write-back reverted an operator approval.

The stranded records then survived hot reload and restart, because no later pass is a baseline pass.

Changes

Runtime: per-server serialization (internal/runtime/tool_approval_lock.go, tool_quarantine.go, lifecycle.go)

Each server gets its own sync.Mutex (in a sync.Map). It is never global, so one large server cannot stall approvals on other servers. The lock covers:

  • The whole checkToolApprovals pass: the gate read, the prior-record snapshot, the isBaselinePass decision and every write. The pass also reads the storage quarantine flag and treats the server as quarantined if either source says so. This fails closed: a storage record that cannot be read or decoded counts as quarantined, and only a missing record defers to the config. QuarantineServer flips storage under the lock before the runtime config catches up, so a server quarantined mid-flight is never auto-baselined.
  • ApproveTools, ApproveAllTools, BlockTools, BlockAllTools, tool toggles and removed-tool deletes.
  • Server-approval baseline promotion. approveToolsLocked re-reads every record under the lock. In pending-only mode it never promotes a record that has become changed, which keeps the rug-pull guarantee.
  • QuarantineServer's storage flip, and only that flip. The lock is never held across SaveConfiguration, LoadConfiguredServers, HandleUpstreamServerChange or UnquarantineServer.
  • The scanner's atomic baseline+blocks commit, through a new optional scanner.ToolApprovalLocker capability on the unquarantiner adapter. This stops a pass holding a stale pending copy from overwriting an operator block.

SSE emits and the reindex goroutine run after the lock is released.

Runtime and storage: keep lock hold time short

  • storage.SaveToolApprovals writes many records in one bbolt transaction. An approval reads the named records in one snapshot and commits them in one transaction. The Spec 105 stamping rule is applied per record, as saveReadToolApproval did.
  • A pending record whose definition has not changed is no longer rewritten on every pass. Before, each pass cost one fsynced transaction per tool.

Effect: end-to-end CLI approval of a 180-tool server takes about 2–7s on a loaded host, against 5.4–6.3s on base and about 20s with the lock but without these two changes.

API and CLI: report what actually applied

Fields are additive. status, server_name and approved keep their meaning.

  • POST /servers/{id}/security/approve and POST /servers/{id}/tools/approve read the records back after the write and return approved_count, blocked_count, still_pending, still_changed and held_tools (capped at 50).
  • tools/approve's approved now counts requested tools whose record actually reads approved, and the rest are listed in not_approved. Before, it echoed the request length.
  • mcpproxy review approve and mcpproxy security approve print the counts in table output. In every output format they print a warning to stderr naming any tool still held.
  • security approve gets the same 30s budget as review approve, since an approval can now wait for an in-flight pass.
  • Docs: docs/api/rest-api.md and docs/cli/review-commands.md.

Compatibility decisions

  • status stays "approved" when the server approval succeeds. Remaining holds are reported through the new counts, not through a new status value, so the Web UI, macOS tray and CLI JSON consumers need no change. ReviewScreen.vue is not touched; another PR owns it.
  • Records already stranded pending in existing databases are not auto-healed. They cannot be told apart from legitimate post-baseline pending tools, so re-approve them through review (mcpproxy review approve <server>).
  • Two Spec 105 legacy-stamp interleaving tests injected an operator write inside a pass. That interleaving is now impossible: the write waits for the pass. The tests now assert that it waits and that the disable still survives.
  • No new dependencies. No built-in MCP tool schemas or descriptions changed.

Verification

Unit and race tests:

  • New deterministic tests (internal/runtime/tool_quarantine_concurrency_test.go) pin both lost-update interleavings through a test-only read hook.
  • Without the lock they fail with the exact live symptom, read_006=pending and read_003=pending.
  • Further tests cover:
    • storage-quarantine fail-closed
    • the quarantine flip waiting for an in-flight pass
    • pending-only promotion skipping changed
    • WithToolApprovalLock being per server
    • no rewrite of unchanged pending records
    • batched approve
    • the scanner locking order
    • REST response counts
    • the CLI warning

Commands run:

go test -race -count=1 -timeout 40m -skip "E2E|Binary|MCPProtocol|TestInfoEndpoint|TestGracefulShutdownNoPanic|TestSocketInfoEndpoint" ./internal/runtime/...
go test -race -count=1 -timeout 30m -skip "<same>" ./internal/storage/... ./internal/security/... ./internal/httpapi/... ./cmd/mcpproxy/...
go test -race -tags server -timeout 30m -skip "<same>" ./internal/serveredition/... ./internal/config/... ./internal/oauth/... ./internal/server/... ./internal/httpapi/... ./internal/storage/...
golangci-lint run --config .github/.golangci.yml ./...                       # only pre-existing issues, none in changed files
golangci-lint run --config .github/.golangci.yml --build-tags server ./...   # same

Live checks used the rebuilt make build binary against scratch daemons with fixture upstreams.

The default-selection approval checks used a fixture variant whose trusted server's tools have a different prefix. In the audit's original fixture both servers serve identical names, so the scan returns 180 hard-tier cross-server shadowing findings and the default approve is correctly refused without --force. That fixture is covered by the diagnosis replay row below.

Check Base (diagnosis) This branch
Fresh trusted 320-tool baseline 10 of 11 runs stranded tools pending 8 of 8 runs at 320/320 approved on the final binary: 5 cold-profile, 3 big-profile. The earlier build also passed 8 of 8
review approve large-review --yes (default selection), 180 tools 2 of 4 runs stranded tools 3 of 3 runs on the final binary (6 of 6 across builds): 165 approved+enabled read, 15 approved+disabled destructive, 0 pending, the same after 120s
Web Review screen approve (Playwright, default selection) not replayed 2 of 2 runs: request blocks 15, response still_pending:0, 165/15 immediately and after 120s
Approval during an overlapping hot reload plus concurrent discover-tools — 3 of 3 runs clean
Diagnosis replay --force --all --except <15 delete_*> on the shadowing fixture 2 of 4 runs stranded tools 4 of 4 runs clean

The approved state also stayed the same after each of these:

  • restart
  • hot reload (config edit)
  • reconnect (disable/enable and restart)
  • 3 overlapping discover-tools plus a rescan

MCP behaviour after approval:

  • call_tool_read executes a formerly stranded tool, and call_tool_destructive on a blocked tool returns TOOL_BLOCKED.
  • The direct surface /mcp/all behaves the same: 165 read and 15 destructive listed, the read call executes and the destructive call is blocked.

The guards still hold:

  • A genuinely new 321st tool on the trusted server stays pending and its call is refused.
  • A changed description makes that tool changed and its call is refused.
  • After re-quarantine, restart and re-approval, the changed tool is still changed. The CLI prints Warning: 1 tool(s) on server 'large-review' still need review (0 pending, 1 changed): read_customer_record_005.
  • An explicitly quarantined server never auto-admits: all 180 tools are pending before approval in every run.

Cross-review

OpenCode github-copilot/gpt-6.1-sol (variant high), run read-only:

  • Round 1: one Medium finding. The storage quarantine lookup failed open on read and decode errors. Fixed, with a regression test.
  • Round 2, at 1ea86db55: VERDICT: PASS.

The last commit only applies a lint-requested De Morgan rewrite.

🤖 Generated with Claude Code

…X-02)

Concurrent discovery passes (reactive connect, boot and post-reload sweeps,
HandleUpstreamServerChange, approval reindex) and operator approvals wrote the
same approval records without mutual exclusion, so a stale pass overwrote a
fresher record and tools were stranded pending across reload and restart.

- per-server mutex covers checkToolApprovals (gate read, baseline decision and
  all writes), ApproveTools/ApproveAllTools/baseline promotion, BlockTools/
  BlockAllTools, tool toggles, removed-tool deletes and the quarantine flip
- passes also consult the storage quarantine flag (fail closed) so a server
  quarantined mid-flight is never auto-baselined
- baseline promotion re-reads each record under the lock and never promotes a
  record that turned changed
- the scanner's baseline+blocks commit runs under the lock via an optional
  ToolApprovalLocker capability
- events and the reindex goroutine run after the lock is released
…-02)

tools/approve and security/approve read the approval records back after the
write and add approved_count, blocked_count, still_pending, still_changed and
held_tools. tools/approve's approved now counts requested tools that actually
read approved and lists the rest in not_approved. status stays "approved".
review approve and security approve print the counts and warn on stderr when
any tool still needs review.
…p pending rewrites (UX-02)

With approval writes serialized per server, lock hold time matters. A
discovery pass rewrote every unchanged pending record (one fsynced bbolt
transaction per tool), and ApproveTools wrote one transaction per tool, so a
165-tool approval waited seconds behind a pass and then took seconds more.

- storage: SaveToolApprovals writes many records in one transaction
- approveToolsLocked reads the named records in one snapshot and commits
  them in one transaction (Spec 105 stamping rule preserved)
- a pending record that stays pending with an unchanged definition is no
  longer rewritten
Approval now waits for an in-flight discovery pass on the server.
…rator writes

The operator disable injected between a pass's read and its stamp now waits
for the pass (per-server tool-approval lock) instead of landing in the
window; the tests assert it waits and still survives.
… read (UX-02)

An unreadable upstream record now counts as quarantined for the tool-approval
pass instead of letting a stale trusted config auto-baseline the server.
A missing record still defers to the config. Found by cross-review.
… removals (UX-02 review)

- tools/approve and security/approve accept expected_hashes (tool -> review
  current_hash); a changed or unreviewed target fails with 409 and nothing
  is written. The review payload exposes current_hash; the CLI and the Web
  review screen send it.
- A security approval promotes the reviewed pending tools inside the same
  locked operation as the baseline+blocks commit, and the unquarantine no
  longer re-promotes every pending record, so a tool discovered after the
  commit stays held.
- A discovery pass only removes a tool (index entry, hash, approval record)
  while no newer pass or approval write ran since its inventory read
  (per-server inventory generation), so a stale removal cannot delete a
  fresh operator decision.
- tools/approve reports unique applied counts from the runtime and
  outcome_verified; a failed read-back is never reported as complete, and
  the CLI warns about it.
@codecov-commenter

Copy link
Copy Markdown

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Deploying mcpproxy-docs with  Cloudflare Pages  Cloudflare Pages

Latest commit: 51eb8a9
Status: ✅  Deploy successful!
Preview URL: https://d4b6d9af.mcpproxy-docs.pages.dev
Branch Preview URL: https://fix-qa070-ux02-approval-seri.mcpproxy-docs.pages.dev

View logs

…to safety hints, keep force-retry hashes (UX-02 review)

- Inventory tickets are taken before tools/list; a pass whose inventory was
  captured before one already applied for the server writes nothing, so an
  older pass reaching the approval lock last cannot delete a fresh block or
  restore a rug-pulled tool. Last-good snapshots keep the newest ticket.
- The review current_hash is a fingerprint of the definition plus safety
  hints; an annotation-only tier change makes a review-bound approval stale.
  The rug-pull hash is unchanged.
- The Web force retry re-sends the hashes of the decision that triggered it,
  not those of a review reloaded while the dialog was open.
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

📦 Build Artifacts

Workflow Run: View Run
Branch: fix/qa070-ux02-approval-serialization

Available Artifacts

  • archive-darwin-amd64 (31 MB)
  • archive-darwin-arm64 (28 MB)
  • archive-linux-amd64 (19 MB)
  • archive-linux-arm64 (17 MB)
  • archive-windows-amd64 (31 MB)
  • archive-windows-arm64 (27 MB)
  • frontend-dist-pr (0 MB)
  • installer-dmg-darwin-amd64 (27 MB)
  • installer-dmg-darwin-arm64 (24 MB)
  • smart-mcp-proxymcpproxy-goWHS3UP.dockerbuild (0 MB)

How to Download

Option 1: GitHub Web UI (easiest)

  1. Go to the workflow run page linked above
  2. Scroll to the bottom "Artifacts" section
  3. Click on the artifact you want to download

Option 2: GitHub CLI

gh run download 38059742930 --repo smart-mcp-proxy/mcpproxy-go

Note: Artifacts expire in 14 days.

…ixed reviews bound (UX-02 review)

- Apply a discovery pass's approval decisions and the index writes derived
  from them in one critical section under the server's tool-approval lock, so
  an operator approval or newer pass can no longer land in between and be
  undone (blocked-tool eviction, stale definitions/annotations).
- Publish an inventory to the StateView only while it is still the newest
  applied one; a stale or overtaken inventory no longer republishes older
  safety hints that set the dispatch tier.
- Take the sweep's inventory ticket per server, immediately before that
  server's tools/list, so a genuinely newer capture is not dropped as stale.
- Give hashless approval records a non-empty review fingerprint, and make the
  Web UI and CLI fall back to an unbound approval only when no tool carries a
  hash; a single hashless tool no longer unbinds the whole approval.
- Number review loads in the Web UI so a late response for another server,
  or an older reload, cannot replace the review being approved.
The macOS review sheet now sends expected_hashes (tool name -> the review's
current_hash) on POST security/approve and tools/approve, like the CLI and
the Web review screen. The request stays unbound only when no tool in the
review carries a hash (an older core); a single hashless tool on a newer core
is left out of the binding, so the core refuses it unless it is blocked.

A 409 "out of date" reloads the review and shows the core's message as a
notice instead of an error, and never opens the force dialog. A force retry
re-sends the hashes of the attempt that triggered it, so a review that
changed while the dialog was open fails as out of date.
…bind approved tools in server approvals (UX-02 review)

Inventory tickets were taken by each caller before its tools/list, so a pass
that stalled before listing (or was coalesced onto another caller's list)
carried an older ticket than a capture that started after it, and its newer
inventory was dropped as stale. managed.Client.ListToolsTicketed now takes the
ticket in the caller that leads the upstream list, once it owns the list;
coalesced callers get the leader's ticket and never adopt an unticketed list.
The single-server pass, the quarantined capture and the sweep use it.

A review-bound server approval now also checks enabled approved tools, which
the unquarantine activates: one whose safety hints changed since the review
(annotations are not in the approval hash, so it stays approved) fails as out
of date instead of being activated.
…e review sheet per server (UX-02 review)

The sheet is keyed by server, so switching servers drops the previous review,
selection and pending force decision. Review loads are ordered and only this
server's newest response is shown. Server and per-tool approvals capture the
block list and expected hashes when the operator clicks, so a review that
reloads before the request runs cannot change what is sent; the force and
blind confirmations re-send that captured decision.
…proval lock, record empty-review baselines, serialize review approvals (UX-02 review)

- applyInventory re-checks quarantine (config or storage) under the server's
  approval lock, and the config-reload quarantine flip writes storage under
  that lock before the purge, so a paused pass cannot re-publish a
  quarantined server to the shared or profile indexes.
- A server approval records a durable tool-baseline decision, so a tool
  first served after approving an empty reviewed inventory stays pending
  instead of being auto-approved as a first baseline.
- Web review: responses answer only the review session that sent them
  (A -> B -> A no longer lets an old approval clear choices or open force).
- macOS review sheet: one approval at a time; force re-sends the attempt
  that failed.
…provals to the empty snapshot, lock tool selection during approval (UX-02 review)
…swers for a review the screen no longer shows (UX-02 review)
…er approval unquarantines (UX-02 review)

The commit validated the reviewed fingerprints, but the unquarantine ran
after the approval lock was released. Annotations are outside the approval
hash, so a capture in between that only flipped a reviewed tool's safety
hints left it approved and the unquarantine activated the unreviewed
version. UnquarantineServerKeepingToolDecisions now takes the blocked tools
and expected fingerprints and rechecks them under the same lock as the
quarantine flip; on a mismatch the server stays quarantined and the stale
review error (409) reaches the caller.
Dumbris added a commit that referenced this pull request Oct 11, 2026
…vals are complete (UX-02)

Per-server lock around every tool-approval read-modify-write (discovery pass incl. quarantine read and baseline decision, approve/block/toggle, baseline promotion that never promotes a changed tool, removed-tool deletes, quarantine flip, scanner approve-with-blocks), so concurrent writers can no longer strand approved tools as pending. Approval responses and CLI review/security approve report approved/blocked counts and remaining holds. Live-verified on a rebuilt daemon (10/10 fresh 320-tool baselines fully approved; 165 read approved+callable / 15 destructive held via CLI and Web; durable across restart, reload, reconnect, rediscovery; new/changed tools still held). OpenCode gpt-6.1-sol VERDICT: PASS at 3797afb. Supersedes #1555; follow-ups #1565-#1569.
@Dumbris

Dumbris commented Oct 11, 2026

Copy link
Copy Markdown
Member Author

Superseded by #1571 (minimal UX-02 fix: per-server approval lock + applied/held counts), merged as 9d0d3c2. The pre-existing gaps this branch was chasing are filed as #1565, #1566, #1567, #1568 and #1569.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants