Skip to content

Audit MCP and skill reliability across Claude, Codex, and Girder #1369

New issue
New issue
Closed
Closed
Audit MCP and skill reliability across Claude, Codex, and Girder#1369
Labels
plan:donePlanboard status

Description

@currentbits
currentbits
opened on Sep 27, 2026

User reports Claude Planboard failures as well as Codex/Girder failures. Investigate the complete provider-to-tool path and instruction/config drift, with bounded read-only or isolated probes.

Plan:

  • Reproduce provider-specific tool discovery, project binding, and approval failures.
  • Inspect project-scoped failure evidence and MCP config/session lifecycle.
  • Audit Girder's installed tool contract against current source and deployment skills.
  • Audit skill discovery, provider distribution, and contradictory instructions.
  • Record evidence, prioritize minimal repairs, and identify verification needed before rollout.

Existing evidence: #1333 and #1190 cover Codex's rejected MCP approval RPC; #1368 records source-only Girder annotations. This audit does not authorize deployments or bypassing approvals.

Activity

  1. currentbits
    added
    plan:todoPlanboard status
    plan:doingPlanboard status
    and removed
    plan:todoPlanboard status
    on Sep 27, 2026

  2. currentbits commented on Sep 27, 2026

    @currentbits
    currentbits
    on Sep 27, 2026
    OwnerAuthor

    Discovery evidence: isolated actual orch-server with a GitHub-origin project, using each provider's generated config. Claude tools/list returns 23 tools and zero issue_* tools; Codex returns 28 including all five Planboard tools. getClaudeMcpArgs ignores projectId, mcpServersDoc binds only coder-memory, and runner's Claude call only passes projectPath. Server correctly gates Planboard on a bound projectId. 497 existing Claude bound config documents have a coder-threads entry; none carries projectId. No GitHub writes performed by this probe.

    Second finding: registering a new user MCP server changes the generated Claude JSON but leaves getClaudeMcpArgs byte-for-byte unchanged. Runner's Claude session reuse key uses argv only, and settings sync does not retire warm sessions. This permits stale connections/tool catalogs after edits; proof covers config/key behavior, not a new real-model warm-session run.

    Continuing Girder/skills audit. Girder #1368 was source-only; installed binary still lists 72 tools, zero annotations. Local dummy-API probe confirms a slow tool blocks ping and a cancellation notification does not stop the request. A 900ms mock response delayed ping by 877ms. No production Girder calls or deployments.

  3. currentbits commented on Sep 27, 2026

    @currentbits
    currentbits
    on Sep 27, 2026
    OwnerAuthor

    Skills/config evidence:

    • Live Girder deploy skill differs between Claude's source symlink and the copied agents/Codex/Grok/Cursor variants. Solenta listSkills reports all installed, missingFrom=[], and no content-version conflict. An isolated two-provider fixture proves syncSkills copies zero despite differing bodies.
    • Slash expansion has no provider parameter. It selects the first /name in traversal order, normally Claude before Codex. Isolated /demo fixture expands the Claude body even with a distinct Codex body present. Native auto-discovery may therefore use a different version from Solenta's slash expansion.
    • SKILL_DIRS ignores CODEX_HOME/CURSOR_HOME for ordinary skill roots even though plugin discovery honors them. An isolated custom CODEX_HOME-only skill is absent from the palette. Kimi's separate known path/overlay issue remains Kimi skills never reach the CLI: wrong install dir, overlay hides skills, project scan is claude-only #695; no new native Kimi claim is made here.
    • Girder is configured natively in both Claude and Codex, using the same command/env. Solenta's managed MCP list contains only context7 and playwright. Availability is not uniform across providers merely because the tool exists in one CLI's config.
    • Girder skill/tool descriptions drift: app_config implementation accepts branch/kind/dockerfile/previewsTTLHours but its advertised key description omits them; skill directs port configuration through CLI/API despite an app_config tool. Its service instructions recommend empty-commit redeploy while its Rules prefer redeploy for config-only changes. These are instruction/description problems, not proof those API calls fail.
    • services.js still injects a Codex exec/exec-resume-only note although interactive runner uses app-server.

    Planboard result-size/completeness reproduction posted on #1187. Core probes live in /tmp/solenta-mcp-audit (no credentials or real deployment calls). No tracked Solenta files edited; existing Girder source-only changes preserved.

  4. currentbits commented on Sep 27, 2026

    @currentbits
    currentbits
    on Sep 27, 2026
    OwnerAuthor

    Audit complete. No tracked Solenta source or user MCP/skill configuration changed. Prior Codex real-model reproduction plus new isolated discovery/protocol/skill probes establish multiple separate failures. Relevant Solenta implementation files match fetched origin/main.

    Completed investigation:

    • Provider discovery/binding and approval paths.
    • Project-scoped failure evidence and config/session lifecycle.
    • Installed Girder binary vs source and deployment skill contract.
    • Skill discovery, copy drift, home overrides and slash precedence.
    • Prioritized minimal repairs and verification gaps.

    Remediation roadmap (not implemented or shipped):

    Limits: no new native Claude-model hot-reload run; no native Kimi/Grok/Cursor matrix; no Girder production mutation or deployment. Source-level config/reuse evidence is distinguished from live model results. Keeping this umbrella open as plan:todo for remediation, with detailed evidence in earlier comments.

  5. currentbits
    added
    plan:todoPlanboard status
    plan:doingPlanboard status
    and removed
    plan:doingPlanboard status
    plan:todoPlanboard status
    on Sep 27, 2026

  6. currentbits commented on Sep 27, 2026

    @currentbits
    currentbits
    on Sep 27, 2026
    OwnerAuthor

    Implementation in progress. Claude generated-config-to-tools/list now exposes Planboard for its bound GitHub project; unbound/memory-only isolation remains covered. A real warm-process fixture now respawns after MCP token/config changes and reuses again when unchanged. Codex now distinguishes its empty mcp_tool_call confirmation form from arbitrary elicitation and routes it through permission cards; allow/deny/Stop use the correct MCP result shape. A real Codex 0.157.1 model-initiated diagnostic_ping succeeded using the new mapping (one approval, one server call, DIAGNOSTIC_PONG). 119 affected Claude/Codex/Planboard checks pass. Planboard MCP pages default to 50 open issues, with cursor/state/exact-number options; board fixture covers 1,002 open issues plus separately bounded closed history. The GraphQL page was checked against GitHub. Skill provider-selection/custom-home/drift regressions pass. Girder slow-call test reproduced blocked ping; fixed isolated transport passes the race detector. Changes are not committed or shipped yet.

  7. currentbits commented on Sep 27, 2026

    @currentbits
    currentbits
    on Sep 27, 2026
    OwnerAuthor

    Implementation ready for review in draft PR #1370: #1370
    Review head: 61ca86d. The branch contains only this repair commit on current origin/main.

    Completed:

    • Claude gets project-bound coder-threads configuration, so Planboard discovery works. Bound-config write failures propagate instead of falling back to an unbound configuration.
    • Claude warm-process reuse now includes a digest of effective managed MCP configuration, so edits at the same filename replace the process on the next turn.
    • Ordinary Codex MCP confirmation elicitations use Solenta's permission cards and correct accept/decline/cancel responses. Arbitrary forms remain outside this change.
    • MCP issue_list now supports bounded pages, state, cursor and exact issue number. The board fetches all open issue pages plus recent closed issues; older active issues are no longer hidden by the old 1,000-total-issue cap.
    • Slash expansion selects the current provider's skills. Skill inventory shows differing SKILL.md content without overwriting intentional variants.
    • Headless Codex guidance and Girder's configuration-only deployment instructions were corrected.

    Verification:

    • 6,967 checks passed across core (38), renderer (2,631), Electron (4,040), memory (233), and stats (25). The complete Electron suite passed at concurrency 4; at default concurrency two unchanged provider-usage timeout fixtures missed their fake PID files. Inherited app guardrail/config overrides were removed from test environments.
    • Final focused regressions: 55/55. Typecheck and production build passed.
    • Real Codex 0.157.1 model-driven diagnostic: one approval, one harmless MCP call, DIAGNOSTIC_PONG.
    • Live read-only GitHub pagination/state/exact-number probes passed.
    • Hosted macOS, Windows and Linux smoke checks pass. All three full hosted test jobs were still running at this update.

    Girder:

    • Local repair commit 7cbe5b0 on fix/mcp-client-reliability; full Go suite and MCP race checks pass.
    • Stdin control requests remain responsive during tools/call; ordered bounded tool execution supports queued/active cancellation and serialized output.
    • Installed local binary 1.1.3 advertises all 75 tools with annotations. Installed-binary probe confirms ping responds while a slow call runs, cancelled calls emit no late result, and subsequent reads work.
    • All eight installed Girder deploy skill locations now match the corrected source.
    • The Girder commit is not pushed: its local base includes unrelated unpublished commits. No control-plane deployment or client-session restart was performed.

    Remaining:

  8. currentbits commented on Sep 27, 2026

    @currentbits
    currentbits
    on Sep 27, 2026
    OwnerAuthor

    Continuing PR #1370 verification: hosted macOS and Ubuntu full tests pass, and all three platform smoke jobs pass. Windows full tests failed in run 36346962930. Investigating its failure before landing or release; issue remains plan:doing.

  9. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    PR #1370 is now at 1a19ebb. The previous hosted run passed macOS/Linux and all smoke checks. The Kimi Windows cleanup correction held; Windows then exposed a separate Codex steering fixture race, which waited for the fake server recording turn/start before the client had received its reply. A delayed reply reproduced the same failure locally. The test now waits for the real client's canSteer signal via the existing runner injection seam; no production behavior changed. All 40 focused Codex checks pass. Hosted checks are running again; keeping this issue plan:doing until landing and release verification.

  10. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Solenta repair #1370 is merged as fc354cc after all six hosted checks passed at 1a19ebb: full test and smoke jobs on Windows, Linux, and macOS. The merge tree is byte-identical to the tested PR head. Building the nightly from the merged commit in the isolated release checkout; package verification is still pending. The separate Girder repair passes its local full Go suite and MCP race checks, but hosted validation remains unavailable. Arbitrary elicitation forms (#1190) and full closed-history UI (#1187) remain outside this repair.

  11. 11 remaining items

  12. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Completed the previously pending input-card visual/interaction check in a local browser using the shipped component and styles; detailed evidence is recorded on #1190. Required validation, masking, typed values, custom answers, error/retry, disabled state, decline and cancel all passed. Fixture removed; no production code changes needed.

    The remaining live Browser check is an installation-state blocker: the inspected app bundle is still nightly-202609280651-fc354ccb, while the published and platform-verified release is nightly-202609281438-22c8d443. preview.info reports no pane. UI automation could not stage the update because the app changed during interaction. The app needs Update beside Settings, then Restart when active work can be interrupted, before verifying Views → Browser on this thread. Leaving this umbrella open.

  13. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Update-state follow-up: the app bundle on disk now reports nightly-202609281438-22c8d443. Computer Use confirms the live UI shows Restart (Restart to update), so installation is staged and relaunch is still pending. preview.info still reports no accessible Browser pane. Other agent runs are active; no restart was forced. Input-card visual/interaction checks remain passed as documented on #1190. The next verification requires relaunch, then Views → Browser on this thread.

  14. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Post-restart verification: the Restart control is gone. Computer Use screenshot of this thread shows a transcript-only layout with Views at the upper right; no Browser pane/address bar is present. preview.info likewise reports no pane. Both attempted Views clicks, including refresh-and-click, were rejected by Computer Use as changed/stale app state before the menu opened. No new Browser registration defect established. User needs to select Views → Browser on this thread; another update/restart is not requested. Prior component visual checks remain passed.

  15. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Final live-check attempt after the user confirmed the Browser address bar: Computer Use selected the restarted nightly and this exact thread, but its current accessibility tree contained no Browser navigation/address controls. A fresh full-state lookup followed immediately by clicking Views was again rejected as user-changed/stale app state. A subsequent preview.info still reports no bound pane. Stopping the repeated setup/confirmation loop: no further restart requested and no new source defect established. Browser-to-MCP verification remains explicitly unverified; the previously recorded input-card component visual/interaction checks and real Codex protocol probes remain passed. No production changes made in this attempt.

  16. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Continuation verified an additional Browser hypothesis without more live-UI approval loops. An isolated renderer probe on reviewed #1377 head 2974079 opened Browser via Views, answered a persisted question through its real callback, applied the next working detail, passed through detail=null, restored the thread, and unmounted/remounted it. The Browser pane persisted/restored throughout (1/1 pass). This does not reproduce a question-answer layout reset and is not a claim of packaged Electron verification. Temporary probe removed from the worktree; worktree remains clean. Live preview.info still reports no bound pane.

    The separate MCP transport repair remains an open draft at its unchanged reviewed head. One fresh hosted retry (attempt 3) completed with both jobs refused before any test step by the same external account restriction. No further retries or unrelated repairs started.

    Remaining blockers: live Browser-to-MCP verification, and external hosted validation/landing of the separate transport PR. Solenta source repairs, release/package checks, input-card visual checks and native Codex protocol probes remain completed. Broader unsupported input schemas/sandbox grants (#1190), Kimi (#695), closed-history UI (#1187), and discovery diagnostics (#324/#415) retain their separate scope.

  17. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Continuing with the remaining native-Claude verification gap while live preview remains unavailable. Plan: run the production runner against the installed Claude CLI with an isolated temporary store/repository and only harmless fixture MCP tools; change the effective MCP document at the same generated path between turns; verify a new process/tool catalog is used, then verify an unchanged third turn reuses it. No user MCP configuration or production endpoints will be modified. Existing Browser and external CI blockers remain separately recorded.

  18. currentbits commented on Sep 28, 2026

    @currentbits
    currentbits
    on Sep 28, 2026
    OwnerAuthor

    Native Claude MCP-refresh verification now passes, closing the earlier real-CLI gap. Used the production createRunner from reviewed #1377 head 2974079 with installed Claude Code 2.1.283 / claude-opus-5-5 and a temporary repository/store/provider configuration. Strict MCP configuration exposed only a harmless local fixture; built-in tools and hooks were disabled for these probe processes.

    Three actual model-driven turns:

    1. probe_before: one CLI process, one tool call, correct unique marker.
    2. Changed registered MCP command arguments while retaining the same generated config filename. Runner spawned a second CLI with --resume; probe_after was discovered/called and returned the correct marker.
    3. Unchanged configuration: the same second CLI handled probe_after successfully, with no third spawn.

    Assertions passed for exactly one call per turn, two native processes total, changed-config respawn, and unchanged-config reuse. No user MCP settings or production endpoints were modified. Successful probe and earlier sandbox-blocked attempts were cleaned up; no probe children or temporary runtime directories remain. Runnable probe/evidence retained outside the repository in the existing temporary audit directory; production worktree remains clean.

    Remaining recorded blockers are unchanged: live Browser-to-MCP verification and external hosted validation/landing of the separate transport PR. No additional CI retry or live UI approval loop was started.

  19. currentbits commented on Sep 29, 2026

    @currentbits
    currentbits
    on Sep 29, 2026
    OwnerAuthor

    Recheck 2026-09-29:

    • Girder PR Worker blocked on a permission prompt stalls the orchestration invisibly #31: no longer blocked. It is merged and all four checks are green (bash shellcheck+bats, actionlint, compose validate, go build+test+lint).

    • Browser → MCP preview: preview info still returns "Browser pane is not open". I checked the chain statically against the running nightly (out/Solenta Nightly.app, process started 2026-09-28 20:11):

      • renderer bundle has src:"about:blank" on the webview (Initialize Browser pane before binding MCP preview tools #1377 fix present)
      • packaged preload exposes preview.bind/unbind/...
      • ipc.js preview:* handlers are present
      • orchServer runs in the main process and requires the same preview.js module, so the IPC bind and the MCP info share one sessions map

      No drift found. The remaining live check needs the pane open on this thread in the current app session.

    • Local branch codex/thread-worker-ux commit 4e72e06 matches the landed Fix MCP discovery, approvals, and provider skill selection #1370 (fc354cc). The only diff is CSS from later main commits, so nothing on the branch still needs landing.

  20. currentbits commented on Sep 29, 2026

    @currentbits
    currentbits
    on Sep 29, 2026
    OwnerAuthor

    Browser→MCP follow-up, 2026-09-29: no app bug found.

    Every earlier "pane is open but MCP says closed" report came from testing in a browser other than Solenta's Browser pane. The user confirmed this today: "Unable to connect" came from another app, while 127.0.0.1:53833 returns 200. Solenta's BrowserPane.go() only loads through preview.navigate, which uses the same binding MCP reads, so it cannot show that error.

    The static chain in the running nightly is intact: #1377 src:"about:blank", preload preview.*, the ipc.js handlers, and an in-process orch server sharing one preview.js sessions map. The live info/navigate/screenshot check is still not run. It needs Views → Browser opened on this thread inside Solenta Nightly; no restart or code change is required.

  21. currentbits commented on Sep 29, 2026

    @currentbits
    currentbits
    on Sep 29, 2026
    OwnerAuthor

    Ran the Browser→MCP check myself in an isolated Electron 35.7.5 process: the real electron/preview.js and a real <webview> bound on did-attach, same as BrowserPane.

    • bind, info, click, type, screenshot, external-URL refusal, and PANE_CLOSED after the pane is removed: all pass.
    • Real defect found: a navigate issued right after bind returned about:blank (3/3). loadURL resolves on the guest's initial about:blank did-finish-load (41 ms), and the real page commits at 42 ms. So the MCP tool and an immediate screenshot could see a blank page.

    Fix in #1382: navigate waits, bounded to 2 s, for the initial load to stop. The regression test fails without the fix, and the probe is 11/11 on 3/3 runs with it.

  22. currentbits commented on Sep 29, 2026

    @currentbits
    currentbits
    on Sep 29, 2026
    OwnerAuthor

    #1382 has been merged (4423803) and published as nightly nightly-202609290908-44238035 (macos-arm64, linux-x64 and win32-x64 assets). After updating Solenta Nightly, preview navigate right after the Browser pane opens returns the real page instead of about:blank.

  23. currentbits
    added
    plan:donePlanboard status
    and removed
    plan:doingPlanboard status
    on Oct 9, 2026

  24. currentbits commented on Oct 9, 2026

    @currentbits
    currentbits
    on Oct 9, 2026
    OwnerAuthor

    Done: audit fixes landed in fc354cc (#1370) and 65d15d0 (#1371). Closing in the 2026-10-09 board cleanup (#1532).

  25. currentbits
    closed this as completedon Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    plan:donePlanboard status

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions