Skip to content

fix(k8s): ops-runner mounts no service account token - #4179

Merged
lilyshen0722 merged 2 commits into
mainfrom
cto/f539-ops-runner-token
Nov 16, 2026
Merged

lilyshen0722 merged 2 commits into
mainfrom
cto/f539-ops-runner-token

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

@coderabbitai ignore

ops-runner no longer mounts the provisioner's service-account token. It gets its namespace from the Downward API instead (F539).

Why

ops-runner is the exec target for operator scripts, and it runs as agent-provisioner. That account's Role (templates/agent-provisioning/rbac.yaml) covers every Secret in the namespace, plus Deployments and pod exec. Nothing run there calls the Kubernetes API from inside the pod. #4176 left ops-runner on the chart check's allowlist until that was established, and this PR settles it.

Inventory (what runs in ops-runner)

Imports were traced transitively for each source:

  • runbooks: guard-name canary, token-copy scrubs, read-point seed, preset-skill cleanup, usage report, DCR client retire, rollback recipe, ref-links;
  • CI: the canary step in deploy-dev.yml, which execs from the runner with the deploy account's own kubectl;
  • the operator recipes on the ops host.

What they do:

  • Mongo and Postgres: every script reads or writes them.
  • HTTP: some also call the backend or a vendor.
  • Provisioners: a few import a provisioner only indirectly, through agentEventService. None calls one.
  • Token revocation is database-only (routes/registry/tokens.ts).

A provisioner import still loads with no token. @kubernetes/client-node 0.21.0 falls back to http://localhost:8080 when no token file exists (dist/config.js:296-301). Clients build, and only a call fails. The account has no cloud identity (only Helm's annotations on dev and staging), and nothing else in the pod (single container, key-file GCP auth) reads the token.

The namespace file goes too

With no token there is no /var/run/secrets/kubernetes.io/serviceaccount/namespace. backend/scripts/staging-seed.ts reads its namespace from that file or from POD_NAMESPACE, and its guard refuses without either. So ops-runner now sets POD_NAMESPACE from fieldRef: metadata.namespace.

What changes

  • templates/ops/ops-runner-deployment.yaml: automountServiceAccountToken: false and the POD_NAMESPACE Downward API entry.
  • scripts/verify-service-account-tokens-chart.js:
    • It now exempts only the backend and backend-worker Deployments.
    • A backend-image workload that mounts no token must set POD_NAMESPACE from metadata.namespace.
  • backend/__tests__/unit/helm/opsRunnerEnvMirror.test.js: the env-mirror guard parses fieldRef and pins POD_NAMESPACE as ops-runner's one extra variable. It also fails on any entry it can't parse.
  • Docs:
    • ops-runner-and-no-exec-into-backend.md: the no-token rule. It also corrects a stale line, since staging does run ops-runner.
    • agent-computer-steel.md: no longer says ops-runner reads Secrets.
    • ui-validation.md: no longer says staging has no ops-runner.

Proof

  • Test-first:
    • The chart check failed on ops-runner in all 3 lanes when the allowlist dropped it, and again for the missing POD_NAMESPACE. Both pass now: "ok: 12 workloads render as agent-provisioner across 3 lanes; only Deployment/backend, Deployment/backend-worker mount its token".
    • The env-mirror guard failed on the new variable until it was pinned.
  • Still pass: backend/__tests__/unit/helm (112 tests), and the network-policy, pg-roles and dev-provider-env checks. The worker chart check needs yq and runs in CI.

Rollout

ops-runner already rolls on every deploy, because its image tag changes, so this rides Tuesday's roll with no extra restart. After the roll, the consumer checks are:

  • the pod spec shows no token volume;
  • POD_NAMESPACE is set;
  • one ops-runner script still runs.

Tier 2 (access rights), with one review (Opus) below. It touches k8s/, so it's a hand press.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4VWLGaqG2kVEDDn4hY3nm

lilyshen0722 and others added 2 commits November 15, 2026 18:28
ops-runner is the exec target for operator scripts and runs as
agent-provisioner, which can read every Secret in its namespace. The scripts
run there read Mongo and Postgres; none calls the Kubernetes API from inside
the pod (inventoried 11-16: runbooks, the operator recipes, backend/scripts).
Token revocation is database-only. A script that imports a provisioner still
loads, since @kubernetes/client-node falls back to localhost:8080 without a
token file.

The chart check now exempts only the backend and its worker, and failed on
ops-runner before this change. F539.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4VWLGaqG2kVEDDn4hY3nm
…it has no token file

With no service-account token mounted, ops-runner has no namespace file, and
backend/scripts/staging-seed.ts reads its namespace from that file or from
POD_NAMESPACE, refusing without either. ops-runner now sets POD_NAMESPACE from
fieldRef metadata.namespace. The chart check fails a backend-image workload
that mounts no token and lacks it; the env-mirror guard parses fieldRef and
pins POD_NAMESPACE as ops-runner's only extra.

Docs from the review: staging does run ops-runner (values-staging.yaml), the
Steel runbook no longer says ops-runner reads Secrets, and the new bullet says
what the scripts do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4VWLGaqG2kVEDDn4hY3nm

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one review for this change (D218, Tier 2: access rights): an Opus subagent, read-only, at 358ba30. It found nothing Major. The Minor and the nits are fixed at 5732327, with one consequence the review's doc point surfaced, described below.

Checked and true:

  • Inventory: nothing run in ops-runner calls the Kubernetes API from inside the pod. Imports were traced transitively:
    • Never reach a provisioner: guard-name-canary.js, the token-copy scrubs, seed-pod-read-points, remove-orphan-preset-skills, usage-report.js, and the inline release and ref-link recipes.
    • Import one only: retire-hosted-mcp-dcr-client.ts and the operator teardown and capture recipes import it through agentEventService, and none calls it. Pod deletion is database work plus HTTP to the hosted worker (hostedRuntimeService.ts:452-468). Revocation is database-only (routes/registry/tokens.ts:220-226).
    • Every active recipe on the ops host is mongoose, pg or mongodb only.
  • Import is safe: both provisioner modules only loadFromDefault and makeApiClient at import. client-node 0.21.0 (package-lock.json:1671) falls back to http://localhost:8080 (config.js:296-300). A probe with no kubeconfig or token built the clients, and a call got ECONNREFUSED.
  • Nothing else in the pod needs the token: it has a single container, no sidecars or init containers, and GCP auth by key file. The account has only Helm annotations on dev and staging.
  • The only CI exec into ops-runner is the canary step (deploy-dev.yml:494), with the runner's own kubectl. There is no CronJob and no backend reference.
  • Checks: the chart check passes here and fails on main's template in all 3 lanes. __tests__/unit/helm passes, as do the network-policy, pg-roles, dev-provider-env and default-model checks. CI runs it (tests.yml:644).
  • Rollout: RollingUpdate, one replica, no probes. It already rolls on every deploy, so there's no extra restart. No infra identifiers.

Minor and nits, with what was done:

  • agent-computer-steel.md:114-116 said ops-runner reads the namespace's Secrets. It now says only the backend can, and that ops-runner mounts no token.
  • ops-runner-and-no-exec-into-backend.md:60-61 and ui-validation.md:60 said staging has no ops-runner. Both are corrected: values-staging.yaml enables it.
  • The new bullet's "read Mongo and Postgres" now also says some scripts call the backend or a vendor over HTTP.

Found while fixing ui-validation.md:

  • The problem: with no token there is no service-account namespace file. backend/scripts/staging-seed.ts:22-29 reads that file or POD_NAMESPACE, and its guard (guard.ts:66) refuses without either. So the staging seed would refuse in ops-runner. That fails safe, but it breaks the path.
  • The fix: ops-runner now sets POD_NAMESPACE from fieldRef: metadata.namespace.
  • The guards:
    • The chart check fails a backend-image workload that mounts no token and lacks POD_NAMESPACE. It failed first, in all 3 lanes.
    • The env-mirror guard parses fieldRef and pins POD_NAMESPACE as ops-runner's only extra variable. It also fails on any entry it can't parse. 112 of 112 tests pass.
  • The staging-seed recipe itself still runs in the staging backend, where it was walked. The doc says it isn't walked on ops-runner.

GATE: PASS

@lilyshen0722
lilyshen0722 added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit fe2ab0f Nov 16, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant