Repository navigation
feat(tools/adversarial-review): add grok reviewer backend - #1446
Conversation
potiuk
left a comment
There was a problem hiding this comment.
Thanks for picking up the reviewer-backend slice of #1416. The extraction, the self-detection and the tests look solid. But the read-only guarantee, which is what this backend has to provide, rests on a flag that grok does not actually enforce. That needs fixing before this can merge.
Blocking — --permission-mode plan is not wired in grok (tools/adversarial-review/src/adversarial_review/backends.py, _grok)
The PR says the reviewer is "read-only by plan mode". The grok-build sources it cites say otherwise:
docs/user-guide/22-permissions-and-safety.md, Available modes:planis "Accepted for compatibility; use plan mode for gated planning". Plan mode is a separate state, entered with/planor theenter_plan_modetool.crates/codegen/xai-grok-agent/src/config.rs, aboveenum PermissionMode: "OnlyBypassPermissionsis wired at spawn; others are forward-compat."crates/codegen/xai-grok-shell/src/util/config/permissions.rs: the session's effective mode is one ofAsk,AutoorAlwaysApprove.planresolves toAsk.
So grok --permission-mode plan runs a normal headless session in ask mode, with the full default toolset ("executes it with full tool access", per 14-headless-mode.md). The argv snapshot test pins the flag, but that proves only that the flag is passed, not that it restricts anything.
Even real plan mode would not be enough here. 19-plan-mode.md says plan mode blocks the edit tools but "Bash commands are not inspected for file writes", and that subagents "are not covered by the parent session's plan-mode edit gate".
Grok does have switches that remove tools rather than relying on a mode. Something along these lines would give a real guarantee:
argv = [
"grok",
"--tools", "read_file,grep,list_dir", # allowlist: no shell, no edit tools
"--no-subagents",
"--disable-web-search", # web_search is auto-approved; it is an egress channel
"--deny", "Bash",
"--deny", "Edit",
"--deny", "Write",
"--deny", "WebFetch",
"--deny", GROK_DENIED_TOOLS, # MCP meta-tools survive --tools unless denied
"--output-format", "json",
"--prompt-file", str(ctx.brief_path),
*_model("-m", ctx),
](--tools is the allowlist, and --deny rules still apply if the allowlist is ever loosened.) The claude backend denies WebFetch/WebSearch for the same reason, so grok should match it. Please also add grok's own write-granting spellings (--always-approve, acceptEdits) to WRITE_GRANTING in tests/test_backends.py, and update the module docstring and README so they describe the mechanism actually used.
A smoke run against a real grok would help, e.g. a brief that asks it to run touch and to edit a file, to show both are refused.
models.md says the Step 0 gate stops the skill, but it does not (tools/privacy-llm/models.md)
until then the Step 0 gate stops any skill that reads private lists while the Grok reviewer is part of the active stack
The gate described a few lines further down builds the stack from Claude Code plus whatever privacy-llm.md lists under Currently configured LLM stack. Nothing reads the adversarial_review.reviewers list, so adding grok there never enters the stack and never trips the gate. Please say what the adopter has to do (declare xAI under the configured stack and the opt-in list), not that the gate will catch it.
The same gap applies to codex, copilot and gemini, so this paragraph could name reviewer backends in general rather than only xAI. That wider question is fine to leave for a follow-up.
Smaller observations
- Backend lists outside the diff still say four CLIs.
projects/_template/adversarial-review.md("Any of: codex, copilot, gemini, claude") is the one adopters copy, so it matters most. The same list appears inplugins/magpie-pr-management/skills/code-review/SKILL.md(with-reviewers:),selectors.md,prerequisites.md,adversarial.md,plugins/magpie-setup/skills/isolated-setup-install/SKILL.md(Step R),docs/setup/secure-agent-setup.md(the~/.grokcredentials comment) and the acceptance list intools/spec-loop/specs/adversarial-review.md. The spec is covered by the spec-sync pre-check inAGENTS.md. - README read scope.
README.mdsays which CLIs confine reads to their working directory but does not say where grok stands. With--tools read_file,grep,list_dir, reads are bounded by grok'sReadrules, not the working directory, so a--deny 'Read(...)'or a--sandboxprofile is worth a sentence here, as the codex note does. - Registry row.
docs/adapters/registry.mdlists Grok in the Agent harness row asreviewer backend only. That fits #1416, but the issue stays open for the harness work. Worth confirming the row is expected to change once skill loading lands.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.
|
Follow-up to my review. I've pushed Correction on the What the commit changes:
The PR description still says "Documents xAI as an opt-in privacy-LLM entry" and "read-only by plan mode". Could you update those two bullets? The one open item from the review is a smoke run against a real |
Why: xAI's Grok CLI had no adversarial-review backend, so adopters running it had no way to use it as a read-only reviewer (Refs apache#1416). The backend runs `grok --permission-mode plan` (read-only everywhere but the session's own plan file), passes the brief through --prompt-file so the diff never sits in argv, and denies every MCP tool invocation via `--deny MCPTool` — the CLI cannot close its MCP servers from the command line. The answer is extracted from the `--output-format json` envelope, rejecting any stopReason other than end_turn. Self-detection uses the GROK_SESSION_ID marker grok sets on its own child processes, checked before Claude Code's widely inherited CLAUDECODE. The privacy-LLM registry documents xAI as an opt-in entry, and the adapter registry marks Grok as reviewer-backend-only. Generated-by: ZCode (GLM-5.3-Flash)
…lowlist grok accepts `--permission-mode plan` but enforces only `bypassPermissions`, so the grok reviewer ran with its full default toolset. It now runs with `--tools read_file,grep,list_dir`, `--no-subagents` and `--disable-web-search`, plus deny rules for Bash, Edit, Write, WebFetch and MCPTool. The tests pin all of this and treat grok's write-granting spellings as forbidden. The privacy-llm note no longer says the Step 0 gate stops a skill when grok is a reviewer. Reviewer backends sit outside that gate by design, because their input is public-bound. The other places that list the reviewer CLIs now include grok. Generated-by: Claude Opus 5
b1882e8 to
3fe04e3
Compare
potiuk
left a comment
There was a problem hiding this comment.
Approving with the fixes pushed on top, rebased onto current main.
- The grok reviewer is read-only through its tool allowlist (
--tools read_file,grep,list_dir,--no-subagents,--disable-web-search), with deny rules for the shell, edit, web-fetch and MCP tools. Every flag is checked against grok's CLI source (xai-grok-pager/src/app/cli.rs), and any mismatch with a future grok release fails closed as a reviewer error rather than a writable reviewer. models.mdnow matches the design: reviewer backends sit outside the privacy-LLM gate because their input is public-bound.- The backend lists across the repo include
grok.
A smoke run against a real grok is still welcome as a follow-up; it wasn't available on this machine.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.
- adversarial-review: the shared pre-PR review block also covers a skill working from someone else's PR, verifying or importing it. - repo-health-family: how workflow-security-audit gets zizmor's GitHub token without `$(gh auth token)`, and the clone fallback. - security-issue-lifecycle: invalidate's issue read is no longer redirected into a file. apache#1446 (grok reviewer backend) carried its own spec update. This branch's first commit updates vetted-command-surface for the installed-path change. .last-sync moves to the current main tip, 3e11608. Generated-by: Claude Code (Opus 5.5)
Summary
grokreviewer backend totools/adversarial-review, per thereviewer-backend slice of feat(adapter/grok): xAI Grok agent harness and adversarial-review backend #1416:
grok --permission-mode plan --output-format json --deny MCPTool --prompt-file <brief>.rejected outright, per grok's plan-mode contract); the brief goes through
--prompt-fileso the diff never sits in argv (grok's-p/--singletakestext only). Grok cannot close its MCP servers from the CLI, so
--deny MCPToolauto-denies every MCP tool invocation instead; the residualdifference from codex/claude (servers still connect) is documented in the
README next to the copilot/gemini note.
--output-format jsonenvelope (text),rejecting any
stopReasonother thanend_turn. Self-detection usesGROK_SESSION_ID(grok sets it on its own child processes), checked beforeClaude Code's widely inherited
CLAUDECODE.tools/privacy-llm/models.md)and adds Grok to the Agent-harness registry row as
reviewer backend only.Type of change
tools/*/withpyproject.toml)tools/<system>/*.md)Test plan
tests/test_backends.py: argv snapshot forgrok, extract cases(
end_turn/refusal/cancelled/max_tokens/ missing / empty /non-JSON),
-mmodel override,--deny MCPToolpin, backend-orderassertion updated (
grokbeforeclaude).tests/test_detect.py:GROK_SESSION_IDself-detection (including agrok-inside-Claude-Code nesting case) and backend enumeration updates.
uv run --directory tools/adversarial-review --project . python -m pytest,ruff check,ruff format --check,mypyon the package.prekhooks over the changed files: markdownlint, typos, doctoc,check-placeholders, lychee (all added links resolve; feat(adapter/grok): xAI Grok agent harness and adversarial-review backend #1416 reachable),
labeler regeneration (no drift), vendor-neutrality-score (no drift —
neither
**Capability:**nor**Harness:**lines changed).(14-headless-mode.md, 19-plan-mode.md, 22-permissions-and-safety.md, changelog 1.0.4).
RFC-AI-0004 compliance
provider endpoints, same class as the existing reviewer CLIs.
row added; no project-specific strings introduced.
private-list skills until the adopter declares it.
Linked issues
Refs #1416 (reviewer-backend slice; skill loading, relay symlinks, spec-loop
profile and guard adapter stay open).
Notes for reviewers
--deny MCPToolis defense-in-depth, not a disconnect: permission rulesauto-deny every MCP tool invocation, but the servers still handshake. If a
maintainer prefers strict parity with codex/claude wording, dropping the flag
is a one-line change plus two test updates — the README note either way says
what grok can and cannot do.
GROK_SESSION_IDis placed beforeCLAUDECODEinBACKENDSsince grok isClaude-compatible in places and Claude's marker is the most widely inherited;
the nesting test pins that order.
Generative AI disclosure: this change was drafted with AI assistance
(Generated-by trailer on the commit) and reviewed by the human contributor.