Skip to content

feat(tools/adversarial-review): add grok reviewer backend - #1446

Merged
potiuk merged 2 commits into
apache:mainfrom
liwenjie200543:contributor/grok-reviewer-backend
Sep 28, 2026
Merged

potiuk merged 2 commits into
apache:mainfrom
liwenjie200543:contributor/grok-reviewer-backend

Conversation

@liwenjie200543

Copy link
Copy Markdown
Contributor

Summary

  • Adds a grok reviewer backend to tools/adversarial-review, per the
    reviewer-backend slice of feat(adapter/grok): xAI Grok agent harness and adversarial-review backend #1416: grok --permission-mode plan --output-format json --deny MCPTool --prompt-file <brief>.
  • Read-only by plan mode (everything but the session's own plan file is
    rejected outright, per grok's plan-mode contract); the brief goes through
    --prompt-file so the diff never sits in argv (grok's -p/--single takes
    text only). Grok cannot close its MCP servers from the CLI, so --deny MCPTool auto-denies every MCP tool invocation instead; the residual
    difference from codex/claude (servers still connect) is documented in the
    README next to the copilot/gemini note.
  • The answer is extracted from the --output-format json envelope (text),
    rejecting any stopReason other than end_turn. Self-detection uses
    GROK_SESSION_ID (grok sets it on its own child processes), checked before
    Claude Code's widely inherited CLAUDECODE.
  • Documents xAI as an opt-in privacy-LLM entry (tools/privacy-llm/models.md)
    and adds Grok to the Agent-harness registry row as reviewer backend only.

Type of change

  • Python package (tools/*/ with pyproject.toml)
  • Tool / bridge contract (tools/<system>/*.md)

Test plan

  • tests/test_backends.py: argv snapshot for grok, extract cases
    (end_turn / refusal / cancelled / max_tokens / missing / empty /
    non-JSON), -m model override, --deny MCPTool pin, backend-order
    assertion updated (grok before claude).
  • tests/test_detect.py: GROK_SESSION_ID self-detection (including a
    grok-inside-Claude-Code nesting case) and backend enumeration updates.
  • uv run --directory tools/adversarial-review --project . python -m pytest, ruff check, ruff format --check, mypy on the package.
  • prek hooks over the changed files: markdownlint, typos, doctoc,
    check-placeholders, lychee (all added links resolve; feat(adapter/grok): xAI Grok agent harness and adversarial-review backend #1416 reachable),
    labeler regeneration (no drift), vendor-neutrality-score (no drift —
    neither **Capability:** nor **Harness:** lines changed).
  • CLI surface verified against xai-org/grok-build docs at main
    (14-headless-mode.md, 19-plan-mode.md, 22-permissions-and-safety.md, changelog 1.0.4).

RFC-AI-0004 compliance

  • Sandbox — no new unrestricted host access; grok reaches only its own
    provider endpoints, same class as the existing reviewer CLIs.
  • Vendor neutrality — one more backend on the vendor axis; registry
    row added; no project-specific strings introduced.
  • Privacy LLM — xAI documented as opt-in; the Step 0 gate stops
    private-list skills until the adopter declares it.

Linked issues

Refs #1416 (reviewer-backend slice; skill loading, relay symlinks, spec-loop
profile and guard adapter stay open).

Notes for reviewers

  • --deny MCPTool is defense-in-depth, not a disconnect: permission rules
    auto-deny every MCP tool invocation, but the servers still handshake. If a
    maintainer prefers strict parity with codex/claude wording, dropping the flag
    is a one-line change plus two test updates — the README note either way says
    what grok can and cannot do.
  • GROK_SESSION_ID is placed before CLAUDECODE in BACKENDS since grok is
    Claude-compatible in places and Claude's marker is the most widely inherited;
    the nesting test pins that order.

Generative AI disclosure: this change was drafted with AI assistance
(Generated-by trailer on the commit) and reviewed by the human contributor.

@github-actions github-actions Bot added substrate:privacy Tool substrate: PII redaction / approved-LLM gating substrate:review Tool substrate: second-model read of a change before publishing (read-only CLIs, merged findings) labels Sep 28, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking up the reviewer-backend slice of #1416. The extraction, the self-detection and the tests look solid. But the read-only guarantee, which is what this backend has to provide, rests on a flag that grok does not actually enforce. That needs fixing before this can merge.

Blocking — --permission-mode plan is not wired in grok (tools/adversarial-review/src/adversarial_review/backends.py, _grok)

The PR says the reviewer is "read-only by plan mode". The grok-build sources it cites say otherwise:

  • docs/user-guide/22-permissions-and-safety.md, Available modes: plan is "Accepted for compatibility; use plan mode for gated planning". Plan mode is a separate state, entered with /plan or the enter_plan_mode tool.
  • crates/codegen/xai-grok-agent/src/config.rs, above enum PermissionMode: "Only BypassPermissions is wired at spawn; others are forward-compat."
  • crates/codegen/xai-grok-shell/src/util/config/permissions.rs: the session's effective mode is one of Ask, Auto or AlwaysApprove. plan resolves to Ask.

So grok --permission-mode plan runs a normal headless session in ask mode, with the full default toolset ("executes it with full tool access", per 14-headless-mode.md). The argv snapshot test pins the flag, but that proves only that the flag is passed, not that it restricts anything.

Even real plan mode would not be enough here. 19-plan-mode.md says plan mode blocks the edit tools but "Bash commands are not inspected for file writes", and that subagents "are not covered by the parent session's plan-mode edit gate".

Grok does have switches that remove tools rather than relying on a mode. Something along these lines would give a real guarantee:

argv = [
    "grok",
    "--tools", "read_file,grep,list_dir",   # allowlist: no shell, no edit tools
    "--no-subagents",
    "--disable-web-search",                 # web_search is auto-approved; it is an egress channel
    "--deny", "Bash",
    "--deny", "Edit",
    "--deny", "Write",
    "--deny", "WebFetch",
    "--deny", GROK_DENIED_TOOLS,            # MCP meta-tools survive --tools unless denied
    "--output-format", "json",
    "--prompt-file", str(ctx.brief_path),
    *_model("-m", ctx),
]

(--tools is the allowlist, and --deny rules still apply if the allowlist is ever loosened.) The claude backend denies WebFetch/WebSearch for the same reason, so grok should match it. Please also add grok's own write-granting spellings (--always-approve, acceptEdits) to WRITE_GRANTING in tests/test_backends.py, and update the module docstring and README so they describe the mechanism actually used.

A smoke run against a real grok would help, e.g. a brief that asks it to run touch and to edit a file, to show both are refused.

models.md says the Step 0 gate stops the skill, but it does not (tools/privacy-llm/models.md)

until then the Step 0 gate stops any skill that reads private lists while the Grok reviewer is part of the active stack

The gate described a few lines further down builds the stack from Claude Code plus whatever privacy-llm.md lists under Currently configured LLM stack. Nothing reads the adversarial_review.reviewers list, so adding grok there never enters the stack and never trips the gate. Please say what the adopter has to do (declare xAI under the configured stack and the opt-in list), not that the gate will catch it.

The same gap applies to codex, copilot and gemini, so this paragraph could name reviewer backends in general rather than only xAI. That wider question is fine to leave for a follow-up.

Smaller observations

  • Backend lists outside the diff still say four CLIs. projects/_template/adversarial-review.md ("Any of: codex, copilot, gemini, claude") is the one adopters copy, so it matters most. The same list appears in plugins/magpie-pr-management/skills/code-review/SKILL.md (with-reviewers:), selectors.md, prerequisites.md, adversarial.md, plugins/magpie-setup/skills/isolated-setup-install/SKILL.md (Step R), docs/setup/secure-agent-setup.md (the ~/.grok credentials comment) and the acceptance list in tools/spec-loop/specs/adversarial-review.md. The spec is covered by the spec-sync pre-check in AGENTS.md.
  • README read scope. README.md says which CLIs confine reads to their working directory but does not say where grok stands. With --tools read_file,grep,list_dir, reads are bounded by grok's Read rules, not the working directory, so a --deny 'Read(...)' or a --sandbox profile is worth a sentence here, as the codex note does.
  • Registry row. docs/adapters/registry.md lists Grok in the Agent harness row as reviewer backend only. That fits #1416, but the issue stays open for the harness work. Worth confirming the row is expected to change once skill loading lands.

This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. After you've
addressed the points above and pushed an update, an Apache Magpie
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

@potiuk

potiuk commented Sep 28, 2026

Copy link
Copy Markdown
Member

Follow-up to my review. I've pushed b1882e8a onto this branch to address it, and one of my findings was wrong.

Correction on the models.md finding. I asked you to spell out what an adopter must declare for xAI. That was wrong. The adversarial-review design (docs/designs/2026-09-23-adversarial-review.md, decision 5 and the rejected-alternatives list) keeps every reviewer backend outside the privacy-LLM gate on purpose, because a reviewer only ever sees public-bound content. Grok shouldn't be the one backend treated differently. The commit replaces the paragraph with a short note saying that for all five backends.

What the commit changes:

  • grok argv: --permission-mode plan is replaced by --tools read_file,grep,list_dir --no-subagents --disable-web-search plus --deny for Bash, Edit, Write, WebFetch and MCPTool. The module docstring and README now explain why.
  • Tests: the argv snapshot is updated, a new test checks that grok is read-only through the allowlist, and --always-approve and acceptEdits are added to WRITE_GRANTING.
  • Backend lists: grok is added to the lists in the adopter template, the pr-management-code-review skill files, isolated-setup-install Step R, secure-agent-setup.md and the spec's acceptance list.

The PR description still says "Documents xAI as an opt-in privacy-LLM entry" and "read-only by plan mode". Could you update those two bullets? The one open item from the review is a smoke run against a real grok: a brief that asks it to run touch and to edit a file, to show both are refused.

liwenjie200543 and others added 2 commits September 28, 2026 12:00
Why: xAI's Grok CLI had no adversarial-review backend, so adopters
running it had no way to use it as a read-only reviewer (Refs apache#1416).
The backend runs `grok --permission-mode plan` (read-only everywhere
but the session's own plan file), passes the brief through
--prompt-file so the diff never sits in argv, and denies every MCP
tool invocation via `--deny MCPTool` — the CLI cannot close its MCP
servers from the command line. The answer is extracted from the
`--output-format json` envelope, rejecting any stopReason other than
end_turn. Self-detection uses the GROK_SESSION_ID marker grok sets on
its own child processes, checked before Claude Code's widely
inherited CLAUDECODE. The privacy-LLM registry documents xAI as an
opt-in entry, and the adapter registry marks Grok as
reviewer-backend-only.

Generated-by: ZCode (GLM-5.3-Flash)
…lowlist

grok accepts `--permission-mode plan` but enforces only
`bypassPermissions`, so the grok reviewer ran with its full default
toolset. It now runs with `--tools read_file,grep,list_dir`,
`--no-subagents` and `--disable-web-search`, plus deny rules for Bash,
Edit, Write, WebFetch and MCPTool. The tests pin all of this and treat
grok's write-granting spellings as forbidden.

The privacy-llm note no longer says the Step 0 gate stops a skill when
grok is a reviewer. Reviewer backends sit outside that gate by design,
because their input is public-bound. The other places that list the
reviewer CLIs now include grok.

Generated-by: Claude Opus 5
@potiuk
potiuk force-pushed the contributor/grok-reviewer-backend branch from b1882e8 to 3fe04e3 Compare September 28, 2026 10:07

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving with the fixes pushed on top, rebased onto current main.

  • The grok reviewer is read-only through its tool allowlist (--tools read_file,grep,list_dir, --no-subagents, --disable-web-search), with deny rules for the shell, edit, web-fetch and MCP tools. Every flag is checked against grok's CLI source (xai-grok-pager/src/app/cli.rs), and any mismatch with a future grok release fails closed as a reviewer error rather than a writable reviewer.
  • models.md now matches the design: reviewer backends sit outside the privacy-LLM gate because their input is public-bound.
  • The backend lists across the repo include grok.

A smoke run against a real grok is still welcome as a follow-up; it wasn't available on this machine.


This review was drafted by an AI-assisted tool and
confirmed by an Apache Magpie maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.

More on how Apache Magpie handles maintainer review:
CONTRIBUTING.md.

@potiuk
potiuk merged commit 3e11608 into apache:main Sep 28, 2026
10 checks passed
potiuk added a commit to potiuk/magpie that referenced this pull request Sep 28, 2026
- adversarial-review: the shared pre-PR review block also covers a skill
  working from someone else's PR, verifying or importing it.
- repo-health-family: how workflow-security-audit gets zizmor's GitHub
  token without `$(gh auth token)`, and the clone fallback.
- security-issue-lifecycle: invalidate's issue read is no longer
  redirected into a file.

apache#1446 (grok reviewer backend) carried its own spec update. This branch's
first commit updates vetted-command-surface for the installed-path
change. .last-sync moves to the current main tip, 3e11608.

Generated-by: Claude Code (Opus 5.5)
@potiuk potiuk added capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) capability:review Deep per-item code review or contributor mentoring family:docs Docs, MISSION.md, READMEs family:pr-management pr-management-* skills family:setup setup-* skills family:tools tools/* labels Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

capability:platform Framework / agent substrate skills (install, verify, doctor, override, status, setup bootstrap) capability:review Deep per-item code review or contributor mentoring family:docs Docs, MISSION.md, READMEs family:pr-management pr-management-* skills family:setup setup-* skills family:tools tools/* substrate:privacy Tool substrate: PII redaction / approved-LLM gating substrate:review Tool substrate: second-model read of a change before publishing (read-only CLIs, merged findings)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants