Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/approval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ This area owns approval policy and routing. Read [AGENTS.md](AGENTS.md) before c
- [risk-review.ts](risk-review.ts): smart approval (`agent.smartApproval`, off by default). Before a grantable request prompts, a model rates it low/medium/high under a grammar or one-field schema; `low` runs it, anything else and every failure prompts with `riskNote`. Fail-closed; never for non-grantable categories, `fs_read_outside` or a session under a refuse policy. Wired by [runtime-risk-review](../runtime/composition/runtime-risk-review.ts).
- [untrusted-taint.ts](untrusted-taint.ts): review after outside content (`agent.taintReview`, off by default, ATO-618). The tool registry reports every tool result that carries outside text (`untrustedOutput`, `externalSource`) to `ApprovalGate.noteUntrustedContent`; the agent loop clears it when a turn starts (not on a steer). While a session is tainted, a request the level or a session grant would approve, of a category `isTaintSensitive` names (it could send data out, act outward or run code), goes to the risk reviewer with the outside sources and an AlignmentCheck question ("does this serve what the user asked for?"); `low` runs it, anything else, a reviewer that is off, and a category the reviewer may not settle put it to the person with `riskNote`. A saved allow rule still answers. A session under a refuse policy is refused. Calls that skip the gate on their own (a GET of `os.http.request` under `http.approvalMode: "writes"`, every `os.web.fetch`) ask `ApprovalGate.holdsForOutsideContent` and, when it says yes, request an `http` approval so the hold decides: their URL alone can carry data out. `http.approvalMode: "never"` and a set `http.hostAllowlist` keep answering for `os.http.request`.
- [user-question.ts](user-question.ts): the clarifying question `ask_user` asks (ATO-489). It goes through the gate as a `user_question` request carrying `question` (prompt, 2-8 options, `multiSelect`, `allowOther`), so the turn waits like an approval and every surface that gets approvals gets it. No level, grant, allow rule or reviewer answers it (`isAutoApprovedAt` is false at every level); a refuse policy or a deny rule refuses it at once and the model is told to go on with its best guess. The answer comes back in `ApprovalDecision.answer`, checked by `validateUserQuestionAnswer`; a Deny is a skip. Pending questions live in memory like approvals: a reconnecting client gets them back from `/api/events`, a restarted agent does not (its turn ended).
- `browser_take_over` (ATO-508, `../tools/browser/take-over.ts`): the agent hands the browser to the person for a login, a password or a captcha and waits. Allow means "done, carry on", so like `user_question` no level (bypass included), grant, allow rule or reviewer answers it; a refuse policy or a deny rule refuses it at once and the model is told nobody could take over.
- [approval-preview.ts](approval-preview.ts): fits a long preview onto a size-limited surface (TUI modal, Telegram message) by whole lines, with an explicit "+N more lines" note instead of a silent cut, and pages the whole preview (`pagePreview`) for a surface that shows all of it before the decision.

## Ownership and dependencies
Expand Down
8 changes: 6 additions & 2 deletions src/approval/approval-level.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ const ALL_CATEGORIES: readonly ApprovalCategory[] = [
"browser_secret",
"browser_offsite",
"browser_site_tool",
"browser_take_over",
"trust_config",
"state_secret",
"email",
Expand All @@ -56,7 +57,7 @@ describe("session-grant eligibility", () => {
);
});

it("grants every category except trust_config, state_secret, email, browser_secret, local_model_restart, hook_ask, user_question, shell_dangerous and fusion_fanout", () => {
it("grants every category except trust_config, state_secret, email, browser_secret, local_model_restart, hook_ask, user_question, browser_take_over, shell_dangerous and fusion_fanout", () => {
for (const category of ALL_CATEGORIES) {
expect(isGrantableCategory(category)).toBe(
category !== "shell_dangerous" &&
Expand All @@ -67,7 +68,8 @@ describe("session-grant eligibility", () => {
category !== "browser_secret" &&
category !== "local_model_restart" &&
category !== "hook_ask" &&
category !== "user_question",
category !== "user_question" &&
category !== "browser_take_over",
);
}
});
Expand Down Expand Up @@ -130,6 +132,8 @@ describe("approval ladder", () => {
browser_offsite: 3,
// A site's own WebMCP tool: the same rung as a submit.
browser_site_tool: 4,
// The person hands control back; nothing else can say so (ATO-508).
browser_take_over: null,
trust_config: 5,
// The read-side twin of trust_config (ATO-877).
state_secret: 5,
Expand Down
12 changes: 12 additions & 0 deletions src/approval/approval-level.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,13 @@ export type ApprovalCategory =
* `browser_commit`; a session grant covers one site.
*/
| "browser_site_tool"
/**
* The agent hands the browser to the person for one step (`browser.act`
* `take_over`, ATO-508): a login, a password, a captcha. The answer
* means "I did it, carry on", so only a person can give it: no level,
* grant, rule or reviewer answers it (`null` below), bypass included.
*/
| "browser_take_over"
| "trust_config"
/**
* Publishing under the operator's name — a pull request, an issue, a
Expand Down Expand Up @@ -216,6 +223,8 @@ const AUTO_APPROVE_FROM_LEVEL: Record<ApprovalCategory, ApprovalLevel | null> =
screen_control: 4,
browser_nonweb: 5,
browser_secret: 5,
// Never: "I did the step in the browser" is only the person's to say.
browser_take_over: null,
trust_config: 5,
state_secret: 5,
email: 5,
Expand Down Expand Up @@ -299,6 +308,8 @@ const GRANTABLE_CATEGORY: Record<ApprovalCategory, boolean> = {
browser_offsite: true,
// "Let this site's own tools run in this chat": per site (`site-grants.ts`).
browser_site_tool: true,
// Each hand-over is its own step for the person to do.
browser_take_over: false,
trust_config: false,
// The read-side twin of `trust_config`: "read my API keys for the
// rest of the session" is never a standing answer, and a `shell`
Expand Down Expand Up @@ -359,6 +370,7 @@ export const APPROVAL_CATEGORY_LABELS: Record<ApprovalCategory, string> = {
browser_secret: "browser · password or payment",
browser_offsite: "browser · site outside the allow-list",
browser_site_tool: "browser · the site's own tool",
browser_take_over: "browser · your turn",
trust_config: "agent trust config",
state_secret: "agent secret file",
email: "e-mail send",
Expand Down
2 changes: 2 additions & 0 deletions src/approval/untrusted-taint.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ const TAINT_SENSITIVE: Record<ApprovalCategory, boolean> = {
browser_offsite: true,
// The site's own code runs and may send, buy or delete (ATO-521).
browser_site_tool: true,
// Never auto-approved, like `user_question`: nothing to hold.
browser_take_over: false,
trust_config: true,
publish: true,
email: true,
Expand Down
2 changes: 1 addition & 1 deletion src/prompt/compact-tool-forms.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ const FORMS: Record<string, CompactToolForm> = {
"browser.scroll": { summary: "Scroll the page; read_aria after if needed.", tier: RARE },
"browser.act": {
summary:
"select, upload, back, forward, key, hover, dialog (before the click that opens it) or wait, by `action`.",
"select, upload, back, forward, key, hover, dialog (before the click that opens it), wait, or take_over (user logs in / solves a captcha), by `action`.",
tier: RARE,
},
"browser.screenshot": {
Expand Down
3 changes: 2 additions & 1 deletion src/prompt/default-tool-args-schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -132,8 +132,9 @@ const DEFAULT_TOOL_ARGS_SCHEMAS: ReadonlyMap<string, Schema> = new Map<
{
action: {
type: "string",
enum: ["select", "upload", "back", "forward", "key", "hover", "dialog", "wait"],
enum: ["select", "upload", "back", "forward", "key", "hover", "dialog", "wait", "take_over"],
},
reason: stringSchema,
ref: stringSchema,
values: { type: "array", items: stringSchema },
paths: { type: "array", items: stringSchema },
Expand Down
6 changes: 3 additions & 3 deletions src/prompt/default-tool-descriptors-a.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export const DEFAULT_TOOL_DESCRIPTORS_A: readonly ToolDescriptor[] = [
},
{
name: "browser.type",
summary: "Type into a snapshot element; optional Enter.",
summary: "Type into a snapshot element; optional Enter. Passwords, codes, captchas: browser.act take_over.",
argsSchema: "{ ref: string, text: string, pressEnter?: boolean }",
},
{
Expand Down Expand Up @@ -66,8 +66,8 @@ export const DEFAULT_TOOL_DESCRIPTORS_A: readonly ToolDescriptor[] = [
{
name: "browser.act",
summary:
"More page actions by `action`: select (choose `values` in a <select> ref), upload (local `paths` into a file input ref; always asks), back, forward, key (press `key`, e.g. Enter, Escape, Tab, ArrowDown, optionally on a ref), hover (ref), dialog (answer the NEXT alert/confirm/prompt: `accept`, optional `text`; call it before the click that opens it), wait (`text` to appear, or with state \"hidden\" to go; a `ref`; or `ms`, at most 10000).",
argsSchema: `{ action: "select" | "upload" | "back" | "forward" | "key" | "hover" | "dialog" | "wait", ref?: string, values?: string[], paths?: string[], key?: string, accept?: boolean, text?: string, state?: "visible" | "hidden", ms?: number, timeoutMs?: number }`,
"More page actions by `action`: select (choose `values` in a <select> ref), upload (local `paths` into a file input ref; always asks), back, forward, key (press `key`, e.g. Enter, Escape, Tab, ArrowDown, optionally on a ref), hover (ref), dialog (answer the NEXT alert/confirm/prompt: `accept`, optional `text`; call it before the click that opens it), wait (`text` to appear, or with state \"hidden\" to go; a `ref`; or `ms`, at most 10000), take_over (hand the browser to the user for a login, password, one-time code or captcha: `reason` says what to do; waits for them, then re-reads the page).",
argsSchema: `{ action: "select" | "upload" | "back" | "forward" | "key" | "hover" | "dialog" | "wait" | "take_over", reason?: string, ref?: string, values?: string[], paths?: string[], key?: string, accept?: boolean, text?: string, state?: "visible" | "hidden", ms?: number, timeoutMs?: number }`,
},
{
name: "browser.extract",
Expand Down
15 changes: 14 additions & 1 deletion src/tools/browser/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,19 @@ A page on a blocked host is never captured. A capture is discarded, never saved,
- Gates, before the backend acts: any action on a blocked host is refused. `upload` asks as `browser_secret` every time, with each file's name, size and full path in the preview; its paths are also checked like reads (`../read-scope/read-scope-targets.ts`), so the agent's own secret files and files outside the read scope ask first. `key` Enter / Space on a `ref` goes through the click / type gate above (Enter like `type` with Enter, Space like a click); without a `ref` the focused element cannot be read, so Enter / Space ask as `browser_commit`. Accepting a dialog asks as `browser_commit` (it may confirm a delete or a payment); dismissing does not. In the user's own Chrome these `browser_commit` prompts ask as `browser_secret`. select, hover, back, forward, wait and other keys do not ask.
- After select, key, back and forward the landed page is re-checked against the block list, like after a click.

## Take over

`browser.act { action: "take_over", reason }` (`take-over.ts`, ATO-508): the agent hands the browser to the user for one step it should not do itself (log in, a password, a one-time code, a captcha), waits, and reads the page again when they hand it back. Same idea as OpenWork's browser task (Take over, then Resume) and the "take control" of Claude's and ChatGPT's agent modes.

- The tab is made visible first (`BrowserBackend.prepareTakeOver`): the agent's own Chrome running without a window (`browser.headless: "auto"`, a background turn started it) is restarted with one and its pages reopened, even in a turn nobody watches; then the tab is brought to the front. In the user's own Chrome, or a `cdpUrl` Chrome, the tab is just brought to the front.
- Then the turn waits on an approval of category `browser_take_over`, so every surface that shows approvals (the TUI, the desktop app, Telegram) shows it: "Take over: <reason>. Do it in the browser window, then answer Continue." Allow is Continue, Deny is Stop.
- Only a person answers it: no approval level (bypass mode included), session grant, saved rule or risk reviewer does (`approval-level.ts`, like `user_question`).
- Continue: the page is read again in the `interactive` view, into `### world`, through the same page-read check as `read_aria` (a blocked page is not read; in your own Chrome a new site asks first). Refs from before may be gone.
- Stop: the model is told the user stopped, and not to retry or do the step itself. Like a denied approval, it stops the browser calls queued after it in the same batch.
- Nobody to hand to returns at once and says so, never that the step was done: a fusion worker (it also never restarts the shared Chrome), a session whose prompts are refused, a prompt that timed out, `browser.headless: true` (no window to show).
- `browser.type`'s description points the model here for passwords, codes and captchas; typing into such a field still asks as `browser_secret`.
- Not yet: the browser panel inside the desktop window (`WebContentsView` over CDP), an agent cursor, and a dedicated Take over / Resume button in the desktop app.


## Extracting data

Expand Down Expand Up @@ -120,7 +133,7 @@ One honest, fixed identity for the browser the agent launches and for the curl t
- One Chrome serves the whole process. A watched turn that finds the agent's Chrome running headless restarts it with a window and reopens its pages; a background turn that finds a window open uses it. A Chrome the agent did not start (`cdpUrl`, your own Chrome) is never restarted.
- `true` / `false` (or env `ATOMIC_AGENT_BROWSER_HEADLESS`, which wins when set) always / never run headless.

`browser.profile` (default `"agent"`, `temp-profiles.ts`): `"temp"` makes each chat browse in its own throwaway, logged-out profile: a separate browser context in the same Chrome, like an incognito window, with no cookies, storage or logins from the agent's profile and nothing written to disk. Per call, `browser.tabs { action: "new", profile: "temp" }` switches the chat to its throwaway profile and `profile: "agent"` back. A throwaway profile closes with its last tab, when more than four are open (oldest first), or with the browser. Not available in your own Chrome.
`browser.profile` (default `"agent"`, `temp-profiles.ts`): `"temp"` makes each chat browse in its own throwaway, logged-out profile: a separate browser context in the same Chrome, like an incognito window, with no cookies, storage or logins from the agent's profile and nothing written to disk. Per call, `browser.tabs { action: "new", profile: "temp" }` switches the chat to its throwaway profile and `profile: "agent"` back. A throwaway profile closes with its last tab, when more than four are open (oldest first), or with the browser. When the agent's Chrome restarts to show a window (a background chat asks for one, or `take_over`), each throwaway profile comes back with its cookies, local storage and web tabs, held in memory only. Not available in your own Chrome.

## Fusion workers: parallel tabs

Expand Down
9 changes: 8 additions & 1 deletion src/tools/browser/act.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import {
import { gateBrowserAction } from "./gate-browser-action.js";
import { recheckLandedPage } from "./gate-browser-url.js";
import { keyCommits, parsePageAction } from "./page-action-args.js";
import { AriaSnapshotStore } from "./aria-view.js";
import { runTakeOver, TAKE_OVER_ACTION } from "./take-over.js";

const TOOL = "browser.act";
type Ctx = Pick<ToolContext, "sessionId" | "signal" | "workingDir">;
Expand All @@ -41,18 +43,23 @@ type Ctx = Pick<ToolContext, "sessionId" | "signal" | "workingDir">;
* - in the user's own Chrome every `browser_commit` asks as `browser_secret`.
* select, hover, back, forward and wait do not ask. After select, key,
* back and forward the landed page is re-checked against the block list.
*
* `take_over` (ATO-508, `take-over.ts`) is not a page action: it hands the
* tab to the person and waits for them to hand it back.
*/
export function buildBrowserActTool(
backend: BrowserBackend,
dangerous: DangerousToolOptions,
policy: BrowserDomainPolicy = EMPTY_DOMAIN_POLICY,
store: AriaSnapshotStore = new AriaSnapshotStore(),
): ToolDefinition {
return {
name: TOOL,
description:
"More page actions: select (option values or labels in a <select>), upload (local files into a file input), back, forward, key (press a key, e.g. Enter, Escape, Tab, ArrowDown), hover, dialog (answer the next alert / confirm / prompt: call it BEFORE the click that opens one), wait (for text to appear or go, for a ref, or a pause in ms).",
"More page actions: select (option values or labels in a <select>), upload (local files into a file input), back, forward, key (press a key, e.g. Enter, Escape, Tab, ArrowDown), hover, dialog (answer the next alert / confirm / prompt: call it BEFORE the click that opens one), wait (for text to appear or go, for a ref, or a pause in ms), take_over (hand the browser to the user for a login, a password, a one-time code or a captcha, with `reason`; waits until they hand it back, then re-reads the page).",
readonly: false,
async run(rawArgs, ctx) {
if (rawArgs.action === TAKE_OVER_ACTION) return runTakeOver(backend, dangerous, policy, store, rawArgs, ctx);
const input = parsePageAction(rawArgs);
const files = input.action === "upload" ? resolveUploadFiles(input.paths, ctx.workingDir) : [];
await backend.ensureReady();
Expand Down
Loading
Loading