Skip to content

Browser: Take over, the user finishes a step and hands back (ATO-508) - #1679

Draft
sosidudku1 wants to merge 3 commits into
mainfrom
feat/ato-508-browser-take-over
Draft

sosidudku1 wants to merge 3 commits into
mainfrom
feat/ato-508-browser-take-over

Conversation

@sosidudku1

@sosidudku1 sosidudku1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

The agent can now hand the browser to the user for one step it should not do itself (log in, a password, a one-time code, a captcha), wait, and pick up again when the user hands it back.

browser.act { action: "take_over", reason: "log in to example.com" }:

  1. Makes the agent's tab visible. If the agent's own Chrome runs without a window (browser.headless: "auto" and a background turn started it), it is restarted with one and its pages are reopened, even in a turn nobody watches (the user may answer from Telegram and walk to the machine). Then the tab is brought to the front. The user's own Chrome and a cdpUrl Chrome are used as they are.
  2. Pauses the turn on an approval of a new category, browser_take_over: "Take over: log in to example.com. Do it in the browser window, then answer Continue." Allow is Continue, Deny is Stop. It goes through the normal approval gate, so the TUI, the desktop app and Telegram already show it with no UI change.
  3. Continue: the page is read again (the compact interactive view, into ### world, through the same page-read check as read_aria), so the model sees what the user did.
  4. Stop: the model is told the user stopped, not to retry and not to do the step itself.

Only a person can say "done"

  • browser_take_over is never auto-approved: no approval level (bypass mode is level 5, it still asks), no session grant, no "always" rule, no risk reviewer. Same treatment as ask_user's user_question.
  • When nobody can answer, it returns at once and says so, never that the step was done: a fusion worker (it also never restarts the shared Chrome), a session whose prompts are refused, a prompt that timed out, browser.headless: true (no window to show), and the test seam that skips the gate.
  • Stop and every "nobody" outcome mark the result as not approved, so browser calls queued after it in the same batch (a click planned for "after the login") do not run.

browser.type's description and the prompt catalog now point the model to take_over for passwords, codes and captchas. Typing into such a field still asks as browser_secret, as before.

Precedent

  • OpenWork's browser task: when the page reaches a password field the agent asks the user to "Take over", the run pauses, the user acts, presses Resume, and the agent re-reads the page.
  • Claude's and ChatGPT's agent modes: "Take control" / "you're in control", then hand back.
  • AionUi does the same.

Left for later

  • The browser panel inside the desktop window (WebContentsView) and the CDP bridge to it.
  • The animated agent cursor.
  • A dedicated Take over / Resume button in the desktop UI (today the desktop shows the generic approval card).

Tests

  • src/tools/browser/take-over.test.ts: the prompt and its wording, Continue re-reads the page, Stop, refusal by nobody, no window, fusion worker, the gate seam, blocked host before and after, the user's own Chrome, reason parsing; plus a real ApprovalGate at level 5 that still asks and ignores a session grant, and one under a refuse policy.
  • src/tools/browser/background-browser.test.ts: a background turn's headless Chrome is restarted with a window and its page reopened; an existing window is only brought to the front; headless: true reports no window; a cdpUrl Chrome is never restarted.
  • src/approval/approval-level.test.ts: the new category in the ladder and grant tables.

Verification

  • Review found: in a background turn whose chat browses in a throwaway profile (browser.profile temp or tabs new profile temp), take_over restarted the headless Chrome with a window. The restart reopened only the agent profile's tabs, so the person was handed a blank window and the login page was gone.
  • Changed (product decision 18.11: carry the pages over rather than refuse): before the restart, TempProfiles.snapshot() takes each throwaway profile's cookies and local storage (storageState, in memory only, never on disk) and its web tabs; after it, restore() opens each session's context again with that state and its tabs at the same URLs. take_over then hands over the same page in a window. This applies to every restart with a window, not only take_over. Tests in background-browser.test.ts.
  • Not carried over: session storage, in-page state (half-filled forms), and tabs that were not on http(s). The page is loaded again from its URL.
  • MBP run must check: the new test passes; a cron turn with browser.profile: "temp" that opens a login page and calls take_over shows that page in a window, and its cookies are still there (still logged-out from the agent's own profile).

Not yet checked on the second Mac.

browser.act { action: "take_over", reason } makes the agent's tab
visible (a headless Chrome the agent started is restarted with a window),
brings it to the front and waits on a browser_take_over approval:
Allow is Continue, Deny is Stop. The new category is never answered by a
level (bypass included), a grant, a saved rule or the risk reviewer.
Continue re-reads the page into ### world; Stop, or nobody to answer,
says so and stops the browser calls queued after it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant