Repository navigation
Browser: Take over, the user finishes a step and hands back (ATO-508) - #1679
Draft
sosidudku1 wants to merge 3 commits into
Draft
sosidudku1 wants to merge 3 commits into
sosidudku1 wants to merge 3 commits into
Conversation
browser.act { action: "take_over", reason } makes the agent's tab
visible (a headless Chrome the agent started is restarted with a window),
brings it to the front and waits on a browser_take_over approval:
Allow is Continue, Deny is Stop. The new category is never answered by a
level (bypass included), a grant, a saved rule or the risk reviewer.
Continue re-reads the page into ### world; Stop, or nobody to answer,
says so and stops the browser calls queued after it.
…browses in a throwaway profile (ATO-508)
… window (ATO-508)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
The agent can now hand the browser to the user for one step it should not do itself (log in, a password, a one-time code, a captcha), wait, and pick up again when the user hands it back.
browser.act { action: "take_over", reason: "log in to example.com" }:browser.headless: "auto"and a background turn started it), it is restarted with one and its pages are reopened, even in a turn nobody watches (the user may answer from Telegram and walk to the machine). Then the tab is brought to the front. The user's own Chrome and acdpUrlChrome are used as they are.browser_take_over: "Take over: log in to example.com. Do it in the browser window, then answer Continue." Allow is Continue, Deny is Stop. It goes through the normal approval gate, so the TUI, the desktop app and Telegram already show it with no UI change.interactiveview, into### world, through the same page-read check asread_aria), so the model sees what the user did.Only a person can say "done"
browser_take_overis never auto-approved: no approval level (bypass mode is level 5, it still asks), no session grant, no "always" rule, no risk reviewer. Same treatment asask_user'suser_question.browser.headless: true(no window to show), and the test seam that skips the gate.browser.type's description and the prompt catalog now point the model totake_overfor passwords, codes and captchas. Typing into such a field still asks asbrowser_secret, as before.Precedent
Left for later
WebContentsView) and the CDP bridge to it.Tests
src/tools/browser/take-over.test.ts: the prompt and its wording, Continue re-reads the page, Stop, refusal by nobody, no window, fusion worker, the gate seam, blocked host before and after, the user's own Chrome, reason parsing; plus a realApprovalGateat level 5 that still asks and ignores a session grant, and one under a refuse policy.src/tools/browser/background-browser.test.ts: a background turn's headless Chrome is restarted with a window and its page reopened; an existing window is only brought to the front;headless: truereports no window; acdpUrlChrome is never restarted.src/approval/approval-level.test.ts: the new category in the ladder and grant tables.Verification
browser.profile temportabs new profile temp), take_over restarted the headless Chrome with a window. The restart reopened only the agent profile's tabs, so the person was handed a blank window and the login page was gone.TempProfiles.snapshot()takes each throwaway profile's cookies and local storage (storageState, in memory only, never on disk) and its web tabs; after it,restore()opens each session's context again with that state and its tabs at the same URLs. take_over then hands over the same page in a window. This applies to every restart with a window, not only take_over. Tests inbackground-browser.test.ts.browser.profile: "temp"that opens a login page and calls take_over shows that page in a window, and its cookies are still there (still logged-out from the agent's own profile).Not yet checked on the second Mac.