Skip to content

fix(browser): keep the Annotate note box out of the page's reach (ATO-1543) - #1608

Draft
sosidudku1 wants to merge 1 commit into
mainfrom
fix/ato-1543-annotate-note-outside-page
Draft

sosidudku1 wants to merge 1 commit into
mainfrom
fix/ato-1543-annotate-note-outside-page

Conversation

@sosidudku1

Copy link
Copy Markdown
Collaborator

Linear: ATO-1543 (follow-up to ATO-518 / #1510)

What leaked

Annotate mode put a small comment box on the page itself. The overlay runs in an isolated world inside a closed shadow root, so the page cannot read the saved pins. But typing is different: key and input events in the page's document are composed, and a page script listening on window in the capture phase runs before anything in the overlay can stop them. So any site could read every key the user typed into the note box. #1510 only added a "don't type passwords here" line.

What changed

  • The overlay no longer has a text box. A click on an element drops a numbered pin right away (click a pin to remove it, same as before). Nothing on the page takes focus and the overlay has no key listeners, so keys go wherever the user had them.
  • The user writes what to change for each pin number in the agent's own chat ("1: bigger, 2: drop it"). The page has no way to reach that.
  • browser.annotate results say so: start tells the model to ask the user to write per-pin notes in the chat, not on the page; read returns pins without text and tells the model to pair pin numbers with the user's chat messages and to ask when a pin has no words yet.
  • Pins no longer carry a comment field; anything a script sends back under that name is dropped.
  • Tool descriptions, compact form, README and code comments updated.

Precedent

Same split as Playwright codegen's element picker and Claude in Chrome: pick the element in the page, take the text in the host's UI. I looked at keeping the box in the page with an isolated world plus stopping propagation, and it does not work: a page capture listener on window registered earlier always runs first.

Tests

  • annotate-in-page.test.ts: the start script has no textarea, input, contentEditable, .focus( or key listeners, and builds no such element; a real click on a page element drops a numbered pin with no comment; a repeat click on the same element and a synthetic (untrusted) click add nothing; read returns pins without comment.
  • annotate-pins.test.ts: pins without a comment are kept; a comment field from a script is dropped and never rendered.
  • annotate.test.ts: start sends the user to the chat; read renders pins as [n] <tag> "text" and points the model at the chat messages.

Not yet checked on the second Mac.

…-1543)

The annotate overlay had a comment box inside the page. Key and input
events in the page's document are composed, so a page script listening
on window in the capture phase saw every key the user typed there, even
with the overlay in an isolated world and a closed shadow root.

The overlay now only picks elements: a click drops a numbered pin at
once, nothing on the page takes focus or listens to keys. The user says
what to change for each pin number in the agent's chat, and the tool
results tell the model to pair pins with those chat messages (and to ask
when a pin has no words yet). Pins no longer carry a comment field.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant