Skip to content

Screen control: numbered app elements and background press (ATO-998, ATO-530) - #1680

Draft
sosidudku1 wants to merge 3 commits into
mainfrom
feat/ato-998-screen-elements
Draft

sosidudku1 wants to merge 3 commits into
mainfrom
feat/ato-998-screen-elements

Conversation

@sosidudku1

@sosidudku1 sosidudku1 commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

Two new macOS screen tools. Together they let the agent work in an app by number instead of by pixels, while the user keeps their mouse and focus.

os.screen.elements { app, window?, maxElements? } (ATO-998, item 2)

  • Reads one app window's Accessibility tree and returns its controls as a short numbered list, like the browser tool's ARIA refs:
    [3] button "Save" (812,440,60,24) press
    [4] textfield "To:" value="ann@example.com" (160,130,500,22) confirm
  • Only controls are listed: buttons, text fields, checkboxes, radio buttons, pop-ups, menu items, links, rows, cells, tabs, sliders, plus images, texts or groups that answer AXPress. Containers and plain text are walked through. Hidden (zero-sized) elements are skipped.
  • Limits like the existing window-text walk: depth 14, 800 elements looked at, 60 listed by default, 200 at most.
  • Read-only. It asks as screen_capture, like a window snapshot with its text.
  • Each element's child-index path, role and name are kept per session and app (ScreenElementStore, bounded like ScreenGeometryStore), so a later call can say element: 3.

os.screen.press { app, element, window?, value? } (ATO-530)

  • Runs AXPress on element n. With value, it sets a text field's AXValue instead (folded into this tool rather than a separate set_value).
  • Background delivery: no mouse or keyboard event is posted, and the app is not brought to the front. If the app raises itself in response, the result says so.
  • At action time the script finds the same window by its title, walks the stored path again, and refuses unless the same role and name are still there. A restarted app or a changed window means "list again".

Both run inside the same constant JXA script as the other screen ops (new elements and press ops). The path, the text and the pinned app all travel in the JSON env payload, never in the script source.

Safety rules

  • No silent fallback. If an element doesn't support AXPress, or a value can't be set, the tool refuses. It never clicks or types instead. The refusal points to os.screen.click / os.screen.type, which move the mouse, bring the app forward and ask the user again.
  • Same app tiers as click and type (screen-protected.ts):
    • A press counts as a click. It is refused in web browsers, Atomic Agent itself, settings, password managers and script runners. It is allowed in terminals and IDEs.
    • Setting a value counts as typing, so it is also refused in terminals and IDEs.
  • What can be listed. Password and login windows are never read (WINDOW_CAPTURE_REFUSED_BUNDLE_IDS, re-checked by pid inside the script). Apps where a press would be refused anyway are not listed either: blocked apps, and browsers, where browser.read_aria is the list to use. Password fields are listed without their value.
  • Approval. A press asks as screen_control and the prompt names the element, the window and the app. Any text being set is shown in full. The script checks again that the process still belongs to the approved app and isn't protected.
  • Both tools exist only with tools.screen.enabled on macOS. They sit in the same serial resource class as the other screen tools, and Fusion workers never see them.

Precedent

  • Hermes / cua-driver background delivery (delivery_mode: background). We copy its rule that a requested action the driver can't do is a refusal, never a quiet switch of delivery mode.
  • OpenClaw cua-computer: AX element lists for desktop apps.
  • Claude computer use: per-app tiers (browsers read-only, terminals click-only), already mirrored in screen-protected.ts.
  • This repo's browser ARIA ref list (browser.read_aria): the numbered format small models already handle.

What's left

  • Windows and Linux (ATO-527): both tools give the existing mac-only message there.
  • ATO-998 item 3 (a stop button and a screen border while the agent is in control) isn't in this PR. I found nothing for it in the tree yet.
  • The elements list isn't drawn by os.screen.point yet. That would be a small follow-up: frames are already global screen points.

Tests

screen-elements.test.ts uses the screen-host.fixture.ts pattern. No JXA runs in it. It covers:

  • parsing the script's JSON (elements, outcome, focus)
  • line format and the maxElements bounds
  • the element store and how it evicts old entries
  • the env payload (model text never reaches the script)
  • a source check that pressElement has no bring-to-front, no CGEvent and no click
  • listing, storing, and refusals by app class
  • press with the background pin, value setting, no-fallback refusals, element-changed and stale-pid refusals
  • a declined approval, and terminals (press allowed, value refused)

Catalog wiring is updated too: descriptors, schemas, compact forms, grammar, resource classes, the gate filter, worker exclusions, and the related pinned lists and counts.

Verification

  • Review found two ways a press could hit a control other than the approved one:
    1. The press re-check compared only role and name at the stored path. After a re-sort or an inserted row, the next row's "Delete" (or an unnamed icon button) sits at that path and passed.
    2. Windows were found by title only, so with two "New Message" windows a press could land in the other one if the z-order changed during the approval wait.
  • Changed: the listing now walks the window the probe picked (by its frame) and stores that window's frame. The press sends the window frame and the element frame. In the script, findWindow picks the same-titled window at the stored frame. With one window of that title it uses it even if moved; with several it refuses window-changed when none sits at the stored frame (window-ambiguous when no frame was stored). pressElement also refuses element-changed unless the element sits at the same place relative to its window (4 pt tolerance; moving the whole window is fine). New tests in screen-elements.test.ts.
  • MBP run must check: with two Mail compose windows titled "New Message", list one, click the other to bring it forward, press: expect window-ambiguous or window-changed, nothing sent. In a list with several "Delete" buttons, list, re-sort, press: expect element-changed. Moving the window between list and press still presses. The new System Events calls position() / size() on windows run fast enough.

Not yet checked on the second Mac.

…ATO-530)

os.screen.elements lists one app window's controls as a numbered
Accessibility list ([n] role "name" value? (x,y,w,h) actions) and keeps
the element paths per session and app. os.screen.press presses element n
(AXPress) or sets a text field's value in the background: no mouse event,
no bring-to-front, no silent fallback to a click. Same protected-app rules
and screen_control approval as click/type; the element is re-resolved by
path, role and name at action time.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant