Repository navigation
Screen control: numbered app elements and background press (ATO-998, ATO-530) - #1680
Draft
sosidudku1 wants to merge 3 commits into
Draft
sosidudku1 wants to merge 3 commits into
sosidudku1 wants to merge 3 commits into
Conversation
…ATO-530) os.screen.elements lists one app window's controls as a numbered Accessibility list ([n] role "name" value? (x,y,w,h) actions) and keeps the element paths per session and app. os.screen.press presses element n (AXPress) or sets a text field's value in the background: no mouse event, no bring-to-front, no silent fallback to a click. Same protected-app rules and screen_control approval as click/type; the element is re-resolved by path, role and name at action time.
…nly role and name (ATO-998, ATO-530)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Two new macOS screen tools. Together they let the agent work in an app by number instead of by pixels, while the user keeps their mouse and focus.
os.screen.elements { app, window?, maxElements? }(ATO-998, item 2)[3] button "Save" (812,440,60,24) press[4] textfield "To:" value="ann@example.com" (160,130,500,22) confirmAXPress. Containers and plain text are walked through. Hidden (zero-sized) elements are skipped.screen_capture, like a window snapshot with its text.ScreenElementStore, bounded likeScreenGeometryStore), so a later call can sayelement: 3.os.screen.press { app, element, window?, value? }(ATO-530)AXPresson element n. Withvalue, it sets a text field'sAXValueinstead (folded into this tool rather than a separateset_value).Both run inside the same constant JXA script as the other screen ops (new
elementsandpressops). The path, the text and the pinned app all travel in the JSON env payload, never in the script source.Safety rules
AXPress, or a value can't be set, the tool refuses. It never clicks or types instead. The refusal points toos.screen.click/os.screen.type, which move the mouse, bring the app forward and ask the user again.screen-protected.ts):valuecounts as typing, so it is also refused in terminals and IDEs.WINDOW_CAPTURE_REFUSED_BUNDLE_IDS, re-checked by pid inside the script). Apps where a press would be refused anyway are not listed either: blocked apps, and browsers, wherebrowser.read_ariais the list to use. Password fields are listed without their value.screen_controland the prompt names the element, the window and the app. Any text being set is shown in full. The script checks again that the process still belongs to the approved app and isn't protected.tools.screen.enabledon macOS. They sit in the same serial resource class as the other screen tools, and Fusion workers never see them.Precedent
delivery_mode: background). We copy its rule that a requested action the driver can't do is a refusal, never a quiet switch of delivery mode.screen-protected.ts.browser.read_aria): the numbered format small models already handle.What's left
os.screen.pointyet. That would be a small follow-up: frames are already global screen points.Tests
screen-elements.test.tsuses thescreen-host.fixture.tspattern. No JXA runs in it. It covers:maxElementsboundspressElementhas no bring-to-front, no CGEvent and no clickCatalog wiring is updated too: descriptors, schemas, compact forms, grammar, resource classes, the gate filter, worker exclusions, and the related pinned lists and counts.
Verification
findWindowpicks the same-titled window at the stored frame. With one window of that title it uses it even if moved; with several it refuseswindow-changedwhen none sits at the stored frame (window-ambiguouswhen no frame was stored).pressElementalso refuseselement-changedunless the element sits at the same place relative to its window (4 pt tolerance; moving the whole window is fine). New tests inscreen-elements.test.ts.window-ambiguousorwindow-changed, nothing sent. In a list with several "Delete" buttons, list, re-sort, press: expectelement-changed. Moving the window between list and press still presses. The new System Events callsposition()/size()on windows run fast enough.Not yet checked on the second Mac.