Repository navigation
Screen capture on Windows and Linux (ATO-527) - #1675
Draft
sosidudku1 wants to merge 3 commits into
Draft
sosidudku1 wants to merge 3 commits into
sosidudku1 wants to merge 3 commits into
Conversation
os.screen.capture now takes a whole-screen PNG on Windows (constant PowerShell CopyFromScreen script, DPI aware, path in env) and Linux (grim, gnome-screenshot, spectacle, scrot or import, first that works). Click, type, key and point stay macOS-only; their off-mac message says capture works here.
…red /tmp (ATO-527)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First part of ATO-527: the agent can now look at the screen on Windows and Linux. It cannot click or type there yet.
What changes
With
tools.screen.enabledon,os.screen.captureis now registered on Windows and Linux too, not just on macOS. Called with no arguments, it takes a PNG of the whole screen (every display) and saves it in the same temp folder as the macOS screenshots (<tmp>/atomic-agent-screens/<session>/). It asks the user first under the samescreen_capturecategory. The result has the same shape as on macOS: path, pixel size, scale, origin, and the session remembers the geometry.System.Windows.Forms/System.DrawingCopyFromScreen. The script is sent as-EncodedCommand, so nothing gets quoted. The output path and the 1280 px size cap go in through environment variables (ATOMIC_SCREEN_OUT,ATOMIC_SCREEN_MAX_SIDE) and never into the script text. It callsSetProcessDPIAwarebefore it reads any screen size, so a 150% display is captured in full, in physical pixels. It also shrinks the image itself and prints a small JSON report. A locked PC, the UAC prompt, or a session with no desktop (a service, SSH) gets a clear error.sh -c 'command -v …'loop finds the installed tools, with the tool names passed as argv. Then it tries, in order:grim(wlroots Wayland),gnome-screenshot -f,spectacle -b -n -f -o(KDE), and on X11 onlyscrotandimport -window root. If one tool produces no PNG, the next one is tried. The size is read from the PNG header. Ifmagickorconvertis installed, they shrink the picture to 1280 px. Otherwise it stays full size, with a note.app,windowortextis refused on Windows and Linux. Window snapshots stay macOS-only for now.os.screen.click,type,keyandpointstay macOS-only and are not registered on Windows or Linux. Their off-mac message now says that capture works on this OS and control is macOS-only for now.filterToolDescriptorsByConfig) gets an optionalscreen.controlflag. Withcontrol: false, onlyos.screen.captureis kept. The runtime catalog and/toolspasscontrol: process.platform === "darwin", so the prompt matches what is registered.Precedent
cua-driverfor computer use on macOS, Windows and Linux.wmctrlon Linux foros.window.*(src/tools/os/window.ts). This change follows that pattern, using the same command runner.Next slice
Input on Windows and Linux (SendInput on Windows, xdotool / ydotool on Linux, or
cua-driveras an external MCP server) is the next step, and it needs checks on real machines.cua-perception(AGPL) andcuabotare not used.Files
src/tools/os/screen/screen-capture-win-linux.ts(the script, argv, tool choice, parsers) and its test.screen-capture.ts: the Windows/Linux flow (choose a tool, ask, capture, remember the geometry).screen-host.ts:screenCaptureSupported/screenControlSupported, areadHeaddependency for the PNG header, and the new off-mac message.index.ts: registration per platform.filter-disabled-tools.ts,runtime-tool-catalog.ts,tools-listing.ts: thecontrolgate.src/tools/os/README.md. The prompt descriptor summary mentions the Windows/Linux limits.screen-jxa.tsandscreen-input.tsare untouched.screen-input.test.tshas one changed expectation: Linux now registers capture.No real capture runs in tests. Not run on real Windows or Linux yet. The
tools.screen.enabledtext in the config docs still says macOS only, because changing it means regeneratingsettings-docs.generated.ts.Verification
os.tmpdir()is the shared/tmp, and the screenshot folder/tmp/atomic-agent-screens/<session>/was created with default permissions under a predictable name. Another local user could read a full-screen capture, or pre-create the parent (or a symlink) and own what lands inside.ensurePrivateDirinscreen-host.tscreates the folder 0700, checks the parent and the session folder are plain folders (not symlinks) owned by this user, sets both back to 0700, and refuses the capture otherwise. On Windows (no uid, per-user temp) only the symlink check applies. Tests inscreen-capture-win-linux.test.ts.Not yet checked on the second Mac.