Skip to content

Screen capture on Windows and Linux (ATO-527) - #1675

Draft
sosidudku1 wants to merge 3 commits into
mainfrom
feat/ato-527-screen-capture-win-linux
Draft

sosidudku1 wants to merge 3 commits into
mainfrom
feat/ato-527-screen-capture-win-linux

Conversation

@sosidudku1

@sosidudku1 sosidudku1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

First part of ATO-527: the agent can now look at the screen on Windows and Linux. It cannot click or type there yet.

What changes

With tools.screen.enabled on, os.screen.capture is now registered on Windows and Linux too, not just on macOS. Called with no arguments, it takes a PNG of the whole screen (every display) and saves it in the same temp folder as the macOS screenshots (<tmp>/atomic-agent-screens/<session>/). It asks the user first under the same screen_capture category. The result has the same shape as on macOS: path, pixel size, scale, origin, and the session remembers the geometry.

  • Windows: one fixed PowerShell script that copies the virtual screen with System.Windows.Forms / System.Drawing CopyFromScreen. The script is sent as -EncodedCommand, so nothing gets quoted. The output path and the 1280 px size cap go in through environment variables (ATOMIC_SCREEN_OUT, ATOMIC_SCREEN_MAX_SIDE) and never into the script text. It calls SetProcessDPIAware before it reads any screen size, so a 150% display is captured in full, in physical pixels. It also shrinks the image itself and prints a small JSON report. A locked PC, the UAC prompt, or a session with no desktop (a service, SSH) gets a clear error.
  • Linux: one fixed sh -c 'command -v …' loop finds the installed tools, with the tool names passed as argv. Then it tries, in order: grim (wlroots Wayland), gnome-screenshot -f, spectacle -b -n -f -o (KDE), and on X11 only scrot and import -window root. If one tool produces no PNG, the next one is tried. The size is read from the PNG header. If magick or convert is installed, they shrink the picture to 1280 px. Otherwise it stays full size, with a note.
  • Honest limits: GNOME on Wayland blocks all of these tools (its screenshot API is private and we don't use the desktop portal yet), and the message says so and suggests a "GNOME on Xorg" session. When no tool is found, the message lists what to install, and nothing is asked first. With no graphical session at all (SSH, a container), the message says that. A finished picture over 16 MB is refused.
  • A region, app, window or text is refused on Windows and Linux. Window snapshots stay macOS-only for now.
  • Every child process gets an allow-listed environment, so no provider keys or tokens reach it.
  • os.screen.click, type, key and point stay macOS-only and are not registered on Windows or Linux. Their off-mac message now says that capture works on this OS and control is macOS-only for now.
  • The prompt descriptor gate (filterToolDescriptorsByConfig) gets an optional screen.control flag. With control: false, only os.screen.capture is kept. The runtime catalog and /tools pass control: process.platform === "darwin", so the prompt matches what is registered.

Precedent

  • Hermes and OpenClaw use the MIT cua-driver for computer use on macOS, Windows and Linux.
  • ChatGPT's computer use on Windows works on the foreground screen only, which is the same "look at the whole screen" starting point.
  • This repo already uses PowerShell on Windows and wmctrl on Linux for os.window.* (src/tools/os/window.ts). This change follows that pattern, using the same command runner.

Next slice

Input on Windows and Linux (SendInput on Windows, xdotool / ydotool on Linux, or cua-driver as an external MCP server) is the next step, and it needs checks on real machines. cua-perception (AGPL) and cuabot are not used.

Files

  • New: src/tools/os/screen/screen-capture-win-linux.ts (the script, argv, tool choice, parsers) and its test.
  • screen-capture.ts: the Windows/Linux flow (choose a tool, ask, capture, remember the geometry).
  • screen-host.ts: screenCaptureSupported / screenControlSupported, a readHead dependency for the PNG header, and the new off-mac message.
  • index.ts: registration per platform. filter-disabled-tools.ts, runtime-tool-catalog.ts, tools-listing.ts: the control gate.
  • Docs: src/tools/os/README.md. The prompt descriptor summary mentions the Windows/Linux limits.
  • Tests: new fake-runner tests for each OS (the Windows script and env payload, argv per Linux tool, Wayland/X11 choice, fallback, ImageMagick downscale, missing-tool and no-display messages, report parsing, size refusal). Three existing tests that expected nothing on Linux were updated.

screen-jxa.ts and screen-input.ts are untouched. screen-input.test.ts has one changed expectation: Linux now registers capture.

No real capture runs in tests. Not run on real Windows or Linux yet. The tools.screen.enabled text in the config docs still says macOS only, because changing it means regenerating settings-docs.generated.ts.

Verification

  • Review found: on Linux os.tmpdir() is the shared /tmp, and the screenshot folder /tmp/atomic-agent-screens/<session>/ was created with default permissions under a predictable name. Another local user could read a full-screen capture, or pre-create the parent (or a symlink) and own what lands inside.
  • Changed: ensurePrivateDir in screen-host.ts creates the folder 0700, checks the parent and the session folder are plain folders (not symlinks) owned by this user, sets both back to 0700, and refuses the capture otherwise. On Windows (no uid, per-user temp) only the symlink check applies. Tests in screen-capture-win-linux.test.ts.
  • MBP run must check: the new tests pass; on macOS a capture still saves (its tmpdir parent is per-user and owned by the user, so the checks pass).

Not yet checked on the second Mac.

os.screen.capture now takes a whole-screen PNG on Windows (constant
PowerShell CopyFromScreen script, DPI aware, path in env) and Linux
(grim, gnome-screenshot, spectacle, scrot or import, first that works).
Click, type, key and point stay macOS-only; their off-mac message says
capture works here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant