Vercel Labs has released agent-desktop, an open-source command-line tool (CLI) that allows AI agents to control desktop applications on macOS, Windows, and Linux. In its version 1.0.2, released on October 30, 2026 under the Apache-2.0 license, it does not rely on screenshots but on the operating system’s accessibility tree.
A structured rather than visual reading
The accessibility tree is a structured description of interface elements, made available by the system to assistive tools. agent-desktop provides the agent with a JSON-formatted snapshot, in which each actionable element is given a reference, or ref. These references are qualified by the snapshot identifier, as in the example @s8f3k2p9:e8: an incomplete reference, like @e8, is rejected with the INVALID_ARGS code.
When the interface changes, an action may return STALE_REF or AMBIGUOUS_TARGET, and the agent must then establish a new snapshot. Errors also indicate whether a retry is safe. The tool presents itself as a component called by the agent, not as an agent itself: it does not incorporate any language model and has no graphical interface. Written in Rust, it comes with a C library that programs in Python, Swift, Go, Ruby, or Node can call directly.
It works on macOS 13+, Windows 10 1809+ or Server 2019+, Ubuntu 24.04+, on x64 and ARM64 architectures, without a GPU. On macOS, accessibility permission must be granted, using the command agent-desktop permissions –guide.
Token savings announced by the publisher
To limit the volume of data sent to the model, the –skeleton option reads only three levels deep. Unopened containers indicate their child count and a reference for exploring a specific area. According to its designer, the reduction reaches 78% to 96% of tokens (text units processed by models) compared to a full traversal of the tree in Slack, VS Code, and Notion. For Apple Maps, it reports 3,811 tokens per observation, versus 273 with this method. On a simulated 52-step task, consumption drops from 198k to 31k tokens. These figures are self-reported and not from an independent benchmark.
Anthropic’s documentation indicates that a screenshot used by its computer use tool costs about 1,000 to 1,800 input tokens, and its tool definition adds about 4,500 tokens per request. OpenAI, for its part, offers click, type, and scroll actions that user code executes. Microsoft UFO² also uses UI Automation but presents itself as a complete agent for Windows.
Actions on references go through the accessibility API without moving the pointer or taking focus, allowing the agent to work while the user continues to use their machine. The global –headed option allows physical input if necessary. Each action also waits up to 5,000 ms for a target to be visible, active, and stable.
The project notes several limitations. An application that does not publish an accessibility tree remains inaccessible to the tool. A cold-started Chromium or Electron application may take 10 to 25 seconds to expose its web tree. Finally, snapshots, clipboard reads, and captures can contain private data, which the documentation recommends keeping out of public logs. The code is available on the project’s GitHub repository. Installation is done with the command npm install -g agent-desktop, then agents discover the tool via agent-desktop skills get desktop.
