computer-use-sway

computer-use-sway is a local MCP stdio server for inspecting and operating the current Wayland session through Sway-native tools. It turns the desktop into a small explicit capability surface instead of an opaque remote-control channel.

Screen and window inspection Pointer control Text and key chords Clipboard access Local stdio transport

What the client can see and do

Observe the session

Read

Outputs, seats, focused windows, and required binaries

The server returns the state needed to reason about the current desktop before acting. A simplified Sway tree provides stable window identity without exposing the full compositor structure as raw noise.

Capture the screen

Image

Screenshots as MCP image content or data URLs

The client can receive a current visual surface through grim, then combine it with structured focus and window information.

Operate deliberately

Write

Focus, point, click, drag, scroll, type, and send chords

Window targeting works by container ID, app ID, class, or title. Pointer and keyboard actions use native Wayland tools rather than an X11 compatibility path.

Exchange text

Clipboard

Read and write the text clipboard

Clipboard capabilities use wl-clipboard and remain separate from keyboard entry, so the client can choose the least disruptive mechanism for a task.

A narrow local bridge

There is no HTTP listener and no remote account. The MCP host launches the server over stdio. When that host starts with a sanitized environment, the server reconstructs XDG_RUNTIME_DIR, SWAYSOCK, and WAYLAND_DISPLAY from the active local session, then diagnoses the required tools before accepting desktop work.

sequenceDiagram
    participant Client as Trusted MCP client
    participant Server as computer-use-sway
    participant Sway as Sway / Wayland tools
    Client->>Server: Inspect session
    Server->>Sway: swaymsg / grim
    Sway-->>Server: Tree, focus, screenshot
    Server-->>Client: Structured state + image
    Client->>Server: Explicit input action
    Server->>Sway: wtype / pointer / clipboard

Project identity

Repository

Source and setup