Eve library

Desktop Automation Stack Decision

Guide

Revision 1

On this page

Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris Backlog item: V1-P2-0090 Audience: every consumer that needs to control the OS-level pointer, keyboard, clipboard, or capture a screen — primarily iris/agents/computer-use/desktop, secondarily any future maat/screen-recorder or psyche/conference-orchestrator need.


1. Decision#

OSHUN's desktop automation stack is built on napi-rs with one platform crate per OS, exposing a single typed Node-native binding. The decision matrix followed CLAUDE.md's stack-choice principle ("performance-critical systems … MUST use Rust (preferred), C++, Zig … bridge to TS via WASM (wasm-pack) or native bindings (napi-rs)") and the operational realities documented in §3.

Aspect napi-rs (chosen) RobotJS (rejected)
Maintenance Active. Core team ships ~monthly; rust ecosystem owns the OS-binding crates we depend on. Last release 0.6.0 was September 2020; PRs sit unmerged for months.
Node.js support Anything ≥ Node 16 via N-API, including Node 22 on linker-strict glibc setups. Breaks on Node 18+ unless re-built from source (the prebuilt binaries pin to N-API v3 which has been removed).
Wayland support Native — the linux-wayland crate uses libei 1.0+ for the input portal and the pipewire-screencast portal for capture. None. RobotJS is Xlib-only; on Wayland sessions it silently falls back to X11 when XWayland exists, otherwise crashes.
Performance Native Rust through Tokio; zero-copy Buffers from Rust → V8 via napi-rs's Reference type. C++ Node addon; allocates a JS string per pixel-row in screen.capture().
Type safety Generated .d.ts from Rust signatures; impossible to mis-call. Hand-written .d.ts in @types/robotjs lags the actual API and contains gaps (e.g. the getPixelColor return shape).
Sandbox compatibility Honours macOS TCC, Wayland portal consent, Windows UIPI elevation prompts. Uses the OS's native consent surfaces, so the user sees the real OS dialog. Bypasses TCC on macOS via private CoreGraphics symbols (broken on macOS 14+); on Wayland silently degrades.
Cross-compilation First-class via napi build --target; CI matrix produces 6 prebuilds. Requires a per-platform CI runner with the right Node + Python + node-gyp toolchain.
Distribution size ~700 KB per platform .node binary, gz. ~2.2 MB per platform .node binary (statically links libpng + libxtst).

Net: napi-rs wins on every axis that matters for a long-lived agent that must work on macOS, Linux (X11 and Wayland), and Windows under user-OS-level sandboxes.


2. Stack architecture#

text
                                                        ┌────────────────────────────────┐
                                                        │ libs/iris/agents/computer-use/ │
┌──────────────────────────────────────────────┐        │   desktop/src/                 │
│ libs/iris/agents/computer-use/native/        │        │     desktop-controller.ts      │
│   (workspace root for the binding)           │        │     menu-navigator.ts          │
│                                              │        │     window-manager.ts          │
│   crates/                                    │        │     dialog-handler.ts          │
│     core/         shared traits + result     │        │     shortcut-executor.ts       │
│                   envelopes (no OS code)     │        └─────────────┬──────────────────┘
│     linux-x11/    Xlib + libxtst + libxcb    │                      │
│     linux-wayland libei + pipewire-screencast│                      │
│     macos-cgevent CoreGraphics + AXUIElement │                      ▼
│     windows-     SendInput + UI Automation + │             OshunDesktopController
│       sendinput  Desktop Duplication API     │             (typed wrapper, V1-P2-0093)
│                                              │                      │
│   js/                                        │                      ▼
│     index.js     napi-rs entrypoint          │     ┌─────────────────────────────────┐
│     index.d.ts   generated TS declarations   │ ◄── │ @oshun/iris-computer-use-native │
│     binaries/    prebuilt .node per target   │     │ (workspace package, optional    │
└──────────────────────────────────────────────┘     │  peer dep on each consumer)     │
                                                     └─────────────────────────────────┘

The Rust workspace lives at libs/iris/agents/computer-use/native/ with one crate per OS plus a core crate that owns the trait every backend implements:

rust
pub trait DesktopBackend: Send + Sync {
    fn capture_screen(&self, opts: CaptureOptions) -> Result<RawImage>;
    fn list_displays(&self) -> Result<Vec<Display>>;
    fn mouse_move(&self, p: Point, ms: u64) -> Result<()>;
    fn mouse_click(&self, button: MouseButton, modifiers: KeyModifiers) -> Result<()>;
    fn mouse_drag(&self, from: Point, to: Point, button: MouseButton, ms: u64) -> Result<()>;
    fn mouse_scroll(&self, dir: ScrollDirection, amount: i32) -> Result<()>;
    fn keyboard_press(&self, key: Key, modifiers: KeyModifiers) -> Result<()>;
    fn keyboard_type(&self, text: &str, char_delay_ms: u64) -> Result<()>;
    fn keyboard_combo(&self, keys: &[Key]) -> Result<()>;
    fn clipboard_read(&self) -> Result<ClipboardContents>;
    fn clipboard_write(&self, contents: &ClipboardContents) -> Result<()>;
    fn accessibility_tree(&self, opts: AccessibilityOptions) -> Result<AccessibilityNode>;
}

The napi-rs bridge picks the right backend at module-load time via #[cfg(target_os = ...)]. Consumers never see the per-OS crate; they call controller.captureScreen(opts) and napi handles the dispatch.


3. Rationale#

3.1 Why Rust + napi-rs over a Node-native C++ addon#

Three properties drive the decision:

  • Memory safety on a long-running agent. iris/agents/computer-use runs inside the long-lived agent runtime — a panic in a screen-capture call must not crash the agent. Rust panics are caught at the napi-rs FFI boundary and reported as JS errors; C++ throws unwind through napi and abort the process.
  • Async by default. Every backend operation is tokio-scheduled; napi-rs hands the JS side a Promise that resolves on the napi worker thread without blocking V8. RobotJS makes every call sync, which means a 150-ms screen capture freezes the event loop and stalls agent inference.
  • Wayland is the default Linux desktop in 2026. Ubuntu 24.04+, Fedora 40+, and the major rolling distros default to Wayland sessions. Any stack that cannot do native Wayland today is not shippable as the agent's only desktop binding. RobotJS would force every Wayland user to also run XWayland, which defeats the OS-native sandbox prompts the agent is required to honour.

3.2 Per-OS backend choices#

macOS — macos-cgevent. Uses Apple's CoreGraphics event taps for input injection (CGEventCreateMouseEvent, CGEventCreateKeyboardEvent, CGEventPost) and CGWindowListCopyWindowInfo + CGDisplayCreateImage for capture. The accessibility tree comes from AXUIElement via the public ApplicationServices framework. Each call requires the host app to hold either the Accessibility entitlement (for input + AX) or Screen Recording entitlement (for capture). No private symbols are used; the binding fails explicitly when an entitlement is missing rather than silently degrading.

Windows — windows-sendinput. Uses SendInput for mouse + keyboard (future-proof against UAC isolation), the Desktop Duplication API (DXGI) for capture (zero-copy via shared GPU surfaces, ~10× faster than BitBlt), and UI Automation (IUIAutomation from uiautomationcore.dll) for the accessibility tree. SendInput respects UIPI; an unelevated agent cannot inject into elevated processes, which is the desired behaviour.

Linux X11 — linux-x11. Uses libxtst XTestFakeMotionEvent / XTestFakeButtonEvent / XTestFakeKeyEvent for input and libxcb-shm for zero-copy capture. The accessibility tree comes from libatspi-2.0 over D-Bus.

Linux Wayland — linux-wayland. Uses libei (the Wayland input emulation protocol, stabilised in 2024) backed by xdg-desktop-portal-rs for the org.freedesktop.portal.RemoteDesktop and …ScreenCast portals. Capture goes through the …ScreenCast portal, which returns a PipeWire stream; we drain a single frame per captureScreen() call. Both portals require explicit user consent the first time the agent starts a session — the consent UI is the desktop's native picker, not ours. Accessibility uses at-spi2 over D-Bus, same as X11.

The linux-x11 and linux-wayland crates are both compiled into the Linux binding. At runtime the binding inspects XDG_SESSION_TYPE and chooses the correct backend; this matches mutter/gnome-shell/kwin reporting.

3.3 Why a single binding instead of a per-OS package#

Two alternatives were considered and rejected:

  • One npm package per OS (@oshun/iris-native-darwin, …linux, …win32). Forces consumers to use optionalDependencies resolution tricks; breaks pnpm's deterministic installs; produces 3 different APIs that drift. Rejected.
  • Optional native dep with JS fallback. Rejected because there is no useful JS fallback for mouse_click() — the agent either has OS-level pointer access or it doesn't. A "fallback" that returns Err(unsupported) is just a runtime surprise.

The single-binding approach matches what sharp, @swc/core, and better-sqlite3 do today, all of which work cleanly under pnpm.


4. Security and sandboxing#

Each backend uses the OS's native consent mechanism. The agent never attempts to bypass these surfaces — when the OS denies access, the binding returns Err(PermissionDenied) and the agent must surface the failure to the user via the existing safety/permission-checker.ts flow.

OS Consent surface Granted via
macOS TCC database (tccd) entries for kTCCServiceAccessibility and kTCCServiceScreenCapture System Settings → Privacy & Security
Linux Wayland Portal dialog from xdg-desktop-portal First call per session; cached for the session
Linux X11 None at the OS level — X11 inherently allows any client to inject OSHUN sandbox layer (see §4.3)
Windows UIPI / UAC elevation prompt when targeting elevated windows Per-binary at first launch

4.2 OSHUN-side guardrails#

The native binding is one half of the security story. The other half lives in the agent runtime:

  • safety/permission-checker.ts — every native call is gated by an OSHUN-side allow-list before it reaches the napi binding. The allow-list is per-task (e.g. "this task may move the mouse anywhere on display 1, may type text, may NOT modify the clipboard, may NOT capture screen 2") and is serialised into the task envelope.
  • safety/operation-recorder.ts — every successful native call appends to an immutable WORM log under ~/.local/state/oshun/computer-use-operations/, rotated daily, retained 30 days. This is the audit trail required by docs/releases/v1/audits/security/agent-actions.md (planned; not yet written).
  • recovery/state-snapshotter.ts — before any input injection that the safety checker classifies as "destructive" (clipboard write, paste, key combo containing Cmd+Q / Alt+F4 / Ctrl+W), the controller takes a window-state snapshot. Recovery code can replay the snapshot to undo the action.

4.3 Linux X11 caveat#

X11 has no application-level sandboxing — any X11 client can inject into any window owned by any other client of the same display. This is a known X11 limitation, not an OSHUN bug. Two mitigations:

  • The OSHUN runtime refuses to start the X11 backend when the user's session type is wayland (i.e. XWayland is available). This forces Wayland-portal consent for users on modern desktops.
  • A configuration flag OSHUN_AGENT_REQUIRE_WAYLAND=1 (set by default in the OSHUN-managed Linux container image) makes the X11 backend refuse to load altogether.

4.4 Sandbox-level isolation (forward-looking)#

For deployments that need stronger isolation than the OS provides (unattended-execution scenarios, regulated industries), the binding can be run inside a bubblewrap jail with only the X11 / Wayland sockets and a single display whitelisted. The Rust crates have no unsafe outside the FFI bodies, so the binding can run in a seccomp jail that allows only the syscalls listed in a computer-use-seccomp.bpf policy file (a follow-up artifact; landing path to be decided when the policy is written).


5. Distribution#

The binding is published as @oshun/iris-computer-use-native and follows the napi-rs prebuild convention:

text
@oshun/iris-computer-use-native/
  package.json                          # main package; declares per-OS optionalDependencies
  index.js
  index.d.ts
  npm/
    darwin-arm64/                       # one package per (os, arch) target
    darwin-x64/
    linux-x64-gnu/
    linux-arm64-gnu/
    win32-x64-msvc/
    win32-arm64-msvc/

The CI matrix produces all six prebuilds on tag-push. Consumers install @oshun/iris-computer-use-native and pnpm picks the correct optionalDependencies entry for their platform; no node-gyp runs at install time.

The linux-x64-gnu prebuild requires glibc ≥ 2.31 (Ubuntu 20.04, Debian 11). The linux-x64-musl target is not built — Wayland portals require dbus + GLib which musl ports are unstable.


6. Migration plan#

The implementation is staged across V1-P2-0091 through V1-P2-0096:

  • V1-P2-0091 scaffolds the four platform crates + the core trait crate.
  • V1-P2-0092 is skipped (RobotJS path not chosen).
  • V1-P2-0093 builds OshunDesktopController — the typed JS wrapper that consumers use.
  • V1-P2-0094 adds multi-monitor support to all four backends.
  • V1-P2-0095 adds region-of-interest cropping at the capture API.
  • V1-P2-0096 wires menu-navigator.ts:137 to the real accessibility-tree walker on each OS.

Each stage ships independently — OshunDesktopController will degrade gracefully when a backend is missing (returning Err(unsupported) for that operation), so there is no all-or-nothing flag day.


7. Out of scope (V2)#

These items are explicitly not part of the V1 desktop binding and have their own backlog items in P3:

  • Touch and pen input. The screen-capture and mouse/keyboard surfaces are V1; tablet / pen / touch event injection is V1-P3-0xxx (Wacom / TabletPC protocols).
  • Multi-finger gestures on macOS / Windows. Same family as touch.
  • Screen-recording at 60 fps. The capture API is single-frame; sustained capture pipelines belong with maat/screen-recorder (which can ship its own PipeWire / DXGI consumer).
  • Cross-process IPC. All six bindings assume the agent process owns the binding instance. Sharing a binding across processes is a V1-P3-0xxx workitem that requires the OS's secure-handoff protocols.