# Desktop Automation Stack Decision

**Status:** Final v1.0 **Date:** 2026-04-26 **Owners:** eng-iris **Backlog
item:** `V1-P2-0090` **Audience:** every consumer that needs to control the
OS-level pointer, keyboard, clipboard, or capture a screen — primarily
`iris/agents/computer-use/desktop`, secondarily any future
`maat/screen-recorder` or `psyche/conference-orchestrator` need.

---

## 1. Decision

OSHUN's desktop automation stack is built on **napi-rs** with one platform crate
per OS, exposing a single typed Node-native binding. The decision matrix
followed CLAUDE.md's stack-choice principle ("performance-critical systems …
MUST use Rust (preferred), C++, Zig … bridge to TS via WASM (`wasm-pack`) or
native bindings (`napi-rs`)") and the operational realities documented in §3.

| Aspect                | napi-rs (chosen)                                                                                                                                       | RobotJS (rejected)                                                                                                      |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| Maintenance           | Active. Core team ships ~monthly; rust ecosystem owns the OS-binding crates we depend on.                                                              | Last release `0.6.0` was September 2020; PRs sit unmerged for months.                                                   |
| Node.js support       | Anything ≥ Node 16 via N-API, including Node 22 on linker-strict glibc setups.                                                                         | Breaks on Node 18+ unless re-built from source (the prebuilt binaries pin to N-API v3 which has been removed).          |
| Wayland support       | Native — the `linux-wayland` crate uses `libei` 1.0+ for the input portal and the `pipewire-screencast` portal for capture.                            | None. RobotJS is Xlib-only; on Wayland sessions it silently falls back to X11 when XWayland exists, otherwise crashes.  |
| Performance           | Native Rust through Tokio; zero-copy Buffers from Rust → V8 via napi-rs's `Reference` type.                                                            | C++ Node addon; allocates a JS string per pixel-row in `screen.capture()`.                                              |
| Type safety           | Generated `.d.ts` from Rust signatures; impossible to mis-call.                                                                                        | Hand-written `.d.ts` in `@types/robotjs` lags the actual API and contains gaps (e.g. the `getPixelColor` return shape). |
| Sandbox compatibility | Honours macOS TCC, Wayland portal consent, Windows UIPI elevation prompts. Uses the OS's native consent surfaces, so the user sees the real OS dialog. | Bypasses TCC on macOS via private CoreGraphics symbols (broken on macOS 14+); on Wayland silently degrades.             |
| Cross-compilation     | First-class via `napi build --target`; CI matrix produces 6 prebuilds.                                                                                 | Requires a per-platform CI runner with the right Node + Python + node-gyp toolchain.                                    |
| Distribution size     | ~700 KB per platform .node binary, gz.                                                                                                                 | ~2.2 MB per platform .node binary (statically links libpng + libxtst).                                                  |

Net: napi-rs wins on every axis that matters for a long-lived agent that must
work on macOS, Linux (X11 _and_ Wayland), and Windows under user-OS-level
sandboxes.

---

## 2. Stack architecture

```
                                                        ┌────────────────────────────────┐
                                                        │ libs/iris/agents/computer-use/ │
┌──────────────────────────────────────────────┐        │   desktop/src/                 │
│ libs/iris/agents/computer-use/native/        │        │     desktop-controller.ts      │
│   (workspace root for the binding)           │        │     menu-navigator.ts          │
│                                              │        │     window-manager.ts          │
│   crates/                                    │        │     dialog-handler.ts          │
│     core/         shared traits + result     │        │     shortcut-executor.ts       │
│                   envelopes (no OS code)     │        └─────────────┬──────────────────┘
│     linux-x11/    Xlib + libxtst + libxcb    │                      │
│     linux-wayland libei + pipewire-screencast│                      │
│     macos-cgevent CoreGraphics + AXUIElement │                      ▼
│     windows-     SendInput + UI Automation + │             OshunDesktopController
│       sendinput  Desktop Duplication API     │             (typed wrapper, V1-P2-0093)
│                                              │                      │
│   js/                                        │                      ▼
│     index.js     napi-rs entrypoint          │     ┌─────────────────────────────────┐
│     index.d.ts   generated TS declarations   │ ◄── │ @oshun/iris-computer-use-native │
│     binaries/    prebuilt .node per target   │     │ (workspace package, optional    │
└──────────────────────────────────────────────┘     │  peer dep on each consumer)     │
                                                     └─────────────────────────────────┘
```

The Rust workspace lives at `libs/iris/agents/computer-use/native/` with one
crate per OS plus a `core` crate that owns the trait every backend implements:

```rust
pub trait DesktopBackend: Send + Sync {
    fn capture_screen(&self, opts: CaptureOptions) -> Result<RawImage>;
    fn list_displays(&self) -> Result<Vec<Display>>;
    fn mouse_move(&self, p: Point, ms: u64) -> Result<()>;
    fn mouse_click(&self, button: MouseButton, modifiers: KeyModifiers) -> Result<()>;
    fn mouse_drag(&self, from: Point, to: Point, button: MouseButton, ms: u64) -> Result<()>;
    fn mouse_scroll(&self, dir: ScrollDirection, amount: i32) -> Result<()>;
    fn keyboard_press(&self, key: Key, modifiers: KeyModifiers) -> Result<()>;
    fn keyboard_type(&self, text: &str, char_delay_ms: u64) -> Result<()>;
    fn keyboard_combo(&self, keys: &[Key]) -> Result<()>;
    fn clipboard_read(&self) -> Result<ClipboardContents>;
    fn clipboard_write(&self, contents: &ClipboardContents) -> Result<()>;
    fn accessibility_tree(&self, opts: AccessibilityOptions) -> Result<AccessibilityNode>;
}
```

The napi-rs bridge picks the right backend at module-load time via
`#[cfg(target_os = ...)]`. Consumers never see the per-OS crate; they call
`controller.captureScreen(opts)` and napi handles the dispatch.

---

## 3. Rationale

### 3.1 Why Rust + napi-rs over a Node-native C++ addon

Three properties drive the decision:

- **Memory safety on a long-running agent.** `iris/agents/computer-use` runs
  inside the long-lived agent runtime — a panic in a screen-capture call must
  not crash the agent. Rust panics are caught at the napi-rs FFI boundary and
  reported as JS errors; C++ throws unwind through napi and abort the process.
- **Async by default.** Every backend operation is `tokio`-scheduled; napi-rs
  hands the JS side a `Promise` that resolves on the napi worker thread without
  blocking V8. RobotJS makes every call sync, which means a 150-ms screen
  capture freezes the event loop and stalls agent inference.
- **Wayland is the default Linux desktop in 2026.** Ubuntu 24.04+, Fedora 40+,
  and the major rolling distros default to Wayland sessions. Any stack that
  cannot do native Wayland today is not shippable as the agent's only desktop
  binding. RobotJS would force every Wayland user to also run XWayland, which
  defeats the OS-native sandbox prompts the agent is required to honour.

### 3.2 Per-OS backend choices

**macOS — `macos-cgevent`.** Uses Apple's `CoreGraphics` event taps for input
injection (`CGEventCreateMouseEvent`, `CGEventCreateKeyboardEvent`,
`CGEventPost`) and `CGWindowListCopyWindowInfo` + `CGDisplayCreateImage` for
capture. The accessibility tree comes from `AXUIElement` via the public
`ApplicationServices` framework. Each call requires the host app to hold either
the **Accessibility** entitlement (for input + AX) or **Screen Recording**
entitlement (for capture). No private symbols are used; the binding fails
explicitly when an entitlement is missing rather than silently degrading.

**Windows — `windows-sendinput`.** Uses `SendInput` for mouse + keyboard
(future-proof against UAC isolation), the **Desktop Duplication API** (DXGI) for
capture (zero-copy via shared GPU surfaces, ~10× faster than `BitBlt`), and **UI
Automation** (`IUIAutomation` from `uiautomationcore.dll`) for the accessibility
tree. SendInput respects UIPI; an unelevated agent cannot inject into elevated
processes, which is the desired behaviour.

**Linux X11 — `linux-x11`.** Uses `libxtst` `XTestFakeMotionEvent` /
`XTestFakeButtonEvent` / `XTestFakeKeyEvent` for input and `libxcb-shm` for
zero-copy capture. The accessibility tree comes from `libatspi-2.0` over D-Bus.

**Linux Wayland — `linux-wayland`.** Uses `libei` (the Wayland input emulation
protocol, stabilised in 2024) backed by `xdg-desktop-portal-rs` for the
`org.freedesktop.portal.RemoteDesktop` and `…ScreenCast` portals. Capture goes
through the `…ScreenCast` portal, which returns a PipeWire stream; we drain a
single frame per `captureScreen()` call. Both portals require **explicit user
consent** the first time the agent starts a session — the consent UI is the
desktop's native picker, not ours. Accessibility uses `at-spi2` over D-Bus, same
as X11.

The `linux-x11` and `linux-wayland` crates are both compiled into the Linux
binding. At runtime the binding inspects `XDG_SESSION_TYPE` and chooses the
correct backend; this matches `mutter`/`gnome-shell`/`kwin` reporting.

### 3.3 Why a single binding instead of a per-OS package

Two alternatives were considered and rejected:

- **One npm package per OS** (`@oshun/iris-native-darwin`, `…linux`, `…win32`).
  Forces consumers to use `optionalDependencies` resolution tricks; breaks
  pnpm's deterministic installs; produces 3 different APIs that drift. Rejected.
- **Optional native dep with JS fallback.** Rejected because there is no useful
  JS fallback for `mouse_click()` — the agent either has OS-level pointer access
  or it doesn't. A "fallback" that returns `Err(unsupported)` is just a runtime
  surprise.

The single-binding approach matches what `sharp`, `@swc/core`, and
`better-sqlite3` do today, all of which work cleanly under pnpm.

---

## 4. Security and sandboxing

### 4.1 OS-level consent surfaces

Each backend uses the OS's native consent mechanism. The agent **never**
attempts to bypass these surfaces — when the OS denies access, the binding
returns `Err(PermissionDenied)` and the agent must surface the failure to the
user via the existing `safety/permission-checker.ts` flow.

| OS            | Consent surface                                                                             | Granted via                                    |
| ------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------- |
| macOS         | TCC database (`tccd`) entries for `kTCCServiceAccessibility` and `kTCCServiceScreenCapture` | System Settings → Privacy & Security           |
| Linux Wayland | Portal dialog from `xdg-desktop-portal`                                                     | First call per session; cached for the session |
| Linux X11     | None at the OS level — X11 inherently allows any client to inject                           | OSHUN sandbox layer (see §4.3)                 |
| Windows       | UIPI / UAC elevation prompt when targeting elevated windows                                 | Per-binary at first launch                     |

### 4.2 OSHUN-side guardrails

The native binding is one half of the security story. The other half lives in
the agent runtime:

- **`safety/permission-checker.ts`** — every native call is gated by an
  OSHUN-side allow-list before it reaches the napi binding. The allow-list is
  per-task (e.g. "this task may move the mouse anywhere on display 1, may type
  text, may NOT modify the clipboard, may NOT capture screen 2") and is
  serialised into the task envelope.
- **`safety/operation-recorder.ts`** — every successful native call appends to
  an immutable WORM log under `~/.local/state/oshun/computer-use-operations/`,
  rotated daily, retained 30 days. This is the audit trail required by
  `docs/releases/v1/audits/security/agent-actions.md` (planned; not yet
  written).
- **`recovery/state-snapshotter.ts`** — before any input injection that the
  safety checker classifies as "destructive" (clipboard write, paste, key combo
  containing Cmd+Q / Alt+F4 / Ctrl+W), the controller takes a window-state
  snapshot. Recovery code can replay the snapshot to undo the action.

### 4.3 Linux X11 caveat

X11 has no application-level sandboxing — any X11 client can inject into any
window owned by any other client of the same display. This is a known X11
limitation, not an OSHUN bug. Two mitigations:

- The OSHUN runtime refuses to start the X11 backend when the user's session
  type is `wayland` (i.e. XWayland is available). This forces Wayland-portal
  consent for users on modern desktops.
- A configuration flag `OSHUN_AGENT_REQUIRE_WAYLAND=1` (set by default in the
  OSHUN-managed Linux container image) makes the X11 backend refuse to load
  altogether.

### 4.4 Sandbox-level isolation (forward-looking)

For deployments that need stronger isolation than the OS provides
(unattended-execution scenarios, regulated industries), the binding can be run
inside a `bubblewrap` jail with only the X11 / Wayland sockets and a single
display whitelisted. The Rust crates have no `unsafe` outside the FFI bodies, so
the binding can run in a `seccomp` jail that allows only the syscalls listed in
a `computer-use-seccomp.bpf` policy file (a follow-up artifact; landing path to
be decided when the policy is written).

---

## 5. Distribution

The binding is published as `@oshun/iris-computer-use-native` and follows the
napi-rs prebuild convention:

```
@oshun/iris-computer-use-native/
  package.json                          # main package; declares per-OS optionalDependencies
  index.js
  index.d.ts
  npm/
    darwin-arm64/                       # one package per (os, arch) target
    darwin-x64/
    linux-x64-gnu/
    linux-arm64-gnu/
    win32-x64-msvc/
    win32-arm64-msvc/
```

The CI matrix produces all six prebuilds on tag-push. Consumers install
`@oshun/iris-computer-use-native` and pnpm picks the correct
`optionalDependencies` entry for their platform; no `node-gyp` runs at install
time.

The `linux-x64-gnu` prebuild requires glibc ≥ 2.31 (Ubuntu 20.04, Debian 11).
The `linux-x64-musl` target is **not** built — Wayland portals require dbus +
GLib which musl ports are unstable.

---

## 6. Migration plan

The implementation is staged across `V1-P2-0091` through `V1-P2-0096`:

- `V1-P2-0091` scaffolds the four platform crates + the `core` trait crate.
- `V1-P2-0092` is **skipped** (RobotJS path not chosen).
- `V1-P2-0093` builds `OshunDesktopController` — the typed JS wrapper that
  consumers use.
- `V1-P2-0094` adds multi-monitor support to all four backends.
- `V1-P2-0095` adds region-of-interest cropping at the capture API.
- `V1-P2-0096` wires `menu-navigator.ts:137` to the real accessibility-tree
  walker on each OS.

Each stage ships independently — `OshunDesktopController` will degrade
gracefully when a backend is missing (returning `Err(unsupported)` for that
operation), so there is no all-or-nothing flag day.

---

## 7. Out of scope (V2)

These items are explicitly **not** part of the V1 desktop binding and have their
own backlog items in P3:

- **Touch and pen input.** The screen-capture and mouse/keyboard surfaces are
  V1; tablet / pen / touch event injection is `V1-P3-0xxx` (Wacom / TabletPC
  protocols).
- **Multi-finger gestures on macOS / Windows.** Same family as touch.
- **Screen-recording at 60 fps.** The capture API is single-frame; sustained
  capture pipelines belong with `maat/screen-recorder` (which can ship its own
  PipeWire / DXGI consumer).
- **Cross-process IPC.** All six bindings assume the agent process owns the
  binding instance. Sharing a binding across processes is a `V1-P3-0xxx`
  workitem that requires the OS's secure-handoff protocols.
