Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris Backlog
item: V1-P2-0090 Audience: every consumer that needs to control the
OS-level pointer, keyboard, clipboard, or capture a screen — primarily
iris/agents/computer-use/desktop, secondarily any future
maat/screen-recorder or psyche/conference-orchestrator need.
1. Decision#
OSHUN's desktop automation stack is built on napi-rs with one platform crate
per OS, exposing a single typed Node-native binding. The decision matrix
followed CLAUDE.md's stack-choice principle ("performance-critical systems …
MUST use Rust (preferred), C++, Zig … bridge to TS via WASM (wasm-pack) or
native bindings (napi-rs)") and the operational realities documented in §3.
| Aspect | napi-rs (chosen) | RobotJS (rejected) |
|---|---|---|
| Maintenance | Active. Core team ships ~monthly; rust ecosystem owns the OS-binding crates we depend on. | Last release 0.6.0 was September 2020; PRs sit unmerged for months. |
| Node.js support | Anything ≥ Node 16 via N-API, including Node 22 on linker-strict glibc setups. | Breaks on Node 18+ unless re-built from source (the prebuilt binaries pin to N-API v3 which has been removed). |
| Wayland support | Native — the linux-wayland crate uses libei 1.0+ for the input portal and the pipewire-screencast portal for capture. |
None. RobotJS is Xlib-only; on Wayland sessions it silently falls back to X11 when XWayland exists, otherwise crashes. |
| Performance | Native Rust through Tokio; zero-copy Buffers from Rust → V8 via napi-rs's Reference type. |
C++ Node addon; allocates a JS string per pixel-row in screen.capture(). |
| Type safety | Generated .d.ts from Rust signatures; impossible to mis-call. |
Hand-written .d.ts in @types/robotjs lags the actual API and contains gaps (e.g. the getPixelColor return shape). |
| Sandbox compatibility | Honours macOS TCC, Wayland portal consent, Windows UIPI elevation prompts. Uses the OS's native consent surfaces, so the user sees the real OS dialog. | Bypasses TCC on macOS via private CoreGraphics symbols (broken on macOS 14+); on Wayland silently degrades. |
| Cross-compilation | First-class via napi build --target; CI matrix produces 6 prebuilds. |
Requires a per-platform CI runner with the right Node + Python + node-gyp toolchain. |
| Distribution size | ~700 KB per platform .node binary, gz. | ~2.2 MB per platform .node binary (statically links libpng + libxtst). |
Net: napi-rs wins on every axis that matters for a long-lived agent that must work on macOS, Linux (X11 and Wayland), and Windows under user-OS-level sandboxes.
2. Stack architecture#
┌────────────────────────────────┐
│ libs/iris/agents/computer-use/ │
┌──────────────────────────────────────────────┐ │ desktop/src/ │
│ libs/iris/agents/computer-use/native/ │ │ desktop-controller.ts │
│ (workspace root for the binding) │ │ menu-navigator.ts │
│ │ │ window-manager.ts │
│ crates/ │ │ dialog-handler.ts │
│ core/ shared traits + result │ │ shortcut-executor.ts │
│ envelopes (no OS code) │ └─────────────┬──────────────────┘
│ linux-x11/ Xlib + libxtst + libxcb │ │
│ linux-wayland libei + pipewire-screencast│ │
│ macos-cgevent CoreGraphics + AXUIElement │ ▼
│ windows- SendInput + UI Automation + │ OshunDesktopController
│ sendinput Desktop Duplication API │ (typed wrapper, V1-P2-0093)
│ │ │
│ js/ │ ▼
│ index.js napi-rs entrypoint │ ┌─────────────────────────────────┐
│ index.d.ts generated TS declarations │ ◄── │ @oshun/iris-computer-use-native │
│ binaries/ prebuilt .node per target │ │ (workspace package, optional │
└──────────────────────────────────────────────┘ │ peer dep on each consumer) │
└─────────────────────────────────┘
The Rust workspace lives at libs/iris/agents/computer-use/native/ with one
crate per OS plus a core crate that owns the trait every backend implements:
pub trait DesktopBackend: Send + Sync {
fn capture_screen(&self, opts: CaptureOptions) -> Result<RawImage>;
fn list_displays(&self) -> Result<Vec<Display>>;
fn mouse_move(&self, p: Point, ms: u64) -> Result<()>;
fn mouse_click(&self, button: MouseButton, modifiers: KeyModifiers) -> Result<()>;
fn mouse_drag(&self, from: Point, to: Point, button: MouseButton, ms: u64) -> Result<()>;
fn mouse_scroll(&self, dir: ScrollDirection, amount: i32) -> Result<()>;
fn keyboard_press(&self, key: Key, modifiers: KeyModifiers) -> Result<()>;
fn keyboard_type(&self, text: &str, char_delay_ms: u64) -> Result<()>;
fn keyboard_combo(&self, keys: &[Key]) -> Result<()>;
fn clipboard_read(&self) -> Result<ClipboardContents>;
fn clipboard_write(&self, contents: &ClipboardContents) -> Result<()>;
fn accessibility_tree(&self, opts: AccessibilityOptions) -> Result<AccessibilityNode>;
}
The napi-rs bridge picks the right backend at module-load time via
#[cfg(target_os = ...)]. Consumers never see the per-OS crate; they call
controller.captureScreen(opts) and napi handles the dispatch.
3. Rationale#
3.1 Why Rust + napi-rs over a Node-native C++ addon#
Three properties drive the decision:
- Memory safety on a long-running agent.
iris/agents/computer-useruns inside the long-lived agent runtime — a panic in a screen-capture call must not crash the agent. Rust panics are caught at the napi-rs FFI boundary and reported as JS errors; C++ throws unwind through napi and abort the process. - Async by default. Every backend operation is
tokio-scheduled; napi-rs hands the JS side aPromisethat resolves on the napi worker thread without blocking V8. RobotJS makes every call sync, which means a 150-ms screen capture freezes the event loop and stalls agent inference. - Wayland is the default Linux desktop in 2026. Ubuntu 24.04+, Fedora 40+, and the major rolling distros default to Wayland sessions. Any stack that cannot do native Wayland today is not shippable as the agent's only desktop binding. RobotJS would force every Wayland user to also run XWayland, which defeats the OS-native sandbox prompts the agent is required to honour.
3.2 Per-OS backend choices#
macOS — macos-cgevent. Uses Apple's CoreGraphics event taps for input
injection (CGEventCreateMouseEvent, CGEventCreateKeyboardEvent,
CGEventPost) and CGWindowListCopyWindowInfo + CGDisplayCreateImage for
capture. The accessibility tree comes from AXUIElement via the public
ApplicationServices framework. Each call requires the host app to hold either
the Accessibility entitlement (for input + AX) or Screen Recording
entitlement (for capture). No private symbols are used; the binding fails
explicitly when an entitlement is missing rather than silently degrading.
Windows — windows-sendinput. Uses SendInput for mouse + keyboard
(future-proof against UAC isolation), the Desktop Duplication API (DXGI) for
capture (zero-copy via shared GPU surfaces, ~10× faster than BitBlt), and UI
Automation (IUIAutomation from uiautomationcore.dll) for the accessibility
tree. SendInput respects UIPI; an unelevated agent cannot inject into elevated
processes, which is the desired behaviour.
Linux X11 — linux-x11. Uses libxtst XTestFakeMotionEvent /
XTestFakeButtonEvent / XTestFakeKeyEvent for input and libxcb-shm for
zero-copy capture. The accessibility tree comes from libatspi-2.0 over D-Bus.
Linux Wayland — linux-wayland. Uses libei (the Wayland input emulation
protocol, stabilised in 2024) backed by xdg-desktop-portal-rs for the
org.freedesktop.portal.RemoteDesktop and …ScreenCast portals. Capture goes
through the …ScreenCast portal, which returns a PipeWire stream; we drain a
single frame per captureScreen() call. Both portals require explicit user
consent the first time the agent starts a session — the consent UI is the
desktop's native picker, not ours. Accessibility uses at-spi2 over D-Bus, same
as X11.
The linux-x11 and linux-wayland crates are both compiled into the Linux
binding. At runtime the binding inspects XDG_SESSION_TYPE and chooses the
correct backend; this matches mutter/gnome-shell/kwin reporting.
3.3 Why a single binding instead of a per-OS package#
Two alternatives were considered and rejected:
- One npm package per OS (
@oshun/iris-native-darwin,…linux,…win32). Forces consumers to useoptionalDependenciesresolution tricks; breaks pnpm's deterministic installs; produces 3 different APIs that drift. Rejected. - Optional native dep with JS fallback. Rejected because there is no useful
JS fallback for
mouse_click()— the agent either has OS-level pointer access or it doesn't. A "fallback" that returnsErr(unsupported)is just a runtime surprise.
The single-binding approach matches what sharp, @swc/core, and
better-sqlite3 do today, all of which work cleanly under pnpm.
4. Security and sandboxing#
4.1 OS-level consent surfaces#
Each backend uses the OS's native consent mechanism. The agent never
attempts to bypass these surfaces — when the OS denies access, the binding
returns Err(PermissionDenied) and the agent must surface the failure to the
user via the existing safety/permission-checker.ts flow.
| OS | Consent surface | Granted via |
|---|---|---|
| macOS | TCC database (tccd) entries for kTCCServiceAccessibility and kTCCServiceScreenCapture |
System Settings → Privacy & Security |
| Linux Wayland | Portal dialog from xdg-desktop-portal |
First call per session; cached for the session |
| Linux X11 | None at the OS level — X11 inherently allows any client to inject | OSHUN sandbox layer (see §4.3) |
| Windows | UIPI / UAC elevation prompt when targeting elevated windows | Per-binary at first launch |
4.2 OSHUN-side guardrails#
The native binding is one half of the security story. The other half lives in the agent runtime:
safety/permission-checker.ts— every native call is gated by an OSHUN-side allow-list before it reaches the napi binding. The allow-list is per-task (e.g. "this task may move the mouse anywhere on display 1, may type text, may NOT modify the clipboard, may NOT capture screen 2") and is serialised into the task envelope.safety/operation-recorder.ts— every successful native call appends to an immutable WORM log under~/.local/state/oshun/computer-use-operations/, rotated daily, retained 30 days. This is the audit trail required bydocs/releases/v1/audits/security/agent-actions.md(planned; not yet written).recovery/state-snapshotter.ts— before any input injection that the safety checker classifies as "destructive" (clipboard write, paste, key combo containing Cmd+Q / Alt+F4 / Ctrl+W), the controller takes a window-state snapshot. Recovery code can replay the snapshot to undo the action.
4.3 Linux X11 caveat#
X11 has no application-level sandboxing — any X11 client can inject into any window owned by any other client of the same display. This is a known X11 limitation, not an OSHUN bug. Two mitigations:
- The OSHUN runtime refuses to start the X11 backend when the user's session
type is
wayland(i.e. XWayland is available). This forces Wayland-portal consent for users on modern desktops. - A configuration flag
OSHUN_AGENT_REQUIRE_WAYLAND=1(set by default in the OSHUN-managed Linux container image) makes the X11 backend refuse to load altogether.
4.4 Sandbox-level isolation (forward-looking)#
For deployments that need stronger isolation than the OS provides
(unattended-execution scenarios, regulated industries), the binding can be run
inside a bubblewrap jail with only the X11 / Wayland sockets and a single
display whitelisted. The Rust crates have no unsafe outside the FFI bodies, so
the binding can run in a seccomp jail that allows only the syscalls listed in
a computer-use-seccomp.bpf policy file (a follow-up artifact; landing path to
be decided when the policy is written).
5. Distribution#
The binding is published as @oshun/iris-computer-use-native and follows the
napi-rs prebuild convention:
@oshun/iris-computer-use-native/
package.json # main package; declares per-OS optionalDependencies
index.js
index.d.ts
npm/
darwin-arm64/ # one package per (os, arch) target
darwin-x64/
linux-x64-gnu/
linux-arm64-gnu/
win32-x64-msvc/
win32-arm64-msvc/
The CI matrix produces all six prebuilds on tag-push. Consumers install
@oshun/iris-computer-use-native and pnpm picks the correct
optionalDependencies entry for their platform; no node-gyp runs at install
time.
The linux-x64-gnu prebuild requires glibc ≥ 2.31 (Ubuntu 20.04, Debian 11).
The linux-x64-musl target is not built — Wayland portals require dbus +
GLib which musl ports are unstable.
6. Migration plan#
The implementation is staged across V1-P2-0091 through V1-P2-0096:
V1-P2-0091scaffolds the four platform crates + thecoretrait crate.V1-P2-0092is skipped (RobotJS path not chosen).V1-P2-0093buildsOshunDesktopController— the typed JS wrapper that consumers use.V1-P2-0094adds multi-monitor support to all four backends.V1-P2-0095adds region-of-interest cropping at the capture API.V1-P2-0096wiresmenu-navigator.ts:137to the real accessibility-tree walker on each OS.
Each stage ships independently — OshunDesktopController will degrade
gracefully when a backend is missing (returning Err(unsupported) for that
operation), so there is no all-or-nothing flag day.
7. Out of scope (V2)#
These items are explicitly not part of the V1 desktop binding and have their own backlog items in P3:
- Touch and pen input. The screen-capture and mouse/keyboard surfaces are
V1; tablet / pen / touch event injection is
V1-P3-0xxx(Wacom / TabletPC protocols). - Multi-finger gestures on macOS / Windows. Same family as touch.
- Screen-recording at 60 fps. The capture API is single-frame; sustained
capture pipelines belong with
maat/screen-recorder(which can ship its own PipeWire / DXGI consumer). - Cross-process IPC. All six bindings assume the agent process owns the
binding instance. Sharing a binding across processes is a
V1-P3-0xxxworkitem that requires the OS's secure-handoff protocols.