Eve library

`iris-computer-use-native`

Other

Revision 1

On this page

Native (Rust) backend for the iris/computer-use desktop binding. Each supported OS implements the iris_desktop_core::DesktopBackend trait in its own crate; the napi-rs bridge in crates/iris-desktop-napi selects the right backend at module-load time and exposes the surface to Node via @oshun/iris-computer-use-native.

Crate layout#

Crate Purpose
iris-desktop-core Backend-agnostic traits, types, errors. Compiles everywhere.
iris-desktop-linux-x11 XTest input + XCB capture + AT-SPI accessibility.
iris-desktop-linux-wayland libei input portal + screencast portal + AT-SPI.
iris-desktop-macos-cgevent CoreGraphics events + CGDisplayCreateImage + AXUIElement.
iris-desktop-windows-sendinput SendInput + GDI BitBlt (DXGI in V1-P3) + UI Automation.
iris-desktop-napi napi-rs cdylib that exposes IrisDesktop* JS functions.

Backends use #[cfg(target_os = "...")] heavily so each crate compiles to an empty lib on non-matching hosts. The workspace cargo build succeeds on every supported CI host.

Strategy#

See docs/releases/p2/desktop-automation-strategy.md (V1-P2-0090) for the napi-rs vs RobotJS decision, the full architecture, and the security model.

Build#

text
cd libs/iris/agents/computer-use/native
cargo build --workspace

Linux requires the X11 + XTest dev headers when the linux-x11 backend is in scope (apt install libx11-dev libxtst-dev libxrandr-dev) and a running D-Bus session for the Wayland backend's portal probe.

Status#

Backlog Status Notes
V1-P2-0090 done Strategy doc.
V1-P2-0091 done This scaffold. All 6 crates compile; X11 is fully wired against XTest.
V1-P2-0092 skipped RobotJS path not chosen.
V1-P2-0093 partial OshunDesktopController is present; native clipboard backends are not.
V1-P2-0094 partial X11/macOS/Windows enumerate displays; Wayland capture remains stubbed.
V1-P2-0095 partial X11/macOS/Windows crop regions; Wayland capture remains stubbed.
V1-P2-0096 pending AT-SPI / AXUIElement / UIA accessibility-tree walkers are not wired.

Native fixture#

Task 6.4 adds a browser-independent Linux X11 fixture at tools/eve-everywhere/run-native-desktop-fixture.mjs. It launches an isolated two-screen, 144-DPI Xvfb server and GTK3 target, runs a real AT-SPI preflight, loads the compiled N-API backend through OshunDesktopController, verifies occlusion refusal/recovery, completes a controlled file-and-clipboard action, redacts sensitive screenshot pixels before an ephemeral write, and verifies process/socket/scratch cleanup.

The retained receipt is intentionally pixel-free. Run the reproducible probe from the repository root after building the debug binding:

sh
cargo build --manifest-path libs/iris/agents/computer-use/native/Cargo.toml \
  -p iris-desktop-napi
dbus-run-session -- env NO_AT_BRIDGE=0 GTK_A11Y=always \
  node --import tsx tools/eve-everywhere/run-native-desktop-fixture.mjs
node tools/eve-everywhere/verify-native-desktop-fixture.mjs

The dated Task 6.4 proof covers the Linux X11 backend only. It does not upgrade the Wayland/macOS/Windows rows above or authorize production native use.