Eve library

@psyche/computer-use-core

Other

Revision 1

On this page

Native desktop computer-use primitives and an observe/reason/act loop.

ADR-0077 limits this package to the OS-level boundary: native display/window pixels, accessibility, app/window focus, pointer, keyboard, scrolling, clipboard, and native file dialogs. It is not a browser driver. A recognized browser window must be refused and handed to the real Playwright tools in @oshun/assistant unless a preregistered measured exception transfers the single-owner decision.

At a glance#

  • Path: libs/psyche/computer-use-core
  • Domain: psyche (active)
  • Owner: @GreyChimp
  • Docs: deep-dive · systems one-pager
  • Nx project: psyche-computer-use-core

Admission status#

This library is not admitted to Eve yet. The remaining gates are explicit:

  • Task 6.2 removed the package's direct vendor binding. Planning and vision now require separate model-registry bindings backed by the shared @oshun/ai provider interface. Screenshot bytes cannot reach either provider until an injected redaction policy returns a complete decision record.
  • Task 6.3 completed the two-pass fabrication audit. Missing native capture or input capabilities now raise typed errors; mock capture and action handlers are available only from @psyche/computer-use-core/testing.
  • Task 6.4 proves the native boundary against a real OS-level fixture.
  • Task 6.5 adds and proves mandatory per-run authority and execution controls.
  • Task 6.6 still owns leased-work admission. Drawer exposure remains none.

See ADR-0077.

Provider and screenshot contract#

ComputerUseAgent does not read API keys, construct vendor SDK clients, or choose model names. Its owning application must provide a ComputerUseModelRegistry that resolves:

  • planning: text and image input, text output, and tool use;
  • vision: text and image input and text output (no tool use required).

An absent binding or capability mismatch throws ComputerUseContractError with code COMPUTER_USE_MODALITY_UNBOUND before capture or provider use. Missing registry ownership fails construction with COMPUTER_USE_MODEL_REGISTRY_UNBOUND.

Every image provider call also requires an injected ScreenshotRedactor. There is intentionally no production pass-through implementation. The agent validates the original and redacted image's declared format/MIME, image signature, positive dimensions, exact decoded byte count, byte ceiling, and pixel ceiling before transmission. Empty native-fallback envelopes therefore cannot be sent to a model. A missing policy throws COMPUTER_USE_REDACTION_UNBOUND.

Each accepted image produces a non-pixel ScreenshotModelInputRecord with:

  • purpose, provider, and registry-owned model;
  • format, MIME, width, height, pixel count, and encoded byte size;
  • native-binding/provider-hook/data-URL capture source;
  • policy id, redaction status, region count, explanation where applicable, and timestamps.

Planning records are returned in AgentExecutionResult.screenshotInputs and emitted as screenshot_prepared events. Vision records are also attached to ScreenAnalysisResult.vision. Model descriptions are marked untrusted and are never interpreted as executable instructions.

ts
import {
  createComputerUseAgent,
  createStaticComputerUseModelRegistry,
  type ComputerUseRunControlHooks,
  type ComputerUseRunPolicy,
  type ScreenshotRedactor,
} from '@psyche/computer-use-core';

const modelRegistry = createStaticComputerUseModelRegistry({
  planning: {
    provider: planningProvider,
    model: planningModel,
    inputModalities: ['text', 'image'],
    outputModalities: ['text'],
    supportsTools: true,
  },
  vision: {
    provider: visionProvider,
    model: visionModel,
    inputModalities: ['text', 'image'],
    outputModalities: ['text'],
    supportsTools: false,
  },
});

const screenshotRedactor: ScreenshotRedactor = operatorOwnedRedactionPolicy;
const runControlHooks: ComputerUseRunControlHooks = owningHostRunControls;
const agent = createComputerUseAgent({
  modelRegistry,
  screenshotRedactor,
  runControlHooks,
});

const policy: ComputerUseRunPolicy = authorityIssuedForThisRun;
const result = await agent.run('Complete the bounded native task', policy);

The example deliberately leaves provider construction, credential custody, model selection, redaction, OS target observation, process confinement, confirmation, and end-state verification with the owning application. The package rejects an unbound planning provider, redactor, or run-control host.

Per-run execution controls#

ComputerUseRunPolicy is required for every run call. It contains exact application and window tuples, action types, canonical network origins, canonical absolute task-file read/write entries, and explicit limits for steps, actions, actions per minute, action interval, duration, total/per-call tokens, frame age, thinking time, and interrupt acknowledgement. The legacy agent-config maxSteps value is not authority and is no longer used by the run loop; policy.budgets.maxSteps is the enforced ceiling.

Before capture, the host must attest real process-level network and task-file confinement. logical-check-only is never accepted. Run and action previews are digest-bound and carry monotonic sequence numbers. Interactive runs require an explicit confirmation, external-side-effect actions require a fresh explicit confirmation, and destructive actions require step-up.

Every admission returns the exact executable action with automatic retries disabled. The agent executes that returned primitive, so one preview and one action-budget unit cannot silently become multiple native attempts.

The agent captures a fresh frame before an active action. Changed or old frames produce COMPUTER_USE_STALE_FRAME and the new governed frame is returned for replanning without executing the action. Focus is observed before and after capture and again after confirmation; this package never steals focus. Model completion is provisional until the host's independent verifier returns named, passing, digest-backed checks. The returned controlReceipt is pixel-free and contains authority, confirmation, refusal, budget, interrupt, and end-state facts.

See ADR-0080 for risk rules, the Linux X11 confinement proof, and the remaining admission boundary.

Native failure and testing boundary#

Production factories fail closed when the native controller is unavailable. They never report a successful click, key, scroll, drag, clipboard operation, cursor query, or screenshot unless the corresponding native primitive has completed. Screenshot results contain real encoded bytes and measured dimensions; an empty or synthetic fallback frame is not produced.

Display-free tests must opt into the explicit testing subpath:

ts
import {
  createMockScreenshotProvider,
  createTestActionExecutor,
} from '@psyche/computer-use-core/testing';

The package root does not export those doubles. A real OS fixture and native permission proof remain Task 6.4, so this fail-closed contract is not a native runtime admission claim.

Commands#

bash
pnpm nx test psyche-computer-use-core
pnpm nx lint psyche-computer-use-core
pnpm --dir libs/psyche/computer-use-core typecheck
pnpm --dir libs/psyche/computer-use-core build

<!-- Generated starting point (audit F-3): owners, replace this note with usage examples and API notes. -->