Native desktop computer-use primitives and an observe/reason/act loop.
ADR-0077 limits this package to the OS-level boundary: native display/window
pixels, accessibility, app/window focus, pointer, keyboard, scrolling,
clipboard, and native file dialogs. It is not a browser driver. A recognized
browser window must be refused and handed to the real Playwright tools in
@oshun/assistant unless a preregistered measured exception transfers the
single-owner decision.
At a glance#
- Path:
libs/psyche/computer-use-core - Domain: psyche (active)
- Owner: @GreyChimp
- Docs: deep-dive · systems one-pager
- Nx project:
psyche-computer-use-core
Admission status#
This library is not admitted to Eve yet. The remaining gates are explicit:
- Task 6.2 removed the package's direct vendor binding. Planning and vision now
require separate model-registry bindings backed by the shared
@oshun/aiprovider interface. Screenshot bytes cannot reach either provider until an injected redaction policy returns a complete decision record. - Task 6.3 completed the two-pass fabrication audit. Missing native capture or
input capabilities now raise typed errors; mock capture and action handlers
are available only from
@psyche/computer-use-core/testing. - Task 6.4 proves the native boundary against a real OS-level fixture.
- Task 6.5 adds and proves mandatory per-run authority and execution controls.
- Task 6.6 still owns leased-work admission. Drawer exposure remains none.
See ADR-0077.
Provider and screenshot contract#
ComputerUseAgent does not read API keys, construct vendor SDK clients, or
choose model names. Its owning application must provide a
ComputerUseModelRegistry that resolves:
planning: text and image input, text output, and tool use;vision: text and image input and text output (no tool use required).
An absent binding or capability mismatch throws ComputerUseContractError with
code COMPUTER_USE_MODALITY_UNBOUND before capture or provider use. Missing
registry ownership fails construction with
COMPUTER_USE_MODEL_REGISTRY_UNBOUND.
Every image provider call also requires an injected ScreenshotRedactor. There
is intentionally no production pass-through implementation. The agent validates
the original and redacted image's declared format/MIME, image signature,
positive dimensions, exact decoded byte count, byte ceiling, and pixel ceiling
before transmission. Empty native-fallback envelopes therefore cannot be sent to
a model. A missing policy throws COMPUTER_USE_REDACTION_UNBOUND.
Each accepted image produces a non-pixel ScreenshotModelInputRecord with:
- purpose, provider, and registry-owned model;
- format, MIME, width, height, pixel count, and encoded byte size;
- native-binding/provider-hook/data-URL capture source;
- policy id, redaction status, region count, explanation where applicable, and timestamps.
Planning records are returned in AgentExecutionResult.screenshotInputs and
emitted as screenshot_prepared events. Vision records are also attached to
ScreenAnalysisResult.vision. Model descriptions are marked untrusted and are
never interpreted as executable instructions.
import {
createComputerUseAgent,
createStaticComputerUseModelRegistry,
type ComputerUseRunControlHooks,
type ComputerUseRunPolicy,
type ScreenshotRedactor,
} from '@psyche/computer-use-core';
const modelRegistry = createStaticComputerUseModelRegistry({
planning: {
provider: planningProvider,
model: planningModel,
inputModalities: ['text', 'image'],
outputModalities: ['text'],
supportsTools: true,
},
vision: {
provider: visionProvider,
model: visionModel,
inputModalities: ['text', 'image'],
outputModalities: ['text'],
supportsTools: false,
},
});
const screenshotRedactor: ScreenshotRedactor = operatorOwnedRedactionPolicy;
const runControlHooks: ComputerUseRunControlHooks = owningHostRunControls;
const agent = createComputerUseAgent({
modelRegistry,
screenshotRedactor,
runControlHooks,
});
const policy: ComputerUseRunPolicy = authorityIssuedForThisRun;
const result = await agent.run('Complete the bounded native task', policy);
The example deliberately leaves provider construction, credential custody, model selection, redaction, OS target observation, process confinement, confirmation, and end-state verification with the owning application. The package rejects an unbound planning provider, redactor, or run-control host.
Per-run execution controls#
ComputerUseRunPolicy is required for every run call. It contains exact
application and window tuples, action types, canonical network origins,
canonical absolute task-file read/write entries, and explicit limits for steps,
actions, actions per minute, action interval, duration, total/per-call tokens,
frame age, thinking time, and interrupt acknowledgement. The legacy agent-config
maxSteps value is not authority and is no longer used by the run loop;
policy.budgets.maxSteps is the enforced ceiling.
Before capture, the host must attest real process-level network and task-file
confinement. logical-check-only is never accepted. Run and action previews are
digest-bound and carry monotonic sequence numbers. Interactive runs require an
explicit confirmation, external-side-effect actions require a fresh explicit
confirmation, and destructive actions require step-up.
Every admission returns the exact executable action with automatic retries disabled. The agent executes that returned primitive, so one preview and one action-budget unit cannot silently become multiple native attempts.
The agent captures a fresh frame before an active action. Changed or old frames
produce COMPUTER_USE_STALE_FRAME and the new governed frame is returned for
replanning without executing the action. Focus is observed before and after
capture and again after confirmation; this package never steals focus. Model
completion is provisional until the host's independent verifier returns named,
passing, digest-backed checks. The returned controlReceipt is pixel-free and
contains authority, confirmation, refusal, budget, interrupt, and end-state
facts.
See ADR-0080 for risk rules, the Linux X11 confinement proof, and the remaining admission boundary.
Native failure and testing boundary#
Production factories fail closed when the native controller is unavailable. They never report a successful click, key, scroll, drag, clipboard operation, cursor query, or screenshot unless the corresponding native primitive has completed. Screenshot results contain real encoded bytes and measured dimensions; an empty or synthetic fallback frame is not produced.
Display-free tests must opt into the explicit testing subpath:
import {
createMockScreenshotProvider,
createTestActionExecutor,
} from '@psyche/computer-use-core/testing';
The package root does not export those doubles. A real OS fixture and native permission proof remain Task 6.4, so this fail-closed contract is not a native runtime admission claim.
Commands#
pnpm nx test psyche-computer-use-core
pnpm nx lint psyche-computer-use-core
pnpm --dir libs/psyche/computer-use-core typecheck
pnpm --dir libs/psyche/computer-use-core build
<!-- Generated starting point (audit F-3): owners, replace this note with usage examples and API notes. -->