Eve library

Browser-control backend evaluation — Browser Use (local & cloud) vs. direct CDP

Guide

Revision 1

On this page

Phase: 180.C.20.01 Status: Decision — direct CDP is the verified default; Browser Use is an opt-in, policy-gated, fail-loud provider behind the same interface. Scope: Which backend the Bellona remote-host browser surface routes a task plan to, and the reliability / privacy / latency / cost tradeoffs of each.

This is a real comparison run against the same task fixtures the deterministic browser surface already ships — the five 180.C.19.09 task templates in libs/bellona/remote-protocol/src/browser-task-templates.ts:

Template id Fixture role Mutation posture
task-template:production-board-update open board → focus card → set status → submit safe-mutation + a form submission
task-template:asset-marketplace-search navigate → type query (pressEnter) → wait → snapshot safe-mutation (submit-capable type) + read
task-template:documentation-lookup navigate → search → wait → extract safe-mutation + read
task-template:account-free-research navigate → wait → snapshot read-only throughout
task-template:authenticated-admin-workflow grant profile → navigate → gated admin action authenticated profile + operator-approval-gated mutation

Each template expands (via planRemoteHostBrowserTaskTemplate) into an ordered sequence of typed, schema-validated browser command argument-sets. That expansion is the provider-agnostic plan all three backends consume unchanged — the planner does not change to swap backends (see 180.C.20.02, libs/bellona/remote-protocol/src/browser-agent-provider.ts).

Honesty note (measurement). Direct-CDP numbers below are anchored to the deterministic adapter that is built and exercised on-box by the host test suite (apps/bellona/remote-host/src/browser-adapter.test.ts, browser-agent-direct-cdp.test.ts). The Browser Use cloud figures are projected / uninstrumented — no Browser Use cloud account or API key is wired in this environment, so its provider is an injectable seam that fails loud provider_not_configured until creds are supplied (by design; we do not fabricate a run). Browser Use local figures are likewise projected from the published browser-use architecture (an LLM agent loop driving Playwright over CDP), not measured here. Where a row is projected it is marked (projected).


1. The three backends#

1.1 Direct CDP (first-party, direct_cdp)#

The existing CDP/Playwright host browser adapter (apps/bellona/remote-host/src/browser-adapter.ts). Each plan step is a typed Oshun command (browser.navigate, browser.click_element, browser.submit, …) dispatched to a deterministic handler that drives Playwright over a loopback-only CDP endpoint against an isolated managed profile. There is no LLM in the loop: the plan fully determines the actions. The createRemoteHostBrowserDirectCdpProvider wrapper (apps/bellona/remote-host/src/browser-agent-direct-cdp.ts) exposes this adapter through the common provider interface, so it is selectable as the deterministic default with zero third-party dependencies.

1.2 Browser Use — local (browser_use_local)#

The open-source browser-use Python package run on the same box: an LLM agent loop that observes the page (accessibility tree + screenshots) and decides the next Playwright action. The model call is local-network/remote depending on which LLM is configured, but the browser execution stays on-box. In Oshun this is a RemoteBrowserUseRuntime wired into createRemoteBrowserUseProvider({ mode: 'browser_use_local', runtime }). Without a wired runtime the provider fails loud provider_not_configured.

1.3 Browser Use — cloud (browser_use_cloud)#

The hosted Browser Use Cloud API: the task plan / objective is sent off-box, a remote browser executes it, and step events + screenshots stream back. In Oshun this is the same provider with mode: 'browser_use_cloud' over a runtime that calls the cloud API. Execution leaves the box (offBoxExecution: true), which is the dominant privacy consideration.

A fourth mode, mcp, exists in the interface for "another MCP browser provider" but is out of scope for this measurement; it is gated identically to the Browser Use modes (any non-direct_cdp mode is third-party).


2. Reliability#

Dimension direct_cdp browser_use_local (projected) browser_use_cloud (projected)
Determinism Deterministic — the plan fully specifies the steps; same plan → same actions Non-deterministic — an LLM picks the next action each step Non-deterministic (same loop, remote)
Repeatability of the 5 fixtures Repeatable; asserted in browser-adapter.test.ts (62 tests) + browser-agent-direct-cdp.test.ts Varies run-to-run with model + page changes Varies run-to-run
Failure mode Fails loud with structured evidence (DOM-failure evidence, trace.zip) Agent may "succeed" on the wrong element or loop; needs a verifier Same, plus network/availability failures
Selector robustness to page drift Brittle to large redesigns (selector-bound), but predictable More robust to cosmetic drift (semantic targeting) Same as local
Recovery Compensating actions per mutating command (180.C.19.*) Agent self-correction (unbounded without guardrails) Same
Auditability of what ran Every step is a typed command in the audit log Steps normalized to Oshun events (180.C.20.03), but the decision was the model's Same

Reading. Direct CDP wins decisively on determinism, repeatability, and auditability — exactly the properties an MVP and a regression suite need. Browser Use's value is robustness to page drift and handling under-specified objectives, at the cost of determinism. For the five fixtures, which are fully specified, the non-determinism is a net negative; Browser Use earns its keep on tasks the templates can't pre-express.


3. Privacy#

Dimension direct_cdp browser_use_local browser_use_cloud
Page content leaves the box No Only to the configured LLM (prompt context) Yes — page DOM/screenshots sent to the cloud
Authenticated profile exposure First-party; isolated managed profile by default Third-party agent drives the profile in-process Third-party + off-box
Credential entry Blocked on personal profiles unless explicitly approved; values redacted from evidence (180.C.19.06) Must be gated — agent could be baited into typing secrets Must be gated; higher blast radius
Screenshot/trace redaction Host applies redaction masks before persistence Normalized step screenshots marked redaction: pending until host redacts Same
Step-note leakage n/a (no free-text agent notes) Agent "thoughts" can contain secrets → redacted by redactRemoteBrowserAgentText before persistence (180.C.20.05) Same

Reading. Direct CDP is the only backend where neither page content nor credentials leave the box. Browser Use local sends page context to whatever LLM is configured; Browser Use cloud sends it to a third party. This is why the 180.C.20.04 policy gate refuses to hand a non-direct_cdp provider an authenticated/granted profile, a sensitive origin, a file upload, or a form submission without an explicit, plan-bound operator approval, and why direct_cdp is the only mode exempt from that gate.


4. Latency#

Step type direct_cdp (on-box, measured class) browser_use_local (projected) browser_use_cloud (projected)
navigate (load) network-bound page load; no model call page load + 1 LLM decision (~1–5 s) page load + LLM + round-trip to cloud
click / type / submit low-ms Playwright action + the page's own latency + 1 LLM decision per step + LLM + cloud round-trip per step
snapshot / extract direct DOM/text read model may re-observe before extracting + cloud round-trip
Whole production-board-update (4 steps) ~ sum of 4 deterministic actions + page latencies 4 actions + 4 model decisions + per-step cloud round-trips

Direct CDP adds no model latency: a step costs a Playwright action plus the page's own latency. The host enforces explicit per-class ceilings — read-only standard (≤120 s), safe-mutation short (≤30 s), file-transfer (≤120 s) — in browser.ts. Browser Use adds one LLM decision per step (local) plus a network round-trip per step (cloud). For a 4-step plan that is 4× the model latency stacked on top of the same page latencies. For under-specified objectives where Browser Use replaces many human iterations, that per-step cost can still be a net win; for the pre-expanded templates it is pure overhead.


5. Cost#

Dimension direct_cdp browser_use_local browser_use_cloud
Per-step marginal $ ~0 (CPU/Playwright only) LLM tokens per step (prompt = page context + history) LLM tokens + hosted-browser minutes
Fixed $ none beyond the host a local model endpoint (self-host or API) a Browser Use Cloud subscription / API key
Scaling shape linear in CPU; cheap to fan out linear in tokens; page-context prompts are large linear in tokens + browser-minutes
Cost predictability High (deterministic step count) Lower (step count varies with the agent loop) Lower

Direct CDP is effectively free per step. Browser Use's cost scales with tokens (the page context is a large prompt) and, for cloud, with hosted-browser minutes. A deterministic 4-step template costs ~0 on direct CDP and 4 model calls on Browser Use.


6. Decision#

  1. Direct CDP is the verified default. It is deterministic, on-box, auditable, free per step, and already built + tested on this box. The deterministic MVP runs entirely through direct_cdp with no third-party provider registered — selectRemoteBrowserAgentProvider picks it as the default and refuses to silently downgrade a deterministic task to a third-party agent (no_silent_downgrade).

  2. Browser Use is an opt-in provider behind the same interface, selected only by explicit provider id/mode, for tasks the templates cannot pre-express (under-specified objectives, heavy page drift). It is wired as an injectable seam that fails loud provider_not_configured until a runtime/credentials are supplied — we never fabricate a Browser Use run.

  3. Third-party providers are policy-gated. A non-direct_cdp provider is refused an authenticated/granted profile, a sensitive origin, a file upload, or a form submission without an explicit, plan-bound operator approval (180.C.20.04). The approval must bind to the exact task run (approvedTaskId, single-use), the exact plan hash, the exact provider (id+mode), be unexpired (expiresAt checked against now), and enumerate every detected reason + every detected sensitive origin — so it cannot be replayed against a later run, reused after expiry, retargeted to another provider, or silently widened. Exposure is classified from each step's command name + re-parsed arguments (never the planner's risk labels), the active origin is tracked across steps so a click/type/submit on an already-sensitive page (or a launch_managed that opens directly on a sensitive origin) is caught, and an unknown command or unparseable arguments is denied as maximally dangerous.

  4. Whatever runs, runs through Oshun's schemas. Every Browser Use / external step event is normalized (180.C.20.03) into Oshun command progress, a timeline event, an optional screenshot/trace artifact, and an audit entry — all validated against the existing protocol schemas. An event that cannot be normalized is quarantined fail-loud, never dropped into a fabricated success. Credential-ish substrings in agent step notes are redacted before persistence.

When to reach for each#

  • Direct CDP — the five templates and any task that can be expressed as a deterministic plan; all regression/CI runs; anything touching credentials, sensitive origins, uploads, or submissions on an authenticated profile.
  • Browser Use local — exploratory or under-specified objectives where page context staying on-box matters and a local model is available.
  • Browser Use cloud — only when neither of the above is viable, with explicit operator approval, accepting that page content leaves the box.

7. Evidence pointers#

  • Provider interface, selection, normalization, policy gate, Browser Use seams: libs/bellona/remote-protocol/src/browser-agent-provider.ts
  • Adversarial tests (cannot bypass schema/policy/approval/redaction/audit): libs/bellona/remote-protocol/src/browser-agent-provider.test.ts
  • First-party direct-CDP provider over the real adapter handlers: apps/bellona/remote-host/src/browser-agent-direct-cdp.ts (+ .test.ts)
  • The fixtures (task templates) this evaluation runs against: libs/bellona/remote-protocol/src/browser-task-templates.ts
  • Prompt-injection corpus the gates are also proven against: libs/bellona/remote-protocol/src/browser-prompt-injection.ts