Eve library

Control Room Operator Usability Study Gate

Guide

Revision 1

On this page

Status: local gate contract for 180.C.21.20.

Implementation date: 2026-07-01.

testing/bellona/remote-control/operator-usability-study.ts encodes the structured Control Room usability study that must run before release. The gate does not claim the study has happened. It blocks by default until recruited new operators complete the study, the metrics meet their thresholds, the findings report is published, and the top three friction points have fix evidence.

Required Study#

The study must include at least five operators with no prior Control Room operating experience. Each participant must exercise these scenarios:

  • approve a safe mutation under pressure while the session timeline is updating;
  • explain why a policy decision is allowed, denied, or approval-gated;
  • choose approve or deny for mixed safe, risky, and blocked commands;
  • discover takeover, use it, and release takeover back to the agent.

Success Metrics#

  • Time to approve under pressure: median <= 30 seconds and p90 <= 45 seconds.
  • Policy explanation clarity: average score >= 4/5 and no participant below 3.
  • Approve/deny error rate: <= 5%.
  • Takeover discoverability: at least 80% discovery and median time <= 60 seconds.
  • Findings report: published with participants, observations, ranked friction points, and affected scenarios.
  • Top-three fixes: the three highest-severity friction points have fix evidence before release.

Local Evaluator#

The testing package exposes:

  • createControlRoomOperatorUsabilityStudyPlan()
  • createPassingControlRoomOperatorUsabilityEvidenceSnapshot()
  • evaluateControlRoomOperatorUsabilityStudy()

The evaluator returns blocked without evidence. It only returns passed when new-operator evidence, per-scenario metrics, a published report, and top-three friction fixes are all attached. It rejects duplicate, blank, non-canonical, or placeholder participant identities, including case-variant duplicates, placeholder participant roles, missing or duplicate recruited-participant observations for any scenario, stale, missing, or non-canonical UTC millisecond observation timestamps, metric-only observations without the scenario's required evidence kinds and unique evidence artifacts named by those evidence kinds, duplicated observation evidence including case variants, published reports that omit recruited participants or validated scenario evidence artifacts, published reports that cite invalid, reused, stale, incomplete, or non-contextual observation artifacts, published reports outside the canonical 180.C.21.20 evidence path or escaping that path with traversal/backslash segments, published reports captured before any recruited-participant observation, observation artifacts that do not identify their participant/scenario/evidence-kind context with concrete non-placeholder evidence suffixes, malformed metric observations, stale, non-canonical, or unparseable report timestamps, duplicated friction-point IDs including case variants, invalid severity, reused, whitespace-only, placeholder, or non-contextual fix evidence without concrete non-placeholder suffixes, zero-duration human timing metrics, successful takeover discoveries without a valid time-to-first-action sample, duplicate fix evidence IDs including case variants, and top-three fixes verified before the findings report or without canonical UTC millisecond verification timestamps.

Verification#

bash
pnpm exec vitest run --config vitest.config.ts tests/operator-usability-study.test.ts --reporter=verbose

Focused coverage currently passes with 1 file / 27 tests and proves:

  • the plan lists all four 180.C.21.20 scenarios and report requirements;
  • the gate blocks by default instead of treating implemented UI tests as human study evidence;
  • the gate passes with complete new-operator, metric, report, and fix evidence;
  • the gate blocks when a participant is not new;
  • the gate blocks placeholder recruited-participant identities and roles;
  • the gate blocks case-variant duplicate recruited-participant identities;
  • the gate blocks when a top-three friction point lacks fix evidence;
  • the gate blocks when recruited-participant observations are missing even if stray non-participant rows exist;
  • the gate blocks metric observations without the scenario's required evidence artifacts or evidence-kind coverage;
  • the gate blocks observation artifacts that do not name the required evidence kind they prove;
  • the gate blocks published reports that omit recruited participants, omit validated observation evidence artifacts, or cite invalid observation evidence as if it were valid;
  • the gate blocks published reports outside, escaping, or using placeholder terminal files under the canonical 180.C.21.20 evidence path;
  • the gate blocks scenario evidence artifacts that cannot be traced to their recruited participant and scenario;
  • the gate blocks stale report timestamps and empty contextual observation/fix evidence suffix placeholders;
  • the gate blocks observation and top-three fix evidence that uses placeholder contextual suffixes such as placeholder or todo;
  • the gate blocks stale observation timestamps, reports published before observations, and top-three fixes verified before the findings report;
  • the gate blocks observation, report, and top-three fix timestamps without canonical UTC milliseconds;
  • the gate blocks duplicate recruited-participant scenario observations instead of letting one participant's duplicate rows mask another participant's missing rows;
  • the gate blocks duplicated friction-point identifiers, invalid severity, and reused top-three fix evidence;
  • the gate blocks case-variant duplicate observation and top-three fix evidence identifiers;
  • the gate blocks top-three fix evidence that cannot be traced to its friction point;
  • the gate blocks when approval timing, policy clarity, decision accuracy, or takeover discovery regress.
  • the gate blocks malformed participant metadata, report metadata, fix evidence, and impossible metric values.
  • the gate blocks zero-duration approval and takeover timing observations.
  • the gate blocks takeover discoveries that lack a valid time-to-first-action sample.