Status: local gate contract for 180.C.21.20.
Implementation date: 2026-07-01.
testing/bellona/remote-control/operator-usability-study.ts encodes the
structured Control Room usability study that must run before release. The gate
does not claim the study has happened. It blocks by default until recruited new
operators complete the study, the metrics meet their thresholds, the findings
report is published, and the top three friction points have fix evidence.
Required Study#
The study must include at least five operators with no prior Control Room operating experience. Each participant must exercise these scenarios:
- approve a safe mutation under pressure while the session timeline is updating;
- explain why a policy decision is allowed, denied, or approval-gated;
- choose approve or deny for mixed safe, risky, and blocked commands;
- discover takeover, use it, and release takeover back to the agent.
Success Metrics#
- Time to approve under pressure: median <= 30 seconds and p90 <= 45 seconds.
- Policy explanation clarity: average score >= 4/5 and no participant below 3.
- Approve/deny error rate: <= 5%.
- Takeover discoverability: at least 80% discovery and median time <= 60 seconds.
- Findings report: published with participants, observations, ranked friction points, and affected scenarios.
- Top-three fixes: the three highest-severity friction points have fix evidence before release.
Local Evaluator#
The testing package exposes:
createControlRoomOperatorUsabilityStudyPlan()createPassingControlRoomOperatorUsabilityEvidenceSnapshot()evaluateControlRoomOperatorUsabilityStudy()
The evaluator returns blocked without evidence. It only returns passed when
new-operator evidence, per-scenario metrics, a published report, and top-three
friction fixes are all attached. It rejects duplicate, blank, non-canonical, or
placeholder participant identities, including case-variant duplicates,
placeholder participant roles, missing or duplicate recruited-participant
observations for any scenario, stale, missing, or non-canonical UTC millisecond
observation timestamps, metric-only observations without the scenario's required
evidence kinds and unique evidence artifacts named by those evidence kinds,
duplicated observation evidence including case variants, published reports that
omit recruited participants or validated scenario evidence artifacts, published
reports that cite invalid, reused, stale, incomplete, or non-contextual
observation artifacts, published reports outside the canonical 180.C.21.20
evidence path or escaping that path with traversal/backslash segments, published
reports captured before any recruited-participant observation, observation
artifacts that do not identify their participant/scenario/evidence-kind context
with concrete non-placeholder evidence suffixes, malformed metric observations,
stale, non-canonical, or unparseable report timestamps, duplicated
friction-point IDs including case variants, invalid severity, reused,
whitespace-only, placeholder, or non-contextual fix evidence without concrete
non-placeholder suffixes, zero-duration human timing metrics, successful
takeover discoveries without a valid time-to-first-action sample, duplicate fix
evidence IDs including case variants, and top-three fixes verified before the
findings report or without canonical UTC millisecond verification timestamps.
Verification#
pnpm exec vitest run --config vitest.config.ts tests/operator-usability-study.test.ts --reporter=verbose
Focused coverage currently passes with 1 file / 27 tests and proves:
- the plan lists all four
180.C.21.20scenarios and report requirements; - the gate blocks by default instead of treating implemented UI tests as human study evidence;
- the gate passes with complete new-operator, metric, report, and fix evidence;
- the gate blocks when a participant is not new;
- the gate blocks placeholder recruited-participant identities and roles;
- the gate blocks case-variant duplicate recruited-participant identities;
- the gate blocks when a top-three friction point lacks fix evidence;
- the gate blocks when recruited-participant observations are missing even if stray non-participant rows exist;
- the gate blocks metric observations without the scenario's required evidence artifacts or evidence-kind coverage;
- the gate blocks observation artifacts that do not name the required evidence kind they prove;
- the gate blocks published reports that omit recruited participants, omit validated observation evidence artifacts, or cite invalid observation evidence as if it were valid;
- the gate blocks published reports outside, escaping, or using placeholder
terminal files under the canonical
180.C.21.20evidence path; - the gate blocks scenario evidence artifacts that cannot be traced to their recruited participant and scenario;
- the gate blocks stale report timestamps and empty contextual observation/fix evidence suffix placeholders;
- the gate blocks observation and top-three fix evidence that uses placeholder
contextual suffixes such as
placeholderortodo; - the gate blocks stale observation timestamps, reports published before observations, and top-three fixes verified before the findings report;
- the gate blocks observation, report, and top-three fix timestamps without canonical UTC milliseconds;
- the gate blocks duplicate recruited-participant scenario observations instead of letting one participant's duplicate rows mask another participant's missing rows;
- the gate blocks duplicated friction-point identifiers, invalid severity, and reused top-three fix evidence;
- the gate blocks case-variant duplicate observation and top-three fix evidence identifiers;
- the gate blocks top-three fix evidence that cannot be traced to its friction point;
- the gate blocks when approval timing, policy clarity, decision accuracy, or takeover discovery regress.
- the gate blocks malformed participant metadata, report metadata, fix evidence, and impossible metric values.
- the gate blocks zero-duration approval and takeover timing observations.
- the gate blocks takeover discoveries that lack a valid time-to-first-action sample.