# Control Room Operator Usability Study Gate

Status: local gate contract for `180.C.21.20`.

Implementation date: 2026-07-01.

`testing/bellona/remote-control/operator-usability-study.ts` encodes the
structured Control Room usability study that must run before release. The gate
does not claim the study has happened. It blocks by default until recruited new
operators complete the study, the metrics meet their thresholds, the findings
report is published, and the top three friction points have fix evidence.

## Required Study

The study must include at least five operators with no prior Control Room
operating experience. Each participant must exercise these scenarios:

- approve a safe mutation under pressure while the session timeline is updating;
- explain why a policy decision is allowed, denied, or approval-gated;
- choose approve or deny for mixed safe, risky, and blocked commands;
- discover takeover, use it, and release takeover back to the agent.

## Success Metrics

- Time to approve under pressure: median <= 30 seconds and p90 <= 45 seconds.
- Policy explanation clarity: average score >= 4/5 and no participant below 3.
- Approve/deny error rate: <= 5%.
- Takeover discoverability: at least 80% discovery and median time <= 60
  seconds.
- Findings report: published with participants, observations, ranked friction
  points, and affected scenarios.
- Top-three fixes: the three highest-severity friction points have fix evidence
  before release.

## Local Evaluator

The testing package exposes:

- `createControlRoomOperatorUsabilityStudyPlan()`
- `createPassingControlRoomOperatorUsabilityEvidenceSnapshot()`
- `evaluateControlRoomOperatorUsabilityStudy()`

The evaluator returns `blocked` without evidence. It only returns `passed` when
new-operator evidence, per-scenario metrics, a published report, and top-three
friction fixes are all attached. It rejects duplicate, blank, non-canonical, or
placeholder participant identities, including case-variant duplicates,
placeholder participant roles, missing or duplicate recruited-participant
observations for any scenario, stale, missing, or non-canonical UTC millisecond
observation timestamps, metric-only observations without the scenario's required
evidence kinds and unique evidence artifacts named by those evidence kinds,
duplicated observation evidence including case variants, published reports that
omit recruited participants or validated scenario evidence artifacts, published
reports that cite invalid, reused, stale, incomplete, or non-contextual
observation artifacts, published reports outside the canonical `180.C.21.20`
evidence path or escaping that path with traversal/backslash segments, published
reports captured before any recruited-participant observation, observation
artifacts that do not identify their participant/scenario/evidence-kind context
with concrete non-placeholder evidence suffixes, malformed metric observations,
stale, non-canonical, or unparseable report timestamps, duplicated
friction-point IDs including case variants, invalid severity, reused,
whitespace-only, placeholder, or non-contextual fix evidence without concrete
non-placeholder suffixes, zero-duration human timing metrics, successful
takeover discoveries without a valid time-to-first-action sample, duplicate fix
evidence IDs including case variants, and top-three fixes verified before the
findings report or without canonical UTC millisecond verification timestamps.

## Verification

```bash
pnpm exec vitest run --config vitest.config.ts tests/operator-usability-study.test.ts --reporter=verbose
```

Focused coverage currently passes with 1 file / 27 tests and proves:

- the plan lists all four `180.C.21.20` scenarios and report requirements;
- the gate blocks by default instead of treating implemented UI tests as human
  study evidence;
- the gate passes with complete new-operator, metric, report, and fix evidence;
- the gate blocks when a participant is not new;
- the gate blocks placeholder recruited-participant identities and roles;
- the gate blocks case-variant duplicate recruited-participant identities;
- the gate blocks when a top-three friction point lacks fix evidence;
- the gate blocks when recruited-participant observations are missing even if
  stray non-participant rows exist;
- the gate blocks metric observations without the scenario's required evidence
  artifacts or evidence-kind coverage;
- the gate blocks observation artifacts that do not name the required evidence
  kind they prove;
- the gate blocks published reports that omit recruited participants, omit
  validated observation evidence artifacts, or cite invalid observation evidence
  as if it were valid;
- the gate blocks published reports outside, escaping, or using placeholder
  terminal files under the canonical `180.C.21.20` evidence path;
- the gate blocks scenario evidence artifacts that cannot be traced to their
  recruited participant and scenario;
- the gate blocks stale report timestamps and empty contextual observation/fix
  evidence suffix placeholders;
- the gate blocks observation and top-three fix evidence that uses placeholder
  contextual suffixes such as `placeholder` or `todo`;
- the gate blocks stale observation timestamps, reports published before
  observations, and top-three fixes verified before the findings report;
- the gate blocks observation, report, and top-three fix timestamps without
  canonical UTC milliseconds;
- the gate blocks duplicate recruited-participant scenario observations instead
  of letting one participant's duplicate rows mask another participant's missing
  rows;
- the gate blocks duplicated friction-point identifiers, invalid severity, and
  reused top-three fix evidence;
- the gate blocks case-variant duplicate observation and top-three fix evidence
  identifiers;
- the gate blocks top-three fix evidence that cannot be traced to its friction
  point;
- the gate blocks when approval timing, policy clarity, decision accuracy, or
  takeover discovery regress.
- the gate blocks malformed participant metadata, report metadata, fix evidence,
  and impossible metric values.
- the gate blocks zero-duration approval and takeover timing observations.
- the gate blocks takeover discoveries that lack a valid time-to-first-action
  sample.
