# OPS-003 — Autonomous-content operator metrics and SLO dashboard

Status: **queryable dashboard and ingestion contract implemented; complete live
V1–V9 coverage, human-work, quality, and operations telemetry remains
unproved**.

Implementation source commit: `2db7b53ba10b1a48eb21b22dca9d70aa0604a7f0`.

## Requirement disposition

| OPS-003 requirement                             | Verdict                                          | Evidence                                                                                                                                                                                                                                                                                                                                |
| ----------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Coverage metrics                                | Partially populated; outcome open                | The canonical repository inventory produces exact required/current/missing/stale/blocked quantity records and age samples. The catalog also requires dependency backlog, invalidation backlog, and unregistered-shipped ratio, but no authorized live streams report those values, so they render as unknown rather than zero.          |
| Human-work metrics                              | Contract and dashboard implemented; outcome open | The typed catalog covers untouched release by autonomy tier, active minutes, comments, edit distance, replacements, regenerations, approvals, vetoes, tweaks, takeovers, forks, rollbacks, recurring overrides, appeals, and edit incorporation. No live V1–V9 human-work stream was available, so the family remains missing.          |
| Quality metrics                                 | Contract and dashboard implemented; outcome open | The catalog covers authored/champion pairwise results, hard-gate escapes, pass@k, pass^k, judge calibration/bias/variance/disagreement, novelty/repetition, canon/style drift, and domain outcomes. No authorized evaluation stream was available, so the family remains missing.                                                       |
| Operations metrics                              | Contract and dashboard implemented; outcome open | The catalog covers completion/placement, stage latency/cost, retries/timeouts/failover, duplicate prevention, restart recovery, trace/provenance completeness, kill/incident/withdrawal/rollback time, canary regression, and recertification backlog. No live run/effect/incident stream was available, so the family remains missing. |
| Product/locale/audience distributions and tails | Implemented                                      | Every measured metric has min/mean/p50/p90/p95/p99/max, with queryable breakdowns across product, version, surface, modality, locale, audience, source policy, autonomy tier, stage, and domain. The UI exposes p50/p95/p99 and never substitutes an aggregate mean for a missing subgroup.                                             |
| Exact source records and freshness              | Implemented                                      | Every value, tail, count, and breakdown carries an exact records query. The inspector shows record ID, measurement, dimensions, observed time, stream, source record ID, SHA-256, locator, and evidence references. Empty queries state unknown. Responses are private/no-store and malformed sources block the ledger.                 |

## Metric contract and integrity boundary

`@oshun/contracts/operator-metrics` defines 54 required metric keys, four
families, V1–V9 products, autonomy tiers, measurement kinds, required
dimensions, evidence provenance, per-record freshness SLOs, source errors,
filters, pagination, and report/record-page envelopes.

The contract is fail closed:

- unknown measurements require a reason and are never coerced to zero;
- count, ratio, and distribution metrics reject the wrong measurement kind;
- untouched-release metrics require an autonomy tier, and stage latency/cost
  require a stage;
- ratio numerators cannot exceed denominators;
- evidence references and record IDs must be unique;
- invalid, duplicate, and future-dated records produce integrity errors;
- any source error marks every returned metric and breakdown blocked; and
- freshness derives from each record's observed time plus its declared SLO.

The report retains p90 even though the compact ledger displays p50, p95, and
p99. Exact product, locale, audience, and other dimension queries are encoded
into each breakdown's provenance instead of reconstructed by the browser.

## Sources and serving path

The BFF registers authenticated analytics routes at both `/admin` and
`/v1/admin` for reports and exact records. Query parsing rejects unknown keys,
arrays, invalid enum values, mismatched family/metric pairs, invalid null
sentinels, excessive lengths, invalid limits, and cursors outside the filtered
result.

The built-in adapter maps `content-coverage/content-inventory.v1.json` to **389
records** from **77 inventory rows**: five quantity observations per row plus
four manual-content age samples. Its source references preserve inventory row
IDs, repository line spans and SHA-256 values, lineage references, record
SHA-256, and a seven-day freshness SLO.

Optional live adapters are bounded JSONL files configured through
`OSHUN_AUTONOMOUS_CONTENT_METRIC_SOURCES`. Paths must be absolute regular
non-symlink files; source count, file size, line validity, and combined record
count are bounded. An unreadable, oversized, malformed, duplicate, invalid, or
future-dated source blocks the report instead of silently dropping the fault.

The admin Analytics workspace server-fetches the initial report with the
operator token. Browser refresh and drill-down use same-origin authenticated
Next routes, so the credential remains server-side. The ledger provides
product/modality/locale/audience/freshness filters, four family ledgers,
dimension breakdowns, exact-record pagination, explicit integrity state, and
reduced-motion styling.

## Browser-safe telemetry boundary

Analytics page compilation exposed two pre-existing browser-boundary faults in
dependencies reached by the page:

- the Node-only durable file store is no longer exported from the browser-safe
  `@oshun/contracts/run-protocol` barrel and is available at the explicit
  `@oshun/contracts/run-protocol/file-durable-run-store` subpath; and
- the Egbe browser model no longer statically imports the Node OpenTelemetry
  runtime. Real OTLP export is isolated at
  `@oshun/lilith-agent-welfare/otlp-export`, while the UI truthfully labels an
  unexported batch as an in-memory OTel draft with an intended exporter.

The server adapter still passes its real local OTLP/HTTP export test, including
the content-exclusion assertion.

## Verification

```text
Contract Vitest
  1 file, 7/7 tests passed

BFF Vitest
  2 files, 9/9 tests passed

Admin Vitest
  3 files, 19/19 tests passed
  focused post-copy dashboard rerun: 4/4 passed

Lilith welfare/export-boundary Vitest
  1 file, 20/20 tests passed

Playwright Chromium, one worker
  OPS-003 dashboard provenance/filter/tail/accessibility flow: 1/1 passed
  Egbe desktop accessibility/privacy and mobile-overflow flows: 2/2 passed

TypeScript
  libs/contracts/tsconfig.lib.json: passed
  libs/v6/lilith-agent-welfare/tsconfig.lib.json: passed
  @oshun/admin typecheck: passed
  apps/oshun/bff baseline-ratchet typecheck: passed, 0 new app-own errors

ESLint, Prettier, and git diff --check
  focused changed sources: passed
```

The admin Playwright flow uses the repository's axe harness and found no
serious-or-greater violation in the dashboard. It selects V1, proves the
filtered snapshot settled, opens product breakdowns, drills a populated coverage
metric into canonical repository records, and drills an unpopulated quality
metric into an explicit no-record/unknown state.

A Next production build was not run: this host has 15 GiB RAM, below the
repository's mandatory 32 GiB threshold for that expensive gate without explicit
user approval. Targeted Next development compilation of `/analytics`, both
metrics proxy routes, and `/egbe` passed through Playwright. The pre-commit
hook's Nx ownership calculation unexpectedly selected 287 projects and was
stopped under the same hard resource rule after an unrelated Lilith baseline
failure; all task-owning projects were verified directly as listed above.

## Remaining external evidence

The four family outcome boxes remain open until deployed, freshness-bounded
streams report the missing measures for every required V1–V9 scope. In
particular, this repository does not contain authorized live human-work,
evaluation/judge, run/effect, provider-cost, incident, withdrawal, canary, or
recertification observations. It also does not prove production authentication,
deployment, retention, alerting, or ongoing data-source operation. Those facts
cannot be inferred from a typed catalog, a working dashboard, or synthetic test
records.
