Eve library

OPS-003 — Autonomous-content operator metrics and SLO dashboard

Evidence

Revision 1

On this page

Status: queryable dashboard and ingestion contract implemented; complete live V1–V9 coverage, human-work, quality, and operations telemetry remains unproved.

Implementation source commit: 2db7b53ba10b1a48eb21b22dca9d70aa0604a7f0.

Requirement disposition#

OPS-003 requirement Verdict Evidence
Coverage metrics Partially populated; outcome open The canonical repository inventory produces exact required/current/missing/stale/blocked quantity records and age samples. The catalog also requires dependency backlog, invalidation backlog, and unregistered-shipped ratio, but no authorized live streams report those values, so they render as unknown rather than zero.
Human-work metrics Contract and dashboard implemented; outcome open The typed catalog covers untouched release by autonomy tier, active minutes, comments, edit distance, replacements, regenerations, approvals, vetoes, tweaks, takeovers, forks, rollbacks, recurring overrides, appeals, and edit incorporation. No live V1–V9 human-work stream was available, so the family remains missing.
Quality metrics Contract and dashboard implemented; outcome open The catalog covers authored/champion pairwise results, hard-gate escapes, pass@k, pass^k, judge calibration/bias/variance/disagreement, novelty/repetition, canon/style drift, and domain outcomes. No authorized evaluation stream was available, so the family remains missing.
Operations metrics Contract and dashboard implemented; outcome open The catalog covers completion/placement, stage latency/cost, retries/timeouts/failover, duplicate prevention, restart recovery, trace/provenance completeness, kill/incident/withdrawal/rollback time, canary regression, and recertification backlog. No live run/effect/incident stream was available, so the family remains missing.
Product/locale/audience distributions and tails Implemented Every measured metric has min/mean/p50/p90/p95/p99/max, with queryable breakdowns across product, version, surface, modality, locale, audience, source policy, autonomy tier, stage, and domain. The UI exposes p50/p95/p99 and never substitutes an aggregate mean for a missing subgroup.
Exact source records and freshness Implemented Every value, tail, count, and breakdown carries an exact records query. The inspector shows record ID, measurement, dimensions, observed time, stream, source record ID, SHA-256, locator, and evidence references. Empty queries state unknown. Responses are private/no-store and malformed sources block the ledger.

Metric contract and integrity boundary#

@oshun/contracts/operator-metrics defines 54 required metric keys, four families, V1–V9 products, autonomy tiers, measurement kinds, required dimensions, evidence provenance, per-record freshness SLOs, source errors, filters, pagination, and report/record-page envelopes.

The contract is fail closed:

  • unknown measurements require a reason and are never coerced to zero;
  • count, ratio, and distribution metrics reject the wrong measurement kind;
  • untouched-release metrics require an autonomy tier, and stage latency/cost require a stage;
  • ratio numerators cannot exceed denominators;
  • evidence references and record IDs must be unique;
  • invalid, duplicate, and future-dated records produce integrity errors;
  • any source error marks every returned metric and breakdown blocked; and
  • freshness derives from each record's observed time plus its declared SLO.

The report retains p90 even though the compact ledger displays p50, p95, and p99. Exact product, locale, audience, and other dimension queries are encoded into each breakdown's provenance instead of reconstructed by the browser.

Sources and serving path#

The BFF registers authenticated analytics routes at both /admin and /v1/admin for reports and exact records. Query parsing rejects unknown keys, arrays, invalid enum values, mismatched family/metric pairs, invalid null sentinels, excessive lengths, invalid limits, and cursors outside the filtered result.

The built-in adapter maps content-coverage/content-inventory.v1.json to 389 records from 77 inventory rows: five quantity observations per row plus four manual-content age samples. Its source references preserve inventory row IDs, repository line spans and SHA-256 values, lineage references, record SHA-256, and a seven-day freshness SLO.

Optional live adapters are bounded JSONL files configured through OSHUN_AUTONOMOUS_CONTENT_METRIC_SOURCES. Paths must be absolute regular non-symlink files; source count, file size, line validity, and combined record count are bounded. An unreadable, oversized, malformed, duplicate, invalid, or future-dated source blocks the report instead of silently dropping the fault.

The admin Analytics workspace server-fetches the initial report with the operator token. Browser refresh and drill-down use same-origin authenticated Next routes, so the credential remains server-side. The ledger provides product/modality/locale/audience/freshness filters, four family ledgers, dimension breakdowns, exact-record pagination, explicit integrity state, and reduced-motion styling.

Browser-safe telemetry boundary#

Analytics page compilation exposed two pre-existing browser-boundary faults in dependencies reached by the page:

  • the Node-only durable file store is no longer exported from the browser-safe @oshun/contracts/run-protocol barrel and is available at the explicit @oshun/contracts/run-protocol/file-durable-run-store subpath; and
  • the Egbe browser model no longer statically imports the Node OpenTelemetry runtime. Real OTLP export is isolated at @oshun/lilith-agent-welfare/otlp-export, while the UI truthfully labels an unexported batch as an in-memory OTel draft with an intended exporter.

The server adapter still passes its real local OTLP/HTTP export test, including the content-exclusion assertion.

Verification#

text
Contract Vitest
  1 file, 7/7 tests passed

BFF Vitest
  2 files, 9/9 tests passed

Admin Vitest
  3 files, 19/19 tests passed
  focused post-copy dashboard rerun: 4/4 passed

Lilith welfare/export-boundary Vitest
  1 file, 20/20 tests passed

Playwright Chromium, one worker
  OPS-003 dashboard provenance/filter/tail/accessibility flow: 1/1 passed
  Egbe desktop accessibility/privacy and mobile-overflow flows: 2/2 passed

TypeScript
  libs/contracts/tsconfig.lib.json: passed
  libs/v6/lilith-agent-welfare/tsconfig.lib.json: passed
  @oshun/admin typecheck: passed
  apps/oshun/bff baseline-ratchet typecheck: passed, 0 new app-own errors

ESLint, Prettier, and git diff --check
  focused changed sources: passed

The admin Playwright flow uses the repository's axe harness and found no serious-or-greater violation in the dashboard. It selects V1, proves the filtered snapshot settled, opens product breakdowns, drills a populated coverage metric into canonical repository records, and drills an unpopulated quality metric into an explicit no-record/unknown state.

A Next production build was not run: this host has 15 GiB RAM, below the repository's mandatory 32 GiB threshold for that expensive gate without explicit user approval. Targeted Next development compilation of /analytics, both metrics proxy routes, and /egbe passed through Playwright. The pre-commit hook's Nx ownership calculation unexpectedly selected 287 projects and was stopped under the same hard resource rule after an unrelated Lilith baseline failure; all task-owning projects were verified directly as listed above.

Remaining external evidence#

The four family outcome boxes remain open until deployed, freshness-bounded streams report the missing measures for every required V1–V9 scope. In particular, this repository does not contain authorized live human-work, evaluation/judge, run/effect, provider-cost, incident, withdrawal, canary, or recertification observations. It also does not prove production authentication, deployment, retention, alerting, or ongoing data-source operation. Those facts cannot be inferred from a typed catalog, a working dashboard, or synthetic test records.