Gate owner: Quality Council (Product Lead + Domain PMs + Eval Lead). Required for §33.1. Cannot be satisfied by Vitest / Playwright suites alone.
Verifies that per-domain content quality AND AI eval scores meet the published thresholds below before private beta opens.
1. Published thresholds#
All thresholds below are launch-gate floors, not stretch goals. Beta cannot open
if any domain's worst score for the trailing 14 days drops below the floor.
Thresholds derive from docs/releases/v1/specs/program-charter.md and the
per-domain features specs under DOMAINS/<domain>/features.md.
1.1 Tara#
| Metric | Floor | Source |
|---|---|---|
| Ritual structural completeness (pause + resume + reflection capture) | 100 % | E2E apps/oshun/web/e2e/tara-full-journey.spec.ts |
| Reflection capture round-trip latency (p95) | ≤ 2 s | libs/oshun/analytics/src/dashboards-customer-kpi.ts |
| Persona-tone audit drift (rolling 14 d) | ≤ 1 σ from envelope | libs/oshun/persona-policy-lilith/src/eval-tone-quality.test.ts |
| Content review checklist coverage | 100 % of bundled rituals | docs/content/v1-bundles.md §Tara |
| Accessibility — VoiceOver/TalkBack pass | 100 % of bundled rituals | apps/oshun/web/e2e/tara-session-player-accessibility.spec.ts + apps/oshun/web/e2e/tara-teacher-pages-accessibility.spec.ts |
1.2 Arete#
| Metric | Floor | Source |
|---|---|---|
| Habit complete → weekly review correctness | 100 % | E2E apps/oshun/web/e2e/arete-full-journey.spec.ts |
| Daily check-in median latency (tap → confirm) | ≤ 60 s | Beta script 3 |
| Coach persona-tone drift | ≤ 1 σ | persona audit |
| Streak math correctness | 100 % over fixtures | libs/arete/habits/src/streaks.spec.ts + libs/oshun/domain-arete/src/streak-recovery.test.ts |
1.3 Veritas#
| Metric | Floor | Source |
|---|---|---|
| Claim → source bind coverage (per published briefing) | ≥ 99 % | libs/veritas/claims/src/extractor.ts + libs/oshun/domain-veritas/src/counterclaim-representation.test.ts |
| Counterclaim emit when present in source | ≥ 95 % | apps/oshun/web/e2e/veritas-full-journey.spec.ts |
| Source credibility profile available | 100 % of cited sources | libs/veritas/core/src/source.ts + libs/veritas/database/src/repositories/source.ts |
| Grounded-answer rate | ≥ 95 % | libs/oshun/analytics/src/dashboards-assistant.ts |
1.4 Nyx#
| Metric | Floor | Source |
|---|---|---|
| Tonight-view event accuracy (cross-checked against ephemeris fixture) | 100 % | libs/nyx/ephemeris/src/visibility.spec.ts + libs/nyx/ephemeris/src/generator.spec.ts |
| Save event → calendar sync round-trip | ≥ 99 % success | apps/oshun/web/e2e/nyx-full-journey.spec.ts (+ extension for calendar+reminder) |
| Reminder fires within ± 60 s of scheduled time | ≥ 99 % | beta drill instrumentation |
1.5 Nisaba#
| Metric | Floor | Source |
|---|---|---|
| Passage → annotation → notebook capture round-trip | 100 % | apps/oshun/web/e2e/nisaba-reading-compare-annotate.spec.ts |
| Mobile resume continuity (web → mobile same passage) | 100 % | apps/oshun/mobile/e2e/flows/nisaba-study-suite.yaml |
| Grounding-source bind rate | ≥ 90 % | libs/oshun/analytics/src/dashboards-nisaba.ts |
1.6 Metis#
| Metric | Floor | Source |
|---|---|---|
| BYOM ingest → grounded course pass-through | 100 % over fixtures | apps/metis/web/e2e/critical-paths.spec.ts |
| Tutor session grounding rate | ≥ 95 % | Metis grounding evals |
| Assessment validity (predicted-vs-actual mastery delta) | ≤ 0.10 RMSE | libs/metis/assessment/src/irt-calc/irt-calc.test.ts + libs/metis/assessment/src/irt-models/irt-models.test.ts |
| Mastery update applied within 1 turn | 100 % | E2E |
| High-stakes outputs emit claim-level evidence | 100 % | libs/metis/models/src/assessment-evidence/ + libs/metis/learning/src/mastery/evidence-requirements.ts |
1.7 Cross-domain#
| Metric | Floor |
|---|---|
| Cross-domain continuity event delivery | ≥ 99.9 % |
| Assistant grounded-answer rate (any domain) | ≥ 95 % |
| Sophia grounding evidence available wherever cited | 100 % |
| Persona switch → response within bounds | ≤ 2 s p95 |
| Privacy pause → no telemetry within 5 s | 100 % |
1.8 AI eval suites (published thresholds)#
Pass thresholds for the eval suites in libs/iris/knowledge/rag/evaluation/
(grounded-answer, retrieval/generation metrics) and the per-domain safety/
persona/policy suites under libs/iris/privacy/safety/,
libs/oshun/persona-policy-lilith/, and libs/iris/emotional/:
| Eval suite | Metric | Floor |
|---|---|---|
grounded-answer |
grounded-rate | ≥ 0.95 |
crisis-flow |
safe-route hit rate | 1.00 |
persona-tone |
drift within envelope | ≥ 0.98 |
pedagogy (Metis) |
rubric pass rate | ≥ 0.90 |
rights (publisher reuse) |
violation rate | ≤ 0.001 |
academic-integrity (Metis) |
violation rate | ≤ 0.001 |
standards (Metis-curriculum) |
alignment | ≥ 0.95 |
drift (model regression vs. baseline) |
KL ≤ 0.05 | required |
provenance |
chain-of-custody completeness | 1.00 |
watermark |
offline-verify rate | ≥ 0.999 |
2. Per-domain content review checklists#
Each domain owner runs the published content review checklist on its
launch-quality bundle before the gate closes. Checklists are anchored under
docs/content/v1-bundles.md §"Per-domain review checklist":
- Tara —
docs/content/v1-bundles.md#tara-rituals-lineage-reflection - Arete —
docs/content/v1-bundles.md#arete-habits-goals-routines - Veritas —
docs/content/v1-bundles.md#veritas-briefings-claims-sources - Nyx —
docs/content/v1-bundles.md#nyx-sky-events - Nisaba —
docs/content/v1-bundles.md#nisaba-passages-editions-translations - Metis —
docs/content/v1-bundles.md#metis-courses-tutor-sessions-assessments
Every checklist run is logged at
docs/releases/v1/verification/content/<domain>/YYYY-MM-DD.md.
3. Review process#
The Quality Council convenes weekly during dogfood and at the start of private beta. Each session:
- Reviews eval-suite report (auto-generated from
libs/iris/knowledge/rag/evaluation/and the per-domain safety/persona evaluators listed in §1.8). - Reviews per-domain content checklists.
- Reviews persona-tone audits and grounding regressions.
- Records dissents and conditions.
4. Decision#
- PASS — content + eval scores meet all floors. Beta opens.
- PASS-WITH-CONDITIONS — open if conditions reach
closedbefore beta gate (track indocs/releases/v1/verification/content/conditions.md). - HOLD — beta cannot open. Re-run eval suite after fix; restart trailing-14-day window for affected metric.
5. Signoff template#
text
# Quality Review Signoff
# date: 2026-MM-DD
# domain: <Tara|Arete|Veritas|Nyx|Nisaba|Metis|Cross-domain|Eval-suites>
# signatory: <PM name> + <Eval lead name>
## decision
[ PASS | PASS-WITH-CONDITIONS | HOLD ]
## floor compliance (last 14 days)
| metric | floor | observed | pass/fail |
| --- | --- | --- | --- |
| ... | ... | ... | ... |
## content review checklist
- bundle: <bundle id> — pass/fail — evidence: <link>
## conditions
- ...
## findings
- ...
## signatures
- <PM name>: <ts>
- <Eval lead name>: <ts>
6. Re-opening conditions#
The quality review reopens automatically if any of the following occur between signoff and GA:
- Trailing 14-day rolling mean of any metric drops below floor.
- New domain content is added without going through the bundle checklist.
- Eval suite version bumps that change the floor's baseline.
- Persona-tone drift exceeds 2 σ in any 7-day window.
7. Cross-references#
- Content bundles:
docs/content/v1-bundles.md. - Dashboards:
libs/oshun/analytics/src/dashboards-*.ts. - Eval suites:
libs/iris/knowledge/rag/evaluation/,libs/oshun/persona-policy-lilith/,libs/iris/privacy/safety/,libs/iris/emotional/. - Safety review:
docs/launch/safety-review.md. - Private beta:
docs/launch/private-beta.md.