Status: Canonical Date: 2026-04-26 Owners: OSHUN Quality, OSHUN Reliability,
OSHUN Platform Engineering Tracks: V1-QA-001, V1-QA-002, V1-QA-003,
V1-QA-004, V1-QA-005, V1-QA-006, V1-QA-007 Spec source:
libs/oshun/analytics/src/qa-coverage-matrices.ts
This document records the V1 QA coverage audit and publishes the canonical coverage gap matrix (V1-QA-006) and mandatory automation matrix (V1-QA-007) that downstream CI gates and release readiness reviews compile from.
1. Audit summary (V1-QA-001 – V1-QA-005)#
1.1 Vitest unit/component coverage (V1-QA-001)#
The Oshun-facing apps and libraries publish ~4,000 Vitest test files:
apps/oshun/web/src/**: ~2,074 test files (dominant coverage)apps/oshun/admin/src/**: ~133 test filesapps/oshun/bff/src/**: ~96 test fileslibs/oshun/**: ~327 test fileslibs/contracts/**: ~213 test files- Subsystem libraries:
libs/isis/**~854,libs/iris/**~440,libs/psyche/**~397,libs/lilith/**~133,libs/sophia/**~90.
Conventions: tests colocate with source under __tests__/ directories or
.test.ts siblings. Vitest is the canonical runner (vitest.config.ts under
each library).
1.2 Playwright web/admin coverage (V1-QA-002)#
The web and admin shells publish ~675 Playwright spec files:
apps/oshun/web/e2e/**: ~559 specs (primary web E2E coverage)apps/oshun/admin/e2e/**: ~55 specstesting/e2e/**: ~61 specs (shared infrastructure)
Coverage already includes visual regression snapshots and integrated accessibility checks via E2E support layers.
1.3 Mobile E2E coverage (V1-QA-003)#
The mobile shell publishes ~59 YAML-driven Detox suites under
apps/oshun/mobile/e2e/flows/, including core experience, smoke, regression,
nisaba-study, shell-redesign signoff, offline-and-retry, and shell-specific
flows (notification routing, profile controls, daily passage). Mobile E2E is
automated via the YAML configuration matrix; no free-form TS/JS suites are
declared.
1.4 Accessibility automation coverage (V1-QA-004)#
Accessibility automation references axe-core, jest-axe, and @axe-core/playwright across ~23 files:
- Dedicated tests:
apps/lilith/web/src/__tests__/accessibility/wcag-compliance.test.ts - Shared config:
apps/lilith/web/src/accessibility/testing/axe-config.ts - Vitest integration:
libs/yemaya/ui/src/test/vitest-axe.tsandvitest-axe-extend-expect.ts - E2E support layers across oshun/web, oshun/admin, metis, kalika
- Shared utilities:
serwaa/packages/testing/src/a11y/
1.5 Lighthouse / performance coverage (V1-QA-005)#
Lighthouse and lighthouse-ci references span ~49 files:
apps/oshun/web/oshun-web-lighthouse.yml(CI workflow)apps/tara/web/lighthouserc.js(Lighthouse config)apps/oshun/web/scripts/syncCriticalRouteLighthouseArtifacts.tsapps/oshun/web/src/__tests__/critical-route-lighthouse-artifacts.test.tsapps/lilith/web/src/__tests__/performance/vitals.test.ts
Quality gates are defined in apps/oshun/quality-gates.json.
2. Coverage gap matrix (V1-QA-006)#
The coverage gap matrix lives in machine-readable form at
libs/oshun/analytics/src/qa-coverage-matrices.ts as
OSHUN_V1_COVERAGE_GAP_MATRIX. The matrix declares per-journey coverage
requirements across 10 test signal kinds: unit, integration, contract,
web_e2e, mobile_e2e, accessibility, lighthouse, safety_eval,
visual_regression, load. Each cell is one of mandatory, recommended,
optional, or not_applicable.
2.1 Critical journeys#
Sixteen critical journeys are tracked:
app_open_to_first_value_actionauth_signin_to_session_readyonboarding_to_completionhome_continuation_to_domain_launchcross_domain_recommendation_followthroughuniversal_search_to_result_openactivity_reentry_to_resumeassistant_session_with_grounded_answerassistant_disclosure_indicator_visiblememory_consent_grant_and_revokeadmin_review_decidedadmin_moderation_action_takenadmin_incident_published_to_status_pagesafety_appeal_filed_to_decidedmodel_promoted_to_productionmedia_release_published_with_provenance
2.2 Cross-journey invariants#
- Every journey declares
unit,integration, andcontractas mandatory (locked by tests). - Customer journeys declare
web_e2e,mobile_e2e, andaccessibilityas mandatory. - Assistant journeys additionally declare
safety_evalas mandatory. - Backend-only journeys (model promotion, media release) declare only unit/integration/contract; UI signals are not applicable.
3. Mandatory automation matrix (V1-QA-007)#
The automation matrix lives at the same module as OSHUN_V1_AUTOMATION_MATRIX.
It declares the required test signals per change type:
| Change type | Required signals |
|---|---|
docs_only |
(none) |
shared_typescript |
unit, integration |
web_ui |
unit, web_e2e, accessibility |
pwa |
unit, web_e2e, lighthouse |
mobile_ui |
unit, mobile_e2e |
api_or_bff |
unit, integration, contract |
schema_or_migration |
unit, integration, contract |
search_or_retrieval |
unit, integration, contract |
assistant_or_persona |
unit, integration, contract, web_e2e, mobile_e2e, safety_eval |
media_or_model |
unit, integration, contract, safety_eval |
moderation_or_compliance |
unit, integration, contract |
performance_sensitive |
unit, load, lighthouse |
Tests lock these requirements verbatim and assert that assistant_or_persona
mandates safety_eval, web_ui mandates accessibility, pwa and
performance_sensitive mandate lighthouse, and docs_only requires no
automation.
4. Web Playwright CI coverage inventory (V1-QA-008)#
The web Playwright CI inventory lives in machine-readable form at
libs/oshun/analytics/src/web-playwright-ci-coverage.ts as
OSHUN_V1_WEB_PLAYWRIGHT_CI_COVERAGE. For every web Playwright suite
(playwright.config.ts) in the repo it records:
- the suite identifier and human-readable name
- the
playwright.config.tspath relative to the workspace root - the V1 scope tier (
v1_canonical,v1_corrective, orout_of_v1_scope) - the GitHub Actions workflow files that run the suite, including the Playwright
--projectmatrix entries where applicable - a
ciStatusofcovered(CI runs the suite) ordocumented_no_workflow(intentional gap with a written rationale)
4.1 Per-suite coverage table#
| Suite ID | Config path | Scope | CI status | Workflows |
|---|---|---|---|---|
oshun_web |
apps/oshun/web/playwright.config.ts |
v1_canonical |
covered | oshun-web-e2e.yml, oshun-ci.yml, oshun-web-visual-regression.yml |
oshun_admin_web |
apps/oshun/admin/playwright.config.ts |
v1_canonical |
covered | oshun-admin-e2e.yml (added by V1-QA-008) |
oshun_cross_domain_e2e |
testing/e2e/playwright.config.ts |
v1_canonical |
covered | e2e.yml (yemaya/lilith/veritas/sophia/hathor + lilith API projects), sophia-phase6.yml |
tara_web |
apps/tara/web/playwright.config.ts |
v1_corrective |
covered | tara-ci.yml (chromium/firefox/webkit projects) |
metis_web |
apps/metis/web/playwright.config.ts |
v1_corrective |
documented_no_workflow | (none — tracked under V1-MET-CI sub-track) |
metis_admin |
apps/metis/admin/playwright.config.ts |
v1_corrective |
documented_no_workflow | (none — tracked under V1-MET-CI sub-track) |
sophia_workbench |
apps/sophia/workbench/playwright.config.ts |
v1_corrective |
covered | sophia-phase6.yml, e2e.yml (sophia-chromium project) |
yemaya_studio_web |
apps/yemaya/studio-web/playwright.config.ts |
v1_corrective |
covered | e2e.yml (yemaya-chromium / yemaya-firefox / yemaya-webkit projects) |
bellona_control_room |
apps/bellona/control-room/playwright.config.ts |
out_of_v1_scope |
documented_no_workflow | (none — internal build-orchestration; outside V1 launch path) |
psyche_admin |
apps/psyche/admin/playwright.config.ts |
out_of_v1_scope |
documented_no_workflow | (none — admin shell scaffolded with zero specs; covered by Psyche's own CI chain for non-Playwright surfaces) |
kalika_web |
apps/kalika/web/playwright.config.ts |
out_of_v1_scope |
documented_no_workflow | (none — notebook surface outside V1 launch path) |
hestia_web |
apps/hestia/web/playwright.config.ts |
out_of_v1_scope |
documented_no_workflow | (none — hearth/cooking surface outside V1 launch path) |
demeter_web |
apps/demeter/web/playwright.config.ts |
out_of_v1_scope |
documented_no_workflow | (none — gardening/community surface outside V1 launch path) |
serwaa_admin_dashboard |
serwaa/services/admin-dashboard/playwright.config.ts |
out_of_v1_scope |
documented_no_workflow | (none — admin dashboard scaffolded with zero specs) |
4.2 Cross-suite invariants (locked by tests)#
- Every
v1_canonicalsuite hasciStatus === 'covered'and at least one workflow reference. - Every
documented_no_workflowrow has zero workflow references and a non-emptynotesrationale. - Every referenced workflow file exists on disk with a
name:field that matches the inventory'sdisplayName. - Every
playwright.config.tsreferenced exists on disk and ends withplaywright.config.ts. - Suite ids are unique snake_case strings.
- The
oshun_admin_webrow is covered by.github/workflows/oshun-admin-e2e.yml, added by V1-QA-008 to close the canonical V1 admin Playwright CI gap.
4.3 The OSHUN admin CI workflow added by V1-QA-008#
Before V1-QA-008, apps/oshun/admin published ~55 Playwright specs under
apps/oshun/admin/e2e/** but had no dedicated CI workflow — the closest
coverage was oshun-ci.yml's Run web e2e target step, which only invoked
pnpm nx run @oshun/web:e2e (skipping the admin shell). The new
.github/workflows/oshun-admin-e2e.yml:
- Runs on PRs against
main, push tomain, and a daily 05:30 UTC cron (offset fromoshun-web-e2e.yml's 05:00 UTC cron). - Path-filters on
apps/oshun/admin/**,apps/oshun/bff/**, andlibs/oshun/**so it does not retrigger on unrelated changes. - Spins up Postgres + Redis as services, builds and starts the OSHUN BFF on port
4010, builds and starts the OSHUN admin Next.js server on port 3020, polls
each for readiness, and runs
playwright testsharded 1/2 and 2/2. - Captures Playwright HTML/JSON reports and BFF/admin server logs as artifacts on failure for triage.
5. Mobile E2E CI coverage inventory (V1-QA-009)#
The mobile E2E CI inventory lives in machine-readable form at
libs/oshun/analytics/src/mobile-e2e-ci-coverage.ts as
OSHUN_V1_MOBILE_E2E_CI_COVERAGE. For every mobile E2E suite — Maestro YAML or
Detox — this module records:
- the suite identifier and human-readable name
- the harness (
maestroordetox) - the suite root path relative to the workspace root
- the V1 scope tier (
v1_canonical,v1_corrective,out_of_v1_scope) - the GitHub Actions workflows that run the suite, with the specific Maestro suite filenames or Detox configurations they invoke
- a
ciStatusofcoveredordocumented_no_workflow
5.1 Per-suite coverage table#
| Suite ID | Suite root | Harness | Scope | CI status | Workflows |
|---|---|---|---|---|---|
oshun_mobile |
apps/oshun/mobile/e2e |
maestro | v1_canonical |
covered | oshun-ci.yml (smoke-suite via @oshun/mobile:e2e) |
oshun_admin_mobile |
apps/oshun/admin-mobile/e2e |
maestro | v1_canonical |
covered | oshun-admin-mobile-e2e.yml (smoke / signoff / regression — added by V1-QA-009) |
tara_mobile |
apps/tara/mobile/e2e |
maestro | v1_corrective |
covered | tara-ci.yml (smoke-suite on iOS + Android) |
veritas_mobile |
apps/veritas/mobile |
detox | v1_corrective |
covered | mobile-e2e.yml (Detox veritas.android.emu.debug, veritas.ios.sim.debug) |
cross_mobile_detox |
testing/mobile/e2e |
detox | v1_corrective |
covered | mobile-e2e.yml (shared Detox harness used by veritas) |
lilith_mobile |
apps/lilith/mobile/e2e |
detox | v1_corrective |
documented_no_workflow | (none — Lilith Expo build CI-stability is tracked outside V1-QA) |
iris_mobile |
apps/iris/mobile |
detox | out_of_v1_scope |
documented_no_workflow | (none — no E2E suite declared) |
hestia_mobile |
apps/hestia/mobile |
detox | out_of_v1_scope |
documented_no_workflow | (none — Jest unit tests only) |
demeter_mobile |
apps/demeter/mobile |
detox | out_of_v1_scope |
documented_no_workflow | (none — Jest unit tests only) |
arete_mobile |
apps/arete/mobile |
detox | out_of_v1_scope |
documented_no_workflow | (none — Jest unit tests only) |
nyx_mobile |
apps/nyx/mobile |
detox | out_of_v1_scope |
documented_no_workflow | (none — scaffold without tests) |
5.2 Cross-suite invariants (locked by tests)#
- Every
v1_canonicalmobile suite hasciStatus === 'covered'and at least one workflow reference. - Every
documented_no_workflowrow has zero workflow references and a non-emptynotesrationale. - Every referenced workflow file exists on disk with a
name:field that matches the inventory'sdisplayName. - Every suite root referenced exists on disk.
- The
oshun_admin_mobilerow is covered by.github/workflows/oshun-admin-mobile-e2e.yml, added by V1-QA-009 to close the canonical V1 admin-mobile Maestro CI gap.
5.3 The OSHUN admin-mobile CI workflow added by V1-QA-009#
Before V1-QA-009, apps/oshun/admin-mobile published ~15 Maestro flows under
apps/oshun/admin-mobile/e2e/flows/ (smoke, auth/login, step-up redirect,
review/urgent-review-actions, incidents/incident-acknowledgement,
signoff/incident-signoff, signoff/offline-retry-signoff,
signoff/urgent-queue-signoff, regression-suite, signoff-suite, smoke-suite, plus
visual-captures) but had no CI workflow at all. The new
.github/workflows/oshun-admin-mobile-e2e.yml:
- Builds the admin-mobile Expo app for Android via
expo prebuildand./gradlew assembleDebug, and boots a Pixel 6 (API 34) emulator viareactivecircus/android-emulator-runner@v2with KVM acceleration. - Installs Maestro CLI, then runs the smoke suite on PR/push,
signoff-suite.yamlon main pushes and the daily 09:30 UTC cron, and the fullregression-suite.yamlon workflow_dispatch. - Path-filters on
apps/oshun/admin-mobile/**so unrelated changes do not retrigger the workflow. - Captures Maestro JUnit results,
.maestro/artifacts, and any~/.maestro/screenshots/for triage.
6. Accessibility and Lighthouse CI coverage inventory (V1-QA-010)#
The accessibility and Lighthouse CI inventory lives in machine-readable form at
libs/oshun/analytics/src/accessibility-and-lighthouse-ci-coverage.ts as
OSHUN_V1_ACCESSIBILITY_AND_LIGHTHOUSE_CI_COVERAGE. For every distinct a11y or
Lighthouse check surface in CI it records:
- the surface identifier and human-readable name
- the surface kind:
vitest_a11y,playwright_a11y,axe_cli,pa11y_cli, orlighthouse - the application or aggregate it covers
- the source path of the surface (config, helper, or workflow step)
- the V1 scope tier
- the GitHub Actions workflow files that run the surface
- a
ciStatusofcoveredordocumented_no_workflow
6.1 Per-surface coverage table (Lighthouse)#
| Surface ID | App | Scope | CI status | Workflow(s) |
|---|---|---|---|---|
oshun_web_lighthouse |
@oshun/web |
v1_canonical |
covered | oshun-web-lighthouse.yml, oshun-ci.yml, oshun-web-deploy.yml |
oshun_admin_lighthouse |
@oshun/admin |
v1_canonical |
covered | oshun-admin-lighthouse.yml (added by V1-QA-010) |
tara_web_lighthouse |
@tara/web |
v1_corrective |
covered | tara-web-deploy.yml (production deploy only) |
yemaya_lighthouse |
@yemaya/studio-web |
v1_corrective |
covered | accessibility.yml (root lighthouse.config.js) |
6.2 Per-surface coverage table (axe-core / pa11y CLI)#
| Surface ID | App | Kind | Scope | CI status | Workflow |
|---|---|---|---|---|---|
yemaya_axe_cli |
@yemaya/studio-web |
axe_cli |
v1_corrective |
covered | accessibility.yml |
lilith_axe_cli |
@lilith/web |
axe_cli |
v1_corrective |
covered | accessibility.yml |
yemaya_pa11y_cli |
@yemaya/studio-web |
pa11y_cli |
v1_corrective |
covered | accessibility.yml |
lilith_pa11y_cli |
@lilith/web |
pa11y_cli |
v1_corrective |
covered | accessibility.yml |
6.3 Per-surface coverage table (Playwright a11y helpers)#
| Surface ID | App | Scope | CI status | Workflow(s) |
|---|---|---|---|---|
oshun_web_playwright_a11y |
@oshun/web |
v1_canonical |
covered | oshun-web-e2e.yml, oshun-ci.yml |
oshun_admin_playwright_a11y |
@oshun/admin |
v1_canonical |
covered | oshun-admin-e2e.yml |
lilith_web_playwright_a11y |
@lilith/web |
v1_corrective |
covered | e2e.yml |
metis_web_playwright_a11y |
@metis/web |
v1_corrective |
documented_no_workflow | (none — metis web Playwright suite has no CI workflow yet) |
kalika_web_playwright_a11y |
@kalika/web |
out_of_v1_scope |
documented_no_workflow | (none — out of V1 scope) |
6.4 Per-surface coverage table (Vitest a11y unit/component)#
| Surface ID | App | Scope | CI status | Workflow(s) |
|---|---|---|---|---|
oshun_web_vitest_a11y |
@oshun/web |
v1_canonical |
covered | oshun-ci.yml, ci.yml |
tara_ui_vitest_a11y |
@tara/ui |
v1_corrective |
covered | tara-ci.yml |
yemaya_ui_vitest_a11y |
@yemaya/ui |
v1_corrective |
covered | ci.yml |
lilith_web_vitest_a11y |
@lilith/web |
v1_corrective |
covered | ci.yml |
6.5 Cross-surface invariants (locked by tests)#
- Every
v1_canonicalsurface hasciStatus === 'covered'and at least one workflow reference. - Every
documented_no_workflowrow has zero workflow references and a non-emptynotesrationale. - Every referenced workflow file exists on disk with a
name:field that matches the inventory'sdisplayName. - The five canonical OSHUN admin Lighthouse routes (admin-home,
admin-review-queue, admin-moderation-queue, admin-incidents,
admin-support-cases) are declared in
apps/oshun/admin/lighthouse-routes.json. oshun_admin_lighthouseis covered by.github/workflows/oshun-admin-lighthouse.yml, added by V1-QA-010.- Both
@oshun/weband@oshun/adminhave at least one v1_canonical Lighthouse surface in the inventory.
6.6 The OSHUN admin Lighthouse CI added by V1-QA-010#
Before V1-QA-010, apps/oshun/admin had no Lighthouse coverage. The customer
shell ran @oshun/web:lighthouse via oshun-web-lighthouse.yml and
oshun-ci.yml's lighthouse-web job, but the operator-facing admin shell had
no Lighthouse path of any kind. V1-QA-010 added:
apps/oshun/admin/lighthouse-routes.json— five canonical admin routes (admin-home, admin-review-queue, admin-moderation-queue, admin-incidents, admin-support-cases).apps/oshun/admin/lighthouserc.cjs— Lighthouse CI config asserting 0.85+ performance, 0.95+ accessibility (matching the customer shell bar), 0.95+ best-practices, 0.85+ SEO, with FCP <= 2200ms, LCP <= 3000ms, TBT <= 300ms, CLS <= 0.1, and TTI <= 5500ms.lighthouse:*scripts inapps/oshun/admin/package.jsonandlighthouse,lighthouse:collect,lighthouse:assertNx targets inapps/oshun/admin/project.jsonso the Lighthouse run is invokable throughpnpm nx run @oshun/admin:lighthouse..github/workflows/oshun-admin-lighthouse.yml— runs the Lighthouse budgets on PR/push touchingapps/oshun/admin/**orlibs/oshun/**, and on demand via workflow_dispatch.
7. Governance#
Adding or changing a journey, change type, web Playwright suite entry, mobile E2E suite entry, or accessibility/Lighthouse surface entry requires:
- updating
qa-coverage-matrices.ts,web-playwright-ci-coverage.ts,mobile-e2e-ci-coverage.ts, oraccessibility-and-lighthouse-ci-coverage.tswith the new or changed entry - updating tests (lock-style tests fail closed if entries drift)
- updating this document if the change is structurally significant
- signoff from OSHUN Quality plus the affected domain leads
8. Change log#
- 2026-04-26 — Initial canonical V1 QA audit summary, coverage gap matrix, and
mandatory automation matrix published alongside
libs/oshun/analytics/src/qa-coverage-matrices.ts(V1-QA-001/002/003/004/005/006/007). - 2026-04-26 — Published
OSHUN_V1_WEB_PLAYWRIGHT_CI_COVERAGEinventory inlibs/oshun/analytics/src/web-playwright-ci-coverage.tsand added.github/workflows/oshun-admin-e2e.ymlto close the OSHUN admin Playwright CI gap (V1-QA-008). - 2026-04-26 — Published
OSHUN_V1_MOBILE_E2E_CI_COVERAGEinventory inlibs/oshun/analytics/src/mobile-e2e-ci-coverage.tsand added.github/workflows/oshun-admin-mobile-e2e.ymlto close the OSHUN admin-mobile Maestro CI gap (V1-QA-009). - 2026-04-26 — Published
OSHUN_V1_ACCESSIBILITY_AND_LIGHTHOUSE_CI_COVERAGEinventory inlibs/oshun/analytics/src/accessibility-and-lighthouse-ci-coverage.tsand addedapps/oshun/admin/lighthouserc.cjs,apps/oshun/admin/lighthouse-routes.json, Nx + package.json Lighthouse targets, and.github/workflows/oshun-admin-lighthouse.ymlto close the OSHUN admin Lighthouse gap (V1-QA-010).