Eve library

OSHUN V1 QA Audits and Coverage Matrices

Spec

Revision 1

On this page

Status: Canonical Date: 2026-04-26 Owners: OSHUN Quality, OSHUN Reliability, OSHUN Platform Engineering Tracks: V1-QA-001, V1-QA-002, V1-QA-003, V1-QA-004, V1-QA-005, V1-QA-006, V1-QA-007 Spec source: libs/oshun/analytics/src/qa-coverage-matrices.ts

This document records the V1 QA coverage audit and publishes the canonical coverage gap matrix (V1-QA-006) and mandatory automation matrix (V1-QA-007) that downstream CI gates and release readiness reviews compile from.


1. Audit summary (V1-QA-001 – V1-QA-005)#

1.1 Vitest unit/component coverage (V1-QA-001)#

The Oshun-facing apps and libraries publish ~4,000 Vitest test files:

  • apps/oshun/web/src/**: ~2,074 test files (dominant coverage)
  • apps/oshun/admin/src/**: ~133 test files
  • apps/oshun/bff/src/**: ~96 test files
  • libs/oshun/**: ~327 test files
  • libs/contracts/**: ~213 test files
  • Subsystem libraries: libs/isis/** ~854, libs/iris/** ~440, libs/psyche/** ~397, libs/lilith/** ~133, libs/sophia/** ~90.

Conventions: tests colocate with source under __tests__/ directories or .test.ts siblings. Vitest is the canonical runner (vitest.config.ts under each library).

1.2 Playwright web/admin coverage (V1-QA-002)#

The web and admin shells publish ~675 Playwright spec files:

  • apps/oshun/web/e2e/**: ~559 specs (primary web E2E coverage)
  • apps/oshun/admin/e2e/**: ~55 specs
  • testing/e2e/**: ~61 specs (shared infrastructure)

Coverage already includes visual regression snapshots and integrated accessibility checks via E2E support layers.

1.3 Mobile E2E coverage (V1-QA-003)#

The mobile shell publishes ~59 YAML-driven Detox suites under apps/oshun/mobile/e2e/flows/, including core experience, smoke, regression, nisaba-study, shell-redesign signoff, offline-and-retry, and shell-specific flows (notification routing, profile controls, daily passage). Mobile E2E is automated via the YAML configuration matrix; no free-form TS/JS suites are declared.

1.4 Accessibility automation coverage (V1-QA-004)#

Accessibility automation references axe-core, jest-axe, and @axe-core/playwright across ~23 files:

  • Dedicated tests: apps/lilith/web/src/__tests__/accessibility/wcag-compliance.test.ts
  • Shared config: apps/lilith/web/src/accessibility/testing/axe-config.ts
  • Vitest integration: libs/yemaya/ui/src/test/vitest-axe.ts and vitest-axe-extend-expect.ts
  • E2E support layers across oshun/web, oshun/admin, metis, kalika
  • Shared utilities: serwaa/packages/testing/src/a11y/

1.5 Lighthouse / performance coverage (V1-QA-005)#

Lighthouse and lighthouse-ci references span ~49 files:

  • apps/oshun/web/oshun-web-lighthouse.yml (CI workflow)
  • apps/tara/web/lighthouserc.js (Lighthouse config)
  • apps/oshun/web/scripts/syncCriticalRouteLighthouseArtifacts.ts
  • apps/oshun/web/src/__tests__/critical-route-lighthouse-artifacts.test.ts
  • apps/lilith/web/src/__tests__/performance/vitals.test.ts

Quality gates are defined in apps/oshun/quality-gates.json.


2. Coverage gap matrix (V1-QA-006)#

The coverage gap matrix lives in machine-readable form at libs/oshun/analytics/src/qa-coverage-matrices.ts as OSHUN_V1_COVERAGE_GAP_MATRIX. The matrix declares per-journey coverage requirements across 10 test signal kinds: unit, integration, contract, web_e2e, mobile_e2e, accessibility, lighthouse, safety_eval, visual_regression, load. Each cell is one of mandatory, recommended, optional, or not_applicable.

2.1 Critical journeys#

Sixteen critical journeys are tracked:

  1. app_open_to_first_value_action
  2. auth_signin_to_session_ready
  3. onboarding_to_completion
  4. home_continuation_to_domain_launch
  5. cross_domain_recommendation_followthrough
  6. universal_search_to_result_open
  7. activity_reentry_to_resume
  8. assistant_session_with_grounded_answer
  9. assistant_disclosure_indicator_visible
  10. memory_consent_grant_and_revoke
  11. admin_review_decided
  12. admin_moderation_action_taken
  13. admin_incident_published_to_status_page
  14. safety_appeal_filed_to_decided
  15. model_promoted_to_production
  16. media_release_published_with_provenance

2.2 Cross-journey invariants#

  • Every journey declares unit, integration, and contract as mandatory (locked by tests).
  • Customer journeys declare web_e2e, mobile_e2e, and accessibility as mandatory.
  • Assistant journeys additionally declare safety_eval as mandatory.
  • Backend-only journeys (model promotion, media release) declare only unit/integration/contract; UI signals are not applicable.

3. Mandatory automation matrix (V1-QA-007)#

The automation matrix lives at the same module as OSHUN_V1_AUTOMATION_MATRIX. It declares the required test signals per change type:

Change type Required signals
docs_only (none)
shared_typescript unit, integration
web_ui unit, web_e2e, accessibility
pwa unit, web_e2e, lighthouse
mobile_ui unit, mobile_e2e
api_or_bff unit, integration, contract
schema_or_migration unit, integration, contract
search_or_retrieval unit, integration, contract
assistant_or_persona unit, integration, contract, web_e2e, mobile_e2e, safety_eval
media_or_model unit, integration, contract, safety_eval
moderation_or_compliance unit, integration, contract
performance_sensitive unit, load, lighthouse

Tests lock these requirements verbatim and assert that assistant_or_persona mandates safety_eval, web_ui mandates accessibility, pwa and performance_sensitive mandate lighthouse, and docs_only requires no automation.


4. Web Playwright CI coverage inventory (V1-QA-008)#

The web Playwright CI inventory lives in machine-readable form at libs/oshun/analytics/src/web-playwright-ci-coverage.ts as OSHUN_V1_WEB_PLAYWRIGHT_CI_COVERAGE. For every web Playwright suite (playwright.config.ts) in the repo it records:

  • the suite identifier and human-readable name
  • the playwright.config.ts path relative to the workspace root
  • the V1 scope tier (v1_canonical, v1_corrective, or out_of_v1_scope)
  • the GitHub Actions workflow files that run the suite, including the Playwright --project matrix entries where applicable
  • a ciStatus of covered (CI runs the suite) or documented_no_workflow (intentional gap with a written rationale)

4.1 Per-suite coverage table#

Suite ID Config path Scope CI status Workflows
oshun_web apps/oshun/web/playwright.config.ts v1_canonical covered oshun-web-e2e.yml, oshun-ci.yml, oshun-web-visual-regression.yml
oshun_admin_web apps/oshun/admin/playwright.config.ts v1_canonical covered oshun-admin-e2e.yml (added by V1-QA-008)
oshun_cross_domain_e2e testing/e2e/playwright.config.ts v1_canonical covered e2e.yml (yemaya/lilith/veritas/sophia/hathor + lilith API projects), sophia-phase6.yml
tara_web apps/tara/web/playwright.config.ts v1_corrective covered tara-ci.yml (chromium/firefox/webkit projects)
metis_web apps/metis/web/playwright.config.ts v1_corrective documented_no_workflow (none — tracked under V1-MET-CI sub-track)
metis_admin apps/metis/admin/playwright.config.ts v1_corrective documented_no_workflow (none — tracked under V1-MET-CI sub-track)
sophia_workbench apps/sophia/workbench/playwright.config.ts v1_corrective covered sophia-phase6.yml, e2e.yml (sophia-chromium project)
yemaya_studio_web apps/yemaya/studio-web/playwright.config.ts v1_corrective covered e2e.yml (yemaya-chromium / yemaya-firefox / yemaya-webkit projects)
bellona_control_room apps/bellona/control-room/playwright.config.ts out_of_v1_scope documented_no_workflow (none — internal build-orchestration; outside V1 launch path)
psyche_admin apps/psyche/admin/playwright.config.ts out_of_v1_scope documented_no_workflow (none — admin shell scaffolded with zero specs; covered by Psyche's own CI chain for non-Playwright surfaces)
kalika_web apps/kalika/web/playwright.config.ts out_of_v1_scope documented_no_workflow (none — notebook surface outside V1 launch path)
hestia_web apps/hestia/web/playwright.config.ts out_of_v1_scope documented_no_workflow (none — hearth/cooking surface outside V1 launch path)
demeter_web apps/demeter/web/playwright.config.ts out_of_v1_scope documented_no_workflow (none — gardening/community surface outside V1 launch path)
serwaa_admin_dashboard serwaa/services/admin-dashboard/playwright.config.ts out_of_v1_scope documented_no_workflow (none — admin dashboard scaffolded with zero specs)

4.2 Cross-suite invariants (locked by tests)#

  • Every v1_canonical suite has ciStatus === 'covered' and at least one workflow reference.
  • Every documented_no_workflow row has zero workflow references and a non-empty notes rationale.
  • Every referenced workflow file exists on disk with a name: field that matches the inventory's displayName.
  • Every playwright.config.ts referenced exists on disk and ends with playwright.config.ts.
  • Suite ids are unique snake_case strings.
  • The oshun_admin_web row is covered by .github/workflows/oshun-admin-e2e.yml, added by V1-QA-008 to close the canonical V1 admin Playwright CI gap.

4.3 The OSHUN admin CI workflow added by V1-QA-008#

Before V1-QA-008, apps/oshun/admin published ~55 Playwright specs under apps/oshun/admin/e2e/** but had no dedicated CI workflow — the closest coverage was oshun-ci.yml's Run web e2e target step, which only invoked pnpm nx run @oshun/web:e2e (skipping the admin shell). The new .github/workflows/oshun-admin-e2e.yml:

  • Runs on PRs against main, push to main, and a daily 05:30 UTC cron (offset from oshun-web-e2e.yml's 05:00 UTC cron).
  • Path-filters on apps/oshun/admin/**, apps/oshun/bff/**, and libs/oshun/** so it does not retrigger on unrelated changes.
  • Spins up Postgres + Redis as services, builds and starts the OSHUN BFF on port 4010, builds and starts the OSHUN admin Next.js server on port 3020, polls each for readiness, and runs playwright test sharded 1/2 and 2/2.
  • Captures Playwright HTML/JSON reports and BFF/admin server logs as artifacts on failure for triage.

5. Mobile E2E CI coverage inventory (V1-QA-009)#

The mobile E2E CI inventory lives in machine-readable form at libs/oshun/analytics/src/mobile-e2e-ci-coverage.ts as OSHUN_V1_MOBILE_E2E_CI_COVERAGE. For every mobile E2E suite — Maestro YAML or Detox — this module records:

  • the suite identifier and human-readable name
  • the harness (maestro or detox)
  • the suite root path relative to the workspace root
  • the V1 scope tier (v1_canonical, v1_corrective, out_of_v1_scope)
  • the GitHub Actions workflows that run the suite, with the specific Maestro suite filenames or Detox configurations they invoke
  • a ciStatus of covered or documented_no_workflow

5.1 Per-suite coverage table#

Suite ID Suite root Harness Scope CI status Workflows
oshun_mobile apps/oshun/mobile/e2e maestro v1_canonical covered oshun-ci.yml (smoke-suite via @oshun/mobile:e2e)
oshun_admin_mobile apps/oshun/admin-mobile/e2e maestro v1_canonical covered oshun-admin-mobile-e2e.yml (smoke / signoff / regression — added by V1-QA-009)
tara_mobile apps/tara/mobile/e2e maestro v1_corrective covered tara-ci.yml (smoke-suite on iOS + Android)
veritas_mobile apps/veritas/mobile detox v1_corrective covered mobile-e2e.yml (Detox veritas.android.emu.debug, veritas.ios.sim.debug)
cross_mobile_detox testing/mobile/e2e detox v1_corrective covered mobile-e2e.yml (shared Detox harness used by veritas)
lilith_mobile apps/lilith/mobile/e2e detox v1_corrective documented_no_workflow (none — Lilith Expo build CI-stability is tracked outside V1-QA)
iris_mobile apps/iris/mobile detox out_of_v1_scope documented_no_workflow (none — no E2E suite declared)
hestia_mobile apps/hestia/mobile detox out_of_v1_scope documented_no_workflow (none — Jest unit tests only)
demeter_mobile apps/demeter/mobile detox out_of_v1_scope documented_no_workflow (none — Jest unit tests only)
arete_mobile apps/arete/mobile detox out_of_v1_scope documented_no_workflow (none — Jest unit tests only)
nyx_mobile apps/nyx/mobile detox out_of_v1_scope documented_no_workflow (none — scaffold without tests)

5.2 Cross-suite invariants (locked by tests)#

  • Every v1_canonical mobile suite has ciStatus === 'covered' and at least one workflow reference.
  • Every documented_no_workflow row has zero workflow references and a non-empty notes rationale.
  • Every referenced workflow file exists on disk with a name: field that matches the inventory's displayName.
  • Every suite root referenced exists on disk.
  • The oshun_admin_mobile row is covered by .github/workflows/oshun-admin-mobile-e2e.yml, added by V1-QA-009 to close the canonical V1 admin-mobile Maestro CI gap.

5.3 The OSHUN admin-mobile CI workflow added by V1-QA-009#

Before V1-QA-009, apps/oshun/admin-mobile published ~15 Maestro flows under apps/oshun/admin-mobile/e2e/flows/ (smoke, auth/login, step-up redirect, review/urgent-review-actions, incidents/incident-acknowledgement, signoff/incident-signoff, signoff/offline-retry-signoff, signoff/urgent-queue-signoff, regression-suite, signoff-suite, smoke-suite, plus visual-captures) but had no CI workflow at all. The new .github/workflows/oshun-admin-mobile-e2e.yml:

  • Builds the admin-mobile Expo app for Android via expo prebuild and ./gradlew assembleDebug, and boots a Pixel 6 (API 34) emulator via reactivecircus/android-emulator-runner@v2 with KVM acceleration.
  • Installs Maestro CLI, then runs the smoke suite on PR/push, signoff-suite.yaml on main pushes and the daily 09:30 UTC cron, and the full regression-suite.yaml on workflow_dispatch.
  • Path-filters on apps/oshun/admin-mobile/** so unrelated changes do not retrigger the workflow.
  • Captures Maestro JUnit results, .maestro/ artifacts, and any ~/.maestro/screenshots/ for triage.

6. Accessibility and Lighthouse CI coverage inventory (V1-QA-010)#

The accessibility and Lighthouse CI inventory lives in machine-readable form at libs/oshun/analytics/src/accessibility-and-lighthouse-ci-coverage.ts as OSHUN_V1_ACCESSIBILITY_AND_LIGHTHOUSE_CI_COVERAGE. For every distinct a11y or Lighthouse check surface in CI it records:

  • the surface identifier and human-readable name
  • the surface kind: vitest_a11y, playwright_a11y, axe_cli, pa11y_cli, or lighthouse
  • the application or aggregate it covers
  • the source path of the surface (config, helper, or workflow step)
  • the V1 scope tier
  • the GitHub Actions workflow files that run the surface
  • a ciStatus of covered or documented_no_workflow

6.1 Per-surface coverage table (Lighthouse)#

Surface ID App Scope CI status Workflow(s)
oshun_web_lighthouse @oshun/web v1_canonical covered oshun-web-lighthouse.yml, oshun-ci.yml, oshun-web-deploy.yml
oshun_admin_lighthouse @oshun/admin v1_canonical covered oshun-admin-lighthouse.yml (added by V1-QA-010)
tara_web_lighthouse @tara/web v1_corrective covered tara-web-deploy.yml (production deploy only)
yemaya_lighthouse @yemaya/studio-web v1_corrective covered accessibility.yml (root lighthouse.config.js)

6.2 Per-surface coverage table (axe-core / pa11y CLI)#

Surface ID App Kind Scope CI status Workflow
yemaya_axe_cli @yemaya/studio-web axe_cli v1_corrective covered accessibility.yml
lilith_axe_cli @lilith/web axe_cli v1_corrective covered accessibility.yml
yemaya_pa11y_cli @yemaya/studio-web pa11y_cli v1_corrective covered accessibility.yml
lilith_pa11y_cli @lilith/web pa11y_cli v1_corrective covered accessibility.yml

6.3 Per-surface coverage table (Playwright a11y helpers)#

Surface ID App Scope CI status Workflow(s)
oshun_web_playwright_a11y @oshun/web v1_canonical covered oshun-web-e2e.yml, oshun-ci.yml
oshun_admin_playwright_a11y @oshun/admin v1_canonical covered oshun-admin-e2e.yml
lilith_web_playwright_a11y @lilith/web v1_corrective covered e2e.yml
metis_web_playwright_a11y @metis/web v1_corrective documented_no_workflow (none — metis web Playwright suite has no CI workflow yet)
kalika_web_playwright_a11y @kalika/web out_of_v1_scope documented_no_workflow (none — out of V1 scope)

6.4 Per-surface coverage table (Vitest a11y unit/component)#

Surface ID App Scope CI status Workflow(s)
oshun_web_vitest_a11y @oshun/web v1_canonical covered oshun-ci.yml, ci.yml
tara_ui_vitest_a11y @tara/ui v1_corrective covered tara-ci.yml
yemaya_ui_vitest_a11y @yemaya/ui v1_corrective covered ci.yml
lilith_web_vitest_a11y @lilith/web v1_corrective covered ci.yml

6.5 Cross-surface invariants (locked by tests)#

  • Every v1_canonical surface has ciStatus === 'covered' and at least one workflow reference.
  • Every documented_no_workflow row has zero workflow references and a non-empty notes rationale.
  • Every referenced workflow file exists on disk with a name: field that matches the inventory's displayName.
  • The five canonical OSHUN admin Lighthouse routes (admin-home, admin-review-queue, admin-moderation-queue, admin-incidents, admin-support-cases) are declared in apps/oshun/admin/lighthouse-routes.json.
  • oshun_admin_lighthouse is covered by .github/workflows/oshun-admin-lighthouse.yml, added by V1-QA-010.
  • Both @oshun/web and @oshun/admin have at least one v1_canonical Lighthouse surface in the inventory.

6.6 The OSHUN admin Lighthouse CI added by V1-QA-010#

Before V1-QA-010, apps/oshun/admin had no Lighthouse coverage. The customer shell ran @oshun/web:lighthouse via oshun-web-lighthouse.yml and oshun-ci.yml's lighthouse-web job, but the operator-facing admin shell had no Lighthouse path of any kind. V1-QA-010 added:

  • apps/oshun/admin/lighthouse-routes.json — five canonical admin routes (admin-home, admin-review-queue, admin-moderation-queue, admin-incidents, admin-support-cases).
  • apps/oshun/admin/lighthouserc.cjs — Lighthouse CI config asserting 0.85+ performance, 0.95+ accessibility (matching the customer shell bar), 0.95+ best-practices, 0.85+ SEO, with FCP <= 2200ms, LCP <= 3000ms, TBT <= 300ms, CLS <= 0.1, and TTI <= 5500ms.
  • lighthouse:* scripts in apps/oshun/admin/package.json and lighthouse, lighthouse:collect, lighthouse:assert Nx targets in apps/oshun/admin/project.json so the Lighthouse run is invokable through pnpm nx run @oshun/admin:lighthouse.
  • .github/workflows/oshun-admin-lighthouse.yml — runs the Lighthouse budgets on PR/push touching apps/oshun/admin/** or libs/oshun/**, and on demand via workflow_dispatch.

7. Governance#

Adding or changing a journey, change type, web Playwright suite entry, mobile E2E suite entry, or accessibility/Lighthouse surface entry requires:

  1. updating qa-coverage-matrices.ts, web-playwright-ci-coverage.ts, mobile-e2e-ci-coverage.ts, or accessibility-and-lighthouse-ci-coverage.ts with the new or changed entry
  2. updating tests (lock-style tests fail closed if entries drift)
  3. updating this document if the change is structurally significant
  4. signoff from OSHUN Quality plus the affected domain leads

8. Change log#

  • 2026-04-26 — Initial canonical V1 QA audit summary, coverage gap matrix, and mandatory automation matrix published alongside libs/oshun/analytics/src/qa-coverage-matrices.ts (V1-QA-001/002/003/004/005/006/007).
  • 2026-04-26 — Published OSHUN_V1_WEB_PLAYWRIGHT_CI_COVERAGE inventory in libs/oshun/analytics/src/web-playwright-ci-coverage.ts and added .github/workflows/oshun-admin-e2e.yml to close the OSHUN admin Playwright CI gap (V1-QA-008).
  • 2026-04-26 — Published OSHUN_V1_MOBILE_E2E_CI_COVERAGE inventory in libs/oshun/analytics/src/mobile-e2e-ci-coverage.ts and added .github/workflows/oshun-admin-mobile-e2e.yml to close the OSHUN admin-mobile Maestro CI gap (V1-QA-009).
  • 2026-04-26 — Published OSHUN_V1_ACCESSIBILITY_AND_LIGHTHOUSE_CI_COVERAGE inventory in libs/oshun/analytics/src/accessibility-and-lighthouse-ci-coverage.ts and added apps/oshun/admin/lighthouserc.cjs, apps/oshun/admin/lighthouse-routes.json, Nx + package.json Lighthouse targets, and .github/workflows/oshun-admin-lighthouse.yml to close the OSHUN admin Lighthouse gap (V1-QA-010).