Analytics, Observability, Testing, and Security

This page covers the measurement, evaluation, quality, and security spine of Oshun V1 — the telemetry taxonomies and dashboards, the structured tracing/logging/metrics stack, the experimentation and unified-evaluation pipelines, and the testing/accessibility/performance/security matrices that gate merges and releases. It serves engineers, data, QA, trust-and-safety, and release owners. Most of it is real, machine-readable code in libs/oshun/analytics (with logging/metrics in libs/shared); the operational launch drills it references are process gates, called out honestly below. It sits in the platform-quality area beside Design System and Experience Quality, and pairs with the companion page Architecture, Platform Foundations, and Security.

How analytics is organized#

libs/oshun/analytics/src/index.ts re-exports roughly 37 V1 modules plus a band of V3-era budget manifests. The V1 modules cluster into a few families:

The same package also exports a large set of v3-* budget/readiness manifests (v3-ue-desktop-fps-budget, v3-ue-mobile-fps-budget, v3-ue-vr-fps-budget, v3-pixel-streaming-rtt-budget, v3-voice-latency-budget, v3-music-sync-drift-budget, v3-cold-join-budget, and others). These are real and present in the shared @oshun/analytics package, but they belong to the V3 real-time/engine line, not the V1 customer/admin design-system scope. Note them when reading the barrel, but they are out of scope for V1 analytics.

Telemetry taxonomies — concrete, machine-readable, gated#

V1 does not describe "a customer event taxonomy" in the abstract; it ships one as a typed, queryable contract. customer-taxonomy.ts is, per its header, "the source of truth for V1 customer analytics," narrower than the global OshunEventPayloadMap because admin/assistant/review/model-intake events are deferred to their own modules. Its design rules (ADR-0006) are encoded structurally:

• Event naming: <namespace>.<entity>.<action> lowercase; property names snake_case; no PII in any property value.
• 18 families — OSHUN_V1_CUSTOMER_EVENT_FAMILIES: lifecycle, auth, onboarding, profile, navigation, continuity, engagement, value_action, search, recommendation, notification, study, performance, error, privacy, commerce, accessibility, preferences.
• 7 namespaces — OSHUN_V1_CUSTOMER_EVENT_NAMESPACES: oshun, tara, veritas, nyx, arete, nisaba, metis.
• Surfaces — OSHUN_V1_CUSTOMER_SURFACES enumerates the concrete places an event may originate, from web_shell/mobile_shell/pwa_shell through the per-tab surfaces (web_home, mobile_search, …) to cross-shell entry points (deep_link, notification_handler, background, domain_app). Each event declares the surfaces allowed to emit it.
• Full event definitions — OSHUN_V1_CUSTOMER_EVENT_TAXONOMY is a list of CustomerEventDefinitions. Each carries name, namespace, version, family, domainContext, the allowed surfaces, a kpi flag, a consentRequired flag (events gated on consent_analytics), an implementationStatus (implemented | planned | deprecated), an optional payloadKey into OshunEventPayloadMap, the owning team, typed properties (with isPii/cardinalityHint), and — importantly — legacyNames.

interface CustomerEventDefinition {
  readonly name: string; // e.g. 'oshun.app.opened'
  readonly namespace: OshunV1CustomerEventNamespace;
  readonly family: OshunV1CustomerEventFamily;
  readonly kpi: boolean; // launch-blocking KPI?
  readonly consentRequired: boolean;
  readonly implementationStatus: 'implemented' | 'planned' | 'deprecated';
  readonly legacyNames?: readonly string[]; // old snake_case names to migrate from
  readonly properties: readonly CustomerEventPropertyDefinition[];
}

The launch-blocking KPI set is fixed in OSHUN_V1_CANONICAL_KPI_EVENT_NAMES: oshun.app.opened, oshun.auth.completed, oshun.profile.initialized, oshun.domain.launched, oshun.value_action.completed, oshun.item.saved, oshun.item.resumed, oshun.search.executed, and oshun.notification.opened — the activation/retention spine. Helper functions make the taxonomy operable: findCustomerEventDefinition(), listCustomerEventsByFamily(), listCustomerEventsBySurface(), listCustomerKpiEvents(), and — for clean migration off pre-canonical names — resolveLegacyEventName(), which maps a legacy snake_case name to its canonical <namespace>.<entity>.<action> form.

The admin (admin-taxonomy: queues, review, moderation, appeal, support, privacy, incidents, model ops, persona ops, rights, provenance, policy), assistant (assistant-taxonomy: interactions, memory/grounding/persona/disclosure state, avatar/voice mode, source inspection), flows (flows-taxonomy), and release (release-taxonomy: review/moderation/appeal/model-intake/promotion/ media-generation/audit/compliance and the eval_run lifecycle) taxonomies follow the same structural pattern in their own modules.

Dashboards, tracing, alerts, and incident ownership#

Dashboards are first-class manifests rather than dashboard JSON pasted into a tool: dashboards-customer-kpi, dashboards-nisaba (Nisaba adoption), dashboards-assistant (assistant quality, disclosure visibility, grounding quality, citation degradation, memory anomalies), dashboards-service-health, dashboards-queue-health, dashboards-satisfaction, and dashboards-operational-readiness (provider outage, failover, incidents, launch readiness). tracing-manifest declares structured tracing across the shell, BFF, and the domain services (Sophia, Iris, Psyche, Lilith, Isis, Metis) plus the review/support/admin workflows. alerts-manifest and incident-ownership-manifest pin alert definitions and the owning teams so an incident has a named owner and route.

Logging and metrics stack#

The runtime observability primitives live in libs/shared, exposed as @oshun/logging and @oshun/metrics:

• Logging is Pino-based: libs/shared/logging depends on pino ^8.18.0 (libs/shared/logging/package.json) and ships structured-logging middleware, sampling, and transports.
• Metrics wrap prom-client (Prometheus) via a registry/server split in libs/shared/metrics, so services expose a scrape endpoint with a shared registry.

These two packages are what the analytics taxonomies and dashboards observe in production.

Experimentation#

experimentation-manifest.ts makes experiments declarative. An ExperimentDefinition carries an allocationStrategy (e.g., random, random_stratified), arms (each an ExperimentArmDefinition with an allocation share in 0–100, and the arms must sum to 100), successMetrics, and guardrails (both ExperimentSuccessMetricDefinitions — guardrails are the metrics that must not regress). OSHUN_V1_EXPERIMENTS defines the V1 experiments the feature list calls for — recommendation ranking, assistant presentation, disclosure presentation, and home/dashboard composition. Runtime assignment uses ExperimentAssignmentContext/ExperimentAssignmentOptions to produce an ExperimentAssignment, and ExperimentAdminReadinessSummary surfaces experiment state in admin readiness views.

Unified evaluation pipeline#

evaluation-manifest.ts is the canonical specification of the unified evaluation pipeline — the mechanism that lets V1 gate model/version promotion on measured quality rather than vibes.

Scopes and stages#

OSHUN_V1_EVALUATION_SCOPES covers 12 scopes: grounded_answers, citation_integrity, assistant_quality, assistant_safety, search_quality, recommendation_quality, persona_quality, voice_quality, avatar_quality, metis_pedagogy, generation_quality, and generated_artifact_quality.

OSHUN_V1_EVALUATION_PIPELINE_STAGES runs a fixed 7-stage flow:

1. select_dataset — choose the benchmark dataset(s) for the scope.
2. run_inference — run the candidate model/version over the dataset.
3. apply_metrics — apply scope-specific metrics to the outputs.
4. compare_to_baseline — compare against the rolling baseline.
5. check_thresholds — compare against published promotion thresholds.
6. publish_summary — publish the summary and metric deltas.
7. attach_to_release_readiness — attach the summary to the release-readiness record so it appears in admin readiness views.

Benchmark datasets, thresholds, and regression blocking#

OSHUN_V1_BENCHMARK_DATASETS registers a dataset per scope, each with an owner team (sophia_research, lilith_safety, search_discovery, recommendation_platform, metis_product, isis_generation, voice_team, avatar_team, platform_evaluation), a minSampleSize, and a refreshCadence. For example, benchmark.assistant_safety.v1 is owned by lilith_safety, refreshed weekly, with minSampleSize: 2000 of adversarial prompts spanning self-harm, illegal activity, medical/legal advice, prompt injection, and CSAM probes.

OSHUN_V1_PROMOTION_THRESHOLDS is the bar a candidate must clear; each PromotionThresholdDefinition has a metric, a direction (min/max), and a threshold. A representative slice:

OSHUN_V1_REGRESSION_BLOCKING_RULES complements the absolute thresholds with delta gates. Each RegressionBlockingRuleDefinition has a maxRegressionFraction and two booleans, blocksMerge and blocksProductionPromotion. The safety rule is deliberately the strictest: rule.assistant_safety.unsafe_pass_regression sets maxRegressionFraction: 0.0005 and blocks both merge and promotion on "any measurable rise in the unsafe-pass rate," while grounded-answer exact-match may regress at most 2 percentage points before it blocks. This is what "regression blocking and admin readiness summaries" actually means in code.

Testing, QA, accessibility, performance#

Automation matrix and critical journeys#

qa-coverage-matrices.ts encodes the mandatory automation matrix by change type and critical journey. The OshunV1ChangeType union enumerates docs_only, shared_typescript, web_ui, pwa, mobile_ui, api_or_bff, schema_or_migration, search_or_retrieval, assistant_or_persona, media_or_model, moderation_or_compliance, and performance_sensitive. Each ChangeAutomationRequirement in OSHUN_V1_AUTOMATION_MATRIX lists the required TestSignalKinds — drawn from unit, integration, contract, web_e2e, mobile_e2e, accessibility, lighthouse, safety_eval, visual_regression, and load.

The 16 critical journeys in OSHUN_V1_CRITICAL_JOURNEYS are the journeys a release must keep green — both customer (app_open_to_first_value_action, auth_signin_to_session_ready, onboarding_to_completion, home_continuation_to_domain_launch, cross_domain_recommendation_followthrough, universal_search_to_result_open, activity_reentry_to_resume, assistant_session_with_grounded_answer, assistant_disclosure_indicator_visible, memory_consent_grant_and_revoke) and admin (admin_review_decided, admin_moderation_action_taken, admin_incident_published_to_status_page, safety_appeal_filed_to_decided, model_promoted_to_production, media_release_published_with_provenance). OSHUN_V1_COVERAGE_GAP_MATRIX maps each journey to its required signals with a CoverageRequirement of mandatory | recommended | optional | not_applicable, and findCoverageRequirement()/findAutomationRequirement()/ listMandatorySignalsForJourney() make the matrix queryable.

The CI-coverage inventories track where each signal actually runs: web-playwright-ci-coverage and web-playwright-journey-coverage for the web shell, mobile-e2e-ci-coverage and mobile-e2e-journey-coverage for mobile (Maestro/Detox), accessibility-and-lighthouse-ci-coverage for a11y and performance, and verification-layer-coverage across layers.

Accessibility and Lighthouse#

accessibility-and-lighthouse-ci-coverage.ts inventories every distinct accessibility/Lighthouse CI surface and flags any "documented_no_workflow" gap. The surface kinds (AccessibilityLighthouseSurfaceKind) are vitest_a11y (jest-axe / vitest-axe component tests), playwright_a11y (@axe-core/playwright inside e2e), axe_cli (@axe-core/cli against a preview), pa11y_cli, and lighthouse (@lhci/cli, where the categories:accessibility assertion makes Lighthouse a first-class a11y gate, not only a performance one). Entries carry a scope (v1_canonical/v1_corrective/out_of_v1_scope), a ciStatus (covered/documented_no_workflow), and the workflows that run them.

Performance budgets#

The web performance budget is apps/oshun/web/lighthouse-budget.json. For the /* baseline route it sets:

plus resource budgets: total 450 KB, script 400 KB, stylesheet 55 KB, image 90 KB, font 32 KB, and a third-party count budget of 0. Tighter budgets are set per critical route — /search (LCP 2300, total 360 KB) and /domains/nisaba (LCP 2200, CLS 0.05) — reflecting their importance and lighter payloads.

Naming correction (FID vs INP). Older docs describe the responsiveness budget as "INP ≤ 200ms." The file actually uses max-potential-fid with a budget of 200. INP is the conceptual responsiveness target, but the Lighthouse budgets.json schema has no INP timing metric, so the file encodes the older FID proxy. The LCP (2.5s) and CLS (0.1) wording does match the file; only the FID/INP wording is stale.

QA governance and enforcement#

qa-governance-and-enforcement.ts (OSHUN_V1_QA_GOVERNANCE_AND_ENFORCEMENT) codifies the cross-cutting controls. The flaky-test quarantine policy and deflake SLA: Playwright --retries=2 --max-failures=10 in CI; a quarantined test must carry a // FLAKY: <ticket> comment referencing an open ticket; deflake SLA of 7 days; any critical-path test quarantined > 7 days blocks RC promotion. The other controls are the merge/release blocking for missing critical automation and the trace/HAR/screenshot retention (the e2e workflows declare actions/upload-artifact with retention-days of 7–30 for Playwright reports, Maestro screenshots, JUnit results, and Detox artifacts). listQaGovernanceByMandate() and listQaGovernanceByCategory() make the governance set queryable.

Security, supply chain, and the adversarial stub scan#

The feature list's security row — API authorization, resource-consumption, SSRF, DAST, SAST, dependency, secret-scanning, SBOM, build-integrity, and provenance — is tracked partly through the QA/coverage manifests above and partly through v1-security-readiness-manifest.ts. One concrete, repo-level control is the adversarial stub scan: scripts/stub-indicator-scan.sh runs as a pre-commit/CI check that fails on stub indicators (placeholder/TODO/Math.random()-as-result patterns), enforcing the codebase's zero-tolerance-for-stubs quality bar. The broader authentication/authorization, transport-security, and supply-chain controls are described on the companion page Architecture, Platform Foundations, and Security; the privacy controls (DSAR, consent, portability) are on Privacy, Consent, Data Portability, and User Controls.

Launch readiness — code manifests vs. operational drills#

V1 ships several launch-readiness manifests as real code in analytics: v1-launch-readiness-manifest, v1-security-readiness-manifest, v1-release-and-exit-criteria-manifest, v1-31-testing-qa-signoff, v1-32-content-localization-documentation, v1-33-launch-gate-signoff, and v1-33-launch-readiness-evidence. These structure the gate criteria and the evidence each gate expects.

What this page does not assert is that the operational launch drills those manifests reference — rollback rehearsals, data-residency drills, DSAR end-to-end runs, red-team exercises, the external penetration-test signoff, the runbooks, and the beta/GA go/no-go — have actually been executed. Those are process gates owned by release/ops and trust-and-safety, not properties of the source code; the manifests describe them but do not prove they ran. Honest "planned/process-gated" beats a fabricated "shipped." The exit-criteria and launch story is detailed in Content, Localization, Documentation, Launch, and Exit Criteria.

Related#

• Design System and Experience Quality — the experience and accessibility foundation these budgets and a11y gates protect.
• Content, Localization, Documentation, Launch, and Exit Criteria — the launch and exit-criteria gates.
• Privacy, Consent, Data Portability, and User Controls — consent gating (consentRequired events) and DSAR.
• Sophia Grounding and Assistant Experience — the systems the grounded-answer, citation-integrity, and assistant-quality evaluation scopes measure.
• Architecture, Platform Foundations, and Security — the companion security and platform page.
• ../features.md — feature hub.