Oshun Performance Baselines

Authoritative per-domain and cross-domain performance targets, maintained alongside the benchmarks that verify them. Update this file in the same PR as any change to the related SLO threshold in code.

Canonical benchmark sources (do not duplicate budgets in docs alone — a budget that lives only here and not in a test is not enforced):

0. Measurement provenance#

These numbers are test-enforced budgets. They originate from (a) product latency requirements for each interactive loop, (b) a floor analysis of the minimum work each endpoint must do (bcrypt, pgvector search, LLM call, etc.), and (c) measured percentiles in the reference environment below. Every cell in the §2 table is the target.p{50,95,99} constant of the corresponding scenario in testing/performance/benchmarks/cross-domain-latency.test.ts — changing a number here without changing the test is a type-checked impossibility, so the doc cannot drift from the gate.

Where the gate runs:

Harness integrity is itself tested: the percentile math, empty-sample handling, and report shape are asserted by the cross-domain latency harness describe block at the bottom of the benchmark file (tests run in every CI pass regardless of whether endpoints are reachable). On CI's default path, cross-domain budget assertions run against in-cluster staging endpoints when BENCH_*_URL env vars are set, and with BENCH_ALLOW_SKIP=1 otherwise — skipped scenarios are explicit and loggable, never silent passes.

How to refresh this table after a measurement run:

1. Stand up the reference stack (docker compose -f docker/docker-compose.dev.yml up -d
   • the capability-domain services).
2. Run the harness with endpoint env vars set:
   bashCopy

   BENCH_ISIS_GENERATION_URL=http://localhost:8120 \
   BENCH_SOPHIA_URL=http://localhost:8130 \
   BENCH_BELLONA_URL=http://localhost:8140 \
   BENCH_HATHOR_URL=http://localhost:8150 \
   pnpm test:performance:cross-domain
   

3. For each scenario whose measured p50/p95/p99 drifts more than 10% from the current target constant, update the constant in cross-domain-latency.test.ts and the matching row in §2 together in one PR.
4. Bump "Last reviewed" to the date of the run.

1. Public REST endpoint SLOs#

Source of truth: testing/performance/benchmarks/api-benchmark.test.ts.

Per-domain authenticated read/write budgets (search, entity resolution, world create, session create, build fetch, etc.) are documented under §3 and gated by the domain-specific harnesses listed there. Authentication round-trip is not covered here — Oshun has no monolithic /api/auth/login; each domain validates JWTs issued upstream and the token-issuance path is gated by tests/security/cross-domain-auth.security.test.ts.

Review cadence: quarterly. The benchmark asserts against these exact numbers.

2. Cross-domain call latency (Phase 13.6.3)#

Source of truth: testing/performance/benchmarks/cross-domain-latency.test.ts.

All hops are measured as a single outbound HTTP call from the consuming domain's BFF/worker to the provider domain's public API, not through the end-to-end gateway. Measurements include full response body consumption.

Budget rationale per scenario:

• P50 reflects a warm-cache happy path with both domains co-located in the same Kubernetes cluster. If P50 regresses, suspect a code-path or serialisation change in the consumer or provider.
• P95 reflects one in-flight dependency slow-down (database failover, Redis eviction, cold model load). Sustained P95 regressions indicate a capacity or autoscaler issue.
• P99 allows for one transient retry. A P99 regression that is not matched by a P95 regression points at transient cross-AZ network or noisy-neighbour effects rather than a code defect.

Running the benchmark against a live stack#

export BENCH_ISIS_GENERATION_URL=http://localhost:3400
export BENCH_SOPHIA_URL=http://localhost:3500
export BENCH_BELLONA_URL=http://localhost:3006
cd testing/performance/benchmarks
npx vitest run --config vitest.config.ts cross-domain-latency.test.ts

By default, when a BENCH_*_URL env var is missing the benchmark falls back to the local docker-compose port for that domain. If that service is not running the scenario hard-fails rather than passing silently. PR jobs that cannot stand up every domain can opt into skipping by exporting BENCH_ALLOW_SKIP=1, which converts "no endpoint reachable" into a logged skip for that scenario only. Main-branch CI must not set that flag.

3. Per-domain baselines#

Lilith (consciousness / conversation BFF)#

Isis (generative factory)#

Sophia (knowledge / RAG)#

Baseline rationale: Sophia's retrieval path hits pgvector + a reranker; the 800 ms P95 for /v1/ask assumes the reranker is on and reads 3 documents.

Hathor (worldbuilding)#

World exports are large (the P95 budget allows 2 s to accommodate packaging a small-to-medium world). Quest generation includes an LLM call.

Bellona (engine bridge)#

Yemaya (creative studio)#

Aphrodite (talent / live stream)#

Source: testing/performance/k6/aphrodite/README.md and per-scenario scripts.

Veritas (news / citations)#

Source: testing/performance/k6/veritas/config.js.

Iris (AI assistant framework)#

Source: docs/domains/iris/extras/performance-validation-report.md. Summary:

4. Regression policy#

A PR is considered to regress performance if any of the following occur:

1. A benchmark in the table above fails its assertion and the assertion is not concurrently relaxed in the same PR with a written justification.
2. The cross-domain latency test reports an unexpected SKIP for a scenario whose env vars CI has always provided.
3. A new cross-domain hop is added without a corresponding row in §2.

Regressions are blockers. The PR author is expected to either fix the regression, attribute it to a concurrent migration with a tracking issue, or revise the budget with sign-off from the affected domain owners.

5. Update procedure#

When changing a budget:

1. Update the SLO threshold in the canonical benchmark source (§2 lists the files).
2. Update the corresponding row in this document.
3. Re-run the benchmark locally to confirm the new threshold passes against current production numbers. Attach the percentile output to the PR.
4. Reviewers from every affected domain sign off.