Observability, Performance, Eval Gates & Launch

V7 (Mawu) is a creator republic: a hardened UE5.5 client, a Rust realm fleet, and a platform plane that owns identity, the real-money economy, and child safety — composed so that untrusted, community-hosted realm logic runs only inside a capability-sandboxed WASM runtime behind an inviolable trust boundary. For an operations page that means the central question — is this build good enough to ship? — has no single answer, only a composition: a sandbox that must contain a hostile module, a mesh that must hand off authority across a dying node with zero loss, a netcode that must stay playable under packet loss, a payout formula that must settle to the cent, a CSAM hash that must match before any serve, and a launch gate that refuses to go green while any of those is unproven. This page is the operational spine that ties them together — observability and live operations, performance budgets, build/cook/patch/cert, testing with adversarial eval gates and golden replays, and launch readiness — and it is deliberately honest about which parts are running code, which are machine-checked policy, and which are still spec inherited from the prior versions. The posture V7 shares with V2 and V3 is that it invents none of the foundations: it composes the shared @oshun/metrics/@oshun/tracing libraries, reuses the V6 realtime gateway and V1 observability stack, and proves the V7-specific seams with real Rust/TS evals and regenerable evidence. The section hub is ../V7_ARCHITECTURE.md.

What ships, honestly#

The adversarial eval surface is real and large. V7/ADVERSARIAL_EVAL_GATES.md enumerates 29 merge gates, each owning a seeded hostile fixture, an explicit pass bar, and a concrete CI command — cargo test against the apps/v7/libs/v7 workspaces or pnpm exec vitest against the TS services. The owning functions exist on disk and are substantial (the moremi-realm-server crate alone is >14,000 lines): run_moremi_ixchel_sandbox_escape_eval, run_danu_mesh_handoff_eval, run_moremi_golden_replay_ci_gate, run_default_realm_netcode_loss_gate, run_malware_corpus_eval, run_trust_boundary_eval, and the rest. .github/workflows/v7-ci.yml runs an inventory verifier first, then each gate as a named CI step.

The launch gate is real, machine-checked, and honestly negative. V7/release/launch-readiness-review.json reads decision: "not-launch-ready"; all eight signoffs carry status: "fixture-passed" with the verbatim evidenceBasis: "in-repo adversarial fixture; not a production measurement", and scripts/v7/verify-launch-readiness.mjs enforces a no-fabrication invariant — it cannot declare launch-ready while any signoff rests on fixture evidence (verify-launch-readiness.mjs:378–388). The staged-rollout record (V7/release/staged-rollout.json) has all five stages re-marked planned / pending, never executed.

Three honest qualifications carry through the page. First, V7 does not yet wire @oshun/metrics or @oshun/tracing into its Rust services — there is no Prometheus /metrics family and no OTLP exporter the way V3's world server has one. What each service exposes today is a GET /health capability contract (apps/v7/service_contract.rs). Second, the architecture's metric vocabulary (realm population, mesh-node utilization, authority-handoff latency, sandbox-fuel-overrun rate, scan-queue latency, economy inflation) is specified policy plus in-eval telemetry structs, not a runtime gauge family. Third, there is no V7/ue/Build/ cook-profile tree — unlike V3's fifteen checked-in profiles — because V7 reuses the V5/V6 build pipeline and its UE client is a ~2.2k-LOC procedural skeleton, not a content-complete cook.

Observability and live operations#

The live service surface that actually exists#

Every V7 Rust service — Moremi, Danu, Nephthys, Sekhmet, Hera, the Mawu gateway — includes the shared apps/v7/service_contract.rs via a #[path] module, declares a const DESCRIPTOR: ServiceDescriptor (name, owner, port, purpose, capability list), and binds it in run_service() through service_contract::run_health_server(&DESCRIPTOR). The handler answers GET /health with a JSON capability document — {"service","owner","version":"v7","status":"ready","port","purpose","capabilities":[…]} — and 404s everything else (service_contract.rs:88). Ports are resolvable per deployment via V7_<NAME>_PORT or a fleet-wide V7_PORT_OFFSET (runtime_port, :50). This is an honest seam: a readiness/capability probe a load balancer or the Abundantia browser can poll to flag a degraded realm — not a metrics endpoint. The main.rs of each service is a one-line delegation to run_service().

What the spec names, and what is code today#

The architecture's Observability and Live Operations section names the live-ops metric set — realm population and retention, mesh-node utilization and split/merge frequency, authority-handoff latency and failure count, sandbox-fuel-overrun rate, scan-queue latency, payout distribution, per-realm economy inflation, per-client bandwidth — and states that tracing/logging/SLOs reuse the V1 observability stack and the gateway (reused from V6's egbe-realtime-gateway) exports per-realm health. That is the design; the in-repo reality is narrower and should be read as such. The shared libraries themselves are real and platform-wide (libs/shared/metrics/src/ — registry, helpers, presets; libs/shared/tracing/src/ — tracer, W3C propagation, decorators, middleware), but no V7 service imports them yet (a grep for @oshun/metrics/@oshun/tracing across apps/v7 and libs/v7 returns zero hits).

What V7 does encode today is domain telemetry inside its eval reports, not a scrape surface. The §29 cost/fidelity levers emit DanuDegradationTelemetry { degraded, … } records when a hot region sheds fidelity (mesh-tier reduction, AI-creation fallback, Crucible reduced coverage), and the eval asserts that all three degradation paths each emit an observable degraded-state record (danu-mesh-cluster/src/lib.rs:2663+, asserted at :3528). Sekhmet carries SekhmetTelemetryEvidenceInput for anti-cheat evidence attachment. Creator analytics (Roblox/UEFN-class DAU/retention/revenue dashboards) is named in the spec but unimplemented. The honest summary: live ops is a /health contract plus in-eval telemetry structs today; the full metric/trace spine is inherited-but-unwired.

Performance budgets#

V7's budgets live in two honestly-different places. The numeric policy is stated inline in the architecture and is not a runtime budget object: a 60 fps / 16.7 ms client frame on the reference tier with a 30 fps floor on minimum spec (sustained in a meshed realm of 1000+ via interest-managed culling), authority handoff ≤250 ms p99 with no reconnect, netcode playable at 5% loss / graceful at 20%, per-client downstream 64–256 kbit/s in open world, Pheme ≤40 audible streams per listener after AoI culling with p95 mouth-to-ear ≤250 ms at 1000-inhabitant scale, and distribution targets (one-click Collection install p95 ≤120 s for 5 GB on 100 Mbit/s; Sekhmet scan-queue p95 ≤15 min upload-to-verdict).

A meaningful subset is encoded as Rust constants that eval gates assert — which is stronger than a documented number because a regression fails a test:

There are no typed @oshun/analytics V7 budget manifests of the kind V3 ships (v3-*-budget.ts + V3/validation/ evidence); the per-tier frame-time validation V3 has is, for V7, still inline policy plus these constant-backed gates. The honest framing of a budget V2/V3 use applies here too — it is a contract CI and the eval suite hold the build to, sharpened where a constant is load-bearing.

Build, cook, patch, and cert#

V7 continues on the UE5.5 LTS / Maya client path and reuses the V5/V6 build, cook, and patch pipeline rather than re-inventing it; correspondingly there is no V7-specific Build/CookProfiles/ tree. The client project V7/ue/Mawu.uproject pins EngineAssociation 5.5, sets DisableEnginePluginsByDefault: true (minimal attack/dependency surface), and enables exactly five plugins — EnhancedInput, CommonUI, ModelViewViewModel, GeometryScripting, GeometryProcessing — across six runtime/dev modules (MawuCore, MawuRealm, MawuBuilder, MawuVoice, MawuUI, and the DeveloperTool MawuTests). Engine config lives in V7/ue/Config/ (DefaultEngine.ini, DefaultGame.ini, DefaultInput.ini) and a checked-in realm lock at Config/RealmLocks/dedicated-smoke.lock.json that the automation suite boots a dedicated realm from. The client has been built and run on the on-box UE5.5.4 source build (as the ueagent user; the editor refuses root), exercising the V7.Mawu.* automation specs green.

The distribution mechanics are content-addressed and platform-signed by design (per the Ixchel and Abundantia architecture): every mod artifact's id is the hash of its content plus its full dependency closure (the Nix/lock-file model), giving tamper-evidence, dedup, atomic rollback, and an AOT .cwasm cache key; vetted plugins are AOT-compiled to .cwasm at upload and production runs with the JIT disabled; client patches are platform-signed, never operator-distributed; and console cert constraints gate which sandbox tiers a console realm may grant. Honestly labeled: these are spec and reused-pipeline, not V7-specific checked-in cook profiles or a packaged patch artifact. The launch review records the consequence plainly — the UE client is a skeleton, and no realm fleet, packaged build, or store submission exists yet.

Testing, eval gates, and golden replays#

UE automation: small, real, procedural#

V7/ue/Source/MawuTests holds 12 V7.Mawu.* Automation Specs across two suites — far smaller than V2's 441 or V3's ~34, and the page says so. They are real, not scaffolds: MawuRealmGeometryTests.cpp constructs lock files whose primitives carry an OperationKind, applies them through AMawuComposedRealmActor, and asserts the resulting UDynamicMesh triangle/vertex counts and watertightness for voxel and CSG solids — proving Geometry Script ops build geometry rather than classify it. MawuRealmBootstrapTests.cpp parses the smoke realm lock, composes its six render primitives, boots a dedicated realm through AMawuGameMode, validates the exact digest-bound 23-primitive flagship region and a tampered accessibility rejection, drives the builder→realm OperationKind mapping, and checks the Pheme voice mix (a matching radio channel routes voice outside proximity range). This is V7's pure-C++ surface; the heavy logic lives in Rust.

The 29 adversarial gates#

The eval suite is the load-bearing test layer. Each gate in V7/ADVERSARIAL_EVAL_GATES.md "must run in CI, exercise the seeded adversarial fixture, and fail against the seeded adversarial case if the owning implementation is tampered to allow it." Representative bars: sandbox-escape (0 escapes across hostile capability/fuel/memory/host-tamper modules), mesh-handoff (200 entities × 10,000 crossings, zero loss/dup/reconnect, with a mid-handoff node death injected), single-writer (only the authoritative owner advances the ledger), netcode-loss (playable @5%, graceful @20%), anticheat (precision/recall vs a 95% bar on authoritative signals), csam-hash (100% synthetic known-hash match before serve), payout-formula (correct to the cent), trust-boundary/identity-leak (no platform secret or identity reachable from realm code or web view), and the nine gates added 2026-06-12 (webview-escape, event-tamper, auth-tamper, capability-deny, ban-evasion, overload, character-deletion-limit, generation-policy, accessibility). .github/workflows/v7-ci.yml runs a verify-inventory job, then 23 Rust gate steps plus the Mawu Studio safety gates (generation-policy, accessibility) and the Abundantia economy gates (payout-formula, payout-eligibility, fraud-graph, currency-firewall).

Golden replays: the determinism guard#

Determinism is the cross-cutting invariant (rollback prediction, replay-based crash recovery, replayable plugins), so the golden-replay gate is its CI guard. run_moremi_golden_replay_ci_gate (moremi-realm-server/src/lib.rs:9307) runs two halves: the golden gate replays a recorded input log tick-by-tick through moremi_state_hash and asserts the recorded inputs reproduce the exact authoritative state hash, and the nondeterminism probe clones the fixture, flips the seed (config.seed ^= 0xDEAD_BEEF), and requires the hash to diverge — so the gate fails if unrecorded entropy ever leaks in. The test golden_replay_ci_gate_fails_unrecorded_entropy_probe pins that property; the same capture machinery feeds incident-replay clips and photo-mode exports with content hashes signed by the authoritative node.

Three verifiers back the gate machinery: verify-adversarial-eval-gates.mjs (the 29-gate inventory wiring), verify-staged-rollout.mjs, and verify-launch-readiness.mjs, which additionally greps the source for the eval functions and their key assertions (e.g. assert_eq!(report.escaped_fixtures, 0);, assert_eq!(report.client_reconnects, 0);) so a hand-edited evidence file cannot pass while the code that would produce it is absent.

Launch readiness#

V7's go/no-go aggregates everything above, and its defining property is that it is honestly not ready and structurally cannot lie about it. V7/release/launch-readiness-review.json reads decision: "not-launch-ready". All eight signoffs — trust boundary, Ixchel sandbox, Danu scale/handoff, Sekhmet child-safety/IR, creator economy, anti-cheat, moderation/minor-protection, the all-realms safety floor — carry status: "fixture-passed" and the explicit evidenceBasis: "in-repo adversarial fixture; not a production measurement". The checked-in correctionNote is unsparing: the original 2026-06-02 "launch-ready" decision predated any adversarial audit (it was recorded three days after the feature spec was written); the 100%/100% precision-recall figures are "fixture-toy signatures" (6 true positives, 0 FP, 0 FN on seeded cases), not classifier performance against the 95% bar; the UE client is a ~2.2k-LOC skeleton; and PhotoDNA, PDQ, CSAI Match, Thorn Safer, Lantern, StopNCII, and EAC are not integrated. The staged-rollout record corroborates: all five stages (creator-alpha, rp-beta, meshing-beta, economy-launch, GA) are status: "planned" / decision: "pending", and the incident-response drills recorded in them were code-level simulations, not operational drills.

The machinery enforces this honesty. verify-launch-readiness.mjs encodes a no-fabrication invariant: if any signoff is fixture-passed, the decision cannot be launch-ready (:378–388); the rollout verifier enforces advance-iff-met so a stage cannot claim it advanced without being marked met; and the review's own verify:v7 launch-readiness script, which once asserted the fabricated "launch-ready" record, now fails against it intentionally and must be reworked to verify the honest state before it is re-enabled as a gate. V7 will be launch-ready — per the architecture's checklist — only when the trust boundary is enforced against a real realm process and web view, the sandbox has zero escapes on a launch-scale corpus, Danu sustains 1000+ with clean handoffs on a real cluster, Sekhmet meets its CSAM/grooming/malware bars with the real vendor integrations and passes an operational IR drill, age-assurance and the ≤48 h NCII pipeline are live, payouts settle through Aje with the anti-fraud gates, and the Lilith/Kuanyin safety floor is proven enforced on every realm regardless of operator configuration. Until each of those flips on real evidence, the same verifier keeps the aggregate negative — which, pre-GA, is worth more than a green that lies.

Where this connects#

• Sideways: Hosting, Fleet, Data & Compliance owns the Agones realm fleet, residency tagging, and DSAR pipeline whose per-realm health this page's /health contract and cost-lever telemetry feed; Thesis, Trust Boundary & Topology owns the platform/realm boundary the trust-boundary, identity-leak, and webview-escape gates defend and the safety floor the launch review signs off.
• Platform foundations it composes: the shared platform observability libraries (@oshun/metrics, @oshun/tracing — real and platform-wide, inherited but not yet wired into the V7 Rust services), the V1 observability stack the gateway reuses, and the V5/V6 build pipeline V7's cook and patch path inherits.
• The section hub: ../V7_ARCHITECTURE.md.