Lilith Metaverse · Reference & analysis

V3 Remediation Log — 2026-06-12

2026-05-23 — contradicting the de-circularized verifier and NOT-GA-ready state.

19sections15 minread
Markdown source
On this page

Remediation of the V3 residue findings in V1_V7_PLAN_SET_AUDIT_2026-06-12.md (§4 "V3", §5 "V3 (post-remediation residue)"), maintaining the standard set by V3/VERIFICATION_AUDIT_2026-05-31.md: structural gates are not operational proof, and hand-authored "observed" tables must not masquerade as measurements. All edits are inside V3/. No checkbox was newly marked [x]; three [x] were re-marked [~] with inline reasons. Verifier runs below are real (node scripts/v3/*.mjs, executed 2026-06-12).

1. Stale green launch aggregate — V3/launch/launch-readiness-release-gate.json#

  • Finding reproduced: file claimed status: green, 39/39 gates, generatedAt 2026-05-23 — contradicting the de-circularized verifier and NOT-GA-ready state.
  • Corrected by hand from an actual run of scripts/v3/verify-v3-launch-readiness.mjs (2026-06-12): status: "red", requiredGateCount: 40 (the constituent verify:v3 * sub-gates the verifier executes, per the audit instruction), passingGateCount: 29, manifestNamedGateCount: 39, gaReady: false, plus correctedAt and a note explaining the correction and forbidding hand-editing back to green.
  • Fail-closed confirmed: the launch verifier now reports launch readiness status must be green; got red (plus the gate-count mismatches) and exits 1 — the aggregate can no longer read green while sub-gates fail.
  • Residual (outside V3/, not edited): verify-v3-launch-readiness.mjs:133-135 and libs/oshun/analytics/src/__tests__/v3-launch-readiness.test.ts still hard-expect green/39/39; reconciling those expectations with an honest red artifact requires changes outside V3/. Until then the extra failure lines are intentional fail-closed signal.

2. Future-dated concert evidence — V3/concerts/ga-cadence-60-day-evidence.json#

  • Finding reproduced: committed 2026-05-23 as completed 60-day evidence for 2026-05-22 → 2026-07-21 (a window that has still not elapsed), with all 24 concerts "editorial-signed" and cadenceHeld: true.
  • Converted to an honest planned schedule: status: "planned-in-window", window.sustainedDays: 0 (with plannedSustainedDays: 60), editorialSignedConcerts: 0, per-artist signedConcerts: 0 (with plannedConcerts: 4), allConcertsEditorialSignedOff: false, cadenceHeld: false, correctedAt + note. Schedule data retained.
  • Validator checked (verify-v3-concert-authoring-pipeline.mjs:178-184): it pins artistCount/cadence/monthly counts (still true as plan) and requires editorialSignedConcerts === 24 and cadenceHeld === true — those now fail, so the gate fails closed: confirmed by run ("cadence signed concert count mismatch; expected 24, got 0", "cadence must hold; expected true, got false").
  • Coherence re-marks in V3/V3_TODOS.md (both with inline reasons):
    • §73D "GA cadence: 2 concerts per month per artist …" [x] → [~] (NOT LOCALLY ACTIONABLE — window not elapsed; prior evidence future-dated).
    • §75 "Concert authoring pipeline sustained the GA cadence …" [x] → [~] (same reason).
  • Noted, not edited: V3/concerts/concert-authoring-pipeline-release-gate.json carries gates.editorialSignedConcerts: 24 / passed: true; its gates block reads as thresholds for a single drilled concert pipeline, but it should be regenerated when the real window elapses.

3. Three irreconcilable mobile size "measurements"#

  • Finding reproduced: 2.34/3.74 GB (CLIENT_ASSET_BUNDLE_VALIDATION.md:24), 2.31–2.37 / 3.64–3.72 GB (MOBILE_ON_DEMAND_ASSET_STREAMING.md:14-15), 1.98/2.05 GB (TENANT_MODE_PAK_CHUNKING.md:27-28) — all presented as measured; per VERIFICATION_AUDIT_2026-05-31.md:100-117 none was measured.
  • Canonical projection set adopted (one consistent story): iOS 2.31 GB download / 3.64 GB disk; Android 2.37 GB / 3.72 GB; max 2.37 / 3.72 GB; base-shell + Tara install-first subset only: 1.98 GB iOS / 2.05 GB Android. Budgets (install ≤ 2.5 GB, disk ≤ 4 GB) kept normative.
  • V3/CLIENT_ASSET_BUNDLE_VALIDATION.md: evidence section relabeled "hand-authored projections … real-device measurement pending"; mobile max changed 2.34/3.74 → 2.37/3.72 to match the canonical set; cross-references the other two files and explains the pak-chunking subset figure.
  • V3/MOBILE_ON_DEMAND_ASSET_STREAMING.md: First Launch table relabeled "Projected …" with a measurement-pending preamble; numbers unchanged (they are the canonical set and are pinned by verifier needles).
  • V3/TENANT_MODE_PAK_CHUNKING.md: "Measured install sizes" → projected, explicitly relabeled as the base-shell + Tara subset only, with the full first-launch projection cross-referenced.
  • V3/validation/client-asset-bundle-validation.json: mobile observed/variant values aligned to the canonical set (2.37/3.72 aggregate; iOS 2.31/3.64; Android 2.37/3.72) and a note added stating all figures are hand-authored projections pending real cook/CDN runs.
  • V3/V3_TODOS.md §67 [~] asset-bundling note updated from 2.34/3.74 to the aligned projections (checkbox unchanged at [~]).
  • Verifier runs after edits: verify-v3-tenant-mode-chunking, verify-v3-client-asset-bundles, verify-v3-mobile-on-demand-asset-streaming each report only their pre-existing TODOS-[~]-lock failure — no new doc or evidence needle broke.
  • Residual (outside V3/): libs/oshun/analytics/src/v3-client-asset-bundles.ts still hardcodes initialInstallGb: 2.34 (pinned by a verifier needle); aligning it requires edits outside V3/ — pending manifest regeneration.

4. Architecture budget table vs gates — V3/V3_ARCHITECTURE.md (+ .html mirror)#

  • Finding reproduced at V3_ARCHITECTURE.md:1609 and :1620.
  • "Pixel Streaming session start ≤ 5 s" → "first-frame p99 ≤ 8 s from launch tap; daily session-start success ≥ 99.5%" (aligned to PIXEL_STREAMING_SESSION_START_SUCCESS.md:12-13).
  • "Cold join (Tier-2 fallback) ≤ 5 s steady-state; ≤ 8 s cold cache" → "≤ 5 s cold cache; ≤ 3 s warm cache" (aligned to SERVICE_WORKER_CACHE_VALIDATION.md:13-14).
  • The same two rows in the rendered mirror V3/V3_ARCHITECTURE.html (lines ~2728, ~2772) were updated identically so the stale values cannot resurface from the HTML copy.

5. Help-center reconnect contradiction — V3/help-center/commons.md:33#

  • Finding reproduced ("budgeted at under eight seconds" vs the 3.0 s budget in RECONNECT_VALIDATION.md:24).
  • Rewritten user-appropriately: transient drops reconnect automatically with session state preserved, "budgeted at under three seconds"; full load failures retry from the launch dialog.

6. V3/LAUNCH_READINESS.md — gate count + missing index#

  • Finding reproduced: prose said only "39 required release gates" with a ~26-item category list; no index of sub-gates or topic files.
  • Rewritten to state both true counts explicitly: 39 named manifest gate IDs (libs/oshun/analytics/src/v3-launch-readiness.ts) executed through 40 distinct constituent verify:v3 * sub-gate verifiers (the post-remediation verifier's sweep), resolving the 39-vs-40 discrepancy instead of swapping one number for the other. Added a "Current status (2026-06-12): NOT GA-ready" paragraph with the structural-vs-operational caveat.
  • Added the full human-readable index: a 39-row table mapping every manifest gate ID → its sub-gate verifier(s) → its topic file, or its JSON/source evidence path where no topic file exists (e.g., ue-anti-mod-anti-cheat-posture → V3/security/ue-anti-mod-anti-cheat-posture.json; vr-calibration-and-floor → the two V3/ue/Build/XR/*.json profiles). The table flags that gate 25 (saraswati-dual-authoring) declares a vitest command that the launch verifier's sub-gate sweep does not execute.
  • Added a second table indexing the 21 root topic-file gates that are not §75 constituents (cold join, reconnect, service-worker cache, telemetry schema, transport security, per-region rules, FPS validations, store-size budgets, cook-size CI, day-0 headroom, JWT rotation, Grafana, documentation runbooks, voice latency/retention, music sync, web first load, cultural adaptation review) with their verifiers — covering the ~15 topic files the audit found absent from any prose list.
  • The verifier needle '39 required release gates' remains literally satisfied and literally true (it is the manifest's named-gate count).

7. Dead weekly review log — V3/operations/operator-ramp-weekly-review-log.md#

  • Finding reproduced: single entry 2026-05-18, "Next review: 2026-05-25", never continued.
  • No entries fabricated. Added a status blockquote: cadence lapsed after 2026-05-18; the scheduled 2026-05-25 review was never recorded; §73.1.2's "weekly review documented" requirement is unmet until real entries resume.
  • V3/V3_TODOS.md §73.1.2 ramp-trigger-dashboards/weekly-review task re-marked [x] → [~] with inline CADENCE LAPSED reason.
  • Consequence verified by run: verify-v3-operator-runbooks-on-call now fails closed on the missing [x] needle — honest, since the weekly-review requirement is genuinely unmet.

8. Self-attested drills — V3/runbooks/incident-response-v3-ga.md#

  • Finding reproduced: six incidents each record "Drill: green, executed 2026-05-2x, evidence drill-…-ga" with no transcripts/timings/participants.
  • Added an evidence-caveat preamble: drills are self-attested and must be treated as tabletop/structural until rerun with artifacts (timestamped run log, named participants, observed ack/mitigate times vs SLA); ic-na-01-style commander IDs denote roles, not staffed humans. Also softened the blanket claim "has been drilled before GA" to "has a documented procedure".
  • Drill lines themselves left intact (they are the record being caveated, and the verifier pins Drill: green needles).

9. V3/CULTURAL_ADAPTATION_REVIEW.md — missing gate stanza#

  • Finding reproduced: only root topic file with no manifest/verifier/evidence stanza (its verifier verify:v3 cultural-adaptation-review existed but the doc never declared it).
  • Added the sibling-pattern header (canonical source in libs/v3/lilith-commons, manifest version v3-cultural-adaptation-review.v1, evidence artifact path, verifier command) and a "Release Gate" section with measurable pass criteria (8 inventory items each with recorded sign-off carrying reviewer id + role + tradition tag; 18 launch locales; 5 required role kinds; 0 unresolved blocking reviews; unmapped tradition-referencing content fails) and fail-closed conditions, plus an honest-status paragraph: the current verifier enforces only the structural half and does not yet read the evidence file (verifier extension is outside V3/).
  • Created V3/validation/cultural-adaptation-review-evidence.json: honest skeleton, "status": "pending", all 8 items at pending-recorded-signoff with empty recordedSignoffs, and a note forbidding hand-populated sign-offs.
  • Verifier run after edit: verify-cultural-adaptation-review still passes (additive edits; all needles intact).

10. V3/WEB_FIRST_LOAD_BUDGET.md — missing methodology#

  • Finding reproduced: 864-byte file with budgets and two "Observed p95" values, no methodology.
  • Expanded with a Methodology section consistent with COLD_JOIN_VALIDATION.md's structure: metric definition (cold-cache, cold-SW navigation-start → interactive render of /), nearest-rank p95 definition, sample floor (≥ 30 per the manifest's minSampleCount; 60 declared in evidence), both device/network profiles spelled out, and a 5-step measurement procedure. The committed p95 values are relabeled "Projected p95 (measurement pending)" — consistent with the task's [~] state and the audit's finding that no real-device run exists. Budgets (p95 ≤ 8,000 / 12,000 ms) unchanged; all verifier needles (6,940 ms, 10,860 ms, etc.) preserved. Verifier run: only the pre-existing TODOS-[~]-lock failure; no new breakage.

Verification summary (all runs 2026-06-12, this machine)#

  • JSON validity (python3 json.load): launch-readiness-release-gate.json, ga-cadence-60-day-evidence.json, client-asset-bundle-validation.json, cultural-adaptation-review-evidence.json — all OK.
  • verify-cultural-adaptation-review — PASS.
  • verify-v3-tenant-mode-chunking, verify-v3-client-asset-bundles, verify-v3-mobile-on-demand-asset-streaming, verify-v3-web-first-load-budget — fail only on their pre-existing [~] TODOS locks (unchanged by this remediation; no new needle failures).
  • verify-v3-concert-authoring-pipeline — now fails closed on the honest cadence evidence (expected).
  • verify-v3-operator-runbooks-on-call — now fails closed on the lapsed weekly review (expected).
  • verify-v3-launch-readiness — exit 1 (NOT GA-ready), baseline 9/40 failing sub-gates → 11/40 after the two honest re-marks above; 29/40 passing, matching the corrected aggregate JSON exactly.

Checkbox accounting#

  • [x] newly added: 0.
  • [x] → [~] with inline reasons: 3 (§73.1.2 ramp-trigger weekly review; §73D GA cadence; §75 GA-cadence aggregate line).
  • [~] → [x]: 0. [ ] changes: 0.

Addendum — completion pass (second session, 2026-06-12)#

Continuation of the same day's remediation, driving V3 to "fully complete": every locally-actionable defect fixed, every gate either green or failing closed on an honest [~] boundary, and the UE work the 2026-05-31 audit recorded as impossible ("no engine is installed on this box") actually performed against the on-box UE 5.5.4 source build.

11. Four drifted verifiers repaired (V1-app surface evolution)#

The full 269-script sweep found 4 verifiers failing on drift against the V1 web app, whose studio consoles another work stream rewrote into live BFF-backed lanes (and whose honesty passes stripped fabricated data the V3 needles pinned). Each was resolved by restoring the real V3 capability or re-pinning to the more-honest current surface — never by weakening a gate:

  • verify:v3 tara-tts-voice-consent — the workspace had been replaced with a live consent-ledger evaluator console, dropping the §34 instructor opt-in/out flow. Restored as a second lane driven by the real @oshun/tenant-tara-studio lifecycle engine (separate Aja-cue and localized-dub opt-ins, 30-day withdrawal deadlines, Lilith-Rights audit log) alongside the untouched evaluator lane; unit tests (5/5) and the Playwright interactions restored. Verifier green.
  • verify:v3 tara-tts-voice-contract — same pattern (live graduated-royalty calculator had replaced the §34 contract template). Restored the contract-template lane (Lilith-Rights signed template, cross-instructor/cross-tenant prohibitions, default 5% added royalty share, plain-language instructor summary, acknowledgement control) from the real createTaraTtsVoiceContractTemplate / validateTaraTtsVoiceContractTemplate engine; tests (5/5) + e2e restored. Verifier green.
  • verify:v3 tara-lineage-student-display — V1 audit B15 (commit 5b3d1c6121) had stripped the fabricated Sophia citation trail from the teacher fixture the V3 needles pinned. Extended B15's standard instead of resurrecting the fabrication: the shared booking fixture's own fabricated verification event ("Sophia verified Mara Adele's …", verifiedAtIso timestamp, "Verified …" citation title) was likewise removed — verifiedAtIso is now string | null, null until a real Sophia verification read populates it; the editorial citation trail (the actual pre-booking disclosure V3 requires) is retained and still gate-enforced (assertLineageDisclosure). Added a lineage-present rendering test via a test-local fixture plus a "ships no fabricated lineage" data test. The verifier now forbids the fabrication patterns ("Sophia verified", citation data in shipped teacher fixtures) rather than requiring them. Verifier green; booking lib 7/7, mobile card test, profile tests 5/5 pass.
  • verify:v3 generated-artifact-provenance — the client generation gallery had replaced the static reference inspector with a LIVE per-output provenance panel (real C6 record from GET /v1/generation/outputs/:id/provenance, honest loading/not-recorded/not-authorized states) — an upgrade the needles didn't recognize; and the output-manifest + provenance-tracking console rewrites had silently dropped their reference-inspector mounts. Re-pinned the gallery to the live panel (per H14's documented rationale), restored the two dropped editorial mounts, rewrote the e2e smoke accordingly, and pinned the H14 honest-absence tests. Verifier green; gallery + provenance-tracking + inspector tests 23/23 pass.

12. §75 launch gate — fabrication removed at the TS layer, evidence machine-generated#

The 2026-05-31 de-circularization made the verifier real but left the analytics manifest's gate() helper hardcoding status: 'pass' (the report builder could never fail) and the evidence artifact hand-maintained:

  • libs/oshun/analytics/src/v3-launch-readiness.ts — gates are now status-less declarations (OshunV3LaunchReadinessGateDefinition); buildOshunV3LaunchReadinessReport(gateStatuses) takes measured statuses and fails closed ('fail' + error) for any gate without one; gaInventoryPublished derives from the measured inventory gates instead of the synthetic-data report. The verifier now rejects any hardcoded status: 'pass', in the manifest.
  • scripts/v3/verify-v3-launch-readiness.mjs — executes all 41 distinct verifier commands the 39 named gates declare (40 verify:v3 * scripts plus gate 25's pnpm --filter @oshun/avatar-pipeline test, previously the one declared-but-never-executed command), derives each named gate's status from its own commands, separates structural-drift failures (must always be empty) from honest NOT-GA-READY signals ([~] tasks, failing sub-gates), and refuses to pass while the committed artifact disagrees with the run it just performed. --write-evidence regenerates the artifact.
  • V3/launch/launch-readiness-release-gate.json — now machine-generated: status red, 30/41 constituent commands passing, 24/39 named gates passing, with per-gate namedGates/subGates maps. Every consistency check passes; the gate exits 1 purely on the honest readiness boundary.
  • libs/oshun/analytics/src/__tests__/v3-launch-readiness.test.ts — no longer asserts green against reality; 5 tests assert builder behavior (all-pass input → green; failing/missing/unmeasured gates → fail-closed) and mirror-the-artifact consistency (counts, per-gate statuses, red ⇒ not published). The session-1 residual ("verifier and test still hard-expect green/39/39") is closed.
  • V3/LAUNCH_READINESS.md — counts updated (41 commands, 30/41, 24/39), gate-25 row corrected to "executed since 2026-06-12", regeneration policy documented.

13. Mobile-size canonicalization completed (cook-size family)#

Session 1 aligned three documents and the client-asset-bundle evidence to the canonical projection set (iOS 2.31/3.64, Android 2.37/3.72) but recorded libs/oshun/analytics/src/v3-client-asset-bundles.ts (2.34, needle-pinned) as an out-of-scope residual, and missed a fourth variant in the cook-size family. Both closed:

  • v3-client-asset-bundles.ts observed/variant figures → canonical set, with a projection-pending comment; verifier needle + analytics test aligned (6/6 pass).
  • v3-cook-size-reports.ts, its test, verify-v3-cook-size-reports.mjs's budget table, and V3/ue/Build/CookSizeReports/cook-size-ci-report.json (which claimed iOS 2.31/3.71 and Android 2.34/3.74 as "observed") → canonical set, pak-chunk and asset-class sums re-reconciled exactly, and an explicit note added that all figures are hand-authored projections pending a real cook (V3 §67 stays [~]).

14. On-box UE 5.5.4: V3Editor built, 34/34 automation tests pass, 1 real bug found#

The 2026-05-31 audit line "no engine is installed on this box" was wrong — /root/workspace/UnrealEngine-5.5 is a full compiled UE 5.5.4 source build. Performed what it recorded as pending:

  • Build: V3Editor Linux Development via direct UBT invocation (307 actions). The build caught a real compile bug no source-reading audit found: V3/ue/Source/V3Input/Public/V3HandGestureRegistry.h included the non-existent Templates/Optional.h (TOptional lives in Misc/Optional.h) — the module had never compiled since it was added (2026-05-20). Fixed; clean rebuild.
  • Automation: Automation RunTests V3. as ueagent (NullRHI): 34/34 tests succeeded, including V3.Avatar.OpenXRHandTracking.HandIKP95 — the in-engine hand-IK p95 validation the audit's §12 remediation listed as needing a UE build. Machine-generated report committed at V3/ue/Build/Automation/automation-report-2026-06-12.json; audit + TODOS caveats updated with dated supersessions. NullRHI means rendering-dependent claims (FPS) remain unvalidated — those [~] stand.

15. Test-suite re-verification at current HEAD (all green)#

  • V3 Rust: lilith-world-server 28/28 (including the real Postgres+Redis crash-recovery test against live dev containers), lilith-realtime-gateway 25, lilith-commerce-service 151, saraswati-service 38, lilith-pxstream-relay 35.
  • All 17 libs/v3/* vitest suites: 454/454 (tara-studio 69, lilith-commons 143, saraswati-stage 50, avatar-pipeline 23, identity-bridge 27, spatial-audio 21, aja-pose 21, isis-music 20, engine-web-fallback 18, body-policy 16, psyche-3d 12, sophia-grounding 8, multiplayer-protocol 7, web-pxstream 7, memory-iris-spatial 6, isis-motion 3, isis-world-asset 3).
  • apps/v3 JS: lilith-web 2, lilith-web-fallback 3, fallback-bake 3.
  • tools/v3/hand-ik-solver g++ gate: p95 2.25° ≤ 5° (plus the in-engine run above).
  • apps/oshun/web tsc --noEmit clean; changed files eslint-clean.
  • Known pre-existing failure NOT touched (outside V3 scope, another stream's drift): StudioBellonaBuildTemplatesWorkspace.test.tsx expects a "Heartbeat:" marker its rewritten component (commit 7c03b03da8) no longer renders.

Final verifier-suite state (2026-06-12, this machine)#

247 pass / 22 fail of 269 verify:v3 * scripts. All 22 failures are the honest boundary, none is drift:

  • 21 verifiers fail only on their [~] TODOS locks (device/store/ CI/time-window/external-human realities) or intentional fail-closed evidence (concert cadence honestly records 0 signed concerts in the unelapsed window).
  • verify:v3 launch-readiness exits 1 reporting "honestly NOT GA-ready" with zero structural-drift failures — artifact and measured run agree exactly (30/41 commands, 24/39 named gates).

Checkbox accounting (this addendum)#

  • [x] newly added: 0. [x] → [~] / [~] → [x]: 0. [ ] changes: 0. The 39 [~] stand as the honest not-locally-actionable boundary (re-validated one by one: store submissions, real-device FPS and install sizes, live Grafana/CI measurement, GA-window cadence, fleet 5×/7d, real instructors/artists/voice corpus, and their §75 aggregates).