Audit date: 2026-07-15
Scope: V1 through V10, shared agent/content substrates, generation, evaluation, iteration, human intervention, publication, deployment evidence, and operating feedback loops
Scope amendment (2026-07-22): V10 joined the product registry after this dated audit began. Its independent product review, ledger, bounded verification suite, content-coverage seed rows, and version-specific findings are now part of the same generated cadence. Historical V1–V9 observations below retain their original evidence date rather than being silently rewritten as V10 evidence.
Goal audited: a system in which AI agents autonomously fill every required content surface with exceptional content, while humans may curate, audit, critique, directly author, replace, or take over at any time
Assessment type: repository and architecture audit, not a production certification
Executive verdict#
The goal is not yet achieved end to end.
The repository contains a large and often excellent collection of real agentic components. V1 has the strongest general autonomous director and quality plane; V6 has a substantial real-time character cognition and story stack; V8 has the clearest solve-first self-authoring vertical; and V9 has a strong learning-domain orchestration and seven-gate foundation. Across the portfolio, fail-loud provider seams, deterministic verification, provenance, safety policy, best-of-N search, judge panels, preference capture, editorial gates, and honest remediation of fabricated-success paths are materially ahead of a typical prototype.
That is not the same as the requested outcome:
| Goal clause | Verdict | Why |
|---|---|---|
| Agents autonomously fill everything | No | There is no machine-readable, portfolio-wide inventory of every required content slot and no closure controller that proves each slot is generated, placed, current, and released. V1's live autonomy route schedules four launch categories; V3–V7 still depend on major provider, engine, asset, or deployment seams. |
| Output is top quality / amazing | Not demonstrated | Code tests and synthetic gates prove mechanics and honesty, not audience preference. There is no portfolio-wide corpus of live rendered artifacts with blinded expert/user comparison against the best human-authored baseline. V8 lacks its authored 25-case regression anchor; V9 lacks real-learner efficacy evidence. |
| Agents work autonomously | Partially | V1 and V8 demonstrate strong locally closed loops. RUN-001 defines the common run protocol, RUN-002 adds a durable reference spine with cross-process recovery, and RUN-003 adds a provider-aware idempotent-effect coordinator. These remain portable contract/reference-adapter proofs: products still use separate orchestration around absent inference, rendering, simulation, or live-service stages; no product database adapters, live provider-effect consumers, or common cross-version scheduler are evidenced. |
| Humans can curate, audit, and critique | Tested V1 component; partial portfolio-wide | V1 exposes approve, reject, veto, tweak, standing direction, feedback, provenance, and review windows through a real, tested API. No in-repository operator UI consumer was discovered, and no deployed external consumer was evidenced. These controls are not one common operator plane across V1–V9. |
| Humans can fill or take over whenever | No | Rich pause/resume/fork/retry/manual-step and policy-override libraries exist, as do authoring primitives, but they are not bound to the autonomous review API or product runtimes. The current V1 review item accepts agent output and tweak direction; it does not support claiming a step, editing tool arguments, replacing an artifact with a human-authored version, or resuming the same durable run from that edit. |
| System is production-proven | No | The repository correctly keeps live providers, remaining release-required cooks, deployed transports, human panels, beta cohorts, and real-world outcomes external or pending. Bounded local cook records are not packaged release proof; production telemetry and user-outcome evidence are absent from this audit. |
The accurate portfolio label is:
SOTA-shaped and unusually strong at the component and local-verification layers; not yet a unified, production-observed autonomous content operating system.
This distinction matters. The next leap is not another isolated generator or judge. It is composition: a canonical coverage contract, one durable run and intervention protocol, live artifact evidence, calibrated human-quality data, and a portfolio feedback loop.
A second, source-level pass over the executable prompts, decision algorithms, agent registries, tool dispatchers, memory systems, verification gates, and human-control paths materially sharpens that verdict. Several packages contain excellent governance primitives, but the production-facing path does not always invoke them. In particular, the V1 Agentic Studio route does not consume its budget meter or record tool calls/costs after dispatch, V9's agent runtime API plans and evaluates DAGs but does not execute them, most prompt paths lack the explicit untrusted-content fencing used by the stronger V1 content-quality judge, and multiple release signals prove citation or provenance shape rather than the underlying source/right. Sections 15–19 record the prompt and algorithm audit, the exact enforcement boundaries, and the resulting revised priorities.
1. Audit standard and evidence model#
1.1 What counts as “filled”#
A content requirement is filled only when all of the following are true:
- The requirement exists in a canonical inventory with owner, modality, locale, audience, quantity, freshness, rights, safety, and quality policy.
- A complete artifact exists, rather than only a plan, contract, fixture, request, provider seam, path, or metadata record.
- The artifact passed the content-type-specific correctness, quality, safety, rights, provenance, accessibility, and runtime gates.
- The artifact was published and placed in the consumer experience.
- Its lineage and current release state can be queried.
- It remains inside its freshness and quality-drift service level.
- A human can inspect, replace, edit, take over, or withdraw it without leaving the governed pipeline.
By this definition, a generated brief is not a generated asset; a provider request is not a provider result; a package path is not a cooked game artifact; a structural gate is not an observed user result; and a passing model judge is not proof that users find an experience exceptional.
1.2 Evidence ladder#
The scorecard uses an ordinal evidence ladder, not a percentage or maturity average:
| Level | Meaning |
|---|---|
| 0 — absent | No implementation or the claim is contradicted. |
| 1 — contract | Spec, interface, plan, fixture, deterministic fallback, or explicit external seam. |
| 2 — component | Real domain logic and tests exist, but the path is not closed end to end. |
| 3 — locally closed | A reproducible local path runs through generation/orchestration, verification, governance, and release or placement with honest external boundaries. |
| 4 — production-observed | The path runs with real providers/users, durable operations, calibrated human evaluation, SLOs, and measured outcomes. |
“SOTA” in this report requires both technique and evidence. A sophisticated algorithm at level 2 can be SOTA-shaped, but it is not a level-4 SOTA system.
1.3 Method#
The audit used four evidence classes:
- Current implementation and production-consumer searches, particularly the V1 autonomy route, plane factory, scheduling/persistence, review store, publication bridge, AgentRun lifecycle, Agentic Studio run controls, Lilith operator-override governance, and Yemaya human-override package.
- The current per-version autonomous-content audits and current V8/V9 completion ledgers, cross-checked against their referenced packages.
- Open portfolio ledgers and architecture proposals, including the content capability matrix and platform-consolidation proposal.
- Current primary sources on agent workflows, durable human interrupts, evaluation, observability, security, provenance, multimodal evaluation, and iterative improvement.
The follow-up deep pass traced every in-scope autonomous-content version through its executable entry point or, where no executable generator exists, through the provider seam that terminates the local path. For each path it inspected: prompt construction and sampling, output parsing and validation, planning and selection algorithms, retry/refinement behavior, memory and learning state, tool/budget/authority enforcement, human interruption, persistence, and release evidence. “Direct” in sections 15–19 means source read in this pass; “corroborated” means the current version audit was checked against its referenced source; and “seam” means the operation is intentionally external or absent. This is a content-system audit, not a claim that every unrelated monorepo subsystem was line-read.
The audit does not treat a checked box, passing unit test, or prior report as production proof. It also does not penalize intentionally human-authored policy surfaces merely for not being autonomous; it asks whether the source policy is explicit and whether humans and agents share one governed lifecycle.
1.4 Current verification record#
<!-- AUD-003:BEGIN generated; edit the record/manifest, not this section -->
This is a historical, reproducible observation generated from
evidence/v1-v9/AUD-003/verification-record.json, not a timeless statement
about the current host or a production certification. The tested source commit
is b856ddd2fe3ff0f7ead8751554c21d7d8a010a20.
Recorded environment#
| Field | Recorded value |
|---|---|
| Run interval | 2026-07-22T11:13:14.314Z through 2026-07-22T11:21:42.073Z |
| Tested HEAD | b856ddd2fe3ff0f7ead8751554c21d7d8a010a20 (branch oshun-strong-verify-codex) |
| pnpm lockfile SHA-256 | f0061e1a5951c82f1f8d5297d08bc6ed2e777107c8d6c35cb87a820c91b71b22 |
| Manifest SHA-256 | c2d87ca802554f8bc44972020db0a69e4ce642748361c11a6c09837ca68e1624 |
| Verification contract SHA-256 | ec2281c7c638b60aa7ed7c889f81ada87ef2ebab3b72c1d7d921ff889cce452e |
| Runtime | Node.js v24.15.0; pnpm 10.25.0; Vitest vitest/4.1.7 linux-x64 node-v24.15.0; git version 2.43.0 |
| Host/OS | app-dev-server; linux 6.8.0-134-generic x64; 4 CPUs |
| Start resource observation | 15.24 GiB RAM total; 11.69 GiB available; 32.00 GiB swap total; 30.73 GiB swap free; 5.29 GiB root disk available |
| End resource observation | 12.14 GiB RAM available; 30.73 GiB swap free; 5.29 GiB root disk available |
| Relevant environment gates | CI=false; NODE_OPTIONS=false; OPENAI_API_KEY=false; ANTHROPIC_API_KEY=false; RUNPOD_API_KEY=false; DATABASE_URL=false; TZ=unset; LANG=C.UTF-8; LC_ALL=C.UTF-8 |
| Resource policy | Sequential check/configuration execution; Vitest --maxWorkers=1; abort below 2 GiB available memory |
The RAM, swap, disk, and process values above are observations at the stated run timestamps. They are not properties of every checkout or future host.
Verification results#
| Check | Included/excluded boundary | Recorded result | Bounded log |
|---|---|---|---|
| V1 @oshun/creative-autonomy | Included: libs/oshun/creative-autonomy/vitest.config.ts. Excluded: Every other V1 and workspace Vitest configuration. | 284/284 tests passed across 1 configuration | log |
| V1 @oshun/agentic-studio | Included: libs/oshun/agentic-studio/vitest.config.ts. Excluded: Every other V1 and workspace Vitest configuration. | 348/348 tests passed across 1 configuration | log |
| V1 @oshun/studio-authoring | Included: libs/oshun/studio-authoring/vitest.config.ts. Excluded: Every other V1 and workspace Vitest configuration. | 138/138 tests passed across 1 configuration | log |
| V1 BFF autonomy bindings, E2E/route, and run lifecycle | Included: apps/oshun/bff/src/agentic/autonomy-bindings/autonomy-bindings.test.ts; apps/oshun/bff/src/agentic/autonomy-bindings/autonomy-plane-e2e.test.ts; apps/oshun/bff/src/agentic/autonomy-route.test.ts; apps/oshun/bff/src/agentic/run-lifecycle-store.test.ts. Excluded: BFF engagement polling, media critic/best-of-N, governed-run route, and unrelated route tests. | 66/66 tests passed across 1 configuration | log |
| V8 apps and case/canon packages | Included: 3 apps/v8 configurations; 3 libs/v8 configurations; libs/yemaya/canon-graph; 11 libs/yemaya/case-* configurations. Excluded: libs/yemaya/canon-enforcement (30 tests); All other Yemaya packages and applications. | 404/404 tests passed across 18 configurations | log |
| V8 docs/contract drift | Included: scripts/verify-v8-docs-drift.mjs; libs/yemaya/case-contracts/tools/gen-v5-target.mjs; current V5 public headers; libs/yemaya/case-contracts/src/v5-target.generated.ts. Excluded: All unrelated documentation and contract generators. | 1/1 commands passed | log |
| V9 packages plus V9 canonical contracts | Included: 12 Vitest configurations under libs/v9/* (168 tests); libs/contracts/src/v9/__tests__/v9-contracts.spec.ts (20 tests). Excluded: All non-V9 contract tests; All Metis service/app tests outside libs/v9. | 188/188 tests passed across 13 configurations | log |
| V10 Rail apps, packages, and canonical contracts | Included: 3 Vitest configurations under apps/v10/*; 15 Vitest configurations under libs/v10/*; 9 canonical V10 contract specification files. Excluded: Playwright and native shell execution; BFF integration outside the V10 product's bounded source and route contracts; shared substrate suites already owned by their canonical packages. | 782/785 tests passed across 19 configurations | log |
| Report formatting and whitespace | Included: V1_V9_AUTONOMOUS_CONTENT_SOTA_AUDIT_2026-07-15.md; V1_V9_SOTA_AUDIT_REVIEW_2026-07-15.md; V10/V10_PRODUCT_REVIEW_2026-07-22.md; V10/V_SERIES_AMBIENT_RAIL_TODOS_2026-07-16.md; current worktree diff. Excluded: All unrelated repository files from the Prettier command. | 2/2 commands passed | log |
The V8 drift result is a measured red product gate, not a failure to execute this verification record. Every required-pass test and formatting command must be green before the record writer succeeds.
Exact commands and package boundaries#
V1 @oshun/creative-autonomy#
Included: libs/oshun/creative-autonomy/vitest.config.ts.
Excluded: Every other V1 and workspace Vitest configuration.
pnpm exec vitest run --config libs/oshun/creative-autonomy/vitest.config.ts --reporter=json --maxWorkers=1
V1 @oshun/agentic-studio#
Included: libs/oshun/agentic-studio/vitest.config.ts.
Excluded: Every other V1 and workspace Vitest configuration.
pnpm exec vitest run --config libs/oshun/agentic-studio/vitest.config.ts --reporter=json --maxWorkers=1
V1 @oshun/studio-authoring#
Included: libs/oshun/studio-authoring/vitest.config.ts.
Excluded: Every other V1 and workspace Vitest configuration.
pnpm exec vitest run --config libs/oshun/studio-authoring/vitest.config.ts --reporter=json --maxWorkers=1
V1 BFF autonomy bindings, E2E/route, and run lifecycle#
Included: apps/oshun/bff/src/agentic/autonomy-bindings/autonomy-bindings.test.ts; apps/oshun/bff/src/agentic/autonomy-bindings/autonomy-plane-e2e.test.ts; apps/oshun/bff/src/agentic/autonomy-route.test.ts; apps/oshun/bff/src/agentic/run-lifecycle-store.test.ts.
Excluded: BFF engagement polling, media critic/best-of-N, governed-run route, and unrelated route tests.
pnpm exec vitest run --config apps/oshun/bff/vitest.config.ts src/agentic/autonomy-bindings/autonomy-bindings.test.ts src/agentic/autonomy-bindings/autonomy-plane-e2e.test.ts src/agentic/autonomy-route.test.ts src/agentic/run-lifecycle-store.test.ts --reporter=json --maxWorkers=1
V8 apps and case/canon packages#
Included: 3 apps/v8 configurations; 3 libs/v8 configurations; libs/yemaya/canon-graph; 11 libs/yemaya/case-* configurations.
Excluded: libs/yemaya/canon-enforcement (30 tests); All other Yemaya packages and applications.
(cd apps/v8/daedalus-compiler && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd apps/v8/loom-service && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd apps/v8/minos-asp-sidecar && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/v8/case-bundle && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/v8/case-csp && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/v8/case-gates && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/canon-graph && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-assets && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-compiler && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-contracts && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-director && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-engine && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-eval && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-localization && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-pipeline && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-suspects && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-verifier && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd libs/yemaya/case-writers-room && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
V8 docs/contract drift#
Included: scripts/verify-v8-docs-drift.mjs; libs/yemaya/case-contracts/tools/gen-v5-target.mjs; current V5 public headers; libs/yemaya/case-contracts/src/v5-target.generated.ts.
Excluded: All unrelated documentation and contract generators.
node scripts/verify-v8-docs-drift.mjs
V9 packages plus V9 canonical contracts#
Included: 12 Vitest configurations under libs/v9/* (168 tests); libs/contracts/src/v9/__tests__/v9-contracts.spec.ts (20 tests).
Excluded: All non-V9 contract tests; All Metis service/app tests outside libs/v9.
pnpm exec vitest run --config libs/v9/aletheia/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/atlas/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/chiron/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/cross-cutting/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/experience/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/governance/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/hephaestus/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/lesson-explorables/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/lesson-gates/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/mnemosyne-glue/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/prometheus/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v9/theia/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/contracts/vitest.config.ts src/v9/__tests__/v9-contracts.spec.ts --reporter=json --maxWorkers=1
V10 Rail apps, packages, and canonical contracts#
Included: 3 Vitest configurations under apps/v10/*; 15 Vitest configurations under libs/v10/*; 9 canonical V10 contract specification files.
Excluded: Playwright and native shell execution; BFF integration outside the V10 product's bounded source and route contracts; shared substrate suites already owned by their canonical packages.
(cd apps/v10/case-files-cook && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd apps/v10/rail-cli && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
(cd apps/v10/web && pnpm exec vitest run --config vitest.config.ts --reporter=json --maxWorkers=1)
pnpm exec vitest run --config libs/v10/rail-channel-aphrodite/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-arete/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-case-files/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-motion/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-nyx/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-ori-presence/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-stage/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-tara/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-v4-match/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-veritas-live/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-channel-wonder-recall/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-kernel/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-phase-a/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/rail-thread/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/v10/test-channel/vitest.config.ts --reporter=json --maxWorkers=1
pnpm exec vitest run --config libs/contracts/vitest.config.ts src/v10/adult-ring.spec.ts src/v10/case-files.spec.ts src/v10/channel-contract-fields.spec.ts src/v10/channel-contract.spec.ts src/v10/mobile-surface.spec.ts src/v10/ori-presence.spec.ts src/v10/shell-bridge.spec.ts src/v10/tv-surface.spec.ts src/v10/watch-surface.spec.ts --reporter=json --maxWorkers=1
Report formatting and whitespace#
Included: V1_V9_AUTONOMOUS_CONTENT_SOTA_AUDIT_2026-07-15.md; V1_V9_SOTA_AUDIT_REVIEW_2026-07-15.md; V10/V10_PRODUCT_REVIEW_2026-07-22.md; V10/V_SERIES_AMBIENT_RAIL_TODOS_2026-07-16.md; current worktree diff.
Excluded: All unrelated repository files from the Prettier command.
pnpm prettier --check V1_V9_AUTONOMOUS_CONTENT_SOTA_AUDIT_2026-07-15.md V1_V9_SOTA_AUDIT_REVIEW_2026-07-15.md V10/V10_PRODUCT_REVIEW_2026-07-22.md V10/V_SERIES_AMBIENT_RAIL_TODOS_2026-07-16.md
git diff --check
<!-- AUD-003:END generated -->
2. Portfolio scorecard#
| Version | Content coverage | Autonomous loop | Quality evidence | Human control | Live proof | Overall finding |
|---|---|---|---|---|---|---|
| V1 Oshun | 2 | 3 | 3 | 2 | 1 | Strongest general-purpose director, judge, learning, review, and release plane. Human-control level 2 credits the tested API component, not end-user operability: no in-repository operator UI consumer was discovered. Four scheduled categories are not “all content.” |
| V2 fighting/esports | 2 | 2 | 2 | 1 | 1 | Real commentary orchestration, rollback-safe hints, deterministic fallback, persona governance, render math, gRPC bindings, and post-match verification. Live generation/rendering and plan-to-executor paths remain deploy-bound. |
| V3 Lilith metaverse/concert | 1 | 2 | 2 | 1 | 0 | Strong governance and real C2PA/pose/quality components around mostly external music, lyric, TTS, motion, world, and UE execution. Its last machine launch record was red at 30/41 and GA-ready false; that observation now exceeds its 24-hour evidence SLO. |
| V4 tactical game | 2 | 2 | 2 | 1 | 0 | Real director, procgen, roguelike, crowd, and schedules. Commentary prose/audio, bias review, watermarking, and live review queues remain provider/operations seams. |
| V5 open-world universe | 2 | 2 | 2 | 1 | 0 | Deterministic AI Director, procgen, replay/highlight logic, authenticated editor execution, and a bounded Linux binary-asset cook are real. Live model calls, packaged and launch-smoked flagship content, and human receipts remain pending. |
| V6 Egbe companions | 3 | 3 | 2 | 2 | 1 | Substantial real cognition, biography, story, agent foundry, world, gateway, and authoring code. Live WebTransport/SFU, ASR/TTS, OTLP, and Themis/V1 operations remain external. |
| V7 Mawu | 2 | 2 | 2 | 1 | 0 | Real procedural, economic, statistical, co-creator, and fail-loud governance cores. Headline neural/RL/vision/3D/texture/world/NPC backends are explicit seams or stand-ins. |
| V8 Ariadne | 3 | 3 | 3 | 2 | 1 | Best bounded self-authoring reference: solve-first case generation, CSP/solver checks, an authoritative eight-gate ReleaseDecision, a separate seven-gate platform content harness, and reproducible acceptance. Journal sequencing and HITL edit/resume remain process-level integration gaps. |
| V9 Metis | 3 | 3 | 3 | 2 | 0 | Strong verification, knowledge/adaptive models, governance, and release-gate libraries. The Python source forge is template-based, the rich LLM verification path is optional/separate, and real learner efficacy remains unproven. |
No row reaches level 4. V1, V6, V8, and V9 should be treated as reference implementations to converge, not as four additional portfolio runtimes to maintain independently.
<!-- AUD-006:START generated -->
2.1 Generated capability-truth summary#
Canonical machine truth: scripts/audit/capability-truth-registry.json. The
generated product-status and capability views are checked against source
comments, TODO ledgers, test counts, provider probes, artifacts, placement
receipts, and release records. Manual analysis is retained only in fields
labelled commentary; it cannot change a machine status.
| Product state | Count |
|---|---|
| seam-only | 3 |
| blocked | 5 |
| red | 1 |
| stale | 1 |
Generated views: evidence/v1-v9/AUD-006/product-status.generated.md and
evidence/v1-v9/AUD-006/audit-summary.generated.md.
<!-- AUD-006:END generated -->
<!-- AUD-007:START generated -->
2.2 Generated test-coherence summary#
The 54 credited Vitest configurations at source
b856ddd2fe3ff0f7ead8751554c21d7d8a010a20 each have one domain-value sample.
Proof-class distribution: unit=35, e2e=4, integration=9, contract=6;
real-provider=0, engine=0, live=0. Weak assertions and uncovered claims
are published beside pass counts in
evidence/v1-v9/AUD-007/test-coherence.generated.md.
<!-- AUD-007:END generated -->
<!-- AUD-008:START generated -->
2.3 Adversarial review reconciliation#
The follow-up review is retained as adversarial provenance, not as a parallel current truth document. Its 11 validated deltas are folded into this audit with explicit dispositions: 7 accepted, 4 corrected, and 0 rejected.
Generated disposition ledger: evidence/v1-v9/AUD-008/reconciliation-status.generated.md (HTML). Open implementation work remains owned by the linked gap-closure tasks.
<!-- AUD-008:END generated -->
3. What is already strong#
3.1 Honest, fail-loud engineering#
The July audits repeatedly found and removed the most dangerous class of defect: systems that stamped generated, translated, trained, reviewed, routed, watermarked, measured, signed, or production-ready without the corresponding operation or evidence. Current implementations commonly require real upstream results or return typed unavailable/pending states. This is a significant strength because it makes the remaining boundary visible and governable.
Examples include:
- V2's translation path refusing an absent provider rather than returning a bracket-tagged pseudo-translation.
- V3's lyric, mastering, fingerprint, rehearsal, registration, motion, and world paths requiring inputs/evidence or labeling drills and fixtures.
- V4's commentary review, watermark, consent, synthesis, and localization records starting false and becoming true only through evidence.
- V5's translation, replay, approval, loudness, and highlight provenance being derived from actual inputs and receipts.
- V6's template, telemetry, environment-promotion, audit, ASR, latency, and live transport boundaries being explicitly labeled.
- V7 replacing fabricated training, win rates, parity, simulation, and quality claims with measurements, validated ingestion, or fail-loud seams.
- V8's formal case constraints and V9's gate evidence refusing signature or release claims that are not backed by real material.
This “truth before throughput” posture should become a portfolio invariant.
3.2 V1’s autonomous quality loop is substantive#
V1 is not merely a single LLM call behind a cron job. Its current stack includes:
- evidence-bearing signals and contextual opportunity selection;
- a bandit with exploration floors;
- multi-idea synthesis and multi-plan search;
- plan judging before expensive generation;
- node-level best-of-N and critique/revise loops;
- pairwise tournaments with both orderings;
- heterogeneous judge panels, reconciliation, calibration, and reward-hack/slop checks;
- content-type publication gates and Isis media governance;
- durable snapshots for review, learning state, direction, and cycle reports;
- human approval, review windows, veto, rejection, targeted tweak requests, standing direction, and post-publication feedback;
- preference-pair and gold-set machinery, champion/challenger testing, staleness checks, and post-publication monitoring primitives.
The BFF creates the plane using live provider availability, advertises only releasable media domains, fails closed without the text provider, hydrates the plane from a durable store when configured, and starts a non-reentrant periodic cycle driver. That is a credible local autonomous system.
The present live scope is nevertheless much narrower than the portfolio goal.
apps/oshun/bff/src/agentic/autonomy-route.ts defines four launch categories:
veritas.story-explainernyx.sky-event-briefnisaba.study-passagetara.seasonal-ritual
The V1 documentation inventories 28 generation tool surfaces across audience tiers. COV-001 supplies the strict, versioned canonical content-slot schema, deterministic identity, ownership-revision rules, and fail-closed v0 migration. COV-002 now supplies a deterministic source seed containing all 28 surface facts, the four current scheduled-category candidates, V1–V9 launch/open/blocked requirements, and V8/V9 gate/artifact/placement constraints. It reports zero complete canonical slots because source policy, locale, quantity, freshness, rights, accessibility, placement, risk, business ownership, and lifecycle facts remain unproven. COV-003 now supplies a strict portfolio dependency DAG, artifact/evidence/slot integrity diagnostics, bounded product projections, fail-closed transitive invalidation, and required-frontier scheduling. It does not populate that graph with product records. COV-004 now adds a deterministic 77-row locale/modality/accessibility/manual-content inventory: all 54 COV-002 surface/category/artifact/placement targets, nine directly sourced product locale observations, and 14 intentionally manual records. Every unknown count, single-locale source, missing locale declaration, source-policy gap, lifecycle gap, and placement gap remains explicit and owned; none becomes a zero or a release claim. COV-005 now supplies a strict release-candidate import/migration contract, complete shipped-observation reconciliation, 100% required-slot, launch-registration, and active-artifact-evidence targets, a target-zero unknown-shipped metric, dependency/invalidation blockers, and a typed fail-closed assertion. No product release path is shown invoking that assertion, and no complete product candidate is populated. Tool availability, seed registration, graph/inventory/closure contracts, and reviewed modality classifications do not prove that all required product content has been created, localized, made accessible, placed, refreshed, or released.
3.3 V8 demonstrates the right bounded-agent pattern#
V8’s design is the portfolio’s clearest example of combining generative flexibility with deterministic ground truth:
The language model proposes; the constraint solver disposes.
Cases pass uniqueness, deductive completeness, fair-play, in-game solve, quality, safety, and canon gates. The pipeline has a durable journal, cost envelopes, adversarial corpora, a reproducible end-to-end acceptance path, and real DPLL/CSP/solver behavior. This is much stronger than relying on an LLM judge to decide whether its own mystery is solvable.
Its three remaining unchecked items are also informative:
- tone parity against a real authored V5 case;
- the 25-case authored V5 regression corpus;
- GA sign-off dependent on those anchors.
The code can establish formal validity without those assets. It cannot establish that generated cases match the intended authored experience. That is exactly the difference between correctness and “amazing.”
3.4 Domain-specific deterministic verification is a portfolio advantage#
The repository often uses the right verifier for the domain rather than asking an LLM to grade everything:
- V2 rollback legality, consent/cook gates, deterministic fallback selection, lipsync and audio math;
- V3 loudness/peak evidence, pose geometry, bone/contact metrics, C2PA signing;
- V4 BFS playability and bounded adaptive-director rules;
- V5 seeded reproducibility, replay hashes, continuity, loudness, rank scoring;
- V6 event-sourced Ori histories, vector-clock reconciliation, policy checks;
- V7 Wilson intervals, Gini/entropy, covering arrays, erosion, measured simulation inputs;
- V8 SAT/CSP/solver and fair-play rules;
- V9 knowledge-graph, lesson, explorable, mastery, and seven-gate contracts.
This aligns with current agent-evaluation guidance: use code graders for verifiable outcomes, model graders for nuanced open-ended qualities, and humans for expert preference and calibration rather than substituting one grader for all three.
3.5 Provenance and policy are designed into the pipelines#
The portfolio consistently treats provenance, consent, rights, grounding, safety, policy, and release as generation concerns rather than post-processing decorations. Real C2PA work exists, especially in V1/V3/V8/V9, and version audits have removed cryptographic labels from structural fingerprints where a real signature did not exist.
That direction matches the current C2PA requirement that newly generated media record a creation action and appropriate digital source type. The remaining task is uniform coverage: every artifact mutation, including a human edit or replacement, must append to the same signed lineage rather than begin an unrelated manual path.
4. End-to-end architecture as it exists#
The repository has several strong but parallel operating loops:
V1 signal -> creative-autonomy director -> creative-orchestrator
-> judges/release gates -> review queue -> editorial publisher
-> preference learner / cycle snapshot
V2–V7 product request -> version-specific planner or deterministic generator
-> version-specific evidence and launch gates
-> provider / service / Unreal seam
V8 request -> Loom / case pipeline -> writers/assets/compiler
-> solver + eight-gate ReleaseDecision
-> separate seven-gate platform content harness -> V5 placement
V9 wonder -> Prometheus / Atlas / Chiron / Theia / Hephaestus
-> lesson + explorable gates -> release evidence
Agentic Studio -> AgentRun envelopes + controls + governance primitives
Studio Authoring -> human blocks, roles, review, branches, suggestions, lifecycle
Yemaya human override -> takeover/checkpoint/emergency-stop primitives
The horizontal libraries are not yet the single runtime under the vertical
loops. The repository's own V_SERIES_PLATFORM_CONSOLIDATION.md remains
Proposed, and its first common GPS contract milestone is unstarted. That is
consistent with code searches: there are multiple orchestrators, journals,
gates, status vocabularies, human evidence records, and product-specific release
models.
Fragmentation is now the primary architecture risk. Adding another capable version-specific agent will increase surface area without closing the portfolio goal.
5. Human authority audit#
5.1 What humans can do today#
The V1 autonomy API provides a credible artifact-level review loop:
- inspect queue items and full provenance;
- approve or reject;
- veto review-window content;
- request a whole-item or node-addressed tweak;
- update standing creative direction;
- submit post-publication feedback;
- observe parked, failed-gate output instead of losing it silently.
The broader libraries also contain:
- pause, resume, cancel, kill, branch selection, fork, retry-with-changes, and operator-step functions over AgentRun envelopes;
- operator action authorization for pause/kill/escalate/override-gate/review/ defer/reassign;
- scoped, expiring, role-bound Lilith policy overrides with recurring-override review triggers;
- Studio Authoring roles, drafts, block editing, branches, suggestions, comments, lifecycle, checklists, replacement recommendations, and human review concepts;
- Yemaya takeover, checkpoints, and emergency-stop primitives.
These are real assets and should be composed rather than rewritten.
The V1 review API is therefore credited at the component layer. Repository-wide
consumer searches found no web, mobile, admin, or other application calling
/v1/agentic/autonomy/*. An external private client is possible, but no
repository or deployed-consumer evidence in this audit proves one.
5.2 What is not wired#
Production-consumer searches found no application binding for the Agentic Studio
authorizeOperatorAction, retryWithChanges, or operatorStep functions
outside their package, and no application import of @yemaya/human-override.
The Lilith override evaluator is pure governance code; the autonomous content
route does not expose it as a durable intervention.
The resulting human boundary is mostly:
agent finishes candidate -> human accepts / refuses / asks agent to retry
The goal requires:
human observes live run
-> pauses before a consequential tool or stage
-> edits brief, plan, tool arguments, source set, or artifact node
-> claims a step and authors it directly, or replaces the artifact
-> runs the same gates on the mixed human+agent lineage
-> resumes, forks, rolls back, publishes, or withdraws
There is also no evidence that human-authored replacement content enters the
autonomy queue's exact ProducedVariant/provenance/learning path. A human can
author in Studio primitives, and a human can critique an autonomous item, but
the two are not one durable co-authoring transaction.
5.3 Required human-intervention contract#
Every portfolio run should implement the following actions through one durable, role-scoped API:
| Action class | Required actions |
|---|---|
| Observe | Stream plan, current step, artifacts, sources, tool calls, budgets, gate state, and lineage. |
| Control | Pause, resume, cancel, emergency stop, choose branch, fork, rewind, retry, roll back publication. |
| Critique | Comment at run/artifact/node/rubric dimension; request changes; attach a reference; change standing direction. |
| Co-author | Edit brief/plan/tool arguments; claim a step; edit or replace an artifact; author from blank; merge a human branch. |
| Govern | Approve/reject/veto; grant a scoped exception; require extra review; change autonomy level prospectively; never bypass non-overridable safety/rights laws. |
| Recover | Resume the exact persisted checkpoint after process restart; preserve idempotency and the full intervention audit. |
An intervention must not be an annotation beside the run. It must change the run state and provenance atomically.
6. Version-by-version findings#
6.1 V1 — reference director, incomplete coverage#
Proven strengths
- Real self-scheduling director with durable review/learning snapshots.
- Strong search, critique/revision, judge, calibration, preference, publishing, and staleness machinery.
- Real provider discovery and fail-closed operation.
- Best portfolio implementation of output-level human review.
- Strong media governance and content-type release gates.
Material gaps
- Four scheduled content categories do not cover all V1 product content, let alone V1–V9.
- The documented 28 generation surfaces describe access/tooling, not required content completion.
- Rich AgentRun controls and human authoring do not bind to the autonomy plane.
- The review API is real and tested, but no in-repository operator surface was found to drive approve/reject/veto/tweak/direction end to end.
CycleReport.realTokenUsageis persisted partial evidence derived from producer planning usage; generator, research-loop, critic, judge, tool, media/GPU, and monetary cost are not accounted end to end.- COV-004 inventories the currently declared locale surfaces and reviewed modality accessibility minimums, but it is not live release evidence. The 28 generation surfaces and four live autonomy categories still declare no directly linked locale counts, and current artifact accessibility conformance remains unproven. COV-005 provides the shared fail-closed reconciliation contract, but no V1 release path or complete candidate is evidenced as using it.
- The repository declares a single-Hetzner-box production topology, but this audit has no live topology probe proving the deployed environment matches it; process restart is not a whole-box recovery test.
- Live provider artifacts, human-gold calibration, engagement attribution, panel weighting, and A/B outcomes remain deployment/data-bound.
- Some stores have in-process lifetimes even though core plane/job snapshots have durable adapters.
Recommended role: canonical director, review, quality, preference, and release control plane. Generalize its contracts; do not copy the implementation into each version.
6.2 V2 — strong real-time plans, incomplete rendered execution#
Proven strengths
- Real commentary orchestration and provider chain, bounded latency, deterministic labeled fallbacks, rollback-safe AI director hints, persona and consent governance, lipsync/ducking calculations, accessibility contracts, post-match selection, and evidence-based factual verification.
- Three AI service gRPC contracts and bindings were subsequently closed after the original audit called the tier spec-only.
Material gaps
- Live commentary, translation, TTS, face rendering, NPC dialogue providers, and service operations remain deploy-bound.
- Isis lipsync/cosmetic/venue plans do not yet prove executor-produced, cooked, in-game assets.
- Human-authored MOTD and patch notes are correctly manual by policy, but their lifecycle is not unified with the agent content ledger.
- No production match cohort proves latency, naturalness, usefulness, or player preference for generated commentary.
Acceptance slice: one real match generates a fact-grounded commentary, localized captions, consented voice, face animation, signed provenance, in-game playback, operator intervention, and player-quality evidence under the actual latency budget.
6.3 V3 — excellent governance shell, missing generation center#
Proven strengths
- Strong Isis music policy, Saraswati tenant/release logic, real c2pa-rs signing, persona grounding, scripted-only instructor containment, pose runtime, concert quality gates, and implemented motion/world validation logic.
- Fabricated lyric/mastering/review/registration claims have been removed.
Material gaps
- Music synthesis, lyric model execution, TTS, motion capture/AI assist, world asset generation, and UE-side experience are mostly provider or deployment seams.
- The current
launch-readiness-release-gate.jsonwas generated at2026-08-02T19:47:48.767Zfrom published source commitfabc9ddc30cbc57bc3e9b2863ce4667dd28c45c9: it is red at 30/41 required commands and 24/39 named gates, withgaReady: false. V3's current product state isredfrom fresh machine evidence; the release claim is not inferred from historical prose. - The current UE 5.5.4 host and
ueagentare available, but V3 has no retained editor modules and therefore requires aV3Editorrebuild before qualified automation or cook execution. - Real pose data, live V1 registration, human sign-offs, and operational concert evidence do not exist locally.
Acceptance slice: produce and stage one original track plus one concert
segment through real providers, rights/fingerprint checks, mix measurements,
human performance review, C2PA, and audience preference testing. UE playback
must run under the designated ue55-linux execution profile against
V3/ue/V3.uproject and retain the evidence identity required by
evidence/v1-v9/AUD-004/unreal-evidence-runbook.md.
6.4 V4 — real procedural gameplay, commentary is orchestration#
Proven strengths
- Real adaptive director, seeded/BFS-validated procgen, roguelike state, crowds, schedules, and compliance-policy validation.
- Commentary pipelines now distinguish deterministic templates from provider output and require real consent, queue, review, and watermark receipts.
Material gaps
- Calliope prose, Psyche voice, native-speaker review, bias review, watermarking, and review queues are not running end to end.
- The final-polish gate remains honestly red on external voice/integration items.
- The 2026-07-15 historical host observation did not compile or run Unreal
automation and recorded both the editor path and
ueagentas absent. Itsnot-run-on-this-hoststatus was never evidence that Unreal is unavailable to the project. Current source-bound evidence now retains a qualified V4 integrated-runtime execution; V2 and V3 separately report exact rebuild-required preflights inevidence/v1-v9/V2-unreal-host-preflight.jsonandevidence/v1-v9/V3-unreal-host-preflight.json. Any further V4 closure still uses the designatedue55-linuxexecution profile againstV4/ue/V4.uprojectand retains the commandlet, target, user, log, report, and output digests defined inevidence/v1-v9/AUD-004/unreal-evidence-runbook.md.
Acceptance slice: one real match produces end-of-match and live-spectator
commentary with provider output, evidence-bound review, audio provenance, UE
automation on the ue55-linux profile, and blinded player/commentator
evaluation.
6.5 V5 — bounded runtime and cook are real; flagship experience remains open#
Proven strengths
- Strong deterministic procgen, run generation, replay storage, highlight ranking, broadcast shot selection, persona adaptation, and daily puzzle rotation.
- A bounded deterministic AI Director with fail-closed policy, authority, rollback, and replay controls passed its selected UE 5.5.4 automation slice.
- A fresh Linux cook produced 448 packages and retained binary
BootMap, AI Director policy, and safety-policy assets with exact digests. - Translation, replay clearance, approvals, continuity, loudness, and generated provenance now fail closed or derive from evidence.
Material gaps
- The retained cook intentionally skipped stage, package, archive, and launch smoke; it does not prove a packaged executable or playable flagship bundle.
- The selected director automation proves bounded technical decisions with a configured authority verifier, not a live external provider or signed human operator decision.
- TTS, recap, STT, and translation inference remain provider seams.
- Human director/cell/rights approvals remain correctly pending.
- V8 depends on authored V5 case data that is not available in the structured form needed for tone and regression comparison.
Acceptance slice: generate one playable side quest or daily detective
experience and one replay highlight bundle, cook them for Linux and validate
them in V5/ue/V5.uproject on the ue55-linux profile, complete human
receipts, and compare them to the best authored baseline.
6.6 V6 — strongest embodied-agent stack, operational seams remain#
Proven strengths
- Real Clio story reconciliation, event-grounded narration boundary, Moirai cognition, Iris gateway, Ninhursag foundry, Ori/legacy histories, welfare and behavior policy, world server, realtime gateway model, and authoring logic.
- Honest template fallback, provenance, environment, telemetry, ASR, and live service labels.
- Substantial Rust and TypeScript domain tests.
Material gaps
- Deployed WebTransport/SFU, vendor ASR/TTS, real V1 promotion, OTLP export, and live Themis dispatch remain external.
- Template fallback remains a meaningful share of the non-provider experience.
- Human authoring exists, but unified in-run takeover and mixed-lineage publishing are not demonstrated.
- No long-lived real-user evidence demonstrates companion coherence, continuity, safety, latency, voice quality, or emotional value.
Acceptance slice: a persistent companion completes a multi-session dialogue/story/biography arc with live speech, durable memory, operator takeover, policy escalation, signed lineage, restart recovery, and human quality/safety assessment.
6.7 V7 — excellent procedural/statistical core, missing headline models#
Proven strengths
- Real economy simulation, statistics, adversarial checks, covering arrays, terrain generation, genomes, LLM co-creator boundary, generation policy, and fail-loud ingestion of measured external results.
- The July audit eliminated 20 fabricated-result seams and clearly separates procedural stand-ins/design priors from trained or measured output.
Material gaps
- RL training, match-based meta discovery, ML terrain generation, reference-image vision, product text-to-3D/texture/world/NPC models, live Themis dispatch, and full-engine parity evidence are not running systems.
- Procedural world generation is not equivalent to the advertised neural creation experience.
- No full content bundle is shown generated, cooked, placed, played, quality judged, and improved from telemetry.
Acceptance slice: one world region with terrain, assets, NPCs, economy, and quests is generated through real backends, cooked, simulated, played, reviewed, signed, and revised from measured failures.
6.8 V8 — closest bounded autonomous vertical, experiential anchor missing#
Proven strengths
- Real case contracts, canon graph, writers' room, pipeline, compiler, CSP,
verifier, solver, asset orchestration, the authoritative eight-gate
ReleaseDecision, the separate seven-gate platform content harness, journal, budgets, adversarial fixtures, and reproducible zero-human acceptance route. - Formal validity does not depend on a model grading itself.
- Current completion source shows 87 checked and 3 unchecked tasks.
Material gaps
- No authored case prose/VO dataset exists for tone parity.
- V5's 25 cases do not have the complete structured deduction graphs/assets required for the intended regression corpus.
- GA sign-off therefore remains honestly blocked.
- Local acceptance and recorded-fixture integrations are not production proof of provider quality, UE placement, player comprehension, fun, or live-service reliability.
- The most recent authenticated bounded record passed 404/404 V8 tests across
exactly 18 configurations and its docs/contract drift command passed. Both
observations now exceed their seven-day evidence SLO, so V8 is
stale; the historical pass is not current release, provider, Unreal-placement, or player-outcome proof.
Recommended role: first portfolio reference slice to take to level 4. It has the smallest semantic gap between code and the stated autonomous-content goal.
6.9 V9 — strong learning engine, no learner-proven product yet#
Proven strengths
- Real V9 packages for knowledge, learner state, pedagogy/tutor, experience, planning, explorables, lesson gates, governance, cross-cutting concerns, and signed release evidence.
- The current V9 checklist reports 46 checked, 11 explicitly waived/external, and 0 unchecked rows; recent changes made G7 signing and gate evidence honest.
- The latest authenticated bounded record passed 188/188 V9 tests across exactly 13 configurations. It now exceeds its seven-day evidence SLO, so it is historical component evidence rather than a current release or learner- efficacy claim.
- Creator steering and a seven-gate product discipline are strong foundations.
Material gaps
- “Ask a wonder,” lesson player, and mastery view are only partial as rendered consumer experiences.
- Closed beta, week-four retention, mobile/offline, embodied face, V2–V8 game bridge, localization at scale, creator payouts, and independent efficacy study remain partial/external.
- A learning system cannot claim top quality from lesson rubrics alone; it needs durable-understanding outcomes with real learners and controlled comparison.
Acceptance slice: a real learner asks a wonder, completes an adaptive lesson and explorable, demonstrates delayed mastery, receives a revised path, and can inspect/correct the lesson through a teacher/operator workflow.
6.10 V10 — strong composition controls, production programming still gated#
Proven strengths
- V10 owns one strict channel contract, policy kernel, daypart engine, batching rules, elevation budget, discretion boundary, and single-holder media lanes. These controls prevent a tenant from manufacturing attention or seizing playback.
- The morning Thread composes exact references to canonical Tara, Veritas, Case Files, Wonder, and Nyx material. It rejects reordered, substituted, cross-date, future, or malformed editions and renders absence explicitly.
- Channel adapters preserve source authority. Stage plans and fixtures cannot become playable catalog rows; Veritas claims retain proof/correction state; first-party media must bind verified bytes and tenant-exact live-media authority.
- Phase A invariants forbid a second timeline, autoplay, completion debt, engagement funnels, and a duplicate live-media stack. V10's autonomous-content role is therefore bounded composition and delivery, not unconstrained generation.
Material gaps
- Local source, render, and ingest gates do not establish deployed daily programming. Several authorities still need production configuration, canonical rows, long-running hosts, or retained runtime receipts.
- Veritas and V3 Stage first-party operations remain rights/business gates. The
production
first-party-liveclass must stay disabled until those exact tenant paths are deployed and observed. - The Phase A release decision depends on the week-long human dogfood exit test. Automated tests cannot prove the companion remains useful and calm after novelty fades.
- Ori is the exceptional generative path. It is separately consented, source-bounded, safety-filtered, budgeted, and default-off; its release truth cannot be generalized to every V10 channel.
Acceptance slice: a retained production-day record shows fresh canonical editions composed into the Thread and channel surfaces, exact unavailable states for missing sources, no unpressed media action, bounded calm metrics, and a human-approved seven-day dogfood decision.
7. SOTA comparison#
7.1 Workflow selection and orchestration#
Current practice distinguishes fixed workflows from agents that dynamically choose steps and tools. Anthropic's Building effective agents recommends simple composable patterns, environment ground truth at each step, human checkpoints, stopping conditions, transparency, and careful tool design. The repository performs well on decomposition, routing, parallel judging, evaluator/optimizer loops, and bounded deterministic workflows.
The portfolio underperforms on one crucial property: a shared environmental ground-truth loop. Each product defines “done” differently, and several stop at a request, plan, receipt contract, or structural evaluator before the rendered environment confirms the result.
Recommendation: retain fixed workflows for well-defined pipelines (V8 case compilation, media governance, V9 lesson gates) and allow dynamic agent planning only inside bounded, observable stages. More autonomy is not automatically better. Agent decisions should be justified by measurable outcome gains over a simpler workflow.
7.2 Multi-agent design#
Anthropic's multi-agent research system report supports orchestrator/worker parallelism for breadth-heavy, independently searchable work, while reporting high token cost and poor fit for tightly coupled tasks. The repository already uses sensible multi-sample, panel, and orchestration patterns.
It should not respond to this audit by spawning more agents by default. Parallelize independent ideation, research, specialist critique, localization, and adversarial evaluation. Keep solver-coupled case construction, shared world state, publication transactions, and safety decisions in deterministic or strongly serialized workflows.
7.3 Iterative improvement and search#
The portfolio already contains critique/revise, best-of-N, plan sampling, replanning, bandits, preference logs, gold sets, and champion/challenger mechanisms. These align with research directions including Self-Refine, Reflexion, Tree of Thoughts, and Language Agent Tree Search.
The gap is closed-loop promotion from real trajectories. V1 has the primitives, but there is no portfolio prompt/policy optimizer that:
- selects failed or high-disagreement live trajectories;
- produces a natural-language diagnosis;
- proposes prompt, rubric, tool, routing, or policy changes;
- evaluates them on capability, regression, safety, and cost suites;
- promotes only a statistically superior Pareto candidate;
- rolls back on production drift.
GEPA is a useful research reference for reflective prompt evolution and Pareto search, but it should be adopted as a sandboxed optimizer, not allowed to rewrite production policy directly.
7.4 Evaluation quality#
Anthropic's current agent eval guidance recommends a mix of code, model, and human graders; separates capability from regression suites; requires repeated trials for nondeterministic systems; and combines offline evals with production monitoring, user feedback, A/B tests, manual transcript review, and systematic human evaluation.
The repository is strong in code graders and increasingly strong in model graders. It is weak in real human graders and production outcomes. That makes the judge plane sophisticated but incompletely calibrated.
Risks that still require evidence:
- same-family generator/judge bias;
- positional and verbosity bias;
- reward-model overfitting and “judge pleasing” artifacts;
- rubric drift as models change;
- low inter-rater agreement hidden by aggregate scores;
- a candidate passing once but failing repeatedly.
AgentRewardBench is relevant because it evaluates automatic trajectory graders against expert judgments and reinforces that no one model judge should be treated as universally reliable. The portfolio should report both pass@k (can it succeed at least once?) and pass^k (does it succeed reliably on every trial?) where nondeterminism matters.
7.5 Multimodal and experiential quality#
FFmpeg probes, image/video/audio measurements, VLM critics, cross-modal checks, and provenance gates are good foundations. They do not establish perceptual or creative excellence.
Add domain-appropriate batteries:
- image composition and prompt alignment informed by GenEval, plus aesthetic, identity, typography, cultural, and brand-specific human evaluation;
- video quality informed by VBench++, plus continuity, direction, edit rhythm, lip sync, story comprehension, and blinded viewer preference;
- speech/music: intelligibility, speaker/consent match, timing, loudness, artifact detection, musical structure, originality/fingerprint safety, and expert/listener preference;
- 3D/game content: topology, rigging, collision, LOD, cook, performance, playability, novelty, accessibility, fun, and player comprehension;
- education: source faithfulness, misconception repair, transfer, delayed retention, and teacher/learner preference.
External benchmarks are diagnostic, not release authority. Each version needs an internal gold set built from its best human-authored and best shipped work.
7.6 Durable human interrupts#
Current agent runtimes treat human approval as a persisted interruption, not an ordinary callback. The OpenAI Agents SDK human-in-the-loop model supports tool approvals and serialized state for pause/approve/reject/resume. LangGraph interrupts require a durable checkpointer and stable thread identifier, and warn that resumed nodes may re-execute—making idempotency essential.
The repository has most control primitives but lacks their common durable host. Every provider/tool side effect needs an idempotency key derived from run, step, attempt, and artifact lineage. Human edits need optimistic concurrency/version checks so an old approval cannot release a newly changed artifact.
7.7 Tracing and operations#
V1 AgentRun evidence and cost ledgers are strong local concepts; V6 explicitly labels OTLP as not yet exported. The portfolio lacks one cross-version trace schema.
Adopt OpenTelemetry's GenAI semantic conventions for agent, workflow, tool, retrieval, generation, provider, model, token, and conversation operations, extended with content slot, artifact, gate, rights, policy, human intervention, and publication identifiers. OpenAI's agent tracing model is a useful reference for connecting generations, tools, handoffs, guardrails, and custom events.
Trace payloads must default to metadata because prompts, source text, likeness, voice, learner data, and private memories may be sensitive.
7.8 Security, authority, and provenance#
The repository has meaningful prompt-injection fences, policy gates, kill switches, scoped roles, evidence trails, and provenance work. Portfolio closure requires a single capability broker for every delegated tool:
- least-privilege grants bound to agent, tenant, run, purpose, resource, and expiry;
- no token passthrough;
- audience-bound tokens and secure storage;
- egress controls and source/content trust zones;
- approval before high-impact tools;
- immutable audit of grant, use, result, and revocation;
- sandboxing for untrusted content and generated code/assets.
The current MCP authorization specification and MCP security guidance are useful minimum references for resource-bound authorization and token-handling risk. NIST's AI RMF Core calls for defined human-AI oversight roles and post-deployment monitoring with appeal, override, incident response, recovery, and change management. The NIST Generative AI Profile adds system inventory, provenance, human-oversight, model-version, and TEVV retention expectations.
For media lineage, conform every exporter to C2PA Content Credentials 2.2, including creation/open actions, digital source type, ingredients, software agents, transformations, and human edits.
8. Prioritized gap register#
8.1 P0 — required before claiming the goal#
| ID | Gap | Current evidence | Required acceptance criterion |
|---|---|---|---|
| P0-01 | Canonical slot schema is not yet a populated closure ledger | COV-001 implements the strict slot lifecycle; COV-002 mechanically preserves the V1 surface/category facts, V1–V9 launch/open/blocked ledgers, and V8/V9 gates/artifacts/placements while reporting zero complete slots; COV-003 implements dependency/invalidation/view/scheduling contracts; COV-004 deterministically reports 77 inventory rows and nullable quantity aggregates by all required dimensions; COV-005 implements fail-closed candidate reconciliation, 100% coverage targets, and target-zero unknown-shipped metrics. Product canonical slots, graph records, complete shipped observations, release refs/evidence, candidates, and release-path integrations remain unpopulated. | Canonicalize the seed; 100% of required V1–V9 slots and dependencies are machine-readable; every slot reports required/current/missing/stale/blocked with artifact and release refs. Unknown/unregistered shipped content is a release failure in every product release path. |
| P0-02 | Common durable reference/effect spine exists; product integration is absent | RUN-001 implements the strict common aggregate. RUN-002 adds hash-chained append/replay, atomic allocation, checkpoint migration, cross-process resume, archive retention, and a six-point SIGKILL matrix. RUN-003 adds tuple-bound effect identity, durable intent/claim/result journals, provider reconciliation before retry, conservative delivery policies, owner-relabel protection, worker leases, and duplicate/lost-ack/timeout/partial/restart/failover/concurrency conformance. V1, Agentic Studio, Yemaya, V8, V9, and other products still use parallel records; no product database adapter, live provider-effect consumer, migration, or deployed common scheduler is evidenced. |
All autonomous pipelines emit one versioned Run/Step/Attempt/Artifact/Intervention schema with durable checkpoints, idempotent effects, lineage, and compatible state transitions. |
| P0-03 | Human takeover/co-authoring not production-wired | V1 review actions are a real tested API component, but no in-repository operator UI consumer was found and no deployed external consumer is evidenced; pause/fork/retry/operator-step and human-override packages have no autonomous-runtime consumer. | A human-usable operator surface drives approve/reject/veto/tweak/direction and can pause a live run, edit/claim/replace a node, run the same gates, resume after restart, and preserve signed mixed human+agent lineage. |
| P0-04 | “Amazing” has no outcome proof | Strong automated tests and judges; missing cross-version blinded human comparison and real user outcomes. | Each flagship slice beats or is non-inferior to the best authored baseline on blinded SME/user preference with predeclared statistics, while all hard gates remain green. |
| P0-05 | Provider/render/cook gaps prevent complete artifacts | V3–V7 audits list actual inference, transport, asset, engine, or deployment seams; V9 consumer UI is partial. | One real provider-backed, published/placed, telemetry-bearing vertical slice per version passes its target runtime and canary. |
| P0-06 | Capability truth coverage is incomplete | AUD-006 now generates V_SERIES.md, the capability matrix, product status, and audit summary from 38 typed evidence sources. That seed covers all ten products and 22 capabilities, not the full content-slot graph; V8's last authenticated test and contract-drift observations passed but are stale beyond their seven-day SLO. |
Extend the generated registry to every required content slot while retaining package contracts, integration tests, provider probes, artifacts, placement, release evidence, freshness SLOs, and contradiction-blocking CI. |
| P0-07 | Human-gold data is missing or incomplete | V8 authored cases unavailable; V9 beta/efficacy pending; V1 learning awaits live human data. | Versioned, licensed gold/preference corpora exist for each content family, with rater policy, disagreement, lineage, retention, and contamination controls. |
| P0-08 | Portfolio-wide tool authority is not proven | Strong local governance but no shared product-spanning capability broker. | Every consequential tool invocation is least-privilege, tenant/run/purpose-bound, auditable, revocable, injection-tested, and approval-gated where policy requires. |
| P0-09 | Declared runtime controls are not enforced end to end | V1's Agentic Studio route checks scopes and a pre-call budget but does not consume it or append tool/cost ledger entries. The autonomy plane persists partial planner realTokenUsage, but lacks generator/research/critic/judge/tool/media/GPU/dollar accounting and hard full-path spend caps. V9 persists plans but has no DAG executor. |
Adversarial integration tests prove that multi-step calls consume budget, persist every attempted effect and complete measured cost, enforce resolved grants/egress/secrets, stop on kill/pause, resume durably, and never repeat a mutating effect. |
| P0-10 | Prompt trust boundaries are inconsistent | The V1 content-quality judge fences untrusted artifacts, but most generators, critics, refiners, source-grounded tutors, and NPC/commentary paths concatenate source, history, prior output, or reviewer text without equivalent trust delimiters or instruction hierarchy. | Every model boundary labels and isolates untrusted text; prompt-injection suites cover direct, indirect, encoded, multimodal, and cross-item memory attacks; policy/tool decisions never depend only on model obedience. |
| P0-11 | Release evidence can overstate what was proved | V1's default Sophia binding scores declared citations without retrieval and auto-satisfies rights; V8's case-report “signature” is an unkeyed hash and its pre-download asset binding is unsigned/locator-hashed; V9's basic lesson validator accepts citation-shaped strings and its publication percentages are structural proxies. | Release evidence is typed by claim: format, retrieval, entailment, rights receipt, byte-bound authenticated signature, human approval, and placement are distinct, non-substitutable proofs; weak proxies cannot satisfy stronger gates. |
8.2 P1 — required for SOTA production operation#
| ID | Gap | Required outcome |
|---|---|---|
| P1-01 | Unified evaluation harness | Standard task/trial/transcript/outcome/grader records; capability and regression suites; repeated trials; code/model/human graders; pass@k and pass^k; model/provider/version slicing. |
| P1-02 | Judge calibration and anti-overfitting | Blind both-order evaluation, heterogeneous providers, human calibration, disagreement routing, position/length/style bias tests, adversarial judge prompts, and held-out release sets. |
| P1-03 | Portfolio learning loop | Human feedback, gate failures, edits, engagement, and downstream outcomes create versioned learning examples; improvements pass offline eval, shadow, canary, and champion/challenger promotion. |
| P1-04 | Safe prompt/policy evolution | A GEPA-like reflective optimizer proposes changes in a sandbox and optimizes a quality/cost/safety Pareto frontier; humans approve policy/rubric changes; automatic rollback is available. |
| P1-05 | Cross-product canon and asset lineage | One entity/source/claim/character/world/asset graph links V1–V9 artifacts, ingredients, releases, rights, versions, and invalidation. Product-specific graphs remain bounded views. |
| P1-06 | Multimodal experiential evaluation | Real image/video/audio/3D/gameplay/education benchmark batteries plus human preference, not only file validity or proxy scores. |
| P1-07 | Cross-version observability | OTel GenAI traces, slot/run/artifact/gate/intervention correlations, cost/latency/retry/SLO dashboards, privacy controls, and replayable incident evidence. |
| P1-08 | Durable side effects and recovery | Verify the live topology against the repository-declared single-Hetzner-box design; add exactly-once publication or safe idempotent retries, checkpoint versioning, stale-approval invalidation, provider reconciliation, and process-loss plus whole-box backup/restore drills. |
| P1-09 | Risk-tiered autonomy policy | Every content slot declares manual, copilot, approval, review-window, or full-auto policy; high-risk rights/safety/claims/children/identity actions cannot become full-auto through a local override. |
| P1-10 | Continuous re-certification | Source, canon, model, rubric, rights, policy, and provider changes invalidate dependent artifacts and schedule selective regeneration/review. |
8.3 P2 — frontier improvements after the control plane is real#
- Dynamic specialist-agent teams only where independent parallel work shows a measured gain over simpler workflows.
- Learned quality/cost/latency routers using production outcomes with hard exploration and safety constraints.
- Hierarchical taste models by product, tenant, locale, expert cohort, and user, with privacy-preserving aggregation and no collapse into majority taste.
- World-model, RL, and embodied generation for V7/V9 only after real simulation and outcome harnesses exist.
- Distilled specialist judges and generators to reduce cost after fidelity to the calibrated reference system is established.
9. Target operating architecture#
CANONICAL CONTENT COVERAGE GRAPH
requirements / slots / locales / rights / freshness / policies
|
v
PORTFOLIO DIRECTOR
opportunity ranking / dependency planning / budget / scheduling
|
+-------------------+-------------------+
| |
v v
BOUNDED WORKFLOWS DYNAMIC AGENT WORKERS
deterministic stages, solvers, research, ideation, critique,
compilers, render/cook pipelines localization, recovery proposals
| |
+-------------------+-------------------+
v
DURABLE RUN / EVENT SPINE
run -> step -> attempt -> tool -> artifact -> intervention
checkpoint / idempotency / budgets / grants / trace / audit
|
+--------------+--------------+
| |
v v
HUMAN CONTROL PLANE QUALITY / SAFETY PLANE
observe, pause, edit, claim, deterministic graders, solvers,
replace, approve, veto, fork, model panels, gold sets, rights,
override, rollback, author C2PA, accessibility, runtime tests
| |
+--------------+--------------+
v
PUBLICATION + PLACEMENT LAW
signed release, canary, rollback, revocation, freshness
|
v
PRODUCT RUNTIMES / REAL USERS / TELEMETRY
|
v
LEARNING & RECERTIFICATION
preference / edits / failures / outcomes / drift / optimizer
|
+-----> coverage graph + director
9.1 Canonical content coverage contract#
A minimal contract should look like this:
interface ContentSlot {
slotId: string;
product: 'V1' | 'V2' | 'V3' | 'V4' | 'V5' | 'V6' | 'V7' | 'V8' | 'V9';
surfaceId: string;
contentKind: string;
modalities: readonly string[];
locale: string;
audiencePolicyRef: string;
requiredQuantity: number;
freshnessSloSeconds: number | null;
sourcePolicy: 'human' | 'agent' | 'hybrid';
autonomyPolicy:
| 'manual'
| 'copilot'
| 'approval'
| 'review-window'
| 'full-auto';
pipelineRef: string;
gateSuiteRef: string;
rightsPolicyRef: string;
placementTarget: string;
}
interface ContentSlotState {
slotId: string;
state:
| 'missing'
| 'planned'
| 'running'
| 'review'
| 'released'
| 'stale'
| 'blocked';
activeArtifactRefs: readonly string[];
latestRunRef: string | null;
blockerRefs: readonly string[];
nextRequiredAt: string | null;
}
COV-003 implements the strict graph above artifact-local ProvenanceBundle
lineage. It represents source, translation, consented voice, generation,
watermark, caption, provenance, cook, render, placement, review, freshness,
canon, entity, claim, character, world, asset, policy, rights, model, rubric,
and provider dependencies. Product projections expand only in-scope records and
one-hop redacted boundary stubs. Source, canon, policy, rights, model, rubric,
provider, or placement changes propagate through every blocking dependency to
invalidate artifacts, revoke approvals, and withdraw shipped placements. The
analyzer rejects cycles, orphan artifacts, unknown shipped content, dangling
endpoints or evidence, duplicate slots, and scope mismatches. The scheduler
selects only the ready missing/stale/invalidated frontier feeding an active
required slot and explicitly defers optional, disconnected, blocked, and
capacity-excess work.
This is a versioned contract and deterministic engine, not a populated live portfolio graph. COV-004 adds a strict source-digest-bound inventory over every COV-002 surface/category/artifact/placement target, nine product locale surfaces, and manual MOTD/patch-note/replacement/news records. It distinguishes enumerated identities, count-only declarations, and undeclared locale coverage; derives reviewed accessibility minimums from each row's modality; and reports required/current/missing/stale/blocked as known sums plus unknown-row counts by product, surface, modality, locale, source policy, and accessibility requirement. It still reports zero complete canonical slots. COV-005 and the product slices must canonicalize the rows, populate graph/release records, and prove deployed invalidation behavior. COV-005 now provides strict release-candidate import/upcast, complete shipped-observation reconciliation, coverage/unknown-content metrics, dependency and invalidation blockers, and a typed fail-closed assertion. Product slices still must build authoritative candidates and invoke that assertion in every release path.
9.2 Common run protocol#
Adopt one envelope with:
- immutable run identity and parent/fork relationships;
- explicit plan version and per-step state;
- attempt identity and deterministic idempotency keys;
- tool request, approval, grant, result, and side-effect reconciliation;
- input/source/canon/policy/model/rubric/provider versions;
- artifact graph with node addresses and content hashes;
- gate verdicts with real evidence refs;
- human comments, edits, replacements, approvals, and override scope;
- cost, latency, token, GPU, and environmental budgets;
- checkpoint schema version and migration;
- publication, placement, canary, rollback, withdrawal, and invalidation state.
Existing V1 AgentRun and creative-autonomy types should seed this contract. V8's durable stage journal and V9's dependency/governance contracts should become adapters/conformance tests, not separate competing standards.
RUN-001 implements that strict common V1 aggregate, including immutable derived identity, content-slot risk/autonomy scope, plan DAG and state history, deterministic attempts, the full request/approval/grant/invocation/result/ side-effect/reconciliation chain, versioned evidence bindings, artifacts, gates, interventions, cost/latency/token/GPU/environment measurements, and publication through recertification lifecycle. RUN-002 adds the portable hash-chained event, checkpoint-migration, deterministic replay, archive-retention, and intent-before-effect protocol plus a crash-recoverable file adapter. RUN-003 adds tuple-bound effect identities, a hash-chained provider-effect journal, intent/claim-before-call ordering, authoritative results, reconciliation before retry, safe partial resume, fail-closed ambiguity, and worker-lease failover. Its exactly-once proof uses a keyed conformance provider, not a live product provider. These common contracts and the shared-filesystem adapter remain reference proof: RUN-004 owns product/database adapters and live provider consumers, while RUN-005 owns the shared scheduler/runtime control plane.
9.3 Source policy and autonomy ladder#
The portfolio should explicitly support five levels:
| Level | Behavior |
|---|---|
| Manual | Human authors; agents may validate only. |
| Copilot | Human owns the draft; agents research, suggest, transform, or generate selected nodes. |
| Approval | Agent creates the complete candidate; human approval is required. |
| Review window | Agent may release after a durable veto window if policy and gates permit. |
| Full auto | Agent releases and monitors automatically; humans retain withdrawal/kill authority. |
Autonomy is attached to the content slot and action, not merely the agent. Named claims, minors, crisis/health, real-person identity, cloned voice, scholarship authority, high-value spend, legal rights, and irreversible external actions retain mandatory human or institutional authority.
10. Evaluation and “amazing” quality program#
10.1 Three independent gate classes#
- Hard correctness and governance: solver validity, factual grounding, safety, rights, consent, provenance, accessibility, schema, cook, performance, placement. One failure blocks release.
- Calibrated quality: craft rubrics, reference comparisons, novelty, coherence, style, cross-modal consistency, judge panels, and uncertainty.
- Human/product outcomes: blinded expert/user preference, task success, comprehension, retention, fun, usefulness, trust, downstream behavior, and longitudinal impact.
The third class cannot be inferred from the first two.
10.2 Recommended launch thresholds#
These are proposed starting targets and should be tuned per domain before use:
- Coverage: 100% of launch requirements registered; 100% of active artifacts have release, placement, gate, rights, and lineage refs; zero silent unknowns.
- Hard gates: 100% pass for safety, rights, consent, provenance, formal validity, and required runtime/accessibility checks.
- Reliability: critical deterministic tasks pass^5 at or above 0.95; lower results block or lower autonomy even if pass@5 is high.
- Human quality: blinded candidate preference is non-inferior to the best authored baseline with the lower 95% confidence bound above 0.50; the “amazing” target should then require a sustained win rate above 0.60 and lower bound above 0.55 on the target audience, not merely a tie.
- Judge calibration: report human-judge rank correlation, per-dimension confusion, calibration error, and inter-rater agreement; route high disagreement to humans. Do not set one universal threshold before collecting domain data.
- Human efficiency: at least 80% of approval-policy artifacts accepted without full regeneration, while measuring human minutes and edit distance so rubber-stamp approval cannot look efficient.
- Recovery: 100% of checkpoint/restart, duplicate-delivery, stale-approval, provider-timeout, kill-switch, and publication-rollback drills preserve lineage and avoid duplicated side effects.
- Observability: every released artifact traces to slot, run, steps, tools, sources, interventions, gates, model/provider versions, spend, and placement.
10.3 Gold-set design#
Each content family needs:
- best authored positives, not average legacy content;
- known-bad adversarial examples for every hard gate;
- borderline/disagreement examples;
- culturally and linguistically diverse cases;
- real provider/render/cook outputs;
- held-out release items never used for prompt or rubric optimization;
- versioned rater instructions and expertise/locale metadata;
- rights and retention records;
- contamination checks against training and prompt examples.
V8's missing V5 corpus and V9's missing learner cohort should be treated as P0 data-acquisition work, not documentation cleanup.
10.4 Learning loop#
release/canary
-> collect explicit preference, edits, vetoes, gate failures, outcomes
-> attribute to slot/run/step/model/prompt/rubric/provider
-> sample hard, novel, high-impact, and high-disagreement trajectories
-> diagnose failure and propose changes
-> capability + regression + safety + cost evaluation
-> human review of policy/rubric changes
-> shadow -> canary -> champion/challenger
-> promote or roll back
-> invalidate/re-certify dependent artifacts when required
Never train directly on raw engagement as a quality target. Retention and clicks can reward sensationalism, repetition, or dependency. Use them as contextual signals constrained by explicit product outcomes, safety, user intent, and human preference.
11. Delivery sequence and exit criteria#
This is a dependency order, not a calendar estimate.
Phase A — establish portfolio truth#
- Create the content coverage contract and registry.
- Seed it mechanically from V1's 28-surface catalog and four launch categories, the V1–V9 TODO ledgers, and V8/V9 gate, artifact, and placement contracts. COV-002 completed the source-preserving seed, COV-003 completed the strict dependency/invalidation/view/scheduling model, COV-004 completed the source-bound locale/modality/accessibility/manual-content inventory with unknown-preserving aggregates, and COV-005 completed the shared fail-closed closure contract and metrics. Product slices still own canonical slot mapping, graph population, resolved quantities, artifact conformance, lifecycle/placement evidence, authoritative shipped observations, and release-path integration.
- Generate
V_SERIES.md, the capability matrix, and portfolio status views from the registry; add contradiction/drift CI. - Define the common run/event/intervention schemas and conformance suite. RUN-001 completed the strict shared schema and focused contract suite; RUN-002 completed the common durability protocol and crash-recoverable reference adapter; RUN-003 completed the portable provider-effect identity, journal, reconciliation, and safe-retry coordinator. RUN-004 and RUN-005 still own product/database/provider adapters, scheduling, approval binding, and deployed runtime integration.
Exit: any operator can answer exactly what remains unfilled, why, which artifact is active, and what evidence made it releasable. No prose ledger can claim more than the registry.
Phase B — compose human authority#
- Bind Agentic Studio run controls to the V1 BFF and common run store.
- Bind scoped Lilith overrides and Yemaya emergency/takeover semantics.
- Add durable actions for edit plan, edit tool request, claim step, replace node/artifact, merge human branch, author from blank, and rollback.
- Run every human-authored or mixed artifact through the same release and C2PA lineage path.
- Add restart, race, stale-approval, duplicate-effect, and authorization tests.
Exit: a human can interrupt a live generation at any stage, contribute content, resume it on another process, and produce a correctly attributed release without bypassing policy.
Phase C — prove one real vertical per version#
Use the acceptance slices in section 6. Prioritize:
- V8, because formal verification and local acceptance are already strong.
- V1, to prove the reusable director, human control, and learning plane.
- V9, to add real learner and teacher outcomes.
- V6, to validate long-lived embodied/memory behavior.
- V2–V5 and V7, closing provider, cooked-asset, Unreal, and simulation gaps with bounded flagship slices.
Each slice must use real providers or actual engine/runtime outputs, not contract fixtures; it must include a human takeover drill, baseline comparison, signed lineage, placement, telemetry, canary, and rollback.
Exit: nine level-4 flagship rows exist even if non-flagship coverage remains lower. The portfolio may claim “production-proven reference slices,” not yet “everything.”
Phase D — calibrate and automate improvement#
- Build expert/user gold and preference datasets.
- Calibrate judge panels and publish model/rubric cards.
- Standardize capability/regression/reliability evaluation.
- Feed live interventions and outcomes into champion/challenger learning.
- Add sandboxed reflective prompt/policy optimization.
Exit: promoted changes show statistically credible quality gain with no hard gate, reliability, cost, or subgroup regression; automatic rollback works.
Phase E — close and maintain “everything”#
- Let the director schedule missing, stale, invalidated, or underperforming content slots.
- Scale locales/modalities and provider redundancy.
- Add continuous recertification on source/canon/model/policy/rights drift.
- Raise autonomy per slot only after sufficient reliability and human evidence.
- Review portfolio health, incidents, overrides, and quality distribution, not only mean scores.
Exit: all required slots are current and released or carry an explicit, owned blocker; quality and human-authority SLOs hold over time; incident and drift exercises pass.
12. Portfolio release gates#
These portfolio gates sit above, and do not replace, V8's authoritative
eight-gate ReleaseDecision, its separate seven-gate platform content harness,
or V9's product-specific gates.
| Gate | Blocks release when |
|---|---|
| R0 Coverage | Slot is unregistered, wrong source/autonomy policy, short of required quantity, stale, or lacks a placement target. |
| R1 Ground truth | Sources, canon, solver/simulation, factual claims, or input evidence are absent, stale, contradictory, or unverified. |
| R2 Artifact completeness | Any required modality, locale, caption, asset, dependency, cook, or runtime placement is only planned or missing. |
| R3 Quality | Calibrated quality floor fails, uncertainty/disagreement exceeds policy, or human baseline evidence is required but absent. |
| R4 Safety and integrity | Safety, child/crisis, cultural, academic, gameplay fairness, prompt injection, or abuse gates fail. |
| R5 Rights and provenance | Consent, likeness/voice, license, fingerprint, ingredient/action lineage, watermark, signature, or attribution evidence fails. |
| R6 Runtime quality | Accessibility, latency, cost, performance, compatibility, deterministic replay, render/cook, or reliability SLO fails. |
| R7 Human authority | Required approval/review window is incomplete, an override is invalid, a stale approval exists after mutation, or takeover/rollback is unavailable. |
| R8 Live canary | Target user outcome, incident, drift, subgroup, or operational canary is red, or rollback/recovery evidence is missing. |
Every verdict must include evidence references rather than a boolean alone. A gate implementation that cannot fail on an adversarial fixture is itself a release failure.
13. Metrics that should be on the operator dashboard#
Coverage#
- required/current/missing/stale/blocked slots by version, surface, modality, locale, and source policy;
- released quantity versus requirement;
- median and tail content age;
- dependency and invalidation backlog;
- percentage of shipped artifacts absent from the registry (target zero).
Autonomy and human work#
- untouched release rate by autonomy tier;
- human minutes, comments, edit distance, replacement rate, and regeneration count per accepted artifact;
- approval, veto, rejection, tweak, takeover, fork, and rollback rates;
- takeover acknowledgment and checkpoint-resume latency;
- percentage of human edits incorporated correctly;
- recurring override and policy-review rates;
- operator disagreement and appeal outcomes.
Quality#
- blinded pairwise win/tie/loss against best authored and previous champion;
- hard-gate pass rate and defect escape rate;
- pass@k and pass^k;
- judge-human correlation, calibration, bias, variance, and disagreement;
- novelty/diversity versus repetition and canon/style drift;
- content-family outcomes: solve/fun, comprehension, mastery/retention, usefulness/trust, audio/visual experience.
Operations#
- end-to-end completion and published-placement rate;
- latency and cost by stage, model, provider, modality, and quality tier;
- retries, timeouts, provider failover, duplicate-effect prevention, and restart recovery;
- trace and provenance completeness;
- kill-switch, incident, withdrawal, and rollback time;
- canary regression and recertification backlog.
Aggregate means are insufficient. Report tails, product/locale/audience slices, and the distribution of quality so a high average cannot hide catastrophic outliers.
14. Documentation and governance finding#
Two current umbrella documents are materially stale:
V_SERIES.mdis dated 2026-06-12 and still describes V8 as spec-first and V9 connective tissue as not built, while current V8/V9 packages and completion ledgers contradict those statements.CONTENT_GENERATION_CAPABILITY_MATRIX.mdis dated 2026-06-18 and lists image, video, podcast, prose, and grounding gaps that later V1 work partially or fully changed.
Stale truth sources are not a cosmetic issue in an autonomous system. Agents will plan against them, humans will audit against them, and release decisions may use them. Capability evidence should be generated from executable probes and artifact records, with manual narrative clearly labeled as commentary.
The proposed V-Series consolidation document is itself valuable evidence: it recognizes that common GPS contracts and platform seams are missing. Its status should remain Proposed until implementation exists, but its first milestone should be elevated to P0 and expanded from game platform services to the content run/intervention/coverage contracts in this report.
15. Deep source-level architecture and enforcement audit#
15.1 Coverage and interpretation#
The tables below distinguish architecture that is declared, logic that is implemented as a reusable component, and behavior that is actually invoked by the product-facing path. This is essential for agentic systems: a robust grant resolver, budget type, verifier, or human-interrupt state does not govern a run merely because both exist in the same package.
The deep pass covered the following direct execution surfaces:
- V1 autonomous director, ideation, portfolio selection, generation DAG, critique/revise, quality judging, media dispatch, publication readiness, AgentRun route, tool governance, lifecycle storage, and human review.
- V2 low-latency and post-match commentary plus the shared Hathor NPC dialogue runtime used by character systems.
- V3's generation-center provider contracts and the local governance, provenance, pose, and concert-quality logic around them.
- V4 and V5 director/procedural/replay/commentary provider boundaries referenced by their current audits.
- V6 cognition gateway, run manager, story narration, context composition, and authoring controls.
- V7 Forge Assist, capability policy, Mawu Studio generation policy, procedural generation, and simulation/statistics cores.
- V8 solve-first case construction, DPLL/ASP gates, player simulation, asset briefs, safety, provenance aggregation, journal, and HITL queue.
- V9 source ingestion/outline/generation, rich LLM content client, tutoring prompt, agent registry/planner, verification stack, learner mastery, BKT/GKT, FSRS, and publication gates.
15.2 Shared V1 autonomous director: actual control flow#
The executable V1 loop in
libs/oshun/creative-autonomy/src/director/director.ts is a bounded,
state-bearing workflow rather than an unconstrained agent:
- It checks the kill switch before work, resolves expired review windows, and attempts publication of approved items. One publication failure is isolated so a later cycle can retry.
- It learns newly resolved human outcomes once, then decays the Thompson
posterior by
0.98so old preferences lose influence. - Human-requested tweak jobs run before new ideation. A tweak is consumed only after it has been judged; provider failures leave it retryable.
- Sensed signals produce deterministic ideas and optional LLM ideas. LLM ideas are admitted only when their category is known, their topic is not banned, and they overlap an observed signal.
- Cooldowns, minimum opportunity score, portfolio selection, and an AIMD
generation governor bound work. Defaults are three items, two variants,
quality bar
0.75, confidence floor0.5, a six-hour review window, at most 24 signals, two angles, one retake, and a seven-day topic cooldown. - High-stakes work adds a candidate and doubles judge samples; low-stakes work removes one candidate. Candidates are generated sequentially, tournament ranked, rubric gated, and retried only on named weak dimensions. If a retry is worse or fails, the best previously judged take is retained.
- Review state is compacted: the queue retains 200 recent records, compacts after seven days, and drops after 90 days. Learned outcome IDs are snapshotted to prevent double learning.
This is a strong workflow-selection pattern: deterministic admission and hard policy surround probabilistic generation. The remaining weakness is not the director algorithm; it is the narrow four-category launch inventory and the separate, incompletely enforced Agentic Studio execution route.
15.3 Agentic Studio: primitives versus the invoked route#
The AgentRun envelope models plans, branches, tool calls, artifacts, evidence,
cost, rationale, output, provenance, and states including pause, approval,
tool-wait, branch, retry, cancellation, and kill. run-controls.ts implements
pause/resume/cancel/kill/fork/retry/operator-step transformations.
tool-grants.ts, grants/resolver.ts, and tool-isolation.ts model expiring
grants, scope specificity, revocation cascades, network egress, SSRF defenses,
and secret isolation.
The product route in apps/oshun/bff/src/agentic/runs-route.ts does less:
- It authenticates, derives actor/tenant/time server-side, checks configured route scopes, enforces run ownership, honors operator kill switches, and only lets client kill/budget settings make policy stricter.
- It dispatches tools but does not append the executed tool call/result to the returned run, update the cost ledger, or call the budget-consumption function. Reusing the same meter on later calls therefore checks the same pre-call balance rather than accumulated spend.
- It does not invoke the general grant resolver, egress guard, secret scope, or capability-isolation engine. Its configured route scopes are useful but are not equivalent to those declared controls.
- It does not attach an idempotency key or side-effect reconciliation record to
mutating tools. Re-executing an owner-controlled
runIdcan repeat an effect. - The lifecycle store persists submit/approve/discard snapshots, but execution and approval are not one atomic state machine. The rich run-control functions are not exposed through this autonomy route.
The revised conclusion is therefore: V1 has a strong AgentRun design system and useful governance libraries, but the currently invoked route is not yet a durable, budget-accounting, least-privilege, exactly-once agent runtime.
AUD-005 makes this boundary mechanically durable. The generated
V1/architecture/agentic-run-execution-fields.generated.md reference and
pnpm verify:audit:machine-adjacent-docs bind the umbrella description to the
enforcement comments and real route tests. AUTH-002 advances that contract:
toolCalls, evidenceTrail, and the measured cost ledger are populated from
the authoritative boundary before the next admission, while an honest empty
evidence list remains valid for tools that emit none. The same drift gate covers
V8's locator-versus-byte hash scope and signed state without semantic promotion.
15.4 Quality and publication semantics#
V1's generic structured-output helper uses AJV, injects the schema into the system prompt, extracts direct/fenced/embedded JSON, retries twice with concrete schema errors, and doubles truncated output budgets up to eightfold before a typed failure. This is strong syntactic enforcement, but semantic truth remains the responsibility of downstream gates.
The default content-quality judge is stronger than most local prompt paths. It fences untrusted artifacts, sanitizes delimiter-like text, requires at least two samples (three by default), records prompt hashes/raw responses/model/rubric provenance, uses forward/reverse pairwise order, and escalates panel disagreement. Its cascade, candidate generation, Reflexion memory, self-refinement, reward-hack diagnostics, and calibration are real. Important boundaries remain:
- The panel can contain multiple logical judges backed by the same provider/model, so sample count does not prove error independence.
- Candidate “diversity” is bag-of-words cosine, not semantic diversity.
- Benchmark examples improve anchoring but introduce a documented contamination/overfitting channel.
- Self-refinement and Reflexion feed prior generated text or remembered weak phrases back into prompts without the judge engine's equivalent trust fence.
- The default Sophia publication binding measures declared-citation coverage; it does not retrieve the source and verify entailment. The default rights check treats first-party AI/provider license as satisfied without an artifact-specific license receipt.
- The current video producer emits no poster/thumbnail/keyframe. The media critic explicitly requires a visual frame for video and will throw, making that routed video-critique path unscorable today.
Those are proof-type and integration gaps, not reasons to discard the quality architecture. They are reasons to prevent a structural or declared-evidence score from being labeled factual, licensed, or production-ready.
16. Prompt architecture audit#
16.1 Prompt inventory and decision behavior#
| Version/path | Executable prompt and sampling | Parsing/gating | Deep finding |
|---|---|---|---|
| V1 ideation | The system asks for observed-signal-grounded, “surprising but true” ideas; JSON schema constrains categories/angles, temperature 0.8, maximum output 2,000 tokens, recent topics capped at 30. |
Schema plus category/topic/signal admission. | Good constraint sandwich; signal text still needs an explicit untrusted-data boundary. |
| V1 plan-and-solve | Planner requests the smallest concrete 3–12-node DAG; best-of-N defaults to temperatures 0.5/0.75/1.0; an optional plan judge may trigger one replan. |
Structured output, node/domain validation, cycle detection, loud ranking failure. | Strong bounded planning. It is not tree search over tool execution and should not be represented as such. |
| V1 text production | Staff-writer prompt requests citations for every claim and forbids invention; temperature 0.7, 4,000 tokens; dependency excerpts are bounded. |
Output schema requires title, at least 80 characters, and citation fields; Isis and downstream judges gate release. | Asking for citations is not source retrieval. Without source-bound citation IDs, citation prose can be fabricated. |
| V1 critique/revise | Critic receives artifact JSON, previous feedback, and domain rubric; revisions receive prior draft plus bounded feedback, at most three rounds, bar 0.8. |
Best draft retained; governance/cancel checked each attempt. | Artifact/prior draft are concatenated without the CQJ's injection fence; quoted “evidence” is not checked against the artifact. |
| V1 CQJ | Explicit untrusted markers and prompt-immunity rule; rubric anchors/examples; 2–3 samples; forward/reverse pairwise prompts. | Balanced-JSON parse, dimension/range checks, median panel, reconciliation, calibration and reward-hack checks. | Best prompt-security reference in the repo. Reuse its delimiter/sanitizer and provenance envelope elsewhere. |
| V1 media | VLM receives actual image content and scores composition, artifacts, text, and prompt fidelity; cross-modal path uses caption/embedding cosine or an LLM verdict. | Aesthetic 0.6 plus consistency 0.4; missing video frame fails loud. |
Real multimodal critique for images; current video output contract cannot satisfy the critic. |
| V2 live commentary | System asks for one concise line and treats audio/subtitle/gameplay as inert; user JSON carries cue, commentator, rights, locale. Temperature 0.7, max 80 tokens. |
Latency/fallback/rollback controller outside prompt. | Raw cue prompt is not fenced; no fact verifier is present on the latency path. |
| V2 post-match commentary | Prompt pins exact scored moments and two hosts and forbids invented plays/demographics. A second prompt asks a bias reviewer for exact quotes. | Expected moments and both hosts checked deterministically; unknown decisive plays blocked; reviewer findings drop quoted lines. | Generated prose is inserted raw into the reviewer prompt; an empty model finding set clears, so injection-resistant deterministic safety review is still needed. |
| V2 Hathor NPC | System concatenates identity, Big Five traits, emotion, location, nearby entities, constraints, speech style, history, and world context across a provider chain. | Intent/emotion/actions inferred lexically after generation. | _safetyConfig is accepted but unused and the dialogue generator does not call its safety filter. confidenceScore is a constant 0.85, memoriesTriggered is always empty, and persona/history/world text are unfenced. |
| V3 generation core | No direct executable content-model prompt was found in the audited V3 music/lyrics/TTS/motion/world center. | Provider plans, governance, C2PA, pose and concert gates surround external generation. | Prompt quality cannot be certified because the generative center is a seam; absence is the key result. |
| V4 commentary | Local code constructs deterministic requests/templates; prose, synthesis, bias review, and localization terminate at provider/evidence contracts. | Director/procgen/gameplay constraints are deterministic. | Do not score a provider request as a content prompt or output. The local strength is procedural control. |
| V5 content inference | Local content algorithms are deterministic; model-backed world/media prose remains a provider seam. | Replay, highlight, HMAC, and procgen checks are local. | Same boundary as V4: there is no local executable prompt whose quality can close the asset gap. |
| V6 cognition | Dialogue brief says reply in locale/objective/persona and includes steward utterance; gateway adds ranked context, story state, memories, feedback, and prior response. | Tiered iteration/tool/token budgets, optional quality gate/refine, best response retention. | Steward text, retrieved context, prior output, and critique are not isolated as untrusted. Full prompts are stored in run events, creating privacy and indirect-injection exposure. |
| V6 Clio | Optional writer receives event-grounded narrative input; deterministic fallback composes from event facts. | Citation IDs must reference known events and prose must be nonempty. | Citation membership does not prove every prose assertion follows from those events. |
| V7 Forge Assist | System declares a realm sandbox and only oshun.invoke capabilities; raw brief/realm context are included, temperature 0.4, 3,000 tokens. |
Parsed JSON, capability policy, optional bounded quality refine, policy recheck. | Static policy recognizes literal oshun.invoke("..."). Computed/dynamic calls rely on a sandbox integration still marked partial, so static admission is not a full capability boundary. |
| V7 Mawu Studio | Backend request carries prompt/model/source IDs/attribution. | Length, disallowed substrings, model allowlist, Sophia source/attribution, backend-required checks. | The reference backend is deterministic procedural output. Source metadata is not semantic source-to-output grounding. |
| V8 case proposer | Optional LLM may propose only surface title/suspect/motive flavor under fair-play/canon constraints; temperature 0.8, 700 tokens, canon first eight entities plus constraints. |
First/last-brace JSON parse; invalid proposal returns null, deterministic skeleton continues. |
Good containment: the solver owns truth. Surface fields are only lightly schema/semantic checked and canon text is not fenced. |
| V8 asset briefs | Short seeded briefs request period-accurate, cinematic, evidence-grade assets and negatives such as anachronism. | Isis job must return output; first file becomes bound asset and downstream gates carry provenance. | Useful deterministic briefs, but no best-of-N/perceptual asset gate exists at this stage. |
| V9 rich content client | System specializes by content type/Bloom/difficulty/audience and prompt version. Sampling ranges from 0.3 for quizzes to 0.85 for discussion; lesson version 2 requests prerequisites/objectives/summary/example/references. |
Minimum word/marker checks, lesson citation-shape check, one stricter retry, length ceiling; optional verify/refine or best-of-N. | Ordinary generation returns the second answer even if still invalid. It asks the model to invent reference text without supplying a source bundle; citation presence is not citation truth. |
| V9 tutor | System identifies Metis, tutor mode, learner/course/progress/mastery/integrity/safety/oversight brief, and optional retrieved evidence; configurable provider temperature/tokens. | Approved-source status can fail closed; post-response confidence bundle may force abstention; safety/integrity handoff paths can bypass generation. | Retrieved source preview and orchestration fields are embedded in the system message without trust zoning; the separate agent-safety scanner is not invoked in this tutor call path. |
| V9 agent runtime | No runtime model prompt: registry entries declare persona, model, tools, scopes, policies, and DAG nodes. | Deterministic plan/hash/evidence/approval/kill-switch metadata. | This is a registry and planner API, not a multi-agent executor. Model/tool declarations do not prove an executed governed run. |
16.2 Cross-cutting prompt-security conclusion#
Prompt injection is treated as a first-class adversarial concern in isolated components, but not as a universal data-flow property. The recurring unsafe pattern is:
trusted system instruction
+ retrieved/source/canon/world/user/generated text
+ critic or tool-capability request
-> same model context without a typed trust boundary
The required fix is architectural, not a larger “ignore malicious text” suffix:
- Label every field by origin and trust zone before prompt rendering.
- Put untrusted text in escaped, length-bounded data blocks with explicit non-instruction semantics; use the CQJ sanitizer as the initial common implementation.
- Keep authorization, tool selection, source scope, rights, and release policy in deterministic code after the model response.
- Validate citations, evidence quotations, and tool arguments against the original data rather than the model's declaration.
- Run direct, indirect, cross-turn, retrieval-poisoning, generated-output, multimodal, and memory-poisoning attacks at every model boundary.
17. Algorithms, learning mechanisms, and system patterns#
17.1 V1 algorithm inventory#
| Component | Implemented algorithm | Consequence/limit |
|---|---|---|
| Opportunity portfolio | Contextual linear Thompson sampling with ridge posterior: A = λI + Σxxᵀ, b = Σxr, sampled θ ~ N(A⁻¹b, v²A⁻¹) via Cholesky; features combine signal kind, timeliness, novelty, priority. Beta Thompson cold-start before five observations. |
Real contextual exploration/exploitation; reward semantics still depend on sparse human outcomes and judge proxies. |
| Rubric aggregation | Seven dimensions: hook, clarity, novelty, emotion, grounding, structure, audience. Uncovered dimensions fail. Confidence is agreement (1 - 2*mean_std, clamped) times coverage and scorer-count factor. |
Honest coverage/confidence; equal default weights and same-model samples can hide correlated error. |
| Tournament | Round-robin pairwise judgments followed by Bradley–Terry MM with 0.1 pseudo-wins, half-wins for ties, 200 iterations, 1e-10 tolerance, deterministic ID tie-break. |
Stronger than scalar self-score; depends on judge ordering consistency and candidate quality. |
| Planning/search | Small DAG Plan-and-Solve; optional plan critique/replan; best-of-N node generation; bounded critique/revise retaining best previous draft. | Appropriate bounded workflow; not autonomous open-ended search. |
| Learning memory | Human outcomes update Thompson posteriors; Reflexion keeps top eight deduplicated recurring weak categories; reward-hack monitor correlates panel reward with length/slop/format. | Useful online adaptation; unfenced generated phrases can poison reflective memory, and correlation thresholds need human calibration. |
| Reliability | Gold-set Pearson/MAE/Cohen-style nearest-anchor agreement, drift threshold 0.1, panel disagreement, cascade escalation, champion/challenger primitives. |
Good instrumentation; no portfolio live human calibration data closes the loop. |
17.2 V2–V5 algorithm inventory#
| Version | Implemented algorithms/patterns | Deep takeaway |
|---|---|---|
| V2 | Commentary deadlines at 420/240/100/40 ms; rollback hashing; deterministic phrase fallback; damage/event moment scoring; first-order Markov hints; phoneme-to-viseme/coarticulation; audio ducking at -14.5 dB; persona-consent controls. |
The real-time system is correctly hybrid: deterministic fallbacks and rollback safety carry the SLO. Model review remains vulnerable to correlated/self-review failure, and Hathor safety/confidence are not actually applied as declared. |
| V3 | MediaPipe/MoveNet pose inference with geometric features and nearest-centroid classification; Isis job/provider plans; concert technical/quality gates; real c2pa-rs provenance. |
Local intelligence is governance and perception around absent music/lyrics/TTS/motion/world generation. “Agentic concert creation” is not locally executable end to end. |
| V4 | AI-director score 0.42*damage + 0.25*deaths + 0.20*detection + 0.13*objective, deadband, percentile nudges, bounded rubber-band changes (0.25 scalar, 120 ms timing, one parry per 45 s); procgen seed perturbation seed + attempt*7919, bounded retries, BFS connectivity and placement validation. |
Strong deterministic gameplay adaptation and topology generation; commentary/media quality is still a provider/operations problem. |
| V5 | Seeded spatial hashes using 73856093/19349663/83492791, weighted frontier over 320 templates, biome/pattern rules, roguelike selection, highlight score 40/30/20/10, replay SHA/HMAC with 900-second validity, daily date rotation, Kade vector logic, and a bounded authority-gated AI Director with rate/step/scalar limits. |
Procedural/replay/director integrity is substantive, and an authenticated Linux cook proves exact binary BootMap and policy assets. The cook is not a staged/packaged/launch-smoked authored world, and no live provider or human release receipt is implied. |
17.3 V6–V7 algorithm inventory#
| Version | Implemented algorithms/patterns | Deep takeaway |
|---|---|---|
| V6 cognition | Context ranked by salience and priority, bounded/fingerprinted; tier budgets of 4k/16k/64k tokens and 2/8/24 iterations; kill switches at agent/family/tenant; child authority must be a subset; quarantine after three identical tool failures or eight consecutive failures; optional quality-refine keeps best score. | Strong real-time bounded-agent kernel. Runs, budgets, and quarantine are in memory, no durable checkpointer/pause takeover is present, and full prompt logging is a sensitive-data risk. |
| V6 Clio | Weighted event significance, batching, deterministic event-grounded narration fallback, cited-event membership, connective reconciliation. | Reliable narrative continuity skeleton; semantic faithfulness of free prose is not formally checked. |
| V7 Forge | JSON proposal, capability allowlist, bounded quality refinement, resource/priority scheduler, policy recheck after refine. | Sensible compiler-like pattern; computed capability calls require a real sandbox, not regex alone. |
| V7 simulation/procedural | Economy simulation, Wilson intervals, Gini/entropy metrics, covering arrays, hydraulic/thermal erosion and procedural world algorithms. | These are valuable measurable cores. Headline neural/RL/vision/3D backends remain seams, so the local stack is not the advertised autonomous creator yet. |
17.4 V8 solve-first algorithm inventory#
V8 is the strongest example of keeping model creativity outside the truth kernel:
TruthWeaverchooses a seeded culprit, creates exonerating constraints for every innocent, gives about 75% of innocents motive-shaped red herrings, adds means/motive/opportunity facts, repairs conflicts, and emits CNF clauses for culprit implications and exactly-one-culprit.ClueSmithworks backward from facts, places essential clues before 90% of reveal, links fair red herrings to eliminations, and constructs a deduction path.- The DPLL solver uses unit propagation, pure-literal elimination, and first-unassigned branching. Minos checks uniqueness, completeness, Knox/Van-Dine fair-play and balance; uniqueness is tested using only player-available clauses under each culprit assumption.
- Completeness verifies references, availability before reveal, non-withholding, required use, step order, and nondecreasing clue availability.
- Theseus models skill as
p = 0.35 + 0.65*s, uses seeded discovery, and treats dead-end edges as refutations; its highest-skill path must solve fully. - The G5 “panel” is deterministic by default. Eight dimensions are estimated from structural proxies: red-herring ratio, clue-gap variance, evidence coverage, prose length/description variety, era-cell voice, and difficulty. PCA-derived nonnegative weights vary across the evaluated batch.
This yields strong formal mystery validity, but G5 is not equivalent to a calibrated human/LLM taste panel. Batch-dependent PCA can change dimension weights and zero negatively correlated dimensions, so it should be diagnostic, not the sole release definition of “amazing.”
17.5 V9 education algorithm inventory#
V9 contains two materially different learning-state implementations:
- The Python service's persisted mastery helper increases mastery by
+0.03after any tutoring turn without lexical misconception markers and decreases it by-0.05when such markers appear. It incrementscorrect_attemptsmerely when no marker is found. Assessment mastery is0.6*prior + 0.4*score, while confidence increases by0.08even on a failed assessment. This is a useful continuity heuristic, not validated knowledge tracing; it can become confidently wrong. - The TypeScript adaptive engine implements BKT with defaults
pL0=0.3,pT=0.1,pS=0.1,pG=0.2, the standard Bayesian posterior plus learning transition, an identifiability guardpS+pG<1, sequence likelihood, and EM fitting. Its graph tracer propagates signed BKT deltas over a hop-bounded BFS with relation weights0.4/0.3/0.25, decay0.6, maximum three hops, and headroom/downroom clamping. - FSRS-5 uses the canonical 19 weights, D/S/R memory state, power forgetting
curve
R(t,S)=(1+19t/(81S))^-0.5, requested retention0.9, and Again/Hard/Good/Easy scheduling. Correctness, response time, and prior mastery derive the rating; BKT mastery and FSRS retention remain separate signals. - The adaptive tutoring loop runs present, collect, evidence-build, mastery-update, personalization, integrity, next-item, and telemetry steps. It records failures but continues subsequent steps; a missing evidence pack falls back to “correct iff already mastered,” which can reinforce a mastered state during upstream failure and should instead abstain or preserve state.
- The verification stack decomposes claims in five-sentence windows, computes a FActScore/SAFE-style score, uses lexical entailment by default with a bounded numeric-contradiction check, scores citation sufficiency, STEM processes and pedagogy, and blocks when required dependencies are absent. It supports best-of-four pessimistic selection and up to three critique-guided attempts.
- Pedagogical judging supports median panels, disagreement-based confidence, ICC consistency, Spearman alignment to human gold, corrected item-total discrimination, and Clopper–Pearson/Hoeffding selective abstention.
The algorithms are genuinely advanced. The integration conclusion is more reserved: the Python source-generation service currently produces deterministic outline/lesson templates and structural percentage gates, while the rich LLM, verification, BKT/GKT, and FSRS packages are separate or optional. A production claim requires one traced path proving which implementation actually governs each learner and release.
17.6 V10 composition and attention-policy inventory#
V10 adds portfolio-level algorithms without becoming a portfolio-level author:
- daypart resolution uses persisted IANA time zones, local calendar dates, overnight windows, calendar clamps, and explicit loudness ceilings;
- the kernel validates fixed ordered daypart programs, source-channel authority, future exclusion, contextual actions, and media/micro-act identities;
- batching is deterministic and bounded, while elevations require explicit eligibility and budget rather than channel priority claims;
- audio/video arbitration admits one effective holder and separates offer, selection, playback, replacement, and release;
- director-cut and channel programming consume verified results from their source products; deterministic selection does not convert metadata into media or a simulation into a production event; and
- Ori's recap/greeting generation is a separate consent and inference-budget path with source membership, grounding, policy, and quality gates.
The audit implication is deliberately conservative: V10 can prove that it composes and presents source material safely, but only the owning product's generation and release receipts can prove that material itself is complete, current, or production-live.
18. Persistence, human authority, provenance, and failure semantics#
18.1 Durable-state truth table#
| System | What persists or is modeled | What is not yet guaranteed |
|---|---|---|
| V1 autonomy review | Snapshot-hydrated queue, learned IDs, standing direction, cycle reports (including partial planner realTokenUsage), approval/reject/veto/tweak and review windows. |
Complete generator/research/critic/judge/tool/media/GPU/monetary accounting, mid-step checkpoint/resume, atomic human edit plus re-gate, and exactly-once publication across restart. |
| V1 Agentic Studio route | Submitted/approved/discarded run snapshots and rich run schema. | Executed tool ledger/cost mutation, resolved grants, pause/takeover route, mutating-call idempotency, atomic approval-to-execution. |
| V6 cognition | In-process run events, prompts, child relationships, budgets, kill/quarantine state. | Durable checkpointing, privacy-minimized trace storage, operator pause/edit/resume after process loss. |
| V8 StageJournal | Durable append occurs before live publish; replay sorts/deduplicates by sequence. | Sequence and attempt counters live only in the StageJournal instance. Restart or concurrent instances need store-coordinated allocation to prevent duplicate/nonmonotonic sequence. |
| V8 HITL | Pending/resolved queue with ground truth, reason, verifier report, reviewer identity, approve/reject/edit-and-approve decision. | “Edits” are free-form recorded data; no code atomically applies them to the case, reruns gates, invalidates stale evidence, and resumes publication. Default queue IDs/counter are process-local. |
| V8 provenance | Every asset manifest entry contributes claim ID, content hash, generator, reuse status, canon hash, model versions, and repro key. The binder honestly labels its local pre-download records hashScope: locator and signed: false. |
The aggregate record drops hashScope and signed, and its field comment calls the hash asset-byte-bound even when it is only sha256(url + seed). It does not verify a C2PA claim. The case-report “signature” is also an unkeyed SHA-256 digest. |
| V9 source/agent service | Filesystem JSON for source packages, generated drafts, agent plans/evals and observability; DB course/review state; deterministic plan/replay hashes. | Agent DAG execution, transactional evidence emission per node, tool enforcement, durable agent checkpoints, and proof that TS verification evidence gates Python promotion. |
18.2 Human control conclusion#
Human authority is widest in schemas and libraries and narrowest at the point of live execution. V1 offers the best content-review operations; V8 exposes the best bounded case adjudication context; V9 has strong admin review/release records. No in-repository operator UI consumer of the V1 autonomy review API was discovered; an external client remains unverified rather than impossible. None yet demonstrates the complete required intervention transaction:
pause live run
-> claim versioned step
-> inspect hidden evidence and pending tool effects
-> edit/replace artifact or arguments
-> invalidate dependent judgments and approvals
-> rerun required gates
-> sign mixed human/agent lineage
-> resume on another process without repeating side effects
Until that trace passes restart, race, stale-approval, and duplicate-delivery tests, “humans can take over whenever” remains unproven.
18.3 Safety and trust conclusions#
- V8's
requireSafetyClearcorrectly fails when no scanner exists and blocks flagged content. Its offline keyword scanner is an honest floor, not a multimodal classifier. - V9's agent safety endpoint detects a small literal threat lexicon, assigns trust zones, and computes tool-isolation decisions. It is a separate endpoint; the tutor and agent planner do not require a successful screening record before their main path.
- V2 Hathor contains a safety-filter module, but its main dialogue generator ignores the supplied safety configuration.
- V7's static capability scan and V1/V9 declarative grants are useful only when the actual executor consumes their decisions.
- Full prompt/source logging in V6 and learner/source context in V9 require redaction, retention, access, and incident policy. Replayability does not justify storing sensitive prompt bodies by default.
- V1's durable review snapshots contain full artifacts, human feedback, standing direction, and cycle reports; they require explicit retention, access, deletion/export, and incident policy too.
19. Revised findings and takeaways from the deep pass#
19.1 Material corrections to the high-level assessment#
- Governance must be scored at the invoked seam. V1 Agentic Studio and V9 agent runtime should not receive execution credit for controls present only in libraries or plan metadata.
- Citation has at least four non-substitutable meanings: a string that looks like a citation, an identifier that exists, a retrieved passage, and a passage that entails the claim. V1 and V9 use all four in different places; reports and release records must name which one passed.
- A hash is not automatically a signature or a byte binding. V8's case-report digest provides reproducible integrity checking, and its current Loom binding protects a URL/seed locator. Neither authenticates the signer or detects substituted media bytes. The aggregate must preserve that weaker scope until a real C2PA signer upgrades and verifies it.
- Model-generated confidence is not evidence. Hathor's literal
0.85, V9's heuristic mastery confidence, and low-disagreement same-family judge samples can all look precise without being calibrated. - Optional SOTA paths do not govern default output. V9's verifier-guided generation and BKT/GKT/FSRS stack are strong, but the default rich content generator and Python source forge can bypass or never invoke them.
- Formal verification and taste evaluation solve different problems. V8's uniqueness/completeness proof is a portfolio exemplar; its heuristic G5 panel and missing authored cases still cannot prove tone, delight, or audience preference.
- Deterministic fallbacks are a strength when labeled. V2 commentary, V6 narration, V8 case skeletons, and V9 source templates preserve availability and truth. They should not be marketed as equivalent to a live generative experience.
19.2 Remediation order implied by the source evidence#
- Bind budget consumption, tool/evidence/cost ledger mutation, grant resolution, isolation, and idempotency into the V1 executed route; either add a V9 DAG executor with the same guarantees or relabel that API as planner.
- Make the strong V1 CQJ trust-block renderer a shared primitive and migrate every prompt that ingests source, history, canon, prior model output, feedback, or tool text.
- Replace proof conflation: retrieve and entail citations, verify C2PA/signature cryptographically, require artifact-specific rights/consent receipts, and prevent structural publication percentages from standing in for quality.
- Close concrete default-path defects: invoke Hathor safety or remove the
unused contract, derive rather than hard-code confidence, populate or remove
memoriesTriggered, provide a video frame contract to the media critic, and make V9's second invalid generation fail closed. - Connect V9's source forge, rich generator, P0 verification gate, adaptive engine, and publication service into one evidence trace. Replace tutoring turn lexical mastery updates with graded observations or abstention.
- Make V8 journal sequence allocation store-coordinated and implement edit-apply-reverify-resume as one versioned HITL transaction.
- Only then calibrate quality on licensed human-authored baselines and live outcomes; otherwise optimization will learn to satisfy proxies and judge formats rather than users.
20. Final assessment#
The portfolio does not need another broad declaration that its agents are SOTA or production-ready. It needs an evidence-bearing operating system that makes such a declaration mechanically hard to fake.
The best route is to converge existing strengths:
- V1 supplies the director, judges, preference loop, review queue, and publication law.
- Agentic Studio supplies the run envelope, controls, tool/governance concepts, and operator vocabulary.
- Studio Authoring supplies human creation, editing, branching, roles, and editorial lifecycle.
- Yemaya supplies media pipelines and takeover primitives.
- V8 supplies the exemplar of bounded generation plus deterministic proof.
- V9 supplies a domain-rich dependency and seven-gate learning model.
- V2–V7 supply the real-time, engine, simulation, embodied, and multimodal constraints that keep the shared protocol honest.
Once those systems share a coverage graph, durable run spine, intervention contract, calibrated eval harness, and production feedback loop, autonomous generation can scale without pushing humans to the edge. Humans become first-class collaborators and governors inside the same lineage, while agents do the repetitive coverage, search, production, testing, repair, refresh, and monitoring work.
Until then, the defensible claim is:
V1–V9 contain several advanced autonomous-content subsystems and one especially compelling bounded self-authoring pattern, but the portfolio has not yet demonstrated complete coverage, consistently exceptional live content, universal human takeover, or production-observed iterative improvement.
Appendix A — repository evidence map#
Portfolio and truth sources#
V_SERIES.mdV_SERIES_PLATFORM_CONSOLIDATION.mdCONTENT_GENERATION_CAPABILITY_MATRIX.mdV1_V9_SOTA_AUDIT_REVIEW_2026-07-15.md— retained adversarial provenance; canonical dispositions are generated under AUD-008AGENTIC_CONTENT_AUTONOMY_GAP_CLOSURE_TODOS_2026-06-17.mdV1_SOTA_AUDIT_REMEDIATION_TODOS_2026-07-02.md
V1 orchestration and human control#
apps/oshun/bff/src/server.tsapps/oshun/bff/src/agentic/autonomy-route.tsapps/oshun/bff/src/agentic/autonomy-bindings/create-plane.tsapps/oshun/bff/src/agentic/autonomy-bindings/plane-persistence.tsapps/oshun/bff/src/agentic/autonomy-bindings/editorial-publisher.tsapps/oshun/bff/src/agentic/run-lifecycle-store.tslibs/oshun/creative-autonomy/src/review/review-queue.tslibs/oshun/agentic-studio/src/runs/run-controls.tslibs/oshun/agentic-studio/src/dashboard/operator-actions.tslibs/oshun/persona-policy-lilith/src/operator-override-governance.tslibs/oshun/studio-authoring/src/libs/yemaya/human-override/src/V1/architecture/generation-audience-tiers.md
V1 prompts, algorithms, and runtime enforcement#
libs/shared/ai/src/agent-loop/structured-output.tslibs/oshun/creative-autonomy/src/director/director.tslibs/oshun/creative-autonomy/src/ideation/llm-synthesizer.tslibs/oshun/creative-autonomy/src/portfolio/contextual-bandit.tslibs/oshun/creative-autonomy/src/quality/rubric.tslibs/oshun/creative-autonomy/src/quality/tournament.tslibs/oshun/creative-orchestrator/src/planner.tslibs/oshun/creative-orchestrator/src/orchestrator.tslibs/oshun/creative-orchestrator/src/critique-revise.tslibs/shared/content-quality-judge/src/judge-engine.tslibs/shared/content-quality-judge/src/judge-panel.tslibs/shared/content-quality-judge/src/cascade.tslibs/shared/content-quality-judge/src/candidates.tslibs/shared/content-quality-judge/src/self-refine.tslibs/shared/content-quality-judge/src/reflexion.tslibs/shared/content-quality-judge/src/reward-model.tslibs/shared/content-quality-judge/src/calibration.tsapps/oshun/bff/src/agentic/autonomy-bindings/generation-producer.tsapps/oshun/bff/src/agentic/autonomy-bindings/media-critic.tsapps/oshun/bff/src/agentic/runs-route.tslibs/oshun/agentic-studio/src/runs/run-controls.tslibs/oshun/agentic-studio/src/capabilities/tool-grants.tslibs/oshun/agentic-studio/src/capabilities/tool-isolation.tslibs/oshun/agentic-studio/src/grants/resolver.ts
V2–V7 prompt and algorithm evidence#
libs/iris/conversation-orchestration/src/match-commentary.tslibs/calliope/match-commentary/src/commentary-generator.tslibs/calliope/match-commentary/src/moment-scoring.tslibs/hathor/llm-npc/src/dialogue/dialogue-generator.tslibs/hathor/llm-npc/src/dialogue/safety-filter.tsV3_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-14.mdV4/features/shared-cross-cell-engine.mdV4/architecture/world-streaming-procgen.mdV5/architecture/modes-streaming-procgen.mdlibs/v6/moirai-kernel/src/cognition-gateway-mount.tslibs/iris/agents/core/src/agentic/cognition-gateway.tslibs/v6/clio-story/src/index.tslibs/maya/forge-assist/src/forge-assist.tslibs/maya/forge-assist/src/capability-policy.tslibs/maya/forge-companion/generation-pipeline/src/orchestration.tslibs/isis/three-d-pipelines/src/mawu-studio-ai-creation.ts
V8 solve-first, gates, and operations evidence#
libs/yemaya/case-engine/src/llm-proposer.tslibs/yemaya/case-engine/src/truth-weaver.tslibs/yemaya/case-engine/src/clue-smith.tslibs/yemaya/case-engine/src/clew.tslibs/yemaya/case-verifier/src/minos.tslibs/yemaya/case-verifier/src/report.tslibs/yemaya/case-assets/src/bind.tslibs/yemaya/case-eval/src/g4.tslibs/yemaya/case-eval/src/judge.tslibs/yemaya/case-eval/src/pca.tslibs/yemaya/case-pipeline/src/journal.tslibs/yemaya/case-pipeline/src/hitl.tslibs/yemaya/case-pipeline/src/provenance.tslibs/yemaya/case-pipeline/src/safety.tslibs/yemaya/case-assets/src/plan.tslibs/yemaya/case-assets/src/orchestrate.ts
V9 generation, verification, agents, and learning evidence#
apps/metis/service/src/metis/services/source_outline_service.pyapps/metis/service/src/metis/services/source_generation_service.pyapps/metis/service/src/metis/services/source_generation_helpers.pyapps/metis/service/src/metis/services/tutoring_service.pyapps/metis/service/src/metis/services/tutoring_orchestrator_helpers.pyapps/metis/service/src/metis/services/source_tutoring_helpers.pyapps/metis/service/src/metis/services/learner_mastery_helpers.pyapps/metis/service/src/metis/services/agent_runtime_helpers.pyapps/metis/service/src/metis/services/agent_runtime_service.pyapps/metis/service/src/metis/services/agent_safety_helpers.pyapps/metis/service/src/metis/services/publication_release_gates.pylibs/metis/llm-client/src/content-generator/content-generator.tslibs/metis/llm-client/src/content-generator/output-schemas.tslibs/metis/llm-client/src/content-generator/citation-extraction.tslibs/metis/verification/src/gate/verification-gate.tslibs/metis/verification/src/gate/generate-verify-refine.tslibs/metis/verification/src/gate/compose.tslibs/metis/verification/src/claims/claim-extractor.tslibs/metis/verification/src/claims/claim-verifier.tslibs/metis/verification/src/citations/citation-sufficiency.tslibs/metis/verification/src/pedagogy/judge-verifier.tslibs/metis/verification/src/pedagogy/judge-reliability.tslibs/metis/verification/src/calibration/conformal-selective.tslibs/metis/verification/src/generation/verifier-guided.tslibs/metis/tutoring/src/loop/adaptive-loop.tslibs/metis/tutoring/src/loop/mastery-updaters.tslibs/metis/adaptive/src/path/bkt.tslibs/metis/adaptive/src/path/graph-knowledge-tracing.tslibs/metis/adaptive/src/path/adaptive-knowledge-engine.tslibs/metis/adaptive/src/path/fsrs.ts
V10 composition, policy, and channel evidence#
V10/V10_PRODUCT_REVIEW_2026-07-22.mdV10/V_SERIES_AMBIENT_RAIL_TODOS_2026-07-16.mdlibs/contracts/src/v10libs/v10/rail-kernel/srclibs/v10/rail-phase-a/srclibs/v10/rail-thread/srclibs/v10/rail-channel-case-files/srclibs/v10/rail-channel-veritas-live/srclibs/v10/rail-channel-stage/srclibs/v10/rail-channel-wonder-recall/srcapps/v10/web/srcapps/v10/shell/src-tauriscripts/audit/v1-v10-verification-manifest.json
Version audit/status evidence#
V1_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-13.mdV2_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-13.mdV3_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-14.mdV4_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-14.mdV5_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-14.mdV6/AUTONOMOUS_CONTENT_AUDIT_2026-07-14.mdV7/AUTONOMOUS_CONTENT_AUDIT_2026-07-14.mdV8/V8_TODOS.mdV9/V9_TODOS.mdV10/V10_PRODUCT_REVIEW_2026-07-22.mdV10/V_SERIES_AMBIENT_RAIL_TODOS_2026-07-16.md
Appendix B — external primary references#
Agent and orchestration practice#
- Anthropic, Building effective agents, 2024.
- Anthropic, How we built our multi-agent research system, 2025.
- OpenAI, A practical guide to building agents.
Evaluation and improvement#
- Anthropic, Demystifying evals for AI agents, 2026.
- Madaan et al., Self-Refine, 2023.
- Shinn et al., Reflexion, 2023.
- Yao et al., Tree of Thoughts, 2023.
- Zhou et al., Language Agent Tree Search, 2023.
- Agrawal et al., GEPA, 2025/2026.
- AgentRewardBench, 2025.
- GenEval, 2023.
- VBench++, 2024.
Human control, operations, security, and provenance#
- OpenAI Agents SDK, Human-in-the-loop.
- LangGraph, Interrupts.
- OpenAI Agents SDK, Tracing.
- OpenTelemetry, Generative AI semantic conventions.
- NIST, AI Risk Management Framework Core.
- NIST, Generative AI Profile.
- Model Context Protocol, Authorization and Security best practices.
- C2PA, Content Credentials 2.2.