Eve library

Review of the V1–V9 Autonomous Content SOTA Audit (2026-07-15)

Research

Revision 1

On this page

Review date: 2026-07-15

Subject: V1_V9_AUTONOMOUS_CONTENT_SOTA_AUDIT_2026-07-15.md (1,857 lines, committed across 5f1397ad54, a1eaae8d97, 9302fa49b3)

Reconciliation status: this review is retained as adversarial provenance, not as a parallel current truth source. Accepted and corrected findings are folded into V1_V9_AUTONOMOUS_CONTENT_SOTA_AUDIT_2026-07-15.md; the generated disposition ledger is evidence/v1-v9/AUD-008/reconciliation-status.generated.md.

Review method: independent re-verification against source. Every check in this review was executed sequentially in this session on this host (macOS checkout, branch oshun-mobile-parity-2026-07-04): source files were read directly, cited constants re-derived, the V8 docs-drift verifier re-run, two of the report's test-count claims reproduced by running the suites, and production-consumer searches repeated from scratch. External references were checked against known paper/spec identifiers.

1. Overall verdict on the report#

The report is accurate, unusually honest, and largely reproducible. Its central thesis — strong components and local verification, not yet a unified production-observed autonomous content operating system, with governance scored at the invoked seam — survives adversarial re-checking. Of the ~30 load-bearing claims re-verified, all but one were exactly correct, including every numeric constant sampled across V1, V2, V4, V8, and V9. The single factual error found (V3 gate count) was inherited from an upstream audit document, which itself validates the report's own warning about prose truth-source drift.

The material contributions of this review are: one inherited factual error, one class of evidence-staleness the report applies to others but not to its own inputs, and six additions. Subsequent source review qualified two additions: no in-repository operator UI consumer of the V1 autonomy review API was found, although an external private client cannot be excluded, and the live plane persists partial planner token usage but lacks complete measured cost accounting. Those corrected boundaries sharpen, rather than contradict, the report's P0 register.

2. Claims re-verified exactly (confirmatory pass)#

Every item below was checked against primary source in this session, not against the report or the July per-version audits.

# Report claim Verified evidence
1 Four V1 launch categories apps/oshun/bff/src/agentic/autonomy-route.ts:69-95 — exactly veritas.story-explainer, nyx.sky-event-brief, nisaba.study-passage, tara.seasonal-ritual
2 authorizeOperatorAction / retryWithChanges / operatorStep have no app consumer Only match outside the package is apps/bellona/remote-host/src/macos-host-support.ts:422 — an unrelated operatorSteps: readonly string[] field (manual setup steps), a false positive
3 @yemaya/human-override has no app/service import Zero matches across apps/ and services/
4 Runs-route checks a pre-call budget but never consumes it, never appends tool/cost ledger entries Full chain read: runs-route.ts → orchestrator.ts → dispatcher.ts → executor.ts. admitToolCall returns run: input.run unmutated; consumeBudget is never called by orchestrator, dispatcher, or route; the same BudgetAdmissionInput[] (same meter) is re-checked per step
5 Runs-route does not invoke grant resolver / egress guard / secret scope / isolation Route imports only runGuardedToolPlan + emptyCostLedger from the studio package
6 Hathor _safetyConfig accepted but unused; confidenceScore constant 0.85 libs/hathor/llm-npc/src/dialogue/dialogue-generator.ts:84,114
7 V8 case-report "signature" is an unkeyed SHA-256 libs/yemaya/case-verifier/src/report.ts:48-58 (signReport = createHash('sha256') over canonical JSON; docstring is honest about it)
8 V8 asset binding is locator-hashed and honestly labeled; the aggregate drops the labels and overclaims byte scope libs/yemaya/case-assets/src/bind.ts:61,71-72 (sha256(url::seed), hashScope: 'locator', signed: false) vs libs/yemaya/case-pipeline/src/provenance.ts:16-19,60 (comments claim the hash covers "the asset bytes"; hashScope/signed not carried)
9 V8 StageJournal sequence/attempt counters are process-local libs/yemaya/case-pipeline/src/journal.ts:48-50 — instance-private Maps
10 V8 HITL edits are free-form records with no apply/re-gate/resume libs/yemaya/case-pipeline/src/hitl.ts:17,37-38 — edits?: Record<string, unknown>, no application code
11 V8 docs-drift gate is red Re-ran node scripts/verify-v8-docs-drift.mjs in this session: still fails, v5-target.generated.ts stale
12 V9 Python mastery heuristic: +0.03/−0.05, correct_attempts on no-marker, 0.6*prior + 0.4*score, +0.08 confidence even on failed assessment apps/metis/service/src/metis/services/learner_mastery_helpers.py:144,153,175,179 — all four exact
13 V9 rich generator returns the retry "as-is even if it also fails" libs/metis/llm-client/src/content-generator/content-generator.ts:1164-1190 (the code comment says so verbatim)
14 V9 agent runtime plans DAGs but has no executor apps/metis/service/src/metis/services/agent_runtime_service.py — logs "planned with N DAG nodes"; no execution path
15 Sophia binding scores declared citations without retrieval; rights auto-satisfied apps/oshun/bff/src/agentic/autonomy-bindings/sophia-judge.ts (hardcodes stance: 'supports', relevance: 0.8; judgeId honestly says declared-citations); publish-readiness-gates.ts:87 unconditionally satisfied.add(EDITORIAL_GATE_IDS.rights)
16 Media critic throws on video without a still frame; no generator populates one media-critic.ts:94-119 (its own comment says "none" populate thumbnailUrl/posterUrl/keyFrameUrl today)
17 CQJ has a real untrusted-content fence; critique/revise does not libs/shared/content-quality-judge/src/judge-engine.ts:43-60 (fence markers + immunity clause + sanitizer) vs zero fence matches in libs/oshun/creative-orchestrator/src/critique-revise.ts
18 V6 runs/budgets/quarantine are in-memory libs/iris/agents/core/src/agentic/agent-run.ts:102,163,225-227,521 — kill switches, failures, runs, controllers, quarantine, cost caps all instance Maps
19 Director bandit decay 0.98; ideation temp 0.8 / 2,000 tokens / 30-topic cap director.ts:189,290; llm-synthesizer.ts:106,110-111
20 Theseus skill model p = 0.35 + 0.65·s libs/yemaya/case-eval/src/solver.ts:83
21 FSRS-5: 19 canonical weights, FACTOR = 19/81, R(S,S) = 0.9 libs/metis/adaptive/src/path/fsrs.ts:11-54 — exact, including the derivation comment
22 V4 director 0.42·damage … and procgen Seed + Attempt*7919 are real UE C++ V4/ue/Source/V4Gameplay/Private/V4AIDirectorSubsystem.cpp:175; V4/ue/Source/V4Procgen/Private/V4ProcgenSubsystem.cpp
23 V2 latency decomposition 420/240/…/40 ms apps/v2/iris-commentary-orchestration/src/v2-ai-commentary.ts:24-27
24 28 documented generation surfaces V1/architecture/generation-audience-tiers.md:71-100
25 V_SERIES.md dated 2026-06-12; capability matrix last updated 2026-06-18 File headers confirm both
26 Verification record: @oshun/creative-autonomy 248/248 Reproduced this session: 16 files, 248 passed
27 Verification record: V9 176 tests vs ledger 173 Reproduced this session: 159 across libs/v9/* + 17 in libs/contracts/src/v9 = 176; V9/V9_TODOS.md:35 still says 173
28 External references All arXiv IDs (Self-Refine 2303.17651, Reflexion 2303.11366, ToT 2305.10601, LATS 2310.04406, GEPA 2507.19457, AgentRewardBench 2504.08942, GenEval 2310.11513, VBench++ 2411.13503) match the real papers; Anthropic/OpenAI/LangGraph/OTel/NIST/MCP/C2PA links are the correct primary sources

The report's most consequential architectural finding — that the guarded tool-plan chain enforces kill switches and pre-call budget checks but the envelope is threaded through unmodified, so the persisted "executed run" never accumulates tool calls, evidence, or cost — is correct at every level of the call stack.

3. Errors and staleness found in the report#

3.1 V3 launch-gate figure is wrong (inherited, not re-derived)#

The report states (§6.3, §3 examples) that V3/launch/launch-readiness-release-gate.json is "red at 29/40 with gaReady: false". The actual file — unchanged in git since commit 8095244e26 on 2026-06-12 — reads:

text
status: red, requiredGateCount: 41, passingGateCount: 30,
manifestNamedGateCount: 39, passingNamedGateCount: 24, gaReady: false

Neither 30/41 (required) nor 24/39 (named) is 29/40. The figure was copied from V3_AUTONOMOUS_CONTENT_AGENTS_AUDIT_2026-07-14.md:19, which is itself wrong about the file it cites. The substance (red, not GA-ready) holds, but this is a concrete failure of the report's "corroborated" evidence class: the upstream audit was trusted without opening the referenced artifact. Both documents should be corrected to 30/41 required (24/39 named).

3.2 The V3 gate evidence is itself a stale truth source#

The same JSON's generatedAt was 2026-06-12T19:03:08Z — over a month old at review time. The report (correctly) flags V_SERIES.md and the capability matrix as stale truth sources, but treated the stored V3 gate file as current fact. That timestamp proved the consumed artifact had not been rewritten; it did not prove that nobody ran the verifier without --write-evidence. Machine-generated evidence rots exactly like prose. The report's P0-06 acceptance criterion should add: evidence artifacts carry generation timestamps and a max-age policy, and a gate consuming evidence older than its freshness SLO is itself red. AUD-001 subsequently regenerated this record and added a 24-hour freshness contract.

3.3 Host-ambiguous remediation advice#

§6.4 says "this repository has a local UE5.5 toolchain and should use it for the closure slice." That is true of the Linux sandbox (/root/workspace/UnrealEngine-5.5/) and false of this macOS checkout, which has no UE toolchain and cannot build one practically. The recommendation is right but should name the execution host; a reader on this Mac cannot act on it.

Resolution: AUD-004 defines the ue55-linux self-hosted execution profile, records missing local prerequisites as not-run-on-this-host, and provides a dispatchable V4 remote job plus a required executed-evidence schema. A missing binary on a Mac or another checkout is no longer treated as project-level unavailability.

3.4 Verification record is not reproducible as written (minor)#

§1.4 reports pass counts without the commands, package boundaries, or the HEAD commit they were measured at. Reproducing "V9 packages plus V9 canonical contracts — 176" required reverse-engineering the composition (12 suites under libs/v9/* = 159, plus libs/contracts/src/v9 = 17). For a report whose thesis is typed, non-substitutable proof, its own evidence table should carry the exact commands and the commit hash. (The counts themselves reproduced exactly; this is a reproducibility gap, not an accuracy one.) The §1.4 environment description ("15 GiB RAM, 31 GiB swap") is likewise unverifiable after the fact on this dynamically-swapped macOS host.

Resolution: AUD-003 now records the immutable source commit, exact commands and package boundaries, runtime and host observations, authenticated bounded logs, and a generated canonical table. Historical resource observations are explicitly not generalized to another host.

4. Material additions the report missed#

4.1 No in-repository operator UI consumer was found#

Production-consumer search across apps/oshun/web, apps/admin-mobile, and every other in-repository app found zero consumers of /v1/agentic/autonomy/*. The matches for "review queue" in the web app (EvidenceReviewQueue.tsx, lilith-studio) are unrelated features. The API and its tests prove a real component, but the repository contains no human-usable operator surface that drives it. An external private client is possible; no repository or deployed-consumer evidence was available to prove one.

The report's own §15.1 principle — "a control does not govern a run merely because both exist in the same package" — applies symmetrically to human authority: a tested approve endpoint with no discovered operator consumer is a component, not proof of an operating human-review capability. The scorecard's "Strong in V1 output review" and V1 Human-control = 2 deserve that qualification, and P0-03 should explicitly require a human-usable operator surface exercised end to end, not only durable takeover semantics at the API layer.

4.2 The live autonomy plane has partial token usage, not full cost accounting#

The report's P0-09 correctly targets the Agentic Studio route's non-consumed budget meter. The live V1 loop has a related but narrower gap:

  • Portfolio selection enforces a knapsack over idea.estimatedCostUnits against cycleBudgetUnits (libs/oshun/creative-autonomy/src/portfolio/selection.ts:180,202) — a planning-time estimate in abstract studio units.
  • CycleReport.realTokenUsage accumulates run.planning.usage for produced variants, is returned by the cycle route, retained in cycleReports, and persisted with the autonomy-plane snapshot. Director tests assert its numeric token/call totals. This is real, but principally planner-side accounting.
  • Generator/research-loop, critic, judge, tool, media-provider, GPU, and monetary cost are not represented end to end, and there is no complete artifact-level provider-cost ledger.
  • The concurrency governor bounds rate (maxConcurrent: 2, launchesPerMinute: 30), not spend.

An always-on autonomous loop whose full-cycle economics remain partly estimated, with no complete measured-cost ledger, no daily/monthly monetary cap, and no feedback from actual full-path cost to estimatedCostUnits, cannot bound its own burn or report complete cost per accepted artifact (§13 requires exactly that dashboard). P0-09 should require complete token, provider, GPU, tool, and monetary accounting, while retaining realTokenUsage as partial evidence.

4.3 In-code truth-source drift: the runs-route documents behavior it does not have#

apps/oshun/bff/src/agentic/runs-route.ts:289-293 says:

"C9: persist the EXECUTED envelope (tool calls, evidence, cost, final status)"

As §2 item 4 above verifies, the envelope never accumulates tool calls, evidence, or cost — the persisted "executed run" always carries empty toolCalls, evidenceTrail, and emptyCostLedger(). The comment is the same defect class as the stale umbrella docs (P0-06), but inside the enforcement code itself, where a future maintainer or auditor will trust it. Same class: libs/yemaya/case-pipeline/src/provenance.ts:19's "sha-256 of the asset bytes the claim covers" over what may be a locator hash (the report caught this one). P0-06's registry/drift-CI remedy should include doc-comments on enforcement seams, or at minimum these two comments should be fixed when P0-09/P0-11 land.

Resolution: AUD-005 now states the route boundary consistently in the executor, dispatcher, BFF route/store, and V1 architecture. AUTH-002 now closes the V1 executed-route runtime gap: its authoritative boundary persists pending and completed/denied attempts, grant decisions, evidence/effects, measured usage, and kill/budget audit events before another admission. V8 case provenance now carries hashScope and signed state so a locator digest cannot become an asset-byte claim by omission. Both boundaries have contract-generated field references, real-path assertions, and one CI drift verifier spanning enforcement comments, schema docs, and umbrella prose.

4.4 Hathor also fabricates memoriesTriggered#

Alongside the constant confidenceScore: 0.85 the report flagged, dialogue-generator.ts:113 returns memoriesTriggered: [] unconditionally — the response contract advertises memory integration that never occurs. Trivial individually, but it belongs on the same remediation line item (§19.2 step 4) so the fix covers the whole response envelope, not one field.

4.5 P0-01 could be seeded today from in-repo material#

The report is right that no machine-readable coverage contract exists, but it under-sells the raw material already present: the frozen 28-surface catalog (GENERATION_SURFACES in V1), the four launch categories, the per-version TODOS ledgers, and V9's gate contracts are all extractable. Phase A step 2 ("import every launch requirement") should start as a mechanical extraction from these sources rather than a blank-page inventory; that converts P0-01 from a design task into a data-migration task with a measurable day-one count of registered vs unregistered surfaces. The report criticizes the absence of a ledger without seeding its first rows.

4.6 Declared production topology is auditable in-repo and bears on P1-08#

The repository declares V1's canonical production substrate as a single Hetzner box (compose + Caddy + GHCR/SSH CI). A live topology probe was not available in this review, so the deployed environment cannot be asserted to match that declaration. The report's P1-08 (exactly-once publication, checkpoint versioning, disaster recovery) and the "Live proof" scorecard column never engage with the declared design: if deployment matches it, the durable stores, the BFF, and the cycle driver share one failure domain. That design risk is auditable without production access, while confirming the live topology still requires a probe. It materially changes what "durable across restart" must mean (box loss, not just process loss). A one-paragraph topology finding plus a backup/restore drill requirement would close this.

4.7 Smaller absences (flagging, not faulting)#

  • Locale coverage of the live loop: the four launch categories generate (as far as the code shows) untranslated single-locale content; the coverage contract models locales but the report never states the current live locale count. First registry import should record it.
  • Accessibility of generated artifacts: R6 lists accessibility, and V2's caption contracts are praised, but no finding states whether V1's published text/image/audio artifacts carry alt text/captions today.
  • Retention/access policy for V1 review snapshots: §18.3 rightly flags V6 prompt logs and V9 learner data; the V1 durable review/learning snapshots (full artifacts + human feedback + standing direction) deserve the same retention/access treatment.
  • Test-coherence sampling: the report's evidence ladder credits passing suites (e.g., 40/40 BFF autonomy tests) without sampling whether those tests assert domain correctness versus shape. Its conclusions don't depend on this, but the repo's own audit standard (adversarial verification, test coherence) would apply it.

5. Judgment calls I checked and endorse#

  • Scoring governance at the invoked seam (§15.1, §19.1-1) is the right standard and the deep pass applies it consistently — the runs-route and V9 agent-runtime findings would be invisible under a package-level audit.
  • The four-way citation taxonomy (§19.1-2: citation-shaped string / existing ID / retrieved passage / entailed passage) matches exactly what the code does in different places (Sophia declared-citations vs V9 lexical entailment vs V9 citation-shape check) and is the correct P0-11 framing.
  • The hash-vs-signature distinction (§19.1-3) is precisely borne out by report.ts and bind.ts vs the aggregate.
  • Not penalizing honest fail-loud seams aligns with the repo's own quality-standards escape-valve doctrine; the report correctly treats V2's refusing translation path and V8's requireSafetyClear as strengths.
  • Phase ordering (A→E) and V8-first for level 4 are sound. One note: the red V8 docs-drift gate is a minutes-scale fix (node libs/yemaya/case-contracts/tools/gen-v5-target.mjs per the verifier output — regenerate and commit); it should be done immediately rather than waiting for Phase A, since it currently blocks the strongest product's release gate for a mechanical reason.

6. Consolidated recommendation deltas#

Concrete edits to the report's register, in priority order:

  1. P0-03 (amend): add "a human-usable operator surface exists and drives approve/reject/veto/tweak/direction end to end" — no in-repository consumer was found, and any external consumer still needs deployed evidence (§4.1).
  2. P0-09 (amend): retain persisted CycleReport.realTokenUsage as partial planner evidence; require complete token/provider/GPU/tool/dollar accounting per cycle/artifact and a hard spend cap on the live plane (§4.2).
  3. P0-06 (amend): correct the V3 figure to 30/41 (both in this report and in the V3 audit doc); add evidence-age/freshness SLOs for machine-generated gate artifacts (§3.1–3.2); include enforcement-seam doc comments in drift scope (§4.3).
  4. §1.4 (amend): record exact verification commands + HEAD commit; note the host each recommendation assumes (UE closure slices run on the Linux box) (§3.3–3.4).
  5. Phase A (amend): seed the coverage registry mechanically from the 28-surface catalog, launch categories, and version TODOS (§4.5).
  6. P1-08 (amend): add the repository-declared single-box topology, require a live topology probe, and add a box-loss restore drill (§4.6).
  7. Immediate mechanical fix: regenerate v5-target.generated.ts to green the V8 docs-drift gate; fix the two overclaiming comments (runs-route.ts:289, provenance.ts:19) and Hathor's fabricated confidenceScore/memoriesTriggered pair (§4.3–4.4).

7. Bottom line#

The audit is the most reliable single document about this portfolio's autonomous-content reality that the repository currently contains. Its verdict ("SOTA-shaped at component and local-verification layers; not yet a unified, production-observed autonomous content operating system") is supported by the evidence and survived every adversarial re-check attempted here except one inherited number. The additions above do not soften that verdict — they tighten it: no in-repository human operator consumer was discovered, and partial planner token counts do not amount to full-path economic governance. Both strengthen the case for the report's central prescription: build the coverage contract, the durable run spine, and the composed human-control plane before adding any new generator.