# Creative-Autonomy Underlying-Systems Audit — 2026-07-02 Scope: every system `@oshun/creative-autonomy` composes with, audited by reading the actual source (not docs), with each actionable gap **closed in this pass** and every remaining gap named honestly. Companion to `V1/architecture/agentic-ai-studio.md` § "The autonomous creative direction plane" and TODOS §18.15. Verdict up front: the underlying stack is substantially real — the weaknesses were almost all **integration seams that existed but were unbound**, plus a handful of design gaps in the plane itself that binding exposed. All of those are now closed. What remains is exactly two kinds of gap: live provider credentials, and deeper integrations that need corpora/infrastructure that do not exist on this box. ## Per-system findings ### 1. `@oshun/content-quality-judge` (libs/shared) — REAL, now bound - **Found:** a serious judging stack: `JudgeEngine.scorePointwise` runs N≥2 samples per rubric dimension and reports mean + variance (judge-stability signal); `comparePairwise` evaluates BOTH orderings and only counts a consistent winner (the standard mitigation for LLM position bias); unparseable responses throw `JudgeParseError` (never a defaulted score); `computeSlop`/`slopPenalty` is a deterministic over-represented-diction detector; `calibration.ts` computes real Cohen's κ / Pearson against human anchors with drift checks. - **Issue found in MY plane (fixed):** the compellingness panel forced every judge to score all 7 dimensions — which would have forced this binding to _invent_ scores for dimensions the quality-judge rubrics do not measure (e.g. grounding). The panel now accepts **partial** scores per judge, aggregates each dimension over the judges that actually scored it, and throws `IncompleteJudgePanelError` when the panel as a whole leaves a dimension uncovered. - **Issue found (fixed):** the engine consumes `@iris/agents-core` `AgenticProvider` while the orchestrator consumes `@oshun/ai` `LLMProviderInterface` — two provider protocols, no bridge. Built `agenticProviderFromLlm` (pure translation) so one env-keyed router powers planner, judges, and engine. - **Bound as:** `createCraftJudge` (explicit per-content-type rubric-key → dimension mapping, verified against the real rubric keys — note `hook` exists only in the quest rubric, not commentary), `createSlopNoveltyJudge` (deterministic, credential-free), `createQualityPairwiseJudge` (tournament judge). - **Remaining (named, not actionable here):** its `calibration.ts` κ pipeline needs a human-anchor gold set to run against; the plane keeps its own per-category calibration EMA until a deployment accumulates anchors. ### 2. Nyx (`@oshun/domain-nyx`) — REAL, now bound, credential-free - **Found:** `buildNyxUpcomingAstronomicalEvents` computes real upcoming events from the ephemeris (principal lunar phases ≈weekly, solar season markers, plus a source-backed featured list — eclipses, meteor showers), deterministic in `now`. - **Bound as:** `createNyxSkyEventSignalSource` — events → the plane's triangular lead-time calendar model. Audience-estimate/ideal-lead priors per event kind are declared editorial constants the bandit then corrects with real human outcomes. This makes `nyx.sky-event-brief` the deployment's strongest end-to-end autonomous path (zero external credentials for sensing). ### 3. Editorial lifecycle (`@oshun/studio-authoring`) — REAL, now bound - **Found:** a pure, tested state machine (idea → … → published → … → archived) with `tryEditorialAdvance` gate enforcement; blocking authoring-evaluation findings genuinely refuse the publish transition. No store ships with it (pure functions). - **Bound as:** `createEditorialLifecyclePublisher` — every autonomous publish walks draft → in-review → approved → published through the real machine under the deterministic Lilith content-QA gate; a blocked gate throws `EditorialGateRefusedError` with the missing gate ids. Autonomous content can no longer bypass editorial law. - **Issue found in MY plane (fixed):** a publisher throw aborted the whole cycle. Publish failures are now isolated per item (`publishFailures` in the cycle report, `publish-failed` disposition); the item stays `approved` and retries next cycle. ### 4. Generation-job pipeline (BFF `src/generation/`) — REAL, now bound - **Found:** enqueue → executor → **Isis release gate** (`evaluateGenerationRelease`, fail-closed `blocked` without governance evidence) → released-output catalog (gallery/lineage/provenance). Real provider executors for illustration/explainer (Stability), music (Suno), narration (ElevenLabs), sky-briefing (Meshy), video (fal LTX-Video), caption-dub, accessibility-pass — all env-credential-gated, fail-closed. - **Gap found (fixed):** the queue drained ONLY via the operator HTTP tick — no programmatic path. Extracted `processQueuedGenerationJobs` (+ `getGenerationJob`) so the autonomy producer drains its own jobs through the identical executor + release-gate + catalog path. - **Gap found (fixed):** no `text` kind exists — text was unproducible. Built `createLlmTextGenerator`: schema-validated `{title, text, citations[]}` structured output with critique feedback threaded into revision passes, so grounding judges read _declared_ citations and the revise loop genuinely revises. - **Bound as:** `createGenerationJobGenerators` — image→`illustration`, audio→`narration` (dependency-text aware), video→`video`. A blocked or held job throws with its real governance reasons; the plane never treats an ungoverned output as produced. - **Remaining (named):** `music` and `caption-dub` kinds are unbound (no launch category uses them; binding is mechanical when one does). Video outputs currently block at the release gate by design until the deploy-bound video safety/watermark step exists — that is the pipeline being honest, not a plane gap. ### 5. Lilith (`@oshun/persona-policy-lilith`) — REAL, now bound - **Found:** deterministic content-QA hooks (`runLilithQaUnsafeClaimScan`, `runLilithQaVoiceAbuseLanguageScan`) with severity-typed findings and a block-publish verdict; real unsafe-claim rule engine behind them. - **Bound as:** `createLilithToneJudge` (partial judge: a blocker zeroes `audience-fit`; each advisory deducts a documented 0.15) and `lilithEditorialGate` (the publisher's required gate). Scans run under the scholar-historian/lectio posture — the least restrictive published-content pairing; category-specific postures are a wiring parameter. ### 6. Sophia (`@oshun/evidence-sophia`) — REAL, bound with an honest scope - **Found:** `buildSophiaEducationalClaimGrounding` — real support/contradiction counting, warning folding, and grounding-status derivation per claim. - **Bound as:** `createSophiaGroundingJudge` — each text-bearing plan node becomes a claim whose citations are the node's own declared `citations` output, run through the real Sophia model; statuses map to documented score constants; `citationPolicy: 'optional'` supports self-grounded categories (ephemeris-derived sky briefs) at a documented 0.6. - **Honest scope limitation (named in the judge's id):** this measures **declared-citation coverage**, not stance verification against source text. Verifying that a cited source actually supports the claim is the `sophia.ground`/`sophia.fact_check` tool path, which needs a retrieval corpus + creds; when a deployment wires those tools, a stance-verifying judge should replace/augment this one. That is the single biggest remaining quality gap for factual content. ### 7. Hathor (`libs/hathor/narrative-generation`) — REAL, deliberately unbound - **Found:** `NarrativeGenerationService` — a deterministic arc planner (tension models, motifs, seeded) with LLM prose per beat, provenance records, lore-consistency checks, and its own review workflow + staged volumes with STOP-on-regression. - **Decision:** NOT bound this pass. Its request shape requires characters, canon, and motifs — none of the launch categories carry a character canon, so a binding today would fabricate inputs. When a narrative category with a real canon exists (e.g. a Hathor lore volume category), the binding is a straightforward `DomainGenerator` and Hathor's own review workflow slots in as the domain-local quality loop _inside_ the generator, with the plane's queue staying the cross-domain human gate. ### 8. Agentic studio governance — REAL, now bound to the plane - **Found:** the admin agentic-operations kill-switch store is live-read by the runs route on every request; gold-set promotion machinery exists behind operator admin routes. - **Bound as:** `createAdminKillSwitchBinding` — an armed global switch or a switch on `agent-family:creative-autonomy` aborts the next cycle. Gold-set promotion stays **operator-owned by design** (auto-promoting approvals to gold sets is a policy decision, not a default). ## Plane-level SOTA gaps found by this audit (all fixed) 1. **No self-correction before parking** — a gate failure parked immediately. Now: bounded gate-retry (`gateRetries`, default 1) re-produces with the failing dimensions and gate notes fed back into the brief; `gateAttempts` reported per item. 2. **Forced-full-panel fabrication pressure** — fixed with partial specialist panels + `IncompleteJudgePanelError` (above). 3. **Publish failures aborted cycles** — fixed with per-item isolation (above). 4. **No shipped LLM defaults** — every deployment had to write its own synthesizer/judges. Now shipped, all fail-loud via `@oshun/ai/agent-loop`: `createStructuredIdeaSynthesizer`, `createLlmRubricJudge`, and `createLlmPairwiseJudge` (with the same both-orderings position-bias mitigation the quality-judge engine uses). 5. **No canonical text extraction** — judges each needed artifact-walking logic. Now `variantText`/`extractCitations` (deterministic, conventional keys, never paraphrases). ## What still stands between this and "absolutely great content" Honest, ordered by impact: 1. **Live credentials** — `OPENROUTER_API_KEY` binds the director end to end (`server.ts` logs BOUND vs fail-closed); media executors additionally need their provider keys (Stability/ElevenLabs/fal/Suno). Everything downstream of the keys is wired and tested against boundary doubles. 2. **Stance-verified grounding** — wire `sophia.ground`/`sophia.fact_check` over a real retrieval corpus and upgrade the grounding judge from declared-citation coverage to verified support (finding §6). 3. **Human-anchor calibration** — accumulate operator gold-set anchors and run the quality-judge κ calibration against the plane's judges; today the per-category EMA calibrates against review outcomes only (finding §1). 4. **Engagement + search demand signals** — the engagement-decay and catalog-gap sources are built and tested but need the analytics warehouse and search query-log feeds wired as inputs at deploy (real data, not new code). 5. **Narrative categories** — bind Hathor when a canon-bearing category exists (finding §7). 6. **Video release path** — the fal video executor produces, but outputs block at the Isis gate until the deploy-bound safety/watermark step lands (finding §4); autonomous video stays parked-by-governance until then. Test evidence for this pass: `@oshun/creative-autonomy` 118/118 (10 files); BFF agentic + bindings + jobs 77/77; both `tsc --noEmit` clean. --- # Deep implementation review + SOTA comparison — 2026-07-02 (second pass) A prompt-and-implementation-level review of the plane itself, benchmarked against the disciplines SOTA agentic-content systems use (judge ensembles with anchoring and bias mitigation, non-stationary/contextual bandits, self-refine loops, injection defense, durable state, self-scheduling). Every issue below was **verified against the source and fixed in this pass**; the closing table names what deliberately still differs from SOTA and why. ## Process bugs found and fixed 1. **Tweak-loss on production failure** — the director consumed a human's tweak directive _before_ producing; a provider outage silently swallowed the direction. Now consumed only when the regeneration actually got judged (any disposition except `production-failed`); otherwise it stays pending and retries. Test: a scripted outage across exactly the regeneration's produce calls. 2. **Dependency-blind generators** — `createLlmTextGenerator` ignored `ctx.dependencies`: a plan node depending on an outline/research node never saw the upstream artifact. Now a bounded `dependencyContext` (per-artifact + total clamps, identification fields first) threads into text prompts, and a one-line excerpt into image/video prompts. 3. **Specialist-panel overconfidence** — partial panels made single-scorer dimensions contribute std 0, inflating confidence to 1.0 with zero cross-checking. Confidence is now `agreement × coverage` (`coverage = mean(min(1, scorers/2))`): a thin panel is honestly less confident; the report exposes `coverage` separately. 4. **Nyx provenance mis-attribution** (caught by the new end-to-end test) — nyx signals carried the generic `calendar-event` sourceId of the mechanism they delegate to; they now carry `nyx-sky-events`. ## Prompt-engineering upgrades (LLM-judge discipline) - **Score anchors** in the rubric ("0.9+ exceptional … 0.5 mediocre"), the standard fix for judge score-compression; **hook judges the OPENING** explicitly; **verbosity guard** ("never word count") on both rubric and pairwise judges — length bias is the best-documented LLM-judge failure after position bias (which both pairwise paths already mitigate via both-orderings consistency). - **Multi-sample rubric judging** (`samples` option, temperature-perturbed, per-dimension means) — the same stability discipline the shared JudgeEngine uses. - **Deterministic judging clamp** (`clampForJudging`, head+tail with an explicit elision marker) — long texts no longer overflow judge context or silently truncate the ending the structure dimension needs. - **Synthesizer dedupe context** — the ideation model now sees recently produced/queued topics ("do NOT propose near-duplicates"), instead of wasting proposals the novelty scorer would kill anyway. - **Deployment-constrained planner** — the bound producer tells the planner which domains actually have live generators, so plans stop naming unproducible nodes. ## Algorithmic upgrades - **Discounted Thompson sampling** (`decayArms`, γ per cycle, default 0.98): taste is non-stationary; old evidence decays toward the prior instead of accumulating forever. - **Two-level (category, angle) arms**: outcomes update an angle arm alongside the category arm; selection blends the angle draw as a bounded factor `(0.5 + 0.5·draw)` — the plane learns that an audience loves quick-takes but rejects deep-dives _within_ a category. (A full contextual bandit over topic embeddings stays out — see the deliberate-differences table.) - **Trigram-augmented novelty** (`topicSimilarity` = max of token and char-trigram Jaccard): catches morphological near-duplicates ('eclipse'/'eclipses') embeddings would catch, without credentials. - **Failed-topic cooldown**: topics humans rejected/vetoed (or the gate parked) within a window (default 7d) are suppressed at candidate stage and reported (`suppressedByCooldown`) — the director stops relitigating fresh failures every cycle. - **Retry-After honoring**: the governor extends its AIMD cooldown to the provider's own `retryAfterSeconds`/`retry-after` hint when longer. - **Topic sanitization at admission** (single-line, 160-char cap): topics can originate from user-shaped input and flow into prompts; they are now data, not a prompt-injection channel (Lilith/editorial gates remain the content-level defense). ## Infrastructure upgrades - **Durable plane state** (`wireDurableAutonomyPlane` over the same DurableSnapshotStore the jobs queue uses): review queue, bandit posteriors, learner calibration, steering document, and cycle reports survive a restart. Without this, every deploy erased everything the humans had taught the plane. Snapshot/restore round-trip is JSON-safe and tested. - **Autonomous cycle driver** (`createAutonomyCycleDriver`, started in `server.ts` when the director is bound; `OSHUN_AUTONOMY_AUTOCYCLE=0` disables): cycles now fire on the plane's schedule (quiet hours included) with no operator trigger — the plane is autonomous in fact, not just in name. Non-reentrancy is a single latch shared with the HTTP trigger; driver errors are logged and never kill the loop. - **Full-stack end-to-end test** with only the model scripted: one content-routing fake provider serves planner, writer, artifact critic, synthesizer, rubric judge, craft judge, and pairwise judge, and the test drives real Nyx ephemeris sensing → planning → writing → the five-judge panel → tournament → gate → review queue, asserting the declared citation the Sophia judge scored. ## SOTA techniques deliberately not adopted (and why) | SOTA practice | Status here | Why | | ----------------------------------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Embedding-based semantic novelty/dedupe | trigram+token similarity | embeddings are creds-gated; the seam is one similarity function; layer it at deploy | | Contextual bandits (LinUCB/logistic TS over features) | two-level Beta arms | data volumes at launch (few items/cycle) cannot fit feature weights honestly; hierarchical arms learn from dozens, not thousands | | Judge fine-tuning / reward models | calibration EMA + κ pipeline (unbound) | needs a human-anchor gold set that does not exist yet; the shared quality-judge lib already ships the κ machinery for when it does | | Swiss/knockout tournaments for large K | round-robin | K=2–3 variants; round-robin is exact and cheaper below K≈5 | | Multi-turn agentic revision (planner replans mid-run) | bounded gate-retry with feedback | unbounded replanning without human anchors compounds judge bias; one honest retake, then humans | Test evidence for this pass: `@oshun/creative-autonomy` **131/131**; BFF agentic **65/65** (incl. the bound-plane e2e, persistence round-trip, and cycle-driver suites); both `tsc --noEmit` clean.