# Phase 0 Triage — Faked ML / Reasoning Stack (Task 0.1) Ledger: `AGENTIC_CONTENT_QUALITY_TODOS_2026-06-13.md` § Phase 0. Date: 2026-06-13. Verifier read every file/line cited below in-session. ## Method - Fabrication scan: `grep -nE 'Math\.(sin|cos|exp|random)'` over each named impl file, then **read each hit in context** to separate real math (DPO sigmoid, cosine-LR schedule, non-secret id suffix) from fabricated training/score curves (a `Math.sin(step·k)` oscillation of a step/round/progress counter with no model behind it). - Caller scan: `grep -rEl "from '@nous/training'|from '@nous/llm'"` across the repo, excluding `node_modules`, `dist`, `.claude/worktrees`, and `*.spec.ts`. Then inspected each importer's exact import list. ## Caller findings (who consumes these packages, non-test) | Importer | Package | Touches a named fake? | | ---------------------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `libs/nous/inference-acceleration/src/index.ts` | `@nous/training` | No — pulls StreamDiffusion / DeepCache / FlowMatching accel only | | `libs/nous/diffusion-alignment/src/index.ts` | `@nous/training` | **Type-only**: re-exports `DiffusionDPOTrainingRequest`, `DiffusionCurriculumDPOTrainingRequest`, `FluxDPOTrainingRequest` types (no runtime use) | | `V2/services/nous-anti-cheat-classifiers/src/nous-anti-cheat-classifiers.ts` | `@nous/training` | No — anti-cheat training-plan symbols only | | `libs/kalika/research-agents/src/nous-integration.ts` | `@nous/llm` | No — agent/tool/delegation **types** only; not the reasoning files | **Runtime callers of the 15 named fakes outside lib+spec: zero.** The only cross-package dependency is `@nous/diffusion-alignment` re-exporting three _request types_ (not values). → Fail-loud must preserve those type exports; deletion would require rewiring `diffusion-alignment` + `training/index.ts`. ## Triage table Decision key: **fail-loud** = keep the typed interface, replace every fabricated metric/score with `throw new NotConfiguredError(...)`; **keep** = honest today, Phase 1 wires it. Deletion was considered and rejected (see Rationale). ### `libs/nous/training/src/` (training metric fakers) | File | Fabrication evidence (file:line) | Non-test caller | Decision | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `rlaif.ts` | `deriveRoundMetrics` `:361` `Math.sin(round*0.41)*0.025`; `:756` `…+Math.sin(phase)*0.04`; `:761` `…+Math.cos(phase)*0.08` — per-round reward/agreement = closed form of `round` + config, no model | none | fail-loud (0.2) | | `reward-modeling.ts` | `:773` `perplexity: Math.exp(min(10, validationLoss))` over a fabricated `validationLoss`; `deriveRewardMetrics` synthesizes accuracy/loss from step | none | fail-loud (0.2) | | `best-of-n-sampling.ts` | `:413` candidate score `phase = round*0.57 + sampleIndex*0.23 + promptId.length*0.04`; `:414` `Math.sin(phase)*0.07+Math.cos(phase*0.61)*0.05` — **scores by id string length, never reads candidate text** | none | fail-loud (0.2); real best-of-N lives in the content stack (Phase 2.2), not here | | `dpo-training.ts` | `:407` `Math.sin(step*0.17)*0.08`; `:464` `Math.cos(step*0.11)*0.02` fake metric curve. (`:203/206` `Math.exp(±value)` = real DPO sigmoid — keep; `:390` cosine-LR — keep) | none | fail-loud (0.2) | | `ppo-training.ts` | `:206` `Math.sin(update*0.15)*0.04` fake metric | none | fail-loud (0.2) | | `constitutional-ai.ts` | `:406` `Math.sin(step*0.16)*0.08`; `:511` `Math.cos(step*0.12)*0.02` fake metric (`:398` cosine-LR — keep) | none | fail-loud (0.2) | | `diffusion-dpo-training.ts` | `:579` `Math.sin(step*0.13)*0.07`; `:676` `Math.cos(step*0.09)*0.02` (`:288/291` DPO sigmoid, `:562` cosine-LR — keep) | type re-export only | fail-loud (0.2); **preserve `DiffusionDPOTrainingRequest` type** | | `diffusion-curriculum-dpo.ts` | `:1425` `baseLoss*Math.exp(-trainingProgress*2.6)+…` fabricated loss curve (`:851` cosine-LR — keep) | type re-export only | fail-loud (0.2); **preserve `DiffusionCurriculumDPOTrainingRequest` type** | | `flux-dpo-training.ts` | `:1148` `0.72*Math.exp(-progress*2.1)+…`; `:1173` `0.58*Math.exp(-progress*2.5)` fabricated metric curves (`:583` cosine-LR — keep) | type re-export only | fail-loud (0.2); **preserve `FluxDPOTrainingRequest` type** | | `iterative-refinement.ts` | `:392` `Math.exp(-(iteration-1)*0.45)` diminishing-returns + `:394` `Math.sin(phase)*0.01` fabricated refinement-gain curve | none | fail-loud (0.2). NB: ledger 0.1 listed this under `llm/` — it is in `training/`. The honest critique→revise refiner is Phase 2.3, built fresh in the content stack | ### `libs/nous/llm/src/` (reasoning result fakers / prompt builders) | File | Nature (evidence) | Non-test caller | Decision | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- | --------------------------------------------------------------------------------- | | `tree-of-thought.ts` | `generateCandidates` `:421` token-substitutes the first 3 tokenized keywords into fixed templates (`"Analyze how ${keyA} drives the core failure pattern…"`). No provider, no `await`. Returns a "reasoning tree" it did not reason → stub | none | fail-loud (0.3) — require a real provider; name reserved for Phase 1–2 | | `graph-of-thought.ts` | `generateCandidates` `:406` same keyword-template fabrication, no provider | none | fail-loud (0.3) — require a real provider | | `multi-step-reasoning.ts` | No `await`/provider/llm seam anywhere; synchronous template logic emits "reasoning steps" | none | fail-loud (0.3) — require a real provider | | `critique-prompts.ts` | **Honest prompt builder** — returns `{system,user,rubric,…}` prompt strings; fabricates nothing. Gap is only "never sent to a model" | none | keep — Phase 1 wires the rubric into the real judge | | `reflection-prompts.ts` | Prompt builder + a heuristic self-reflection path (`:292` word-count gate emitting "completeness" findings). Prompt-build = keep; the heuristic-finding path that asserts a reflection it didn't reason → fail-loud that path | none | keep-and-fix (0.3) — keep prompt build, fail-loud the heuristic "analysis" output | ## Rationale: fail-loud over deletion 1. **CLAUDE.md** is explicit: _"Honest fail-loud seams … are the opposite of a stub and the correct way to represent a real-but-absent integration"_ and _"When unsure: fail loud, or ask."_ A typed method that `throw`s `not_configured` is the sanctioned representation. 2. **Lower blast radius / reversible.** Deletion = remove 15 impl + 15 spec files, strip 15 `export *` lines from `training/index.ts`, and rewire the 3 type re-exports in `@nous/diffusion-alignment`. Fail-loud keeps the public type surface intact, so no out-of-scope cascade into the diffusion libs. 3. **Names + interfaces are reserved.** Ledger 0.3/2.2/5.3 reserve these names and the `*Request`/`*Result` contracts for the real Phase 1–2/5 implementations. Keeping the contract means the real trainer/searcher drops into a defined shape rather than reinventing it. 4. **The honest core survives.** Input validation and pure config arithmetic (prompt/batch/token counts, real DPO sigmoid, cosine-LR schedules) are real and stay; only the fabricated _metric/score emission_ becomes a loud throw. Deletion remains a valid alternative if the maintainer prefers a smaller surface — it is a one-commit revert of the fail-loud husks plus the index/diffusion rewire. Recorded here so the choice is explicit, not silent. ## Execution plan (drives 0.2 / 0.3) - **0.2** — add `NotConfiguredError` to `@nous/training`; in each training file above, make the metric-emitting runtime path throw it; delete fabricated forecast fields from "planning diagnostics"; rewrite each spec to assert the throw + the surviving honest config math. Adversarial grep `Math\.(sin|cos|exp)` in `training/src/` must return only DPO-sigmoid / cosine-LR / real-loss hits afterward. - **0.3** — `NotConfiguredError` in `@nous/llm`; `tree-of-thought` / `graph-of-thought` / `multi-step-reasoning` require a provider and throw without one; `reflection-prompts` keeps prompt-build, fail-louds the heuristic finding path; `critique-prompts` kept as-is. Specs rewritten accordingly. ## 0.2 completion status (2026-06-13) — scope boundary, logged not silent **Done + verified** (the RLHF / preference / alignment training family this ledger targets): `rlaif`, `reward-modeling`, `dpo-training`, `ppo-training`, `constitutional-ai`, `diffusion-dpo-training`, `diffusion-curriculum-dpo`, `flux-dpo-training`, `honesty-training`, `harmlessness-training`, `helpfulness-training`, `ipo-training`, `kto-training`, `orpo-training`, `safety-training`, `self-play-training`, `rejection-sampling`, `preference-learning` — **18 modules failed-loud**. Each: `Math.sin|cos|exp` adversarial grep CLEAN, fabricated `expected*` quality forecasts removed (real config counts kept), spec asserts `NotConfiguredError` + `not_configured` code. Verified by the parent (not subagent self-report): \*\*`@nous/training` tsc clean - 645/645 tests pass; `@nous/diffusion-alignment` tsc clean + 3/3 pass.\** The three `*Request`types re-exported by`@nous/diffusion-alignment` are preserved. Cascade caught by the parent's full-lib build (the 0.1 cross-_package_ scan missed intra-package consumers): `preference-learning` composed `reward-modeling` and read its removed forecasts; `preference-aligned-inference` spec minted a FLUX artifact by running the now-failing trainer (rebuilt as a typed artifact literal — a boundary test-double; impl unchanged); `diffusion-alignment` spec asserted a removed forecast (repointed to a real diagnostic). **NOT done — out of this ledger's scope (the dir-wide grep is over-broad vs 0.2's named target list).** The identical `Math.exp(-progress·k)` fake-loss / oscillation pattern also lives in the **image-diffusion-distillation / base-SFT training stack**: `supervised-fine-tuning`, `consistency-model-training`, `lcm-lora-training`, `flow-matching-lcm-lora`, `scot-training` (+ the ~20 PEFT modules that compose `instruction-tuning` / `supervised-fine-tuning`). These are a different domain (image/SFT training, the 2026-06-12 SOTA integration ledger), and failing them loud cascades through the whole PEFT stack — explicitly **not** the content-quality concern Phase 0 addresses. Tracked here so the remaining grep hits are explained, not silently dropped. `best-of-n-sampling` and `iterative-refinement` are Phase 0.3. Legitimate `Math.*` that correctly remains in `training/src`: real DPO sigmoid in logsigmoid losses, cosine-LR schedules, lognormal `sigma` noise schedules (`consistency-training-data-pipeline`), cosine²/temporal-shape weighting (`deepcache`), and `Math.random().toString(36)` non-secret id suffixes. ## 0.3 completion status (2026-06-13) **Done + verified.** Added `NotConfiguredError` to `@nous/llm`. The reasoning engines that token-substituted templates with no model now require a real LLM provider and throw without one: `tree-of-thought.buildTree`, `graph-of-thought.buildGraph`, `multi-step-reasoning.reason` (all fabrication helpers deleted; exported types/constructor/`getStats` preserved for the real Phase 1–2 impl). `critique-prompts` and `reflection-prompts` keep their honest prompt **builders** and model-output **parsers** (`buildPrompt`, `parseCritique`, `parseReflection`) but fail-loud the heuristic finding paths (`critiqueResponse`, `analyzeDraft`) that invented `severity`/`scoreImpact` judgments from word-counts/regex with no model. In `@nous/training`, `best-of-n-sampling` (scored candidates by id string length) and `iterative-refinement` (exp/sin gain curve) fail loud like the rlaif family. Verified by the parent: **`@nous/llm` tsc clean + 423/423 tests pass; `@nous/training` tsc clean + 647/647 pass.** The real best-of-N / self-refine for content are Phases 2.2 / 2.3 (built fresh in the content stack), not these `@nous/*` husks. ## 0.4 completion status (2026-06-13) **Done + verified.** The four `libs/yemaya/agents/src/quality-assurance/` scorers each exported an LLM-judge provider interface that was **never wired** (declaration-only — the illusion that LLM judging exists) and fell back to a fabricated quality default when the real path was absent: - `narrative-quality-benchmark` aggregated caller-supplied `dimensionScores` but folded a fabricated **`?? 50`** for every missing dimension into the weighted "quality" (7 sites, incl. the subject↔reference comparison the subagent additionally found). Now aggregates only over **present** dimensions with renormalized weights, exposes `scoredDimensions`/`dimensionCoverage`, and **throws** honestly when zero dimensions are scored (no external consumers). - `dialogue-naturalness-scorer.parseLLMScoreResponse` fabricated **50/0** when the real LLM response was unparseable; now returns `{ reason }` and leaves the score undefined so the rule-based score is left untouched. `evaluateWithLLM` already fails loud (throws) without a handler. - `ending-satisfaction-predictor.predictSatisfaction` folded **0.5** pacing/conformity/surprise into `overallScore` when no ending was set; now fails loud (guard) before any analysis (genuine extra violation the subagent found beyond the dead interface). - `mystery-fairness-validator`: adversarial scan confirmed the remaining `50`s are legitimate per-subgenre config thresholds and the other defaults are domain-constant lookups / vacuous-truth divide-by-zero guards — **no** fabricated quality default. Only the dead interface was annotated. All three dead provider interfaces are **kept and annotated** `Reserved for Phase 1 (content-quality-judge) wiring — NOT yet connected`. Verified by the parent: **yemaya/agents tsc clean; 222/222 quality-assurance tests pass.** ### Phase 0 net result The faked ML/reasoning/judge stack no longer emits a single fabricated metric, score, reasoning trace, or quality judgment when its real backend is absent — it fails loud (`NotConfiguredError` / throw) or reports honest absence (`{ scored:false }` / `undefined` + reason). ~30 modules across `@nous/training`, `@nous/llm`, and `yemaya/agents` remediated and individually test-verified. The names + typed interfaces are preserved for the real Phase 1–2 implementations. (Out of scope, logged above: the image-diffusion/SFT distillation training simulators in `@nous/training` share the pattern but belong to the SOTA ledger.)