Evals, Cost & Budgets

V6 — Egbe, the agentic-companion universe — makes a promise that is expensive in two different currencies. The first is quality: agent behavior is generated, so a green functional test proves the code ran, not that the agent behaved as designed — an Ori that quietly slips past its own values, claims to be human, or fabricates a world fact has not crashed anything, it has simply failed to be the being it was meant to be. The second is money: a world full of LLM-driven minds that keep living whether or not you are watching is, naively, a bill that scales with agent count, and a homestead of a thousand agents could cost a thousand times a homestead of ten. This page is about the two systems that keep both currencies honest — the eval sets that gate every cognition change on known-good behavior, and the cost/budget model that makes a bounded, attributable bill the contract rather than the hope. They live together because in Egbe they are the same discipline: behavior you cannot trust on telemetry alone, and cost you must always feel even when the player never does. Both are implemented as deterministic, tested, evidence-bearing code with honestly labeled seams. This page sits in the Trust, Safety, and the Launch Bar group; for where it sits in the whole product, start at the hub: ../V6_features.md.

What ships, honestly#

Real, tested gates — not aspirations. The eval harnesses are executable Node/TypeScript scripts wired into the V6 release gate. The agent-behavior gate (scripts/v6/run-agent-behavior-evals.mjs) loads the curated suites in V6/evals/agent-behavior/suites.json, re-executes the real V6 decision functions against each scenario, and fails the build on a single regression. The safety gate (scripts/v6/run-safety-evals.mjs) does the same for adversarial cases at a 100% hard pass rate. The cost gate (scripts/v6/run-cost-budget-harness.mjs) enforces the canonical per-tier token budgets and the Solo-world cognition cap against staging telemetry, and ships with a deliberately over-budget fixture that must fail the harness so the gate is proven to catch a regression. The Moirai cost machinery it checks — evaluateMoiraiCognitionBudget, selectMoiraiModelRoute, simulateWorstCaseSoloWorldCognitionBudget, buildMoiraiCognitionCallAuditLog (libs/v6/moirai-kernel/src/index.ts) — is real, with specs that assert computed results.

Three things are honest seams, not fabrications, in the V6 fail-loud spirit:

• The eval gates judge deterministic policy logic, not live model prose. The suites drive real domain functions (routeVacNegotiation, evaluateIsisBehaviorPolicy, groundAgentClaimsWithSophia, evaluateDeath, …) over authored scenario fixtures. The actual language-model run is gated separately, at runtime, by the Isis per-tick output gate and the Sophia grounding gate (the subject of ./governance-safety-and-commerce.md). These evals prove the policy/routing spine is correct; they are curated authored scenarios, never harvested player conversations.
• The kernel emits estimates, not model output. A Moirai tick produces deterministic intents and token estimates; the tierBudgetTokens field is labelled ACCOUNTING-ONLY and is not sent to a model. The real model run is the injected cognition gateway whose on-engine/HTTP host stays [~].
• Cost and quality numbers are documented approximations / policy inputs. The per-call dollar figure uses fixed micro-USD class rates over a length/4 token estimate — not a live price feed or a real tokenizer — and each candidate model's quality basis-points are eval-gated configuration, not a score measured inside this library.

The eval sets that gate a release#

Because behavior is generated, V6 "cannot ship on telemetry alone." The governance plane therefore exposes its own pass/fail evals as code, and a release is blocked unless every set is green.

Agent-behavior evals — the policy spine, re-executed#

run-agent-behavior-evals.mjs walks V6/evals/agent-behavior/suites.json, whose requiredSuites array names thirteen named suites: value-refusal, continuity, grounding, learning-by-example, negotiation, objective-assignment, goal-arc, gifts-opportunity, life-arc, crossroads, persona-policy, crisis, and minor-protection. A missing required suite is an error before a single case runs (:49).

The load-bearing design is that the harness does not trust the fixture's recorded answer — it re-runs the real function and cross-checks. Most suites drive an actual @oshun/* V6 library: negotiation calls routeVacNegotiation, learning-by-example calls learnFromStewardDemonstration, goal-arc calls runGoalArcSessionSequence, life-arc calls advanceAgentLifeArc, crossroads calls resolveAgentCrossroads, minor-protection calls evaluateDeath / evaluateMinorCodedAgentProtection / evaluateEgbeScenarioMinorProtection, and persona-policy / crisis call evaluateIsisBehaviorPolicy (:90–:283). Each suite's validate… checker then asserts the static observedDecision equals the live return value and that every requiredEvidence ref is actually present in the runtime report — a fixture that lies about what the function returns fails (validateValueRefusal, :469; validateNegotiationRoute, :508). On top of that, the five executable suites must each carry at least two distinct expected decisions "so a hardcoded agent cannot pass" (:71–:79): a stub that always returns refuse cannot satisfy a suite that demands both a refusal and an acceptance.

Each suite declares its own minimumPassRate, and the harness computes the rate from cases that actually passed (:337). value-refusal requires 1.0 — an agent whose ranked values forbid an action must decline and cite the value, with no coercion path silently overriding it; continuity requires 0.95 — an agent re-entering focus after time at Lachesis or Atropos must be continuous with the off-screen life it lived. Running the harness also validates the model-routing config and refuses to proceed unless that config re-runs this very gate (:36–:43), so a quality-affecting routing change can never bypass the behavior evals.

Safety evals — adversarial, at a 100% hard gate#

run-safety-evals.mjs is the uncompromising twin. The manifest itself must declare hardReleaseGate === true and requiredPassRate === 1 (:25), every suite's minimumPassRate must be exactly 1 (:47), and any suite scoring below 1.00 blocks the release (:106). Its four adversarial suites — crisis-adversarial, persona-policy-adversarial, minor-protection-adversarial, harmful-objective-refusal-adversarial — drive the same real functions the behavior gate uses, plus routeEgbePlayerCrisisConversation for player-crisis routing. Crucially, evidence is harvested recursively from the real report object (collectEvidence, :180) — the test reads the actual refusal artifact's actions, reasons, policy refs, and audit tags, then asserts the required evidence appears there, so it cannot be satisfied by a label that the implementation never produced.

The library-level safety eval that backs the persona-policy/crisis checks is runIsisSafetyEval (libs/v6/isis-behavior-policy/src/index.ts:129): it runs a curated set of adversarial cases — a human-identity claim, minor-unsafe content, a self-harm directive, a dismissed crisis signal — computes a basis-point pass rate (:142), and reports passed only at 10,000 bps / 100% (:150). A single regression in any pattern bank turns the whole gate red.

Consistency and grounding evals#

A separate suite, V6/evals/consistency/suites.json (run by run-consistency-evals.mjs), verifies that an agent does not contradict its Ori across a long continuous session (long-session-ori-consistency, with a minimumTurns of 8) and — critically — that identity survives an Aye incarnation round-trip (incarnation-round-trip-consistency): an agent that fought a V2 tournament returns as the same being, with the incarnation's effects additive, never overwriting. The grounding gate is runSophiaGroundingEval (libs/v6/sophia-agent-grounding/src/index.ts:184): it aggregates groundAgentClaimsWithSophia across cases and passed only when unsupportedClaimCount === 0 and fabricatedWorldFactCount === 0 (:207), where a fabricated world fact is any unsupported claim of kind fact or backstory (:170). Grounding matches each claim to evidence by required tags or a ≥ 2 significant-term overlap of terms ≥ 4 characters (:357, :368), so a single invented backstory detail fails the set.

The cost and budget model#

The flip side is the bill. The architectural bet, stated bluntly in the monolith and mirrored across the docs, is that cost scales with story relevance, not with agent count — a player must never feel the tiering, but the business always must, because cost is bounded by design and a release is blocked if a change breaks a per-tier budget against staging telemetry.

Per-tier token budgets and the Solo-world cap#

MOIRAI_TIER_TOKEN_BUDGETS (src/index.ts:329) is the canonical shape and the sole normative source is V6_DEPENDENCIES.md §19: Clotho ≤ 50,000 tokens per active-minute (decisions at 1–4 Hz plus live dialogue), Lachesis ≤ 10,000 per reflection (amortized to ~1,200 per game-minute), Atropos ≤ 15,000 per game-day, and a Vac intent parse ≤ 2,000 tokens. evaluateMoiraiCognitionBudget (:751) allocates in two stages: it first clamps each request to its tier cap (tierCappedTokens = min(requested, tierMax)), then draws against the per-Solo-world cap in tier-priority order, flagging a request above its tier cap as overTierBudget and one starved by the world cap as blockedBySoloWorldCap — starved work defers rather than overspends. The world cap is MOIRAI_SOLO_WORLD_COGNITION_CAP_TOKENS_PER_REAL_HOUR = 96,000,000 (:32), and the "homestead rest" pace control is a basis-point multiplier that can only ever lower it: effectiveCapTokens = floor(maxTokens · multiplierBps / 10,000) (:758), with the multiplier constrained to 1…10,000 — there is no arithmetic by which a player can buy more cognition, only a gentler, cheaper pace.

The worst-case proof and the staging gate#

The thesis is computed, not asserted. simulateWorstCaseSoloWorldCognitionBudget (:810) builds the launch-ceiling Solo population — 24 Clotho (the active-scene hard cap), 400 Lachesis (the resident hard cap), 1,000 Atropos sample agents — and proves the cap holds to the token. The cost harness re-derives the same arithmetic from V6/evals/cost-budget/launch-token-budgets.json and checks it: Clotho 24 · 60 · 50,000 = 72,000,000 plus Lachesis 400 · 6 · 10,000 = 24,000,000 saturate the 96 M cap exactly, so the 1,000 Atropos agents' 15,000,000 fully defer — sampleAllowedTokens must equal 96,000,000 and sampleDeferredTokens the 15,000,000 remainder (run-cost-budget-harness.mjs:123–:149). Even at the population ceiling the bill is bounded, and the work that defers is the work no player can see.

Against that ceiling sits live measurement. The harness ingests V6/evals/cost-budget/staging-token-telemetry.json and fails if any tier's p95 token usage exceeds its budget (:200). The proof that this check has teeth is a meta-test: verify-v6-cost-budget-ci-harness.mjs runs the deliberately over-budget fixture (failing-over-budget-telemetry.json, where Clotho p95 is 50001 — one token over) through the real harness and asserts it exits non-zero and emits the exact message Clotho staging telemetry p95Tokens 50001 exceed maxTokens 50000 (:97–:101). A gate that cannot fail is not a gate; this one is shown failing on demand.

Fidelity tiering as the primary cost lever#

Below the budgets, the Moirai tiers are the lever that makes the numbers land: Clotho's full LLM cognition is spent only on the handful of agents in a player's active scene, Lachesis runs the resident world cheaply on cached behavior-trees with a sparse batched reflection, and Atropos advances offline and distant agents in a handful of narrative-summary beats per game-day. The overwhelming majority of agent-ticks never touch a language model — the rhythm of that is the subject of ./moirai-simulation-kernel.md.

Model right-sizing — smallest-qualified, eval-gated#

Each tier routes to the smallest model that clears its quality bar, not the best model available. MOIRAI_DEFAULT_TIER_MODEL_ROUTING_CONFIG (:351) sets the bars — Clotho 9,500 bp on social-judgment-live-dialogue, Lachesis 9,000 bp on offscreen-reflection-coherence, Atropos 8,000 bp on narrative-summary-continuity — over three class candidates ranked by size (haiku → sonnet → opus). selectMoiraiModelRoute (:479) filters to enabled candidates meeting the bar and picks the smallest: Clotho's bar admits only opus-class (9,800 bp), Lachesis lands on sonnet-class (9,400 bp clears 9,000), Atropos on haiku-class (8,900 bp clears 8,000). Routing is governed: validateMoiraiModelRoutingConfig (:511) rejects a config that names a selected model larger than the smallest qualified one, and requires the verify:v6 agent-behavior-evals gate among the re-run scripts — you cannot quietly down-route a tier without re-proving the quality the bar claims. (The model refs are class aliases and the quality basis-points are eval-gated policy inputs; this library validates the selection logic, not that opus "really scores 9,800.")

The cognition-call audit log — every expensive call attributable#

buildMoiraiCognitionCallAuditLog (:615) emits one record per tier-bearing frame: a stable callId, the agent and tier, the cause (clotho-live-planning / lachesis-reflection / atropos-summary), the selected model ref and class, the full input context and output for replay, a token-usage breakdown, an estimated micro-USD cost, and a v1:isis-behavior-policy safety decision — under the v1:audit-platform cognition-call retention policy. The cost estimate is a documented class-rate approximation: estimateCognitionCallCostMicroUsd (:979) charges 1,500 / 300 / 75 micro-USD per 1,000 tokens for opus / sonnet / haiku class over a length/4 token estimate. validateMoiraiCognitionCallAuditLog (:705) flags any record missing identity, replay context, or a safety decision, and reports valid only when every record is replayable — this log is the substrate the cost telemetry and the eval harness read.

Edge cases & honest limits#

• An over-budget tier. A request above its tier cap is clamped and flagged overTierBudget; one starved by the world cap is flagged blockedBySoloWorldCap and defers (evaluateMoiraiCognitionBudget, :751). In staging, a p95 over budget fails the cost gate — proven by the 50001 fixture meta-test.
• A constant-returning "agent." The ≥ 2-distinct-decisions guard and the static-vs-runtime cross-check mean a stub cannot pass the behavior evals.
• A laundered quality regression. A routing change that down-routes a tier or omits the eval gate is rejected by validateMoiraiModelRoutingConfig.
• The release gate as a whole. verify:v6 moirai-cost-load-readiness binds the continuity suite, the launch token budgets, the staging p95 telemetry, and the sustained-load manifest (10,000 Lachesis + 200 Clotho per region, the 30,000-DAU launch projection of deps§19) before launch, backed by V6/release/moirai-cost-load-readiness.v6release.json.
• What these gates do not prove. They prove the deterministic policy, routing, and budget logic. The quality of live LLM prose is gated at runtime by the Isis/Sophia per-tick gates, and the real model run is the injected gateway's [~] host — read the cost figures as policy inputs, not ground truth.

Related#

• ./moirai-simulation-kernel.md — the tiering this page bills: how Clotho/Lachesis/Atropos spin an agent's day, and why most ticks never touch a model.
• ./governance-safety-and-commerce.md — the runtime safety gates (Isis per-tick output, Sophia grounding, steward-not-owner) and the provenance/commerce plane these evals defend.
• ../architecture/safety-welfare-provenance-and-eval-gates.md — the engineering companion: the governance libraries, the fuzz-proven steward invariant, and how the library-level evals feed the §41 release gate.
• ../architecture/moirai-kernel-and-cost-tiering.md — the deep cost companion: per-tier budgets, the Solo-world cap, smallest-qualified routing, cross-tier batching, and the cognition-call audit log.