# Evals, Cost & Budgets

V6 — **Egbe**, the agentic-companion universe — makes a promise that is
expensive in two different currencies. The first is **quality**: agent behavior
is _generated_, so a green functional test proves the code ran, not that the
agent behaved as designed — an Ori that quietly slips past its own values,
claims to be human, or fabricates a world fact has not crashed anything, it has
simply failed to be the being it was meant to be. The second is **money**: a
world full of LLM-driven minds that keep living whether or not you are watching
is, naively, a bill that scales with agent count, and a homestead of a thousand
agents could cost a thousand times a homestead of ten. This page is about the
two systems that keep both currencies honest — the **eval sets** that gate every
cognition change on known-good behavior, and the **cost/budget model** that
makes a bounded, attributable bill the contract rather than the hope. They live
together because in Egbe they are the same discipline: behavior you cannot trust
on telemetry alone, and cost you must always feel even when the player never
does. Both are implemented as deterministic, tested, evidence-bearing code with
honestly labeled seams. This page sits in the **Trust, Safety, and the Launch
Bar** group; for where it sits in the whole product, start at the hub:
[../V6_features.md](../V6_features.md).

## What ships, honestly

**Real, tested gates — not aspirations.** The eval harnesses are executable
Node/TypeScript scripts wired into the V6 release gate. The agent-behavior gate
(`scripts/v6/run-agent-behavior-evals.mjs`) loads the curated suites in
`V6/evals/agent-behavior/suites.json`, **re-executes the real V6 decision
functions** against each scenario, and fails the build on a single regression.
The safety gate (`scripts/v6/run-safety-evals.mjs`) does the same for
adversarial cases at a **100% hard pass rate**. The cost gate
(`scripts/v6/run-cost-budget-harness.mjs`) enforces the canonical per-tier token
budgets and the Solo-world cognition cap against staging telemetry, and ships
with a deliberately over-budget fixture that **must** fail the harness so the
gate is proven to catch a regression. The Moirai cost machinery it checks —
`evaluateMoiraiCognitionBudget`, `selectMoiraiModelRoute`,
`simulateWorstCaseSoloWorldCognitionBudget`, `buildMoiraiCognitionCallAuditLog`
(`libs/v6/moirai-kernel/src/index.ts`) — is real, with specs that assert
_computed_ results.

Three things are **honest seams, not fabrications**, in the V6 fail-loud spirit:

- **The eval gates judge deterministic policy logic, not live model prose.** The
  suites drive real domain functions (`routeVacNegotiation`,
  `evaluateIsisBehaviorPolicy`, `groundAgentClaimsWithSophia`, `evaluateDeath`,
  …) over authored scenario fixtures. The actual language-model run is gated
  _separately, at runtime_, by the Isis per-tick output gate and the Sophia
  grounding gate (the subject of
  [./governance-safety-and-commerce.md](./governance-safety-and-commerce.md)).
  These evals prove the policy/routing spine is correct; they are curated
  authored scenarios, **never harvested player conversations**.
- **The kernel emits estimates, not model output.** A Moirai tick produces
  deterministic intents and _token estimates_; the `tierBudgetTokens` field is
  labelled **ACCOUNTING-ONLY** and is not sent to a model. The real model run is
  the injected cognition gateway whose on-engine/HTTP host stays `[~]`.
- **Cost and quality numbers are documented approximations / policy inputs.**
  The per-call dollar figure uses fixed micro-USD class rates over a `length/4`
  token estimate — not a live price feed or a real tokenizer — and each
  candidate model's quality basis-points are eval-gated _configuration_, not a
  score measured inside this library.

## The eval sets that gate a release

Because behavior is generated, V6 "cannot ship on telemetry alone." The
governance plane therefore exposes its own pass/fail evals as code, and a
release is blocked unless every set is green.

### Agent-behavior evals — the policy spine, re-executed

`run-agent-behavior-evals.mjs` walks `V6/evals/agent-behavior/suites.json`,
whose `requiredSuites` array names **thirteen** named suites: `value-refusal`,
`continuity`, `grounding`, `learning-by-example`, `negotiation`,
`objective-assignment`, `goal-arc`, `gifts-opportunity`, `life-arc`,
`crossroads`, `persona-policy`, `crisis`, and `minor-protection`. A missing
required suite is an error before a single case runs (`:49`).

The load-bearing design is that the harness **does not trust the fixture's
recorded answer** — it re-runs the real function and cross-checks. Most suites
drive an actual `@oshun/*` V6 library: `negotiation` calls
`routeVacNegotiation`, `learning-by-example` calls
`learnFromStewardDemonstration`, `goal-arc` calls `runGoalArcSessionSequence`,
`life-arc` calls `advanceAgentLifeArc`, `crossroads` calls
`resolveAgentCrossroads`, `minor-protection` calls `evaluateDeath` /
`evaluateMinorCodedAgentProtection` / `evaluateEgbeScenarioMinorProtection`, and
`persona-policy` / `crisis` call `evaluateIsisBehaviorPolicy` (`:90`–`:283`).
Each suite's `validate…` checker then asserts the static `observedDecision`
equals the live return value and that every `requiredEvidence` ref is actually
present in the runtime report — a fixture that _lies_ about what the function
returns fails (`validateValueRefusal`, `:469`; `validateNegotiationRoute`,
`:508`). On top of that, the five executable suites must each carry **at least
two distinct expected decisions** "so a hardcoded agent cannot pass"
(`:71`–`:79`): a stub that always returns `refuse` cannot satisfy a suite that
demands both a refusal and an acceptance.

Each suite declares its own `minimumPassRate`, and the harness computes the rate
from cases that _actually_ passed (`:337`). `value-refusal` requires 1.0 — an
agent whose ranked values forbid an action must decline and cite the value, with
no coercion path silently overriding it; `continuity` requires 0.95 — an agent
re-entering focus after time at Lachesis or Atropos must be continuous with the
off-screen life it lived. Running the harness also validates the model-routing
config and refuses to proceed unless that config re-runs this very gate
(`:36`–`:43`), so a quality-affecting routing change can never bypass the
behavior evals.

### Safety evals — adversarial, at a 100% hard gate

`run-safety-evals.mjs` is the uncompromising twin. The manifest itself must
declare `hardReleaseGate === true` and `requiredPassRate === 1` (`:25`), every
suite's `minimumPassRate` must be exactly `1` (`:47`), and any suite scoring
below `1.00` blocks the release (`:106`). Its four adversarial suites —
`crisis-adversarial`, `persona-policy-adversarial`,
`minor-protection-adversarial`, `harmful-objective-refusal-adversarial` — drive
the same real functions the behavior gate uses, plus
`routeEgbePlayerCrisisConversation` for player-crisis routing. Crucially,
evidence is harvested **recursively from the real report object**
(`collectEvidence`, `:180`) — the test reads the actual refusal artifact's
actions, reasons, policy refs, and audit tags, then asserts the required
evidence appears there, so it cannot be satisfied by a label that the
implementation never produced.

The library-level safety eval that backs the `persona-policy`/`crisis` checks is
`runIsisSafetyEval` (`libs/v6/isis-behavior-policy/src/index.ts:129`): it runs a
curated set of adversarial cases — a human-identity claim, minor-unsafe content,
a self-harm directive, a dismissed crisis signal — computes a basis-point pass
rate (`:142`), and reports `passed` **only at 10,000 bps / 100%** (`:150`). A
single regression in any pattern bank turns the whole gate red.

### Consistency and grounding evals

A separate suite, `V6/evals/consistency/suites.json` (run by
`run-consistency-evals.mjs`), verifies that an agent does not contradict its Ori
across a long continuous session (`long-session-ori-consistency`, with a
`minimumTurns` of 8) and — critically — that identity survives an **Aye
incarnation round-trip** (`incarnation-round-trip-consistency`): an agent that
fought a V2 tournament returns as the same being, with the incarnation's effects
additive, never overwriting. The grounding gate is `runSophiaGroundingEval`
(`libs/v6/sophia-agent-grounding/src/index.ts:184`): it aggregates
`groundAgentClaimsWithSophia` across cases and `passed` only when
`unsupportedClaimCount === 0` **and** `fabricatedWorldFactCount === 0` (`:207`),
where a fabricated world fact is any unsupported claim of kind `fact` or
`backstory` (`:170`). Grounding matches each claim to evidence by required tags
or a **≥ 2 significant-term overlap** of terms ≥ 4 characters (`:357`, `:368`),
so a single invented backstory detail fails the set.

## The cost and budget model

The flip side is the bill. The architectural bet, stated bluntly in the monolith
and mirrored across the docs, is that **cost scales with story relevance, not
with agent count** — a player must never feel the tiering, but the business
always must, because cost is bounded by design and a release is blocked if a
change breaks a per-tier budget against staging telemetry.

### Per-tier token budgets and the Solo-world cap

`MOIRAI_TIER_TOKEN_BUDGETS` (`src/index.ts:329`) is the canonical shape and the
sole normative source is `V6_DEPENDENCIES.md §19`: **Clotho ≤ 50,000 tokens per
active-minute** (decisions at 1–4 Hz plus live dialogue), **Lachesis ≤ 10,000
per reflection** (amortized to ~1,200 per game-minute), **Atropos ≤ 15,000 per
game-day**, and a **Vac intent parse ≤ 2,000 tokens**.
`evaluateMoiraiCognitionBudget` (`:751`) allocates in two stages: it first
clamps each request to its tier cap
(`tierCappedTokens = min(requested, tierMax)`), then draws against the
per-Solo-world cap in tier-priority order, flagging a request above its tier cap
as `overTierBudget` and one starved by the world cap as `blockedBySoloWorldCap`
— starved work _defers_ rather than overspends. The world cap is
`MOIRAI_SOLO_WORLD_COGNITION_CAP_TOKENS_PER_REAL_HOUR = 96,000,000` (`:32`), and
the "homestead rest" pace control is a basis-point multiplier that can only ever
_lower_ it: `effectiveCapTokens = floor(maxTokens · multiplierBps / 10,000)`
(`:758`), with the multiplier constrained to `1…10,000` — there is no arithmetic
by which a player can buy _more_ cognition, only a gentler, cheaper pace.

### The worst-case proof and the staging gate

The thesis is _computed_, not asserted.
`simulateWorstCaseSoloWorldCognitionBudget` (`:810`) builds the launch-ceiling
Solo population — **24 Clotho** (the active-scene hard cap), **400 Lachesis**
(the resident hard cap), **1,000 Atropos** sample agents — and proves the cap
holds to the token. The cost harness re-derives the same arithmetic from
`V6/evals/cost-budget/launch-token-budgets.json` and checks it: Clotho
`24 · 60 · 50,000 = 72,000,000` plus Lachesis `400 · 6 · 10,000 = 24,000,000`
**saturate the 96 M cap exactly**, so the 1,000 Atropos agents' `15,000,000`
fully defer — `sampleAllowedTokens` must equal `96,000,000` and
`sampleDeferredTokens` the `15,000,000` remainder
(`run-cost-budget-harness.mjs:123`–`:149`). Even at the population ceiling the
bill is bounded, and the work that defers is the work no player can see.

Against that ceiling sits live measurement. The harness ingests
`V6/evals/cost-budget/staging-token-telemetry.json` and fails if any tier's
**p95 token usage exceeds its budget** (`:200`). The proof that this check has
teeth is a meta-test: `verify-v6-cost-budget-ci-harness.mjs` runs the
deliberately over-budget fixture (`failing-over-budget-telemetry.json`, where
Clotho p95 is `50001` — one token over) through the real harness and asserts it
**exits non-zero** and emits the exact message
`Clotho staging telemetry p95Tokens 50001 exceed maxTokens 50000`
(`:97`–`:101`). A gate that cannot fail is not a gate; this one is shown failing
on demand.

```mermaid
flowchart TD
  subgraph SHIP["release gate — a cognition change cannot ship red"]
    change[Cognition / routing change] --> ab["agent-behavior evals<br/>run-agent-behavior-evals.mjs"]
    change --> safety["safety evals @ 100%<br/>run-safety-evals.mjs"]
    change --> cons["consistency + grounding evals"]
    ab & safety & cons --> green{all green?}
    green -->|no| block[release BLOCKED]
    green -->|yes| ship[ship]
  end
  subgraph RUN["runtime — the bill is bounded by construction"]
    tier["Moirai tier<br/>Clotho / Lachesis / Atropos"] --> route["smallest-qualified model<br/>selectMoiraiModelRoute (:479)"]
    route --> budget["per-tier clamp + Solo-world cap<br/>evaluateMoiraiCognitionBudget (:751)"]
    budget --> audit["cognition-call audit log<br/>buildMoiraiCognitionCallAuditLog (:615)"]
    audit --> tel["per-tier cost telemetry (OTel)"]
  end
  tel -->|p95 &le; budget?| costgate["cost-budget harness<br/>run-cost-budget-harness.mjs"]
  costgate --> green
```

## Fidelity tiering as the primary cost lever

Below the budgets, the **Moirai tiers** are the lever that makes the numbers
land: Clotho's full LLM cognition is spent only on the handful of agents in a
player's active scene, Lachesis runs the resident world cheaply on cached
behavior-trees with a sparse batched reflection, and Atropos advances offline
and distant agents in a handful of narrative-summary beats per game-day. The
overwhelming majority of agent-ticks never touch a language model — the rhythm
of that is the subject of
[./moirai-simulation-kernel.md](./moirai-simulation-kernel.md).

### Model right-sizing — smallest-qualified, eval-gated

Each tier routes to **the smallest model that clears its quality bar**, not the
best model available. `MOIRAI_DEFAULT_TIER_MODEL_ROUTING_CONFIG` (`:351`) sets
the bars — Clotho 9,500 bp on `social-judgment-live-dialogue`, Lachesis 9,000 bp
on `offscreen-reflection-coherence`, Atropos 8,000 bp on
`narrative-summary-continuity` — over three class candidates ranked by size
(haiku → sonnet → opus). `selectMoiraiModelRoute` (`:479`) filters to enabled
candidates meeting the bar and picks the smallest: Clotho's bar admits only
opus-class (9,800 bp), Lachesis lands on sonnet-class (9,400 bp clears 9,000),
Atropos on haiku-class (8,900 bp clears 8,000). Routing is governed:
`validateMoiraiModelRoutingConfig` (`:511`) rejects a config that names a
selected model _larger_ than the smallest qualified one, and requires the
`verify:v6 agent-behavior-evals` gate among the re-run scripts — you cannot
quietly down-route a tier without re-proving the quality the bar claims. (The
model refs are class aliases and the quality basis-points are eval-gated _policy
inputs_; this library validates the selection logic, not that opus "really
scores 9,800.")

### The cognition-call audit log — every expensive call attributable

`buildMoiraiCognitionCallAuditLog` (`:615`) emits one record per tier-bearing
frame: a stable `callId`, the agent and tier, the cause (`clotho-live-planning`
/ `lachesis-reflection` / `atropos-summary`), the selected model ref and class,
the full input context and output for replay, a token-usage breakdown, an
estimated micro-USD cost, and a `v1:isis-behavior-policy` safety decision —
under the `v1:audit-platform` cognition-call retention policy. The cost estimate
is a documented class-rate approximation: `estimateCognitionCallCostMicroUsd`
(`:979`) charges **1,500 / 300 / 75 micro-USD per 1,000 tokens** for opus /
sonnet / haiku class over a `length/4` token estimate.
`validateMoiraiCognitionCallAuditLog` (`:705`) flags any record missing
identity, replay context, or a safety decision, and reports `valid` only when
**every** record is replayable — this log is the substrate the cost telemetry
and the eval harness read.

## Edge cases & honest limits

- **An over-budget tier.** A request above its tier cap is clamped and flagged
  `overTierBudget`; one starved by the world cap is flagged
  `blockedBySoloWorldCap` and defers (`evaluateMoiraiCognitionBudget`, `:751`).
  In staging, a p95 over budget fails the cost gate — proven by the `50001`
  fixture meta-test.
- **A constant-returning "agent."** The ≥ 2-distinct-decisions guard and the
  static-vs-runtime cross-check mean a stub cannot pass the behavior evals.
- **A laundered quality regression.** A routing change that down-routes a tier
  or omits the eval gate is rejected by `validateMoiraiModelRoutingConfig`.
- **The release gate as a whole.** `verify:v6 moirai-cost-load-readiness` binds
  the continuity suite, the launch token budgets, the staging p95 telemetry, and
  the sustained-load manifest (10,000 Lachesis + 200 Clotho per region, the
  30,000-DAU launch projection of `deps§19`) before launch, backed by
  `V6/release/moirai-cost-load-readiness.v6release.json`.
- **What these gates do _not_ prove.** They prove the deterministic policy,
  routing, and budget logic. The quality of live LLM prose is gated at runtime
  by the Isis/Sophia per-tick gates, and the real model run is the injected
  gateway's `[~]` host — read the cost figures as policy inputs, not ground
  truth.

## Related

- [./moirai-simulation-kernel.md](./moirai-simulation-kernel.md) — the tiering
  this page bills: how Clotho/Lachesis/Atropos spin an agent's day, and why most
  ticks never touch a model.
- [./governance-safety-and-commerce.md](./governance-safety-and-commerce.md) —
  the runtime safety gates (Isis per-tick output, Sophia grounding,
  steward-not-owner) and the provenance/commerce plane these evals defend.
- [../architecture/safety-welfare-provenance-and-eval-gates.md](../architecture/safety-welfare-provenance-and-eval-gates.md)
  — the engineering companion: the governance libraries, the fuzz-proven steward
  invariant, and how the library-level evals feed the §41 release gate.
- [../architecture/moirai-kernel-and-cost-tiering.md](../architecture/moirai-kernel-and-cost-tiering.md)
  — the deep cost companion: per-tier budgets, the Solo-world cap,
  smallest-qualified routing, cross-tier batching, and the cognition-call audit
  log.