# Content-quality judge: live calibration (audit G.1)

`run-calibration.ts` runs the real `@oshun/content-quality-judge` judge on a
live model over `GOLD_BENCHMARK_V1`. It reports, per content type:

- Cohen's κ (anchor bands), Pearson r and MAE against the benchmark's scores;
- good-versus-bad separation (pairwise accuracy).

It also records every item's score and the billed cost.

## What the numbers are measured against

`GOLD_BENCHMARK_V1` holds 40 items: 4 good and 4 bad for each of prose,
dialogue, quest, commentary and wellness. Their scores are **author-curated
exemplar labels, not human-panel ratings**
(`provenance.source: 'authored-exemplar'`). The κ here is
judge-versus-curated-label agreement.

Judge-versus-human κ needs a human-rated text corpus, and the repository has
none:

- `libs/euterpe/evals` human-eval rates music listening tests, not text;
- the Studio gold-set capture table (`v1_cross_cutting_gold_set_entry`) does not
  exist in the local database, so there are no accept/reject/edit records.

## The judges

- `--judge engine`: one `JudgeEngine` at its default operating point
  (temperature 0.7, 3 samples per item).
- `--judge panel`: a `JudgePanel` of three members on the **same model** at
  temperatures 0.3, 0.7 and 1.0, 2 samples each. The panel score is the members'
  median. This panel varies only the sampling temperature, not the model; no
  model-diverse or stronger panel was run.

Both run on `deepseek/deepseek-v4-flash-0731`, the repository's test and harness
model (CLAUDE.md, "Test & harness model binding"). Exemplars are off: they are
drawn from this same benchmark.

## Run it

Keys come from the env master via a throwaway script (CLAUDE.md, "Secrets &
local stack"):

```bash
OPENROUTER_PROVIDER_SORT=price OPENROUTER_PROVIDER_QUANTIZATIONS=fp8 \
npx tsx --tsconfig tools/content-quality-calibration/tsconfig.json \
  tools/content-quality-calibration/run-calibration.ts \
  --judge engine --out report.json \
  --baseline libs/shared/content-quality-judge/calibration/<the judge's first report>.json
```

- `--baseline` exits non-zero when any of the baseline's content types is
  missing from the run, or its κ falls more than `--tolerance` (default 0.1)
  below the baseline's (`checkKappaDrift`). It refuses a baseline from a
  different judge (kind, model, members, samples) or benchmark (version, item
  hash, or a subset).
- `--per-type N` scores only the first N items of each type, for a smoke run.
  The report records the subset.

The recorded reports live in `libs/shared/content-quality-judge/calibration/`.
`src/recorded-calibration.spec.ts` runs in the library's test target. For each
report it checks that:

- it was measured on the whole benchmark as it stands: same version, and a
  sha256 over every item's id, type, text and label score;
- every item has a judge score in [0, 100];
- its κ, r, MAE and separation are what its own item scores produce, for exactly
  the benchmark's five content types (and, for a panel, each member's numbers
  are what that member's scores produce);
- a later report of the same judge covers the first report's content types and
  holds κ within tolerance of it.

The adapter counts every call's stop reason and serving endpoint in the report,
and fails a sample the model refused, so a refusal never becomes a score.

No CI job holds a provider key, so CI cannot re-measure the judge itself. The
live drift check is the `--baseline` command above.

## Recorded runs, 2026-09-21

Every run used `deepseek/deepseek-v4-flash-0731` with sort=price and fp8. Every
call was priced, and every call stopped on `end_turn`. The recorded reports are
run 2 of each judge, in `libs/shared/content-quality-judge/calibration/`:

- **Engine.** 120 calls, $0.0285, 794 s. OpenInference served 118 calls and
  StreamLake 2.
- **Panel.** 240 calls, $0.0616, 751 s. OpenInference served 239 calls and
  StreamLake 1. The panel sent 3 of 40 items to a human.

κ, r, MAE and separation are against the curated labels, with n = 8 per type:

| content type | engine κ | engine r | engine MAE | panel κ | panel r | panel MAE | separation (both) |
| ------------ | -------: | -------: | ---------: | ------: | ------: | --------: | ----------------: |
| prose        |    0.231 |    0.997 |        8.8 |   0.231 |   0.997 |       9.2 |              1.00 |
| dialogue     |    0.273 |    0.989 |       10.9 |   0.149 |   0.990 |      10.7 |              1.00 |
| quest        |    0.200 |    0.988 |       13.3 |   0.127 |   0.992 |      13.4 |              1.00 |
| commentary   |    0.429 |    0.984 |       16.5 |   0.590 |   0.994 |      16.1 |              1.00 |
| wellness     |    0.429 |    0.975 |        6.5 |   0.429 |   0.986 |       4.9 |              1.00 |
| all 40       |    0.301 |    0.931 |       11.2 |   0.275 |   0.939 |      10.8 |              1.00 |

Run 1 of each judge is kept in `superseded/`. Its adapter mapped a refusal to
`end_turn` and recorded neither stop reasons nor serving endpoints, so it cannot
show that no refusal became a score. It also has no item hash. Codex's read-only
review found these defects; run 2 was made after the fixes.

What the numbers say:

- **Ordering is right.** Within every type, the judge orders the curated items
  as their authors did (r 0.975–0.997). It scores every good item above every
  bad one (separation 1.00).
- **κ is low because of placement, not ordering.** κ is agreement on the
  0/25/50/75/100 anchor bands. The curated labels sit at 12–18 and 81–89, beside
  the 12.5 and 87.5 band edges, and the judge places items differently. Ranges
  below are from the recorded runs:
  - bad prose, dialogue and quest items score 1.7–16.5, where the labels are
    12–18;
  - good quest items score 53.8–68.0, where the labels are 83–88;
  - bad commentary items score 33.9–43.7, where the labels are 12–15.
- **The engine's per-type κ is noisier than the drift tolerance; the panel's was
  not.** Runs 1 and 2 of the same configuration compare as follows:

  |                         | engine                                                        | panel                 |
  | ----------------------- | ------------------------------------------------------------- | --------------------- |
  | per-type κ moved by     | up to 0.21 (dialogue 0.478 → 0.273, commentary 0.619 → 0.429) | at most 0.03          |
  | items that changed band | 5 of 40                                                       | 5 of 40               |
  | per-item score change   | 2.1 mean, 6.6 at most                                         | 2.2 mean, 9.8 at most |
  | r moved by              | no more than 0.008                                            | 0.03 at most          |

  With 8 items per type, one item crossing a band edge moves κ by about 0.2, so
  a re-run of the engine can trip `checkKappaDrift`'s default 0.1 tolerance on
  noise alone. The panel held within 0.03 across its two runs, which is two
  runs, not a noise bound. Before a scheduled gate relies on per-type κ, measure
  its spread over more runs and either widen the tolerance or grow n.

# External-benchmark sanity run (audit G.2)

`run-external-benchmark.ts` checks the judge against two public benchmarks,
using their published data rather than our own taste:

- **EQ-Bench Creative Writing v3.** Pieces by several models, and the piece
  scores EQ-Bench published for them. The reference score is EQ-Bench's own
  piece score: the mean of its 22 criteria, negative criteria inverted, times 5.
  `--check-eqbench-scoring --eqbench-runs creative_bench_runs.json` recomputes
  every completed run's published score from its pieces, with no model calls. On
  the recorded download it matched 106 of 106, with a largest difference of 0.
  **That score is Claude Sonnet 4's judgement, not a human's.** The report gives
  `runExternalBenchmark` (provenance `published-dataset`) over the pieces.
  Across the sampled models it also gives two Spearman ρ:
  - the judge's mean against EQ-Bench's published score;
  - our slop density against the leaderboard's published `slop_score`.
- **LitBench** (`SAA-Lab/LitBench-Test-Release`). Pairs of r/WritingPrompts
  stories under one prompt, where the chosen story had more upvotes: a human
  preference. The report gives `runPreferenceBenchmark` pairwise accuracy and
  length bias, beside the paper's published range for off-the-shelf zero-shot
  judges: 0.581 (llama-3.1-8b) to 0.731 (claude-3.7-sonnet), deepseek-v3 0.700
  (arXiv 2507.00769).

The judge is the shipped default: `JudgeEngine` with exemplars on, temperature
0.7 and 3 samples, on the harness model, with sort=price and fp8. It scores each
text on its own. The published LitBench judges were shown both stories and
picked one, a different protocol.

The texts are read at run time and never written out; the report keeps ids,
scores, lengths and slop densities.

## Data

```bash
curl -sL -o creative_bench_runs.zip https://github.com/EQ-bench/creative-writing-bench/raw/main/creative_bench_runs.zip
unzip creative_bench_runs.zip                        # creative_bench_runs.json
curl -sL -o creative_writing.js https://eqbench.com/creative_writing.js
```

The recorded run used these files, with the repository at `c7c3ceef54c`
(2026-06-24):

| file                                    | sha256            |
| --------------------------------------- | ----------------- |
| `creative_bench_runs.zip`               | `7e1f141f…4759ec` |
| `creative_bench_runs.json`              | `ccbf1252…932eaa` |
| `creative_writing.js` (the leaderboard) | `28cad62a…3e5c23` |

The report records the full json and leaderboard hashes. LitBench is fetched
through the Hugging Face rows API, and the report records its revision
(`53e16186…`).

- **EQ-Bench sample.** The runs that Claude Sonnet 4 judged, whose model is on
  the leaderboard: 36 runs. Eight models are spread evenly over the published
  score range, with 5 pieces each, chosen by seed.
- **LitBench sample.** 60 of the 2,381 pairs, chosen by seed.
- **Checkpointing.** With `--checkpoint scores.jsonl`, a rerun reuses scores
  already paid for, matched by id, text hash and judge configuration.

## Recorded run, 2026-09-21

`libs/shared/content-quality-judge/calibration/external/2026-09-21-deepseek-v4-flash-0731.json`:
480 calls, $0.184, 3,139 s. Every call was priced and ended with `end_turn`.
OpenInference served 475 calls and StreamLake 5.

**EQ-Bench** (40 pieces from 8 models, against Claude Sonnet 4's scores):

|     r |     κ |  MAE | length excess | slop excess | recalibrate? |
| ----: | ----: | ---: | ------------: | ----------: | ------------ |
| 0.909 | 0.264 | 14.2 |         0.142 |       0.055 | no           |

| model                 | published score | Sonnet 4 mean (5 pieces) | judge mean | published slop | our slop density |
| --------------------- | --------------: | -----------------------: | ---------: | -------------: | ---------------: |
| llama-3.2-1b-instruct |            32.4 |                     33.0 |       24.2 |           61.1 |             2.69 |
| llama-3.1-8b-instruct |            44.4 |                     43.9 |       28.4 |           53.3 |             1.25 |
| gpt-4-0314            |            49.5 |                     52.1 |       33.7 |           41.6 |             1.17 |
| mistral-large-2411    |            58.9 |                     54.8 |       31.1 |           48.8 |             1.63 |
| qwq-32b               |            70.9 |                     73.2 |       53.0 |           34.7 |             0.22 |
| minimax-m2.5          |            75.9 |                     76.8 |       69.4 |           26.8 |             0.37 |
| claude-opus-4-5       |            81.8 |                     82.5 |       67.8 |           16.2 |             0.22 |
| gpt-5                 |            83.9 |                     83.1 |       84.0 |           11.4 |             0.00 |

- **Ranking agrees; level does not.** Across the 8 models, the judge's mean
  ranks with the published score at Spearman ρ 0.952. Per piece, r is 0.909. But
  the judge scores 7 of the 8 models 7–24 points below Sonnet 4, level only on
  gpt-5, hence the MAE of 14.2 and the low κ. It is a harsher grader with
  similar ordering, not a drop-in for the published scale.
- **Slop detection agrees.** Our slop density ranks the models as the
  leaderboard's slop score does (ρ 0.952), though its values sit on a different
  scale.
- **No reward hacking flagged.** The judge tracks length a little more than
  Sonnet 4 does (excess 0.142, threshold 0.3), and slop no more (0.055).

**LitBench** (60 pairs, against human upvote preference):

| accuracy | 95% interval | ties | people chose longer | judge preferred longer | length excess | recalibrate? |
| -------: | ------------ | ---: | ------------------: | ---------------------: | ------------: | ------------ |
|    0.733 | 0.610–0.829  |    0 |               0.550 |                  0.617 |         0.067 | no           |

- **Just above the published range.** 0.733 (44 of 60) is just above the
  published zero-shot maximum of 0.731. The interval, 0.610–0.829, spans most of
  that range, so 60 pairs do not show the judge better or worse than any
  published judge.
- **The protocol differs.** Our judge scored each story alone; the published
  judges compared the two.
- **Contamination cannot be ruled out.** The stories are public Reddit posts,
  and the model may have seen them in training.
- **Length bias is small.** The judge prefers the longer story 6.7 points more
  often than people did (flag threshold 0.15).

`src/recorded-external-benchmark.spec.ts` recomputes the report's agreement,
hacking, divergence, per-model and preference numbers from its per-item scores,
lengths and slop densities (`externalAgreement`, `spearman`,
`runPreferenceBenchmark`). It also requires the report to be non-empty and to
have a call count that matches the texts scored. A tampered accuracy,
correlation or item score fails it, and so does an empty report.

The spec cannot check inputs the report did not keep: the texts, the published
scores, the dataset hashes, and the lengths and slop densities themselves. Those
stand as recorded.

The recorded report was produced before `--checkpoint`, the judge-configuration
checkpoint key and the LitBench revision guard were added. None of the three
changes a score:

- the run reused nothing (480 calls = 160 texts × 3 samples);
- its judge block names the sample count `samplesPerText` where the current
  runner writes `samples`.

**Not done.** G.2 asks for a periodic report. Nothing schedules this run: no CI
job holds a provider key. Scheduling it is follow-up G.2.ci, which needs the
owner, like G.1.ci.

# Thinking budget per quality mode (audit K.2)

The `deep` and `exhaustive` modes now give every generation and judging call a
reasoning ("thinking") budget. The budget travels the whole chain:

1. `COMPUTE_PROFILES[mode].thinkingBudgetTokens` sets it: none for fast and
   balanced, 4,096 for deep, 12,288 for exhaustive.
2. It is applied through `withThinkingBudget` / `providerForMode`:
   - `runModeCompute` sends candidate requests, the panel it builds and the
     refine passes through the budgeted writer;
   - `ContentPipelineService` wraps its writer, and so `buildPanel`'s judges;
   - `JudgeModelConfig.thinkingBudgetTokens` sets it on one engine.
3. It rides on `AgenticCompletionRequest.thinkingBudgetTokens`.
4. The bridges carry it: `wrapSharedProvider` in `@iris/agents-core`, and the
   BFF's `agenticProviderFromLlm`.
5. It becomes `@oshun/ai`'s `ChatCompletionRequest.reasoning`, which OpenRouter
   receives as `reasoning.max_tokens`. OpenRouter passes it to Claude as
   `thinking.budget_tokens`, and `Claude4Adapter` sends it the same way.

The budget is added on top of `maxTokens`, so the answer keeps its allowance.

Anything that cannot send a budget refuses it instead of dropping it:

- the native `@iris` provider adapters;
- the xai, google, ollama, codex and opencode providers;
- the OpenAI API, for anything but an effort level;
- the native Anthropic API, which is not wired for current Claude models;
- a `ContentPipelineService` with a prebuilt `panel` under a budgeted mode.

**Not verified:** a Claude model has not been called with a budget, since the
harness binds no frontier model. `Claude4Adapter` drops the temperature when it
sends a budget, as the Anthropic API wants. The judge engine still sends its
sampling temperature, and whether Claude through OpenRouter accepts that with a
budget is unknown.

## Recorded run, 2026-09-21

`run-thinking-budget.ts` calls through the production path: `JudgeEngine` →
`providerForMode` → `wrapSharedProvider` → `@oshun/ai` `OpenRouterProvider`. The
report is `reports/2026-09-21-thinking-budget.json`: 732 calls, $0.218, all
priced, all `end_turn`.

**Does the budget reach the model?** The table gives reasoning tokens for the
two calls at each setting, judging one story:

| model                             | no budget     | budget 512    | budget 4,096  |
| --------------------------------- | ------------- | ------------- | ------------- |
| `qwen/qwen3.7-flash` (Alibaba)    | 2,986 · 2,673 | **512 · 512** | 2,324 · 2,355 |
| `deepseek/deepseek-v4-flash-0731` | 848 · 1,224   | 357 · 942     | 695 · 1,172   |

- qwen3.7-flash keeps to the budget exactly: the 512-token cap binds at 512.
- The deepseek harness model does not keep to a budget on its route: 942
  reasoning tokens under a 512 cap. The probe cannot tell whether it received
  the budget. A direct call made on the same day, which is not in this report,
  showed its route does act on `reasoning`: `enabled: false` took its reasoning
  tokens to 0.
- A budget is a ceiling, not a floor. On this task qwen reasons about 2,300
  tokens unprompted, so deep's 4,096 does not bind.

**Does deep mode help the judge?** Not on this fixture. The fixture was 40
LitBench pairs (seed 2, revision `53e16186`), scored on qwen3.7-flash with 3
samples per story:

| arm                                | accuracy | 95% interval | mean reasoning tokens |   cost |
| ---------------------------------- | -------: | ------------ | --------------------: | -----: |
| fast (no budget)                   |    0.575 | 0.422–0.715  |                 2,340 | $0.092 |
| deep (4,096 via `providerForMode`) |    0.600 | 0.446–0.737  |                 2,355 | $0.092 |
| a 512 cap (not a mode)             |    0.625 | 0.470–0.758  |                   512 | $0.030 |

- **Paired with fast:**
  - deep got 4 pairs right that fast missed, and missed 3 that fast got (sign
    test p = 1.0);
  - the 512 cap went 4 to 2 (p = 0.69).
- **No arm differs measurably.** Deep reasoned no more than fast, because the
  budget never bound. Cutting reasoning to a fifth cost a third as much and lost
  nothing detectable in 40 pairs.
- **No claim of improvement.** On this judge and model, a larger thinking budget
  is not shown to buy quality; a smaller one is shown to save money.
- **Next step:** a model whose natural reasoning exceeds the budget, or a task
  that needs more reasoning, is where the modes' budgets could matter. That is
  follow-up K.2.q.
