Eve library

Content-quality judge: live calibration (audit G.1)

Other

Revision 3

On this page

run-calibration.ts runs the real @oshun/content-quality-judge judge on a live model over GOLD_BENCHMARK_V1. It reports, per content type:

  • Cohen's κ (anchor bands), Pearson r and MAE against the benchmark's scores;
  • good-versus-bad separation (pairwise accuracy).

It also records every item's score and the billed cost.

What the numbers are measured against#

GOLD_BENCHMARK_V1 holds 40 items: 4 good and 4 bad for each of prose, dialogue, quest, commentary and wellness. Their scores are author-curated exemplar labels, not human-panel ratings (provenance.source: 'authored-exemplar'). The κ here is judge-versus-curated-label agreement.

Judge-versus-human κ needs a human-rated text corpus, and the repository has none:

  • libs/euterpe/evals human-eval rates music listening tests, not text;
  • the Studio gold-set capture table (v1_cross_cutting_gold_set_entry) does not exist in the local database, so there are no accept/reject/edit records.

The judges#

  • --judge engine: one JudgeEngine at its default operating point (temperature 0.7, 3 samples per item).
  • --judge panel: a JudgePanel of three members on the same model at temperatures 0.3, 0.7 and 1.0, 2 samples each. The panel score is the members' median. This panel varies only the sampling temperature, not the model; no model-diverse or stronger panel was run.

Both run on deepseek/deepseek-v4-flash-0731, the repository's test and harness model (CLAUDE.md, "Test & harness model binding"). Exemplars are off: they are drawn from this same benchmark.

Run it#

Keys come from the env master via a throwaway script (CLAUDE.md, "Secrets & local stack"):

bash
OPENROUTER_PROVIDER_SORT=price OPENROUTER_PROVIDER_QUANTIZATIONS=fp8 \
npx tsx --tsconfig tools/content-quality-calibration/tsconfig.json \
  tools/content-quality-calibration/run-calibration.ts \
  --judge engine --out report.json \
  --baseline libs/shared/content-quality-judge/calibration/<the judge's first report>.json
  • --baseline exits non-zero when any of the baseline's content types is missing from the run, or its κ falls more than --tolerance (default 0.1) below the baseline's (checkKappaDrift). It refuses a baseline from a different judge (kind, model, members, samples) or benchmark (version, item hash, or a subset).
  • --per-type N scores only the first N items of each type, for a smoke run. The report records the subset.

The recorded reports live in libs/shared/content-quality-judge/calibration/. src/recorded-calibration.spec.ts runs in the library's test target. For each report it checks that:

  • it was measured on the whole benchmark as it stands: same version, and a sha256 over every item's id, type, text and label score;
  • every item has a judge score in [0, 100];
  • its κ, r, MAE and separation are what its own item scores produce, for exactly the benchmark's five content types (and, for a panel, each member's numbers are what that member's scores produce);
  • a later report of the same judge covers the first report's content types and holds κ within tolerance of it.

The adapter counts every call's stop reason and serving endpoint in the report, and fails a sample the model refused, so a refusal never becomes a score.

No CI job holds a provider key, so CI cannot re-measure the judge itself. The live drift check is the --baseline command above.

Recorded runs, 2026-09-21#

Every run used deepseek/deepseek-v4-flash-0731 with sort=price and fp8. Every call was priced, and every call stopped on end_turn. The recorded reports are run 2 of each judge, in libs/shared/content-quality-judge/calibration/:

  • Engine. 120 calls, $0.0285, 794 s. OpenInference served 118 calls and StreamLake 2.
  • Panel. 240 calls, $0.0616, 751 s. OpenInference served 239 calls and StreamLake 1. The panel sent 3 of 40 items to a human.

κ, r, MAE and separation are against the curated labels, with n = 8 per type:

content type engine κ engine r engine MAE panel κ panel r panel MAE separation (both)
prose 0.231 0.997 8.8 0.231 0.997 9.2 1.00
dialogue 0.273 0.989 10.9 0.149 0.990 10.7 1.00
quest 0.200 0.988 13.3 0.127 0.992 13.4 1.00
commentary 0.429 0.984 16.5 0.590 0.994 16.1 1.00
wellness 0.429 0.975 6.5 0.429 0.986 4.9 1.00
all 40 0.301 0.931 11.2 0.275 0.939 10.8 1.00

Run 1 of each judge is kept in superseded/. Its adapter mapped a refusal to end_turn and recorded neither stop reasons nor serving endpoints, so it cannot show that no refusal became a score. It also has no item hash. Codex's read-only review found these defects; run 2 was made after the fixes.

What the numbers say:

  • Ordering is right. Within every type, the judge orders the curated items as their authors did (r 0.975–0.997). It scores every good item above every bad one (separation 1.00).

  • κ is low because of placement, not ordering. κ is agreement on the 0/25/50/75/100 anchor bands. The curated labels sit at 12–18 and 81–89, beside the 12.5 and 87.5 band edges, and the judge places items differently. Ranges below are from the recorded runs:

    • bad prose, dialogue and quest items score 1.7–16.5, where the labels are 12–18;
    • good quest items score 53.8–68.0, where the labels are 83–88;
    • bad commentary items score 33.9–43.7, where the labels are 12–15.
  • The engine's per-type κ is noisier than the drift tolerance; the panel's was not. Runs 1 and 2 of the same configuration compare as follows:

    engine panel
    per-type κ moved by up to 0.21 (dialogue 0.478 → 0.273, commentary 0.619 → 0.429) at most 0.03
    items that changed band 5 of 40 5 of 40
    per-item score change 2.1 mean, 6.6 at most 2.2 mean, 9.8 at most
    r moved by no more than 0.008 0.03 at most

    With 8 items per type, one item crossing a band edge moves κ by about 0.2, so a re-run of the engine can trip checkKappaDrift's default 0.1 tolerance on noise alone. The panel held within 0.03 across its two runs, which is two runs, not a noise bound. Before a scheduled gate relies on per-type κ, measure its spread over more runs and either widen the tolerance or grow n.

External-benchmark sanity run (audit G.2)#

run-external-benchmark.ts checks the judge against two public benchmarks, using their published data rather than our own taste:

  • EQ-Bench Creative Writing v3. Pieces by several models, and the piece scores EQ-Bench published for them. The reference score is EQ-Bench's own piece score: the mean of its 22 criteria, negative criteria inverted, times 5. --check-eqbench-scoring --eqbench-runs creative_bench_runs.json recomputes every completed run's published score from its pieces, with no model calls. On the recorded download it matched 106 of 106, with a largest difference of 0. That score is Claude Sonnet 4's judgement, not a human's. The report gives runExternalBenchmark (provenance published-dataset) over the pieces. Across the sampled models it also gives two Spearman ρ:
    • the judge's mean against EQ-Bench's published score;
    • our slop density against the leaderboard's published slop_score.
  • LitBench (SAA-Lab/LitBench-Test-Release). Pairs of r/WritingPrompts stories under one prompt, where the chosen story had more upvotes: a human preference. The report gives runPreferenceBenchmark pairwise accuracy and length bias, beside the paper's published range for off-the-shelf zero-shot judges: 0.581 (llama-3.1-8b) to 0.731 (claude-3.7-sonnet), deepseek-v3 0.700 (arXiv 2507.00769).

The judge is the shipped default: JudgeEngine with exemplars on, temperature 0.7 and 3 samples, on the harness model, with sort=price and fp8. It scores each text on its own. The published LitBench judges were shown both stories and picked one, a different protocol.

The texts are read at run time and never written out; the report keeps ids, scores, lengths and slop densities.

Data#

bash
curl -sL -o creative_bench_runs.zip https://github.com/EQ-bench/creative-writing-bench/raw/main/creative_bench_runs.zip
unzip creative_bench_runs.zip                        # creative_bench_runs.json
curl -sL -o creative_writing.js https://eqbench.com/creative_writing.js

The recorded run used these files, with the repository at c7c3ceef54c (2026-06-24):

file sha256
creative_bench_runs.zip 7e1f141f…4759ec
creative_bench_runs.json ccbf1252…932eaa
creative_writing.js (the leaderboard) 28cad62a…3e5c23

The report records the full json and leaderboard hashes. LitBench is fetched through the Hugging Face rows API, and the report records its revision (53e16186…).

  • EQ-Bench sample. The runs that Claude Sonnet 4 judged, whose model is on the leaderboard: 36 runs. Eight models are spread evenly over the published score range, with 5 pieces each, chosen by seed.
  • LitBench sample. 60 of the 2,381 pairs, chosen by seed.
  • Checkpointing. With --checkpoint scores.jsonl, a rerun reuses scores already paid for, matched by id, text hash and judge configuration.

Recorded run, 2026-09-21#

libs/shared/content-quality-judge/calibration/external/2026-09-21-deepseek-v4-flash-0731.json: 480 calls, $0.184, 3,139 s. Every call was priced and ended with end_turn. OpenInference served 475 calls and StreamLake 5.

EQ-Bench (40 pieces from 8 models, against Claude Sonnet 4's scores):

r κ MAE length excess slop excess recalibrate?
0.909 0.264 14.2 0.142 0.055 no
model published score Sonnet 4 mean (5 pieces) judge mean published slop our slop density
llama-3.2-1b-instruct 32.4 33.0 24.2 61.1 2.69
llama-3.1-8b-instruct 44.4 43.9 28.4 53.3 1.25
gpt-4-0314 49.5 52.1 33.7 41.6 1.17
mistral-large-2411 58.9 54.8 31.1 48.8 1.63
qwq-32b 70.9 73.2 53.0 34.7 0.22
minimax-m2.5 75.9 76.8 69.4 26.8 0.37
claude-opus-4-5 81.8 82.5 67.8 16.2 0.22
gpt-5 83.9 83.1 84.0 11.4 0.00
  • Ranking agrees; level does not. Across the 8 models, the judge's mean ranks with the published score at Spearman ρ 0.952. Per piece, r is 0.909. But the judge scores 7 of the 8 models 7–24 points below Sonnet 4, level only on gpt-5, hence the MAE of 14.2 and the low κ. It is a harsher grader with similar ordering, not a drop-in for the published scale.
  • Slop detection agrees. Our slop density ranks the models as the leaderboard's slop score does (ρ 0.952), though its values sit on a different scale.
  • No reward hacking flagged. The judge tracks length a little more than Sonnet 4 does (excess 0.142, threshold 0.3), and slop no more (0.055).

LitBench (60 pairs, against human upvote preference):

accuracy 95% interval ties people chose longer judge preferred longer length excess recalibrate?
0.733 0.610–0.829 0 0.550 0.617 0.067 no
  • Just above the published range. 0.733 (44 of 60) is just above the published zero-shot maximum of 0.731. The interval, 0.610–0.829, spans most of that range, so 60 pairs do not show the judge better or worse than any published judge.
  • The protocol differs. Our judge scored each story alone; the published judges compared the two.
  • Contamination cannot be ruled out. The stories are public Reddit posts, and the model may have seen them in training.
  • Length bias is small. The judge prefers the longer story 6.7 points more often than people did (flag threshold 0.15).

src/recorded-external-benchmark.spec.ts recomputes the report's agreement, hacking, divergence, per-model and preference numbers from its per-item scores, lengths and slop densities (externalAgreement, spearman, runPreferenceBenchmark). It also requires the report to be non-empty and to have a call count that matches the texts scored. A tampered accuracy, correlation or item score fails it, and so does an empty report.

The spec cannot check inputs the report did not keep: the texts, the published scores, the dataset hashes, and the lengths and slop densities themselves. Those stand as recorded.

The recorded report was produced before --checkpoint, the judge-configuration checkpoint key and the LitBench revision guard were added. None of the three changes a score:

  • the run reused nothing (480 calls = 160 texts × 3 samples);
  • its judge block names the sample count samplesPerText where the current runner writes samples.

Not done. G.2 asks for a periodic report. Nothing schedules this run: no CI job holds a provider key. Scheduling it is follow-up G.2.ci, which needs the owner, like G.1.ci.

Thinking budget per quality mode (audit K.2)#

The deep and exhaustive modes now give every generation and judging call a reasoning ("thinking") budget. The budget travels the whole chain:

  1. COMPUTE_PROFILES[mode].thinkingBudgetTokens sets it: none for fast and balanced, 4,096 for deep, 12,288 for exhaustive.
  2. It is applied through withThinkingBudget / providerForMode:
    • runModeCompute sends candidate requests, the panel it builds and the refine passes through the budgeted writer;
    • ContentPipelineService wraps its writer, and so buildPanel's judges;
    • JudgeModelConfig.thinkingBudgetTokens sets it on one engine.
  3. It rides on AgenticCompletionRequest.thinkingBudgetTokens.
  4. The bridges carry it: wrapSharedProvider in @iris/agents-core, and the BFF's agenticProviderFromLlm.
  5. It becomes @oshun/ai's ChatCompletionRequest.reasoning, which OpenRouter receives as reasoning.max_tokens. OpenRouter passes it to Claude as thinking.budget_tokens, and Claude4Adapter sends it the same way.

The budget is added on top of maxTokens, so the answer keeps its allowance.

Anything that cannot send a budget refuses it instead of dropping it:

  • the native @iris provider adapters;
  • the xai, google, ollama, codex and opencode providers;
  • the OpenAI API, for anything but an effort level;
  • the native Anthropic API, which is not wired for current Claude models;
  • a ContentPipelineService with a prebuilt panel under a budgeted mode.

Not verified: a Claude model has not been called with a budget, since the harness binds no frontier model. Claude4Adapter drops the temperature when it sends a budget, as the Anthropic API wants. The judge engine still sends its sampling temperature, and whether Claude through OpenRouter accepts that with a budget is unknown.

Recorded run, 2026-09-21#

run-thinking-budget.ts calls through the production path: JudgeEngine → providerForMode → wrapSharedProvider → @oshun/ai OpenRouterProvider. The report is reports/2026-09-21-thinking-budget.json: 732 calls, $0.218, all priced, all end_turn.

Does the budget reach the model? The table gives reasoning tokens for the two calls at each setting, judging one story:

model no budget budget 512 budget 4,096
qwen/qwen3.7-flash (Alibaba) 2,986 · 2,673 512 · 512 2,324 · 2,355
deepseek/deepseek-v4-flash-0731 848 · 1,224 357 · 942 695 · 1,172
  • qwen3.7-flash keeps to the budget exactly: the 512-token cap binds at 512.
  • The deepseek harness model does not keep to a budget on its route: 942 reasoning tokens under a 512 cap. The probe cannot tell whether it received the budget. A direct call made on the same day, which is not in this report, showed its route does act on reasoning: enabled: false took its reasoning tokens to 0.
  • A budget is a ceiling, not a floor. On this task qwen reasons about 2,300 tokens unprompted, so deep's 4,096 does not bind.

Does deep mode help the judge? Not on this fixture. The fixture was 40 LitBench pairs (seed 2, revision 53e16186), scored on qwen3.7-flash with 3 samples per story:

arm accuracy 95% interval mean reasoning tokens cost
fast (no budget) 0.575 0.422–0.715 2,340 $0.092
deep (4,096 via providerForMode) 0.600 0.446–0.737 2,355 $0.092
a 512 cap (not a mode) 0.625 0.470–0.758 512 $0.030
  • Paired with fast:
    • deep got 4 pairs right that fast missed, and missed 3 that fast got (sign test p = 1.0);
    • the 512 cap went 4 to 2 (p = 0.69).
  • No arm differs measurably. Deep reasoned no more than fast, because the budget never bound. Cutting reasoning to a fifth cost a third as much and lost nothing detectable in 40 pairs.
  • No claim of improvement. On this judge and model, a larger thinking budget is not shown to buy quality; a smaller one is shown to save money.
  • Next step: a model whose natural reasoning exceeds the budget, or a task that needs more reasoning, is where the modes' budgets could matter. That is follow-up K.2.q.