Evaluator Card — preference-stability v1.0.0

artifact_kind: evaluator-card artifact_name: preference-stability version: 1.0.0 owner: Nous · preference-inference team last_reviewed: 2026-04-25 next_review: 2026-07-25 evaluator_kind: stability related_phase_179_tasks: [179.3.2.4, 179.3.2.5, 179.8.3.4]#

1. Identity#

• Evaluator name: preference-stability
• What it measures: robustness of the pairwise-preference-ranker under five perturbation kinds (paraphrase, prompt template, model version, A↔B order swap, adversarial framing) and the calibration width of the resulting utility-fitter posterior.
• Harm families covered (H1–H8): H3 (interpretability — agreement under benign perturbation indicates the model is responding to content rather than artifacts), H5 (safety — adversarial-framing resistance closes a coercion vector).
• Consumers: the §179.10.7 release gate (this card); the workbench stabilityFlags annotations on each PairwisePreference; the observation-weight scaler that down-weights unstable pairs in the utility fits.

2. Method#

• Input corpus / fixture: synthetic two-party benchmark scenarios from libs/contracts/concordia/src/__benchmarks__/benchmarks.ts plus the Mediator.ai-style scenarios in libs/contracts/concordia/src/__benchmarks__/mediator-ai-scenarios.ts. All deterministic, all checked into the repo with hashed party statements.
• Procedure:
  1. Run the primary pairwise comparison via buildPairwiseComparisonPrompt + the routed model.
  2. For each perturbation kind in prompt-perturbations.ts (paraphrase, template, model_version, order_swap, adversarial_framing), produce a perturbed prompt and re-run the comparison.
  3. After un-swapping A ↔ B for order_swap perturbations, compute per-kind agreement = (# matching choices) / (# probes for that kind).
  4. For abstention responses, agreement counts only when the primary was also abstain with the same reason class.
  5. Fit utilities on the surviving observations and compute the central credible-interval width at the configured mass (default 95%).
• Metrics emitted:
  • paraphraseAgreement ∈ [0, 1]
  • templateAgreement ∈ [0, 1]
  • modelVersionAgreement ∈ [0, 1]
  • orderSwapAgreement ∈ [0, 1]
  • adversarialFramingAgreement ∈ [0, 1]
  • ciWidth = mean central CI width across alternatives, at 95% mass
• Aggregation: pass when every per-kind agreement ≥ its threshold AND ciWidth ≤ maxCiWidth. Failure on any one kind blocks release; the gate does not average a low score against a high one.

3. Baselines#

• Comparison baselines: llm-mediator (single-LLM, no formal optimization), nash-genetic (Mediator.ai baseline, no uncertainty calibration), static-templates, seed-offer-only.
• Baseline scores: baseline rankers exhibit higher paraphrase / template / order-swap variance because they do not run the perturbation panel; their stability is uncorrected.

4. Thresholds#

The release-gate floors are the defaults declared in libs/nous/preference-inference/src/pairwise-plan.ts StabilityThresholdsSchema:

Regression threshold: any single-kind agreement that drops more than 0.05 from the previous release blocks release pending review.

Domain-specific thresholds: tenants may raise (never lower) the floors in USE_CASE_CLASS_PROFILES overrides — the default profile for legal_claim, medical_sensitive, regulated_financial, family_sensitive, and employment_sensitive raises model-version agreement to 0.90 and CI-width cap to 0.15.

5. Known limitations#

• Synthetic-fixture stability does not necessarily transfer to arbitrary live cases; tenant pilots are tracked separately in §179.10.8.
• The five perturbation kinds are not exhaustive; adversarial framing has many dimensions (framing as moral consensus, framing as inevitability, framing as expert opinion) and the present probe rotates a fixed set rather than searching adversarially.
• Adversarial-framing threshold is the lowest of the panel because resisting framing is the open weak point; the abstention path is intentionally encouraged when framing is detected.

6. Validation#

• Last validation date: 2026-04-25.
• Validation procedure: the perturbation generators in prompt-perturbations.ts are themselves validated by prompt-perturbations.test.ts, which asserts perturbed prompts preserve the underlying alternative content (no semantic drift) while changing surface form. The agreement computation is asserted in stability-probes.test.ts against hand-crafted edge cases.

7. References#

• libs/nous/preference-inference/src/stability-probes.ts
• libs/nous/preference-inference/src/prompt-perturbations.ts
• libs/nous/preference-inference/src/pairwise-plan.ts (StabilityThresholdsSchema)
• libs/nous/preference-inference/src/calibration.ts — credible-interval calibration
• docs/research/bibliography.md §robustness-evaluation