Evaluator Card — baseline-benchmark-gate v1.0.0

artifact_kind: evaluator-card artifact_name: baseline-benchmark-gate version: 1.0.0 owner: Nous · agreement-search team last_reviewed: 2026-04-25 next_review: 2026-07-25 evaluator_kind: benchmark related_phase_179_tasks: [179.8.1, 179.8.2, 179.10.6]#

1. Identity#

• Evaluator name: baseline-benchmark-gate
• What it measures: whether the Concordia stack outperforms each of four reference baselines on at least the configured number of domain suites, using paired Welch's t tests on per-case KPI values.
• Harm families covered (H1–H8): H4 (fairness — measures whether Concordia's outputs are at least as fair as baselines on every domain), H6 (accountability — emits significance, effect size, and win count for audit).
• Consumers: the §179.10.6 release gate; the §179.10.7 governance review (this card); the §179.8.3.5 internal leaderboard.

2. Method#

• Input corpus / fixture: the eight benchmark suites in libs/contracts/concordia/src/__benchmarks__/benchmarks.ts: two_party, multi_party, procurement, legal_low_stakes, dao, creative, restorative, agent_to_agent. Each suite is a deterministic scenario set with checked-in party statements and expected outcomes.
• Procedure:
  1. For every (suite, baseline, KPI) combination, run Concordia and the baseline on the same case set; record paired (concordiaValue, baselineValue) samples.
  2. Compute the paired difference, run Welch's t test, and emit Cohen's d effect size.
  3. KPI direction is encoded per suite — for higher-is-better KPIs (Nash product, max-min utility, agreement rate) a positive paired delta at p < 0.05 is a "win"; for lower-is-better KPIs (latency, cost, dispute rate) a negative paired delta at p < 0.05 is a "win".
• Metrics emitted: per (suite, baseline) — t statistic, p value, Cohen's d, win/loss verdict. Per baseline — total wins across the eight suites. Gate verdict — pass / fail per baseline.
• Aggregation: the gate passes when, for every baseline, Concordia wins on at least minSuiteWinsPerBaseline suites (default 6 of 8).

3. Baselines#

• Comparison baselines (BaselineKindSchema):
  • llm_mediator_baseline — single-LLM mediator with no formal optimization
  • nash_genetic_baseline — Mediator.ai-style Nash-product genetic search without uncertainty calibration
  • static_templates_baseline — rule-based templates with no party-utility inference
  • seed_offer_only_baseline — human-authored seed offer with no automated counterproposals
• Baseline scores: materialized in libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.test.ts — every release runs the full matrix and the test asserts the gate pass.

4. Thresholds#

Regression threshold: loss of one suite for any baseline triggers review. Loss of more than one suite blocks release.

Domain-specific thresholds: tenants may raise minSuiteWinsPerBaseline to 7 or 8 for their environment; not lower.

5. Known limitations#

• Synthetic benchmark scenarios are deterministic and do not capture the full distribution of real cases; pilot KPIs are tracked independently in §179.10.8.
• Welch's t test assumes paired samples are independent within a suite; this holds for the synthetic scenarios but should be re-validated when production traces enter the corpus.
• Cohen's d is reported but not gated; a statistically significant but small-effect win still counts as a "win" under the current rule.

6. Validation#

• Last validation date: 2026-04-25.
• Validation procedure: every Concordia release re-runs the gate via pnpm vitest run libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.test.ts. The reference Welch's t-statistic, t-table, and gamma-function approximations are unit-tested in the same file (the t-table values at the file lines 287–302 are checked against published references).

7. References#

• libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.ts
• libs/contracts/concordia/src/__benchmarks__/benchmarks.ts
• libs/contracts/concordia/src/__benchmarks__/mediator-ai-scenarios.ts
• testing/concordia/benchmarks/benchmarks.ts — re-export surface
• docs/research/bibliography.md §benchmark-gating