Model Card — pairwise-preference-ranker v1.0.0

artifact_kind: model-card artifact_name: pairwise-preference-ranker version: 1.0.0 owner: Nous · preference-inference team last_reviewed: 2026-04-25 next_review: 2026-07-25 concordia_components: [ '@nous/preference-inference', '@iris/concordia-assistant', '@nous/agreement-search', ] related_phase_179_tasks: [ 179.3.2.1, 179.3.2.2, 179.3.2.3, 179.3.2.4, 179.3.2.5, 179.3.2.6, 179.5.1.1, 179.5.1.2, ] related_cards: system: ../systems/concordia-workbench.md evaluators: - ../evaluators/preference-stability.md - ../evaluators/baseline-benchmark-gate.md - ../evaluators/fairness-suite.md#

1. Identity#

• Model name: pairwise-preference-ranker
• Provider / developer: Concordia · preference-inference (this repo)
• Base model and version: Tenant-configurable. Default routing tier is Anthropic claude-opus-4-7 for high-stakes classes, claude-sonnet-4-6 for moderate-stakes, and a local zero-retention reasoning model for low_stakes_personal, community_restorative, and creative_collaboration classes when zero-retention is required.
• Purpose: Given one party's private intake snippets and two candidate agreements (A and B), emits a pairwise preference: choice ∈ {A, B, tie, abstain}, posterior probability, citations to the snippets used, and an optional redline flag. The output is consumed by the utility-fitters (../models/utility-fitters.md) and feeds every search kernel under @nous/agreement-search.
• Intended use cases (Concordia use-case classes): all 14 ConcordiaUseCaseClass values are eligible for some preference inference, but the routing matrix in USE_CASE_CLASS_PROFILES (see libs/contracts/concordia/src/use-case/use-case-class.ts) restricts which base model is allowed per class and which classes block autonomous acceptance.
• Out-of-scope uses: model output is never the basis for any decision the hard-boundary registry refuses (libs/contracts/concordia/src/boundaries/hard-boundary.ts): binding legal claim resolution, custody, domestic-violence or coercive control, employment adverse action, medical settlement, financial suitability, immigration, criminal matters, or rights waiver without qualified human review.

2. Development context#

• Training data summary: the underlying LLMs are vendor-supplied; their training data is whatever the chosen provider documented. Concordia adds no fine-tuning weights to the base models in v1.0.0. The local zero-retention reasoning model is the vendor base with no Oshun fine-tune.
• Fine-tuning: none. Concordia operates the model purely via prompting. Future fine-tuning enters the §179.5.5.5 training-data-governance pipeline (see libs/contracts/concordia/src/training-data-governance/).
• Prompt family: buildPairwiseComparisonPrompt and parsePairwiseComparisonResponse in libs/nous/preference-inference/src/pairwise-prompt.ts. Single-source template; perturbation variants in prompt-perturbations.ts are used only for stability probing, never as the live ranker.
• Concordia-specific customization: strict JSON schema (PairwiseComparisonResponseSchema) for the response, mandatory citations to private-intake snippet ids, citation-quote verification (the post-processor confirms a quoted span exists in the cited snippet), explicit abstention states (low_confidence, unstable, refused, redline_blocked, non_tradeable, insufficient_evidence), and rationale length cap (320 chars) to discourage fabrication.

3. Inputs and outputs#

• Input schema: BuildPairwisePromptInput from libs/nous/preference-inference/src/pairwise-prompt.ts. Owning party's intake snippets only — the function signature physically excludes opposing-party private fields. Two AgreementSummaryForComparison records (10 clause lines each, max).
• Output schema: PairwiseComparisonResponseSchema (Zod-validated): { schemaVersion: 1, choice, probability ∈ [0, 1], rationale ≤ 320 chars, citations: ≥ 1, abstentionReason?, redlineFlag }. Free-form text is treated as a refusal.
• Deterministic settings used in production: temperature 0.0 by default (overridable per use-case class to a maximum of 0.3 for low_stakes_personal only); top-p 1.0; max tokens 512; per-comparison deadline 6000 ms (pairwise-plan.ts default). Seed is always pinned and recorded in the reproducibility manifest (§179.5.3.3).

4. Evaluation#

The release-gate floors are the defaults declared in libs/nous/preference-inference/src/pairwise-plan.ts (StabilityThresholdsSchema) and the minSuiteWinsPerBaseline = 6 default in libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.ts:113. Tenants may raise (never lower) the floors via configuration.

5. Privacy posture#

• Data visibility at the model: the owning party's intake snippets, the two candidate-agreement summaries (≤ 10 clause lines each), the prompt template version, and the use-case class. No opposing-party private fields, no reviewer-only annotations, no consent ledger entries, no audit trail.
• Per-party isolation: §179.5.1.1 physically separate prompt contexts. Enforced at the function-signature level — buildPairwiseComparisonPrompt accepts only the owning party's snippets. Cross-party leakage is impossible through this surface; integration tests in libs/iris/concordia-assistant/src/party-isolated-context.test.ts and libs/iris/concordia-assistant/src/privacy-red-team.test.ts assert that.
• Training-data reuse eligibility: off by default. Reuse only after the trace passes recordDeidentificationReview (§179.5.5.5, libs/contracts/concordia/src/training-data-governance/) AND the granting party has an active training_data_reuse consent purpose.
• Zero-retention mode: required for legal_claim, medical_sensitive, regulated_financial, family_sensitive, employment_sensitive, and any case where the party requested it via the consent ledger. Routing layer enforces the local zero-retention model for these classes.
• Secure-enclave / confidential-compute posture: §179.5.1.4 enclave channel implemented in libs/nous/concordia-sealed-memory/. Used today for utility aggregation only; preference scoring inside enclave is a roadmap item for v1.1.

6. Safety posture#

• Hard-boundary coverage: the candidate-filter gate (libs/nous/agreement-search/src/candidate-filter.ts) refuses any candidate that triggers a hard boundary BEFORE pairwise scoring. The ranker therefore never scores a hard-boundary-violating candidate. Tested by candidate-filter.test.ts and the boundary unit tests in libs/contracts/concordia/src/boundaries/hard-boundary.test.ts.
• Adversarial robustness: prompt-injection, fishing-for-private-info, and redline-bypass tests in libs/iris/concordia-assistant/src/privacy-red-team.test.ts and libs/nous/agreement-search/src/spec-gaming-audit.test.ts. Output schema rejects free-form text, which closes the most common injection path.
• Coercive-choice refusal: coercive-choice-audit.ts in @nous/agreement-search audits emitted candidate sets and explanations for countdown pressure, hidden anchors, asymmetric disclosure, and forced defaults. The ranker itself does not produce explanations to parties; it produces structured comparisons consumed by other components.
• Prompt-injection resistance: schema-gated output + citation verification
  • rationale length cap. The parse failure modes (unparsable, missing_citation, quote_not_in_snippet) are observed in pairwise-prompt.test.ts.

7. Fairness and bias posture#

• Protected-class disparate treatment: the fairness-profile module (libs/nous/preference-inference/src/fairness-profiles.ts) blocks any fairness profile whose configuration would induce protected-class disparate treatment in scoring. Tested in fairness-profiles.test.ts.
• Power-asymmetry handling: BATNA plausibility (batna-plausibility.ts) refuses to take self-reported BATNA as ground truth — this is the structural defense against a high-leverage party gaming the optimizer. Inequity aversion appears as a separable signal in multi-attribute-utility.ts so it never silently dominates utility.
• Language / locale coverage: language-detector + language-drift modules in @iris/concordia-assistant (language-drift.ts) detect internal- reasoning-language drift relative to the party's intake language. Multiple language-summary and language-preference modules preserve party language end-to-end. Locales below the language-acceptance gate route to human mediator review rather than scoring.

8. Known limitations#

• The vendor base models exhibit paraphrase-sensitivity that cannot be removed by prompting alone. The release-gate floor at 0.80 paraphrase agreement is a containment, not a fix.
• Order-effect bias is small but non-zero. The order_swap perturbation in stability-probes.ts is run on every released model version.
• Adversarial framing (loaded analogies, framing as "everyone agrees") can shift the ranker's choice. The 0.70 floor is a known weak point and is mitigated by the adversarial-framing perturbation in stability probing, which causes the abstention path to engage when framing is detected.
• Long-context degradation: the prompt is intentionally short (≤ 10 clause lines per agreement, ≤ 512-char private snippets). Cases with many clauses are decomposed via proposal-decomposition.ts rather than fed to the ranker as a single mega-prompt.
• BATNA is never accepted as self-reported; this is by design. A party-supplied BATNA is treated as a hypothesis until it is corroborated by evidence in batna-plausibility.ts.

9. Human-in-the-loop expectations#

• Required reviewers when this model is used: determined by use-case class via USE_CASE_CLASS_PROFILES. legal_claim, medical_sensitive, regulated_financial, and family_sensitive always require a qualified human reviewer queue (legal_counsel, compliance_officer, safety_reviewer, or domain expert).
• Escalation triggers specific to this model: posterior probability below 0.55 produces a low-confidence signal; abstention with reason unstable, redline_blocked, or refused triggers the §179.1.3 escalation policy via deriveEscalationDecision. Citation-quote verification failure escalates as a model-integrity incident.
• Mediator-assist vs autonomous use: the model is mediator-assist for every class except low_stakes_personal, community_restorative, creative_collaboration, commerce_negotiation (under explicit autonomous- acceptance authority bounded by settlementLimit), and agent_to_agent_contract (within the policy-constrained autonomous- acceptance contract of §179.7.6.3).

10. Incident history#

• 2026-04-25 (v1.0.0 release review): no incidents on file. Future entries reference incident ids from libs/contracts/concordia/src/incidents/.

11. Regulatory posture#

• EU AI Act Article 11 / Annex IV alignment: technical documentation obligations satisfied by this card together with the optimizer cards under ../optimizers/ and the system card at ../systems/concordia-workbench.md. See docs/compliance/eu-ai-act-mapping.md §Art. 11 for the row-by-row mapping.
• JAMS AI Rules expert-inspection readiness: prompt template version, base-model id, deterministic settings, response schema, and citations are all preserved on every comparison via the reproducibility manifest (libs/contracts/concordia/src/reproducibility/). Trade-secret flags permitted on private-intake snippet content; the citation hash is exposed unredacted.
• NIST AI RMF harm coverage: H1 (privacy), H2 (security), H3 (interpretability), H4 (fairness), H5 (safety), H6 (accountability) per docs/compliance/nist-ai-rmf-mapping.md. H7 (environmental) addressed at the system-card level via cost / latency observability.

12. References#

• libs/nous/preference-inference/src/pairwise-prompt.ts — prompt builder & response schema
• libs/nous/preference-inference/src/pairwise-plan.ts — stability thresholds & per-comparison deadline defaults
• libs/nous/preference-inference/src/stability-probes.ts — perturbation runner used for §179.3.2.4 stability tests
• libs/iris/concordia-assistant/src/privacy-red-team.test.ts — privacy red-team coverage
• libs/contracts/concordia/src/baseline-benchmark-gate/baseline-benchmark-gate.ts — release-gate winning-suite arithmetic
• docs/research/bibliography.md §pairwise-preference — Bradley, Terry, Bradley-Terry-Luce, Davidson, ProMediate, LLMediator