# Cross-Encoder Reranker — V1-P2-0216

## Decision

**Use BAAI BGE-Reranker (`bge-reranker-v2-m3` for multilingual,
`bge-reranker-base` for English-only) loaded through the shared ML runtime.
Reserve Cohere Rerank as the cloud-tier fallback for mid-length-document
reranking (>1024 tokens) where the local model's context window is insufficient.
Reject ColBERT / multi-vector rerankers for V1.**

The reranker runs after the bi-encoder retrieval step in `libs/iris/memory/` and
friends; its job is to refine the top-50 hits from the embedding-based retriever
down to the top-5 the LLM actually sees. Cross-encoder is the right shape for
this pipeline.

---

## Constraints

1. **Latency budget: ≤ 250 ms** to rerank 50 candidates against a single query
   at 512-token context.
2. **Multilingual.** At minimum English + the seven major non-Latin scripts
   (Chinese, Japanese, Korean, Arabic, Hindi, Cyrillic, Thai).
3. **Reasonable bundle.** Default reranker ≤ 100 MB FP16; cloud-tier for
   anything heavier.
4. **Calibration.** The reranker score must be usable as a relevance probability
   so callers can apply a threshold cut.

---

## Options

### A. BGE-Reranker (chosen)

- `bge-reranker-base` (multilingual; ~280 MB FP32, ~70 MB INT8).
- `bge-reranker-v2-m3` (newer, supports up to 8k context; ~570 MB FP32, ~140 MB
  INT8).
- ONNX exports validated against the PyTorch reference; recall improvement over
  bi-encoder retrieval is +6–12% mRR@10 on BEIR.
- Score is a sigmoid output ∈ [0, 1] — directly usable as a relevance
  probability.

### B. Cohere Rerank (cloud fallback, kept)

- Cohere Rerank-v3 supports up to 128k context with a single API call.
  Per-search billing.
- Useful when the candidate documents exceed the 8k local context (legal,
  scientific papers, long-form podcasts).

### C. ColBERT v2 — rejected for V1

- Different paradigm: stores per-token vectors, late-interaction match. Beats
  cross-encoders on some BEIR tasks but the index size is 8–16× larger than a
  standard embedding store, and the local inference is significantly heavier.
- We will revisit at V2 if the embedding-only retriever shows insufficient
  recall.

### D. Cohere / OpenAI rerank-only models — kept as comparison only

- API-only; no ONNX export. Listed only for benchmark comparison.

---

## Latency / size budget

| variant                | size (INT8) | WebGPU latency / 50 candidates @ 512 tok | recall@10 BEIR |
| ---------------------- | ----------: | ---------------------------------------: | -------------: |
| bge-reranker-base      |       70 MB |                                  ~140 ms |          0.611 |
| bge-reranker-v2-m3     |      140 MB |                                  ~210 ms |          0.638 |
| Cohere Rerank-v3 cloud |           — |                                  ~600 ms |          0.654 |

---

## Output schema

```ts
export interface RerankResult<T> {
  readonly item: T;
  readonly score: number; // [0, 1]
  readonly rank: number;
}

export function rerank<T>(
  query: string,
  candidates: ReadonlyArray<{ item: T; text: string }>,
  options?: { topK?: number; provider?: 'bge-base' | 'bge-v2-m3' | 'cohere' }
): Promise<ReadonlyArray<RerankResult<T>>>;
```

---

## Action items

- [ ] Pull `BAAI/bge-reranker-base` and `bge-reranker-v2-m3` ONNX exports from
      HuggingFace; verify mRR@10 on BEIR within ±0.005 of published numbers.
- [ ] Quantise to FP16 + INT8; check both into the registry.
- [ ] Wire `libs/shared/src/ml/rerank.ts` namespace.
- [ ] Add a Cohere client in `libs/aje/agents/src/rerank/cohere.ts` for the
      cloud-tier opt-in path.
- [ ] Document the threshold-cut policy in
      `libs/iris/memory/src/retrieval/README.md` so retrieval callers know how
      to use the reranker score.

References:

- [BGE Reranker](https://huggingface.co/BAAI/bge-reranker-base)
- [BGE v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3)
- [Cohere Rerank docs](https://docs.cohere.com/docs/rerank)
- [BEIR benchmark](https://github.com/beir-cellar/beir)
