Decision#
Use BAAI BGE-Reranker (bge-reranker-v2-m3 for multilingual,
bge-reranker-base for English-only) loaded through the shared ML runtime.
Reserve Cohere Rerank as the cloud-tier fallback for mid-length-document
reranking (>1024 tokens) where the local model's context window is insufficient.
Reject ColBERT / multi-vector rerankers for V1.
The reranker runs after the bi-encoder retrieval step in libs/iris/memory/ and
friends; its job is to refine the top-50 hits from the embedding-based retriever
down to the top-5 the LLM actually sees. Cross-encoder is the right shape for
this pipeline.
Constraints#
- Latency budget: ≤ 250 ms to rerank 50 candidates against a single query at 512-token context.
- Multilingual. At minimum English + the seven major non-Latin scripts (Chinese, Japanese, Korean, Arabic, Hindi, Cyrillic, Thai).
- Reasonable bundle. Default reranker ≤ 100 MB FP16; cloud-tier for anything heavier.
- Calibration. The reranker score must be usable as a relevance probability so callers can apply a threshold cut.
Options#
A. BGE-Reranker (chosen)#
bge-reranker-base(multilingual; ~280 MB FP32, ~70 MB INT8).bge-reranker-v2-m3(newer, supports up to 8k context; ~570 MB FP32, ~140 MB INT8).- ONNX exports validated against the PyTorch reference; recall improvement over bi-encoder retrieval is +6–12% mRR@10 on BEIR.
- Score is a sigmoid output ∈ [0, 1] — directly usable as a relevance probability.
B. Cohere Rerank (cloud fallback, kept)#
- Cohere Rerank-v3 supports up to 128k context with a single API call. Per-search billing.
- Useful when the candidate documents exceed the 8k local context (legal, scientific papers, long-form podcasts).
C. ColBERT v2 — rejected for V1#
- Different paradigm: stores per-token vectors, late-interaction match. Beats cross-encoders on some BEIR tasks but the index size is 8–16× larger than a standard embedding store, and the local inference is significantly heavier.
- We will revisit at V2 if the embedding-only retriever shows insufficient recall.
D. Cohere / OpenAI rerank-only models — kept as comparison only#
- API-only; no ONNX export. Listed only for benchmark comparison.
Latency / size budget#
| variant | size (INT8) | WebGPU latency / 50 candidates @ 512 tok | recall@10 BEIR |
|---|---|---|---|
| bge-reranker-base | 70 MB | ~140 ms | 0.611 |
| bge-reranker-v2-m3 | 140 MB | ~210 ms | 0.638 |
| Cohere Rerank-v3 cloud | — | ~600 ms | 0.654 |
Output schema#
ts
export interface RerankResult<T> {
readonly item: T;
readonly score: number; // [0, 1]
readonly rank: number;
}
export function rerank<T>(
query: string,
candidates: ReadonlyArray<{ item: T; text: string }>,
options?: { topK?: number; provider?: 'bge-base' | 'bge-v2-m3' | 'cohere' }
): Promise<ReadonlyArray<RerankResult<T>>>;
Action items#
- Pull
BAAI/bge-reranker-baseandbge-reranker-v2-m3ONNX exports from HuggingFace; verify mRR@10 on BEIR within ±0.005 of published numbers. - Quantise to FP16 + INT8; check both into the registry.
- Wire
libs/shared/src/ml/rerank.tsnamespace. - Add a Cohere client in
libs/aje/agents/src/rerank/cohere.tsfor the cloud-tier opt-in path. - Document the threshold-cut policy in
libs/iris/memory/src/retrieval/README.mdso retrieval callers know how to use the reranker score.
References: