Eve library

Cross-Encoder Reranker — V1-P2-0216

Guide

Revision 1

On this page

Decision#

Use BAAI BGE-Reranker (bge-reranker-v2-m3 for multilingual, bge-reranker-base for English-only) loaded through the shared ML runtime. Reserve Cohere Rerank as the cloud-tier fallback for mid-length-document reranking (>1024 tokens) where the local model's context window is insufficient. Reject ColBERT / multi-vector rerankers for V1.

The reranker runs after the bi-encoder retrieval step in libs/iris/memory/ and friends; its job is to refine the top-50 hits from the embedding-based retriever down to the top-5 the LLM actually sees. Cross-encoder is the right shape for this pipeline.


Constraints#

  1. Latency budget: ≤ 250 ms to rerank 50 candidates against a single query at 512-token context.
  2. Multilingual. At minimum English + the seven major non-Latin scripts (Chinese, Japanese, Korean, Arabic, Hindi, Cyrillic, Thai).
  3. Reasonable bundle. Default reranker ≤ 100 MB FP16; cloud-tier for anything heavier.
  4. Calibration. The reranker score must be usable as a relevance probability so callers can apply a threshold cut.

Options#

A. BGE-Reranker (chosen)#

  • bge-reranker-base (multilingual; ~280 MB FP32, ~70 MB INT8).
  • bge-reranker-v2-m3 (newer, supports up to 8k context; ~570 MB FP32, ~140 MB INT8).
  • ONNX exports validated against the PyTorch reference; recall improvement over bi-encoder retrieval is +6–12% mRR@10 on BEIR.
  • Score is a sigmoid output ∈ [0, 1] — directly usable as a relevance probability.

B. Cohere Rerank (cloud fallback, kept)#

  • Cohere Rerank-v3 supports up to 128k context with a single API call. Per-search billing.
  • Useful when the candidate documents exceed the 8k local context (legal, scientific papers, long-form podcasts).

C. ColBERT v2 — rejected for V1#

  • Different paradigm: stores per-token vectors, late-interaction match. Beats cross-encoders on some BEIR tasks but the index size is 8–16× larger than a standard embedding store, and the local inference is significantly heavier.
  • We will revisit at V2 if the embedding-only retriever shows insufficient recall.

D. Cohere / OpenAI rerank-only models — kept as comparison only#

  • API-only; no ONNX export. Listed only for benchmark comparison.

Latency / size budget#

variant size (INT8) WebGPU latency / 50 candidates @ 512 tok recall@10 BEIR
bge-reranker-base 70 MB ~140 ms 0.611
bge-reranker-v2-m3 140 MB ~210 ms 0.638
Cohere Rerank-v3 cloud — ~600 ms 0.654

Output schema#

ts
export interface RerankResult<T> {
  readonly item: T;
  readonly score: number; // [0, 1]
  readonly rank: number;
}

export function rerank<T>(
  query: string,
  candidates: ReadonlyArray<{ item: T; text: string }>,
  options?: { topK?: number; provider?: 'bge-base' | 'bge-v2-m3' | 'cohere' }
): Promise<ReadonlyArray<RerankResult<T>>>;

Action items#

  • Pull BAAI/bge-reranker-base and bge-reranker-v2-m3 ONNX exports from HuggingFace; verify mRR@10 on BEIR within ±0.005 of published numbers.
  • Quantise to FP16 + INT8; check both into the registry.
  • Wire libs/shared/src/ml/rerank.ts namespace.
  • Add a Cohere client in libs/aje/agents/src/rerank/cohere.ts for the cloud-tier opt-in path.
  • Document the threshold-cut policy in libs/iris/memory/src/retrieval/README.md so retrieval callers know how to use the reranker score.

References: