Eve library

Embedding Provider Stack Decision

Guide

Revision 1

On this page

Status: Final v1.0 Date: 2026-04-26 Owners: eng-iris, eng-isis Backlog item: V1-P2-0030 Audience: every consumer that calls IrisEmbeddingClient (psyche memory retrieval, sophia research, iris semantic memory, veritas citation grounding, …).


1. Decision#

OSHUN's embedding stack is a three-tier cascade:

  1. Tier 1 — high-quality cloud: OpenAI text-embedding-3-large (3 072 dim, 8 191 tok input). Default for English-language short-to-medium documents and for any pipeline whose downstream evaluation is calibrated against OpenAI vectors.
  2. Tier 2 — multilingual / long-document cloud: Voyage AI voyage-3-large (1 024 dim, 32 000 tok input, native multilingual). Default for non-English text, long-form documents, code, and any retrieval surface whose evaluation MUST exceed the nDCG@10 ≥ 0.55 multilingual gate (V1-P2-0035).
  3. Tier 3 — on-prem / local: BAAI BGE-M3 (1 024 dim, 8 192 tok input, native multilingual + multi-granularity). Default when the call must remain inside the tenant's network, when compliance forbids egress, when the cost-per-token quota is exhausted, and for offline batch jobs.

The cascade is not automatic. Consumers explicitly pick a tier when they call IrisEmbeddingClient.embed() via the tier request field; there is no silent failover between tiers because dimensions differ (3 072 vs 1 024) and embeddings from different tiers MUST NOT be co-stored in the same vector index.


2. Rationale#

2.1 Why three tiers, not one#

Public benchmarks (MTEB v1.27, MIRACL, BEIR) consistently show that no single provider dominates across every retrieval surface OSHUN serves:

Surface Best provider on public benchmarks (Apr 2026)
English short-form retrieval OpenAI text-embedding-3-large
Multilingual / cross-lingual Voyage voyage-3-large and BGE-M3 trade ~1 nDCG@10
Code retrieval Voyage voyage-code-3 (out of scope here)
Long-form (8 k–32 k tokens) Voyage voyage-3-large (32 k window)
On-prem / no-egress workloads BGE-M3 (open weights, deploys on a single A10/L40)

Picking only the cloud provider locks out tenants with strict data residency requirements; picking only the local provider sacrifices quality on the English surfaces that drive most consumer-facing workloads. The three-tier stack lets each consumer optimise the right trade-off without forcing a global compromise.

2.2 Why these specific models within each tier#

OpenAI text-embedding-3-large wins Tier 1 over Cohere embed-v4.0 and Voyage v3 on English MTEB by ~1.5 nDCG@10. It also supports Matryoshka dimension truncation (3 072 → 1 024 → 512 → 256), giving callers a graceful cost/quality knob without a model switch. The 8 191-token input is sufficient for >95% of OSHUN's chunked passages.

Voyage voyage-3-large wins Tier 2 over Cohere embed-multilingual-v3.0 on MIRACL (en, es, fr, ar, zh, sw — the languages V1-P2-0035 measures) and accepts a 32 k-token window, the largest of any frontier embedding model in 2026. Voyage publishes native multilingual training data including Arabic and Swahili, both weak points for OpenAI's models. The 1 024-dim output keeps storage cost roughly 1/3 of Tier 1 vectors.

BGE-M3 is the only credible local option whose published nDCG@10 on MIRACL within ±2 points of cloud Tier 2. Its multi-granularity output (dense + sparse + ColBERT) exceeds OSHUN's needs, and operationally it deploys as a single 1.5 GB ONNX model on a single A10/L40 GPU or even a CPU-only fallback for low-volume tenants. The open weights satisfy data-residency / no-egress mandates that the cloud tiers cannot.

2.3 Why NOT cohere or together for Tier 1/2#

Cohere embed-v4.0 is competitive on quality but its API economics are roughly 30% above OpenAI on English and 15% above Voyage on multilingual at OSHUN's typical batch sizes. Together's hosted embeddings are cost-competitive but lag on quality at the surface sizes OSHUN ships. Both remain registered as IEmbeddingProvider implementations in @iris/knowledge-embeddings/providers/ so a tenant can opt in via the tier:'custom' route.

2.4 Why dimensions are NEVER mixed#

A vector index is keyed by dimensionality — a 3 072-dim text-embedding-3-large vector and a 1 024-dim voyage-3-large vector cannot live in the same index. Cross-comparison via cosine similarity between vectors of different dimensions is undefined behaviour. Consumers who change tier MUST re-embed the entire corpus into a fresh index (the migration path is V1-P2-0033).


3. Default tier per consumer#

Consumer Default tier Reason
@psyche/memory-retrieval (semantic memory) Tier 1 English-dominant; user-facing latency budget is 200 ms.
@psyche/tavus-knowledge (avatar knowledge) Tier 2 Multilingual content from end-users worldwide; long-form transcripts.
@iris/integrations/sophia/research-assistant Tier 1 English research corpora; calibrated against OpenAI vectors in evaluation suite.
@iris/concordia-assistant Tier 1 English; latency-sensitive.
@oshun/evidence-sophia (citation grounding) Tier 2 Source documents are multilingual; long-document retrieval is the load-bearing path.
Local-only / no-egress tenants Tier 3 Compliance-driven; performance acceptable for the tenant's volume.

These defaults are routable via consumer config — a tenant can override any consumer to a different tier. The decision does NOT silently change tiers.


4. API surface (forward-looking)#

IrisEmbeddingClient (built in V1-P2-0031) exposes the following shape:

ts
interface IrisEmbeddingClient {
  embed(req: EmbeddingRequest): Promise<EmbeddingResponse>;
  embedBatch(req: BatchEmbeddingRequest): Promise<BatchEmbeddingResponse>;
  warmCache(
    items: readonly { tier: EmbeddingTier; text: string }[]
  ): Promise<void>;
  getStats(): EmbeddingClientStats;
}

type EmbeddingTier = 'tier1_openai' | 'tier2_voyage' | 'tier3_local';

interface EmbeddingRequest {
  readonly tier: EmbeddingTier;
  readonly text: string;
  readonly model?: string; // tier default applies if omitted
  readonly dimensions?: number; // matryoshka truncation (Tier 1 only)
  readonly tenantId: string;
}

interface EmbeddingResponse {
  readonly embedding: number[];
  readonly model: string;
  readonly dimensions: number;
  readonly tokens: number;
  readonly costMicrocents: number;
  readonly cacheHit: boolean;
  readonly latencyMs: number;
}

Existing @iris/knowledge-embeddings continues to ship as the underlying provider toolkit (per-provider HTTP clients, cache, similarity helpers). @iris/embeddings (V1-P2-0031) is the canonical consumer-facing facade; consumers depend on it only.


5. Cost model#

Approximate prices (Apr 2026) per million input tokens:

Tier Model $/M tokens Encoding /1k tok Memory @1M docs
Tier 1 text-embedding-3-large 3072 $0.13 ~3 072 floats ~12 GB
Tier 1 text-embedding-3-large 1024 $0.13 ~1 024 floats ~4 GB
Tier 2 voyage-3-large $0.18 ~1 024 floats ~4 GB
Tier 3 BGE-M3 self-hosted $0 (CPU) ~1 024 floats ~4 GB

The IrisEmbeddingClient records cost in microcents (V1-P2-0034) matching the canonical pricing unit defined in @oshun/contracts/llm/primitives.


6. Migration & versioning#

When OSHUN upgrades a tier's default model (e.g., from text-embedding-3-large to a future text-embedding-4):

  1. The new model is added to the tier's supported-models list.
  2. A flag (embeddings.defaultModel.<tier>) flips the default for new ingest only.
  3. V1-P2-0033 migration utility re-embeds existing corpora in batches; affected vector indices are dual-written until backfill completes; reads switch at completion.

Tier moves (e.g., Tier 1 → Tier 2 for a consumer) are explicit config changes accompanied by a new index — never an in-place re-embed because dimensions change.


7. Verification#

V1-P2-0030 is satisfied by this document. Subsequent items (V1-P2-0031 … V1-P2-0035) produce the implementation, cache, migration, telemetry, and multilingual-quality verification. This document is the cross-reference point for each.

The decision is reviewed at the start of every quarter; any provider swap requires updating this document and the dependent embeddings-provider-decision.md change-log section before implementation begins.