# Embedding Provider Stack Decision

**Status:** Final v1.0 **Date:** 2026-04-26 **Owners:** eng-iris, eng-isis
**Backlog item:** `V1-P2-0030` **Audience:** every consumer that calls
`IrisEmbeddingClient` (psyche memory retrieval, sophia research, iris semantic
memory, veritas citation grounding, …).

---

## 1. Decision

OSHUN's embedding stack is a **three-tier cascade**:

1. **Tier 1 — high-quality cloud:** OpenAI `text-embedding-3-large` (3 072 dim,
   8 191 tok input). Default for English-language short-to-medium documents and
   for any pipeline whose downstream evaluation is calibrated against OpenAI
   vectors.
2. **Tier 2 — multilingual / long-document cloud:** Voyage AI `voyage-3-large`
   (1 024 dim, 32 000 tok input, native multilingual). Default for non-English
   text, long-form documents, code, and any retrieval surface whose evaluation
   MUST exceed the nDCG@10 ≥ 0.55 multilingual gate (`V1-P2-0035`).
3. **Tier 3 — on-prem / local:** BAAI `BGE-M3` (1 024 dim, 8 192 tok input,
   native multilingual + multi-granularity). Default when the call must remain
   inside the tenant's network, when compliance forbids egress, when the
   cost-per-token quota is exhausted, and for offline batch jobs.

The cascade is **not** automatic. Consumers explicitly pick a tier when they
call `IrisEmbeddingClient.embed()` via the `tier` request field; there is no
silent failover between tiers because dimensions differ (3 072 vs 1 024) and
embeddings from different tiers MUST NOT be co-stored in the same vector index.

---

## 2. Rationale

### 2.1 Why three tiers, not one

Public benchmarks (MTEB v1.27, MIRACL, BEIR) consistently show that no single
provider dominates across every retrieval surface OSHUN serves:

| Surface                       | Best provider on public benchmarks (Apr 2026)       |
| ----------------------------- | --------------------------------------------------- |
| English short-form retrieval  | OpenAI `text-embedding-3-large`                     |
| Multilingual / cross-lingual  | Voyage `voyage-3-large` and BGE-M3 trade ~1 nDCG@10 |
| Code retrieval                | Voyage `voyage-code-3` (out of scope here)          |
| Long-form (8 k–32 k tokens)   | Voyage `voyage-3-large` (32 k window)               |
| On-prem / no-egress workloads | BGE-M3 (open weights, deploys on a single A10/L40)  |

Picking only the cloud provider locks out tenants with strict data residency
requirements; picking only the local provider sacrifices quality on the English
surfaces that drive most consumer-facing workloads. The three-tier stack lets
each consumer optimise the right trade-off without forcing a global compromise.

### 2.2 Why these specific models within each tier

**OpenAI `text-embedding-3-large`** wins Tier 1 over Cohere `embed-v4.0` and
Voyage v3 on English MTEB by ~1.5 nDCG@10. It also supports Matryoshka dimension
truncation (3 072 → 1 024 → 512 → 256), giving callers a graceful cost/quality
knob without a model switch. The 8 191-token input is sufficient for >95% of
OSHUN's chunked passages.

**Voyage `voyage-3-large`** wins Tier 2 over Cohere `embed-multilingual-v3.0` on
MIRACL (en, es, fr, ar, zh, sw — the languages V1-P2-0035 measures) and accepts
a 32 k-token window, the largest of any frontier embedding model in 2026. Voyage
publishes native multilingual training data including Arabic and Swahili, both
weak points for OpenAI's models. The 1 024-dim output keeps storage cost roughly
1/3 of Tier 1 vectors.

**BGE-M3** is the only credible local option whose published nDCG@10 on MIRACL
within ±2 points of cloud Tier 2. Its multi-granularity output (dense + sparse +
ColBERT) exceeds OSHUN's needs, and operationally it deploys as a single 1.5 GB
ONNX model on a single A10/L40 GPU or even a CPU-only fallback for low-volume
tenants. The open weights satisfy data-residency / no-egress mandates that the
cloud tiers cannot.

### 2.3 Why NOT cohere or together for Tier 1/2

Cohere `embed-v4.0` is competitive on quality but its API economics are roughly
30% above OpenAI on English and 15% above Voyage on multilingual at OSHUN's
typical batch sizes. Together's hosted embeddings are cost-competitive but lag
on quality at the surface sizes OSHUN ships. Both remain registered as
`IEmbeddingProvider` implementations in `@iris/knowledge-embeddings/providers/`
so a tenant can opt in via the `tier:'custom'` route.

### 2.4 Why dimensions are NEVER mixed

A vector index is keyed by dimensionality — a 3 072-dim `text-embedding-3-large`
vector and a 1 024-dim `voyage-3-large` vector cannot live in the same index.
Cross-comparison via cosine similarity between vectors of different dimensions
is undefined behaviour. Consumers who change tier MUST re-embed the entire
corpus into a fresh index (the migration path is `V1-P2-0033`).

---

## 3. Default tier per consumer

| Consumer                                       | Default tier | Reason                                                                               |
| ---------------------------------------------- | ------------ | ------------------------------------------------------------------------------------ |
| `@psyche/memory-retrieval` (semantic memory)   | Tier 1       | English-dominant; user-facing latency budget is 200 ms.                              |
| `@psyche/tavus-knowledge` (avatar knowledge)   | Tier 2       | Multilingual content from end-users worldwide; long-form transcripts.                |
| `@iris/integrations/sophia/research-assistant` | Tier 1       | English research corpora; calibrated against OpenAI vectors in evaluation suite.     |
| `@iris/concordia-assistant`                    | Tier 1       | English; latency-sensitive.                                                          |
| `@oshun/evidence-sophia` (citation grounding)  | Tier 2       | Source documents are multilingual; long-document retrieval is the load-bearing path. |
| Local-only / no-egress tenants                 | Tier 3       | Compliance-driven; performance acceptable for the tenant's volume.                   |

These defaults are **routable via consumer config** — a tenant can override any
consumer to a different tier. The decision does NOT silently change tiers.

---

## 4. API surface (forward-looking)

`IrisEmbeddingClient` (built in `V1-P2-0031`) exposes the following shape:

```ts
interface IrisEmbeddingClient {
  embed(req: EmbeddingRequest): Promise<EmbeddingResponse>;
  embedBatch(req: BatchEmbeddingRequest): Promise<BatchEmbeddingResponse>;
  warmCache(
    items: readonly { tier: EmbeddingTier; text: string }[]
  ): Promise<void>;
  getStats(): EmbeddingClientStats;
}

type EmbeddingTier = 'tier1_openai' | 'tier2_voyage' | 'tier3_local';

interface EmbeddingRequest {
  readonly tier: EmbeddingTier;
  readonly text: string;
  readonly model?: string; // tier default applies if omitted
  readonly dimensions?: number; // matryoshka truncation (Tier 1 only)
  readonly tenantId: string;
}

interface EmbeddingResponse {
  readonly embedding: number[];
  readonly model: string;
  readonly dimensions: number;
  readonly tokens: number;
  readonly costMicrocents: number;
  readonly cacheHit: boolean;
  readonly latencyMs: number;
}
```

Existing `@iris/knowledge-embeddings` continues to ship as the underlying
provider toolkit (per-provider HTTP clients, cache, similarity helpers).
`@iris/embeddings` (V1-P2-0031) is the canonical consumer-facing facade;
consumers depend on it only.

---

## 5. Cost model

Approximate prices (Apr 2026) per million input tokens:

| Tier   | Model                       | $/M tokens | Encoding /1k tok | Memory @1M docs |
| ------ | --------------------------- | ---------- | ---------------- | --------------- |
| Tier 1 | text-embedding-3-large 3072 | $0.13      | ~3 072 floats    | ~12 GB          |
| Tier 1 | text-embedding-3-large 1024 | $0.13      | ~1 024 floats    | ~4 GB           |
| Tier 2 | voyage-3-large              | $0.18      | ~1 024 floats    | ~4 GB           |
| Tier 3 | BGE-M3 self-hosted          | $0 (CPU)   | ~1 024 floats    | ~4 GB           |

The IrisEmbeddingClient records cost in microcents (`V1-P2-0034`) matching the
canonical pricing unit defined in `@oshun/contracts/llm/primitives`.

---

## 6. Migration & versioning

When OSHUN upgrades a tier's default model (e.g., from `text-embedding-3-large`
to a future `text-embedding-4`):

1. The new model is added to the tier's supported-models list.
2. A flag (`embeddings.defaultModel.<tier>`) flips the default for new ingest
   only.
3. `V1-P2-0033` migration utility re-embeds existing corpora in batches;
   affected vector indices are dual-written until backfill completes; reads
   switch at completion.

Tier moves (e.g., Tier 1 → Tier 2 for a consumer) are explicit config changes
accompanied by a new index — never an in-place re-embed because dimensions
change.

---

## 7. Verification

`V1-P2-0030` is satisfied by this document. Subsequent items (`V1-P2-0031` …
`V1-P2-0035`) produce the implementation, cache, migration, telemetry, and
multilingual-quality verification. This document is the cross-reference point
for each.

The decision is reviewed at the start of every quarter; any provider swap
requires updating this document and the dependent
`embeddings-provider-decision.md` change-log section before implementation
begins.
