Sophia — Grounding Substrate

Sophia is the platform substrate that lets every Oshun domain speak only when grounded: it turns a question plus retrieved sources into a cited answer, scores the credibility of those sources, and refuses to fabricate synthesis it cannot support. It is a substrate, not a shell peer — it has no consumer tab of its own (its only direct surface is an operator/research workbench) and instead backs Veritas, Metis, Tara, Arete, Nyx, Nisaba, Studio, and the assistant through a stable evidence adapter. This page sits among the platform-substrate deep-dives in the V1 architecture set hubbed at ../ARCHITECTURE.md, alongside Iris, Psyche, Lilith, Isis, and Aje.

Read this page for what is shipping vs. spec. Sophia's design space is large, and the V1 doc set historically described the full vision as if it were all live. This page is candid about the line: a real extractive grounded-answer composer and a real source-credibility engine ship today on the live BFF; a real BM25 engine, two real verification loops, and a real evidence adapter exist in the repo but are not wired into the live customer answer path; and the six-stage ingestion pipeline with multi-store indexing is spec-only — the live answer retrieves over the in-process Nisaba public-domain corpus.

Canonical home (§13). Sophia is a cross-product substrate, so its canonical reference home is the domain space docs/domains/sophia and its code-linked entity catalog at systems/sophia. This page is V1's view — how the V1 platform composes Sophia; the substrate itself is documented in full at its canonical home, which this page references rather than duplicates.

What ships on the live path today#

There are exactly two Sophia surfaces wired into the customer BFF (apps/oshun/bff), both real and both deterministic:

The answer route is registered in apps/oshun/bff/src/routes/domain-stubs.ts (the POST /v1/sophia/answer declaration with originGuard + csrfGuard pre-handlers) and delegates to buildSophiaGroundedAnswer, which calls composeExtractiveAnswer from apps/oshun/bff/src/sophia/answer-composer.ts. The grounding route lives in apps/oshun/bff/src/routes/sophia.ts, is scoped to domain:sophia (or domain:*), is fail-closed (401 without an auth context, 403 without the Sophia scope), and calls groundQuery from apps/oshun/bff/src/sophia/grounding-service.ts.

Everything below distinguishes these shipping surfaces from the richer spec-and-library material so the page stays honest about V1.

The grounded-answer composer (live default, no LLM)#

The always-on default for /v1/sophia/answer is composeExtractiveAnswer — a pure, deterministic, no-I/O function. Given the query plus the citations retrieved from Nisaba, it emits one claim per cited passage, where the claim text is the passage summary verbatim and the label is always retrieved. By construction it never synthesizes, never asserts model-only content, and never fabricates a citation.

Why verbatim, not fact-extraction#

This is a real engineering decision the prior docs omit. The @sophia/research-engine exposes extractFactsFromText, which needs subject–predicate–object triples; on the short Nisaba public-domain summaries it returns []. Routing extractive answers through it would yield an empty claim list and therefore a false ungrounded verdict on perfectly good evidence. So the composer takes the honest primitive for short summaries: one verbatim claim per cited passage. The composer's own header comment documents this tradeoff.

Ranking and de-duplication#

Passages are ordered by Jaccard token-set overlap (lexicalOverlap) between the query tokens and each citation's title + summary, with a stable tie-break by original index so equal-scoring results keep input order. Identical rendered summaries are de-duplicated in the prose, but both claims are still emitted — each one traces to a distinct source id. The composed answer is framed explicitly: "Drawing on N grounded sources: … Sophia asserts nothing beyond these sources."

Accuracy note — this is Jaccard overlap, not BM25. The V1 feature docs headline "lexical retrieval is BM25." A real BM25 engine does exist in the repo (see below), but the live answer path does not use it: the composer ranks by lexicalOverlap (Jaccard), and the BFF search route uses a separate hand-weighted lexical scorer in apps/oshun/bff/src/search/ranking.ts. BM25 is unwired in V1's customer answer flow.

Grounding-state thresholds (concrete and previously undocumented)#

The composer's grounding state is a simple, deterministic function of citation count — these exact thresholds were never surfaced in the docs:

(See answer-composer.ts:80-81 for the empty-citations branch and :121-122 for the ≥3 rule.) The route-level confidence band in buildSophiaGroundedAnswer mirrors this: unavailable for 0 citations, cautious for fewer than 3, grounded for 3 or more.

Claim labels — the type vs. the live behavior#

SophiaClaimLabel (in apps/oshun/bff/src/sophia/answer-types.ts:30) is a three-valued union:

type SophiaClaimLabel = 'retrieved' | 'synthesized' | 'model-only';

retrieved means lifted directly from one cited source; synthesized means composed from multiple retrieved passages; model-only is explicitly forbidden for Sophia output (it exists only for type completeness, and the LLM seam coerces any such label to synthesized). The live extractive composer only ever emits retrieved. The synthesized/model-only labels are reachable only through the optional LLM enhancement described next — which is off by default.

Reconciling the doc vs. the code. The architecture sequence diagram below and features.md frame the grounded answer as an LLM synthesis step that labels "retrieval vs. synthesis." In V1 the default branch is non-LLM: every claim is retrieved by construction. The retrieval-vs-synthesis labeling is a real contract, but the live default never exercises the synthesis side of it.

The grounding state enum vs. the spec grounding status#

There are two distinct grounding enums in the codebase, and they are not the same thing:

The retracted-source / retracted_source member is part of the contract but is never produced today — Nisaba results carry no retraction flag, so there is no honest basis to emit it. It is documented-and-never-produced rather than invented (the composer's own comment says exactly this).

Optional LLM abstractive enhancement (real, gated, fail-soft)#

A real configuration seam exists and is undocumented in the feature pages: when OSHUN_LLM_API_BASE, OSHUN_LLM_API_KEY, and OSHUN_LLM_MODEL are all set and there is at least one citation, buildSophiaGroundedAnswer calls app.sophiaAnswerSynthesizer.synthesize(...) over the same retrieved passages to produce abstractive prose with retrieved/synthesized claims attributed to real citation ids. It is hardened:

• A defensive second pass keeps only citationIds that exist in the actual citations, then drops any claim left with zero citations.
• On any synthesizer error or timeout the code falls through to the extractive composer — never a 500, never un-cited prose, never a fabricated citation (domain-stubs.ts ~417-449).

So even with the LLM enabled, the floor is the deterministic extractive answer. This is the only place in the live path where a synthesized label can appear.

The source-credibility engine (live, POST /v1/sophia/sources/grounding)#

The second shipping surface answers a different question: given a query and a set of candidate sources, how credible is the grounding? It wires the real @sophia/research-engine source-scoring engine, which previously had no BFF runtime surface.

groundQuery (grounding-service.ts) scores each source with scoreSource, ranks them with rankSources, and reports overall confidence via calculateAverageSourceScore — all imported from @sophia/research-engine/source-scoring.

Scoring weights (shipping, previously unsurfaced)#

scoreSource is a weighted blend defined by DEFAULT_SOURCE_SCORING_CONFIG:

The domain-tier base scores (TIER_SCORES) are concrete: authoritative 0.95, academic 0.90, reputable 0.75, standard 0.60, unknown 0.50, low 0.30.

The corroboration floor (SOPHIA_GROUNDING_FLOOR = 0.5)#

This is the substrate's sharpest, and previously undocumented, opinion. A query counts as grounded only when its top-ranked source clears a combined floor:

export const SOPHIA_GROUNDING_FLOOR = 0.5;

The rationale is written into the code: even an authoritative domain contributes only 0.95 × 0.4 = 0.38 on domain reputation alone — below 0.5. A source must add recency, citations, or peer-review to ground a claim. Domain reputation by itself is deliberately insufficient, mirroring a sound research stance: a famous domain with a stale, uncited, non-reviewed page does not get to ground an answer.

Each scored citation carries its full breakdown (domainScore, recencyScore, citationBonus, peerReviewBonus, overallScore) plus a meetsFloor boolean, and the response reports grounded, confidence, citationCount, the ranked citations, and the topCitation.

The grounding flow (live shape)#

Every Sophia-backed answer follows the same retrieve → rank → ground → cite shape, and weak-grounding is a first-class branch: Sophia refuses to fabricate synthesis when retrieval can't support a span and returns an envelope flagged ungrounded/abstained instead. The diagram below is the intended substrate shape; the live path differs at three labeled points (see the callouts under it).

Three live-vs-doc reconciliations (the original ARCHITECTURE.md diagram drew these differently):

1. Retriever is the Nisaba corpus, not a vector fan-out. The old diagram showed a "Retriever (pgvector · Qdrant)" with a downstream Elasticsearch/Neo4j index. The live /v1/sophia/answer retrieves over the in-process Nisaba public-domain corpus via app.domainAdapters.nisaba.searchLibrary({ query, limit: 6 }). No Qdrant / Elasticsearch / Neo4j store is in the live answer path. (See grounding-service.ts header.)
2. The default synthesis step is non-LLM. The old diagram presented "LLM synthesize answer with retrieval-vs-synthesis labels" as the primary branch. The default is the deterministic extractive composer; the LLM is an optional, gated, fail-soft enhancement.
3. Weak grounding is ungrounded/abstained, by citation count — the thresholds in the table above — not by a separate refusal subsystem.

Real but unwired in V1's live answer path#

These are full implementations that exist in the repo and are worth knowing about, but grep over apps/oshun/bff/src shows they are imported nowhere on the live customer answer path. They are honest "built but not wired" — not stubs.

BM25 lexical engine — libs/sophia/semantic-search#

A real BM25 implementation lives at libs/sophia/semantic-search/src/bm25/bm25.ts, with a full BM25Config (k1, b, minTokenLength, removeStopWords, stopWords, fieldWeights, maxHighlights, highlightRadius), per-field weights (title/summary/body/keywords/default), IDF, document-length normalization, and highlight extraction. It is the doc's headline lexical method — but it does not run on /v1/sophia/answer. (The only BFF surface that names bm25 is the admin search-discovery console apps/oshun/bff/src/routes/admin-studio-search-ranking.ts, which reports rankingFunction: 'bm25' with k1/b defaults for operators — distinct from the customer answer path.)

Verification loops — libs/sophia/verification#

Two fully-implemented loops gate high-stakes claims in the library:

• FactCheckLoop (fact-check-loop/fact-check-loop.ts): termination states pass | partial | unsupported | contradicted, resolution actions cite | hedge | refuse | escalate, latency modes (assistant-interactive, deep-research, batch-publication), and independent-retrieval tracking.
• runContradictionLoop + DEFAULT_CONTRADICTION_LOOP_CONFIG (contradiction/contradiction-loop.ts): counterclaim records with relations (direct-contradiction, temporal-conflict, statistical-conflict, …), resolution actions (prefer-source, surface-both, hedge, refuse, escalate-to-expert), and source-quality bands high | mixed | low | contested.

Accuracy + contradiction note. The feature docs say the fact-check and contradiction loops "gate high-stakes claims … on live output," and the publication gate criteria assume a contradiction backlog and unsupported-claim backlog computed by these loops. In V1 neither loop runs on any live customer answer — grep -rln 'contradiction-loop\|fact-check-loop' apps/oshun/bff/src returns nothing, and the live composer emits one retrieved claim per passage and never invokes either loop. The publication gate as described is therefore aspirational for /v1/sophia/answer, not enforced on it. The loops are production-grade code awaiting wiring.

The evidence adapter — @oshun/evidence-sophia#

@oshun/evidence-sophia is the canonical, versioned evidence model shared across consumers. Its index.ts re-exports types, evidence-model, source-set, source-lifecycle, educational-claim-grounding, adapter, and canonical-adapter. The contract is pinned:

SOPHIA_EVIDENCE_ADAPTER_CONTRACT = {
  contract: 'oshun.evidence.sophia.adapter',
  version: '1.0.0',
  minimumCompatibleVersion: '1.0.0',
};

The canonical metadata (getCanonicalSophiaEvidenceMetadata) enumerates the substrate's role and reach:

• consumers: metis, assistant, veritas, nisaba, studio, admin, tara, arete, nyx
• capabilities: semantic_search, citations, evidence_packs, notebooks, source_graph, claim_verification, grounded_answers, trace_export
• productPosition: substrate_not_shell_peer; directSurface: workbench_only

SophiaEvidenceAdapter is the full read/write interface (search, ground, assembleEvidencePack, verifyClaims, saveToNotebook, exportEvidenceTrace, plus contract/metadata/availability). A concrete createCanonicalSophiaEvidenceAdapter binding over @sophia/client exists in canonical-adapter.ts — but it is not the implementation wired into the live grounding surface. The BFF instead builds a role-scoped read registry over app.domainAdapters (see RBAC below). So in V1 the canonical @sophia/*-bound adapter is present in the library but unwired on the live customer path.

Role-based access control on evidence (concrete, not abstract)#

The adapter ships a real RBAC model the docs describe only as "Customer/Admin surfaces." SophiaEvidenceAdapterReadRole = 'grounding' | 'review' | 'admin', and SOPHIA_EVIDENCE_ADAPTER_ROLE_CAPABILITIES gives each role an explicit capability list:

The live routes/sophia.ts enforces this end-to-end: each adapter endpoint parses ?role= (defaulting to grounding), calls requireCapability(role, capability, reply), and returns a 403 with reason: sophia_read_capability_unavailable when the role lacks the capability — e.g. grounding cannot trace_export, and only admin/review can. The notebook adapter is backed by the user's real Nisaba notebooks (mapped into the Sophia record shape) rather than an invented one.

Source-set lifecycle primitives (richer than the prose)#

The same package carries real source-set lifecycle machinery the doc only gestures at:

• evaluateSophiaSourceSetReadiness(sourceSet, asOf) → a SophiaSourceSetReadiness with usableForGrounding, freshnessStatus, rightsStatus, retractionState, and an explicit blockers list.
• computeSophiaSourceSetHash(sourceSet) → a deterministic sha256:… content hash via @noble/hashes/sha2 (sha256) over a stable payload — the identity used to detect material change.
• planSophiaSourceLifecycleInvalidation(...) and materializeSophiaSourceSetArtifacts(...) → the change-detection → invalidation plan over derived artifacts, with freshness/rights/retraction blockers that prevent an unusable source set from grounding.

Trace export#

SophiaTraceExportKind = bibliography | evidence_table | grounded_report | review_packet and SophiaTraceExportFormat = json | csv | markdown | pdf (types.ts:75-80) define the evidence-trace export contract surfaced through the review/admin roles.

Aspirational / spec-only (described concretely, not yet a live flow)#

The original architecture page drew an ingestion pipeline and source lifecycle as if orchestrated end-to-end. In V1 these are a specification, not a running flow on the customer path. The live answer retrieves over the in-process Nisaba public-domain corpus; it does not pass sources through ingestion or multi-store indexing first. The spec describes:

• A six-stage ingestion pipeline: parse → chunk → enrich → embed → index → ingestion-quality eval, as a single orchestrated flow, with below-threshold sources parked in a remediation queue.
• A per-source-type SourceAdapter set: PDF, HTML, RSS, YouTube, audio, IIIF, API, LMS, and BYOM parsers feeding semantic / hierarchical / table-aware / code-aware / time-aligned chunking.
• Multi-store indexing across pgvector, Qdrant, Elasticsearch, and Neo4j, per-tenant.
• Mandatory human checkpoints, an operator workbench, and a downstream invalidation cascade into Veritas, Metis, Nisaba, Tara, and Living Scenes.

These remain the target shape (the lifecycle primitives above are real building blocks for it), but none of them run on a live customer answer in V1. See V1/features.md § Ingestion Pipeline, Source Lifecycle, and Per-Type Adapters and the §9 backlog extensions for the full specification.

Adjacent: the search offline-evaluation gate#

One related real surface is worth a pointer because it shares Sophia's "measure-don't-fabricate" discipline. buildSearchReleaseGateSummary (from @oshun/search-discovery) is mounted live at POST /v1/search/offline-eval (apps/oshun/bff/src/search/offline-eval-route.ts), computing per-slice NDCG@10, MAP@10, recall@100, coverage, diversity, and serendipity, and failing a candidate ranker/recommender that regresses. It is part of the search/knowledge-graph story rather than the grounded-answer path — see Search, Discovery, and Knowledge Graph.

Where to point readers (path correction)#

The prior architecture page named apps/sophia/* plus libs/sophia/{client, database, ingestion, indexing, evaluation} as the service backing for the customer grounding surface. The shipping V1 customer surface lives in the BFF:

• apps/oshun/bff/src/sophia/{answer-composer.ts, answer-synthesizer seam, answer-types.ts, grounding-service.ts}
• apps/oshun/bff/src/routes/{sophia.ts, domain-stubs.ts}
• apps/oshun/bff/src/adapters/sophia-read-adapters.ts (the role-scoped read registry)

The @sophia/* libraries (research-engine, semantic-search, verification) are the engines the BFF composes — source-scoring is wired live; BM25 and the verification loops are not. The @oshun/evidence-sophia adapter files are evidence-model.ts, source-set.ts, types.ts, source-lifecycle.ts, educational-claim-grounding.ts, adapter.ts, and canonical-adapter.ts — note that source-set.ts and types.ts (omitted from the older listing) carry the bulk of the real source-set / freshness / rights / retraction contracts.

Consumers#

Sophia has no consumer tab; it backs other domains through the evidence adapter: Veritas (every claim), Metis (every grounded answer to a student or teacher), Tara (explanatory notes), Arete (habit-science explainers), Nyx (event explainers), Nisaba (the source library and notebooks), Studio, the operator admin surfaces, and the assistant. The full registry is in Customer-Facing Domains; the substrate adapter contract (EvidencePack, CitationTrail, GroundedAnswerEnvelope) is tracked under § 1.2 cross-cutting in ../TODOS.md.

Related#

• High-Level Architecture
• Customer-Facing Domains
• Search, Discovery, and Knowledge Graph
• Trust, Safety, and Privacy
• Subsystem Glossary
• Iris — Assistant Memory Substrate
• V1/features.md § Ingestion Pipeline, Source Lifecycle, and Per-Type Adapters
• Hub: V1 Architecture