Sophia Grounding

Sophia is OSHUN's grounding substrate — the subsystem that any product output leans on when correctness or attribution matters. It scores source credibility, composes grounded answers that never assert beyond their evidence, models evidence packs and citation trails, and (in spec) gates the publication of high-stakes content. Sophia is explicitly a substrate, not a shell peer: it has no consumer-facing tab of its own (its directSurface is workbench_only), and its evidence flows into the assistant, Veritas, Nisaba, Metis, Tara, Arete, and Nyx. This page distinguishes, candidly, between what ships live in V1, what exists in the repo as real-but-unwired engines, and what is specified for later — because Sophia's documentation has historically described the aspiration as though it were the running system.

Reality posture (read this first). Two things are live and shipping on the BFF: the extractive grounded-answer composer behind POST /v1/sophia/answer, and the source-credibility engine behind POST /v1/sophia/sources/grounding. Several richer engines are real code in the repo but not wired into the live customer path: the BM25 lexical index, the fact-check loop, the contradiction loop, and the @oshun/evidence-sophia adapter contract. The full ingestion pipeline, multi-store indexing (Qdrant/Elasticsearch/Neo4j), per-type source adapters, mandatory human checkpoints, the operator workbench, and the downstream invalidation cascade are spec-only in V1. Where the prose below describes those, it says so.

Where the live customer path actually runs#

The shipping grounding surface is in the BFF, not in a separate apps/sophia service. Two routes carry it:

Both core routes are domain-scoped to domain:sophia (or the wildcard domain:*) and fail closed: a missing auth context returns 401, a missing domain scope returns 403 with reason: 'domain_scope_missing'. The answer journey retrieves over the in-process Nisaba public-domain corpus through the searchLibrary adapter — there is no fan-out to an external vector or graph store in the live path.

The grounded-answer composer (/v1/sophia/answer)#

What it does, end to end#

buildSophiaGroundedAnswer(app, query) is the single entry point behind both GET /v1/sophia/grounded and POST /v1/sophia/answer (the POST variant is guarded by originGuard and csrfGuard). The data flow is:

1. Retrieve. If the query is non-empty, call app.domainAdapters.nisaba.searchLibrary({ query, limit: 6 }). Any retrieval error is swallowed into an empty result set — Sophia never throws prose at the caller on a retrieval failure; it degrades to "no grounded sources."
2. Map to citations. Each Nisaba result becomes a SophiaCitation ({ id, title, src, summary, weight, citation? }) where id is the Nisaba resultId, src is the result kind, and weight comes from sophiaCitationWeight(kind).
3. Set an evidence-availability confidence. Independently of the grounding state, the envelope carries a coarse confidence: 'unavailable' for zero citations, 'cautious' for one or two, 'grounded' for three or more.
4. Compose. The default, always-on path calls composeExtractiveAnswer. If app.sophiaAnswerSynthesizer is configured and there is at least one citation, an LLM abstractive layer is attempted first, failing soft back to the extractive composer (see Optional LLM enhancement below).

Extractive by default — and why#

The default composer in answer-composer.ts is deterministic, no-LLM, no-I/O. It emits exactly one claim per retrieved passage: the claim text is the passage summary verbatim (falling back to the title when the summary is empty), and every claim is labeled retrieved. The composed answer string wraps those passages with inline [k] citation markers and the framing "Sophia asserts nothing beyond these sources."

Using verbatim passage summaries rather than fact extraction is a real engineering decision documented in the source: the research engine's extractFactsFromText needs subject–predicate–object triples and returns [] on the short Nisaba summaries. Running it would yield an empty claim list and a false ungrounded verdict. The honest extractive primitive for short summaries is therefore one verbatim claim per cited passage — a claim that is, by construction, fully traceable to a single source.

Ranking is by Jaccard token-set overlap (lexicalOverlap) between the query tokens and the combined title-and-summary tokens, with a stable tie-break by original index, so the most on-topic evidence leads. Stop words are stripped via an inlined tokenize that mirrors the research engine's tokenizer.

Accuracy note — this is not BM25. The headline "Methods: lexical (BM25)" in the source doc is misleading for the live path. A real, full BM25 implementation exists at libs/sophia/semantic-search/src/bm25/bm25.ts (BM25LexicalIndex, IDF, length-normalized term scoring), but it is not wired into the live answer path. The composer ranks by Jaccard overlap; the separate BFF search route uses a hand-weighted lexical scorer (apps/oshun/bff/src/search/ranking.ts). BM25 is unwired in V1's customer flow. See Real-but-unwired: BM25 below.

The claim label vocabulary#

// apps/oshun/bff/src/sophia/answer-types.ts:30
type SophiaClaimLabel = 'retrieved' | 'synthesized' | 'model-only';

All three labels exist in the type, but the live extractive composer only ever emits retrieved. synthesized is reserved for the (optional) LLM layer; model-only is forbidden for Sophia output entirely — the type comment notes the synthesizer coerces any such label to synthesized. This is the honest version of the doc's "retrieval-vs-synthesis label": by construction, the default answer is all retrieval, never synthesis.

Grounding-state thresholds (concrete and undocumented elsewhere)#

The composer's SophiaGroundingState is decided by a small, exact rule set that the prose docs do not surface:

(answer-composer.ts:121-122 for the grounded/partial split; :80-81 for the empty-vs-ungrounded distinction.) Note the deliberate contrast: an empty query is a declined request (abstained), while a real question that found nothing is an honest ungrounded. The retracted-source state is part of the SophiaGroundingState enum but is never produced today — Nisaba results carry no retraction flag, so emitting it would be an invention; it is documented and left unproduced rather than faked.

The answer envelope#

// apps/oshun/bff/src/sophia/answer-types.ts
interface SophiaGroundedAnswer {
  generatedAt: string;
  query: string;
  answer: string;
  confidence: 'unavailable' | 'cautious' | 'grounded';
  citations: readonly SophiaCitation[];
  claims: readonly SophiaClaim[];
  citationMap: Readonly<Record<string, readonly string[]>>; // claimId → citationIds
  groundingState: SophiaGroundingState;
  abstained: boolean;
}

This is the live realization of the doc's "grounded answer envelope": prompt (query) + retrieved passages (citations) + generated answer + claim list + citation map + abstention flag + a confidence signal. The citationMap is a redundant projection of each claim's citationIds, kept for UI convenience.

Optional LLM abstractive enhancement (a real, gated seam)#

When app.sophiaAnswerSynthesizer is set — driven by the OSHUN_LLM_API_BASE / OSHUN_LLM_API_KEY / OSHUN_LLM_MODEL configuration seam — and there is at least one citation, buildSophiaGroundedAnswer attempts an abstractive answer over the same Nisaba passages (domain-stubs.ts ~417-449). The LLM result is sanitized hard before it is trusted:

• Each synthesized claim's citationIds are filtered against the set of real citation ids (validIds); a claim with no surviving id is dropped entirely — a fabricated citation cannot reach the response.
• If the synthesizer returns abstained, or every claim was dropped, the envelope is abstained; otherwise the same ≥3 / 1–2 grounded/partial split applies.
• Any synthesizer error or timeout falls through to the deterministic extractive composer. There is no 500, no un-cited prose, and no fabricated citation on the failure path — fail-soft, never fail-fabricated.

This means the LLM is an enhancement over real grounding, not the primary branch. Architecture diagrams that present "LLM synthesize answer" as the main step overstate it: the live default is the no-LLM extractive composer, and the LLM layer only runs when explicitly configured and only over already-retrieved, already-cited passages.

The source-credibility engine (/v1/sophia/sources/grounding)#

This is the second live route. Where /v1/sophia/answer retrieves and composes, /v1/sophia/sources/grounding judges the credibility of caller-supplied sources and reports a grounding verdict — without inventing any prose.

Request and scoring#

The handler in routes/sophia.ts accepts { query (or question/q), sources: [{ url, title?, publishedDate?, citationCount?, isPeerReviewed? }] }, requires at least one source with a url (else 400 invalid_payload), and calls groundQuery (grounding-service.ts). Each source is scored by the real scoreSource from @sophia/research-engine/source-scoring, ranked by rankSources, and the overall confidence comes from calculateAverageSourceScore (arithmetic mean of overall scores).

scoreSource is a weighted blend across four dimensions — the weights documented in the grounding-service header and shipped in DEFAULT_SOURCE_SCORING_CONFIG:

Domain tiers and their TIER_SCORES are concrete:

The corroboration floor — SOPHIA_GROUNDING_FLOOR = 0.5#

The shipping behavior the docs omit entirely: a query is only "grounded" when its strongest source clears a combined-credibility floor of 0.5 (grounding-service.ts:34). This is a deliberate research stance — domain reputation alone is not enough. Even an authoritative domain contributes only 0.95 × 0.40 = 0.38 from its tier; that is below 0.5, so a citation must add recency, citation volume, or peer review to ground a claim. Every scored citation reports a meetsFloor boolean, and the top citation must clear the floor for grounded to be true. The result shape:

// apps/oshun/bff/src/sophia/grounding-service.ts
interface GroundingResult {
  query: string;
  grounded: boolean; // topCitation.overallScore >= 0.5
  confidence: number; // calculateAverageSourceScore(ranked)
  citationCount: number;
  citations: readonly ScoredCitation[]; // url, domain, domainTier, overallScore + per-dimension breakdown, meetsFloor
  topCitation: ScoredCitation | null;
}

The per-citation breakdown (domainScore, recencyScore, citationBonus, peerReviewBonus) is surfaced so a caller can see why a source did or did not ground a query, not merely the verdict.

Real, but unwired to the live path#

These engines are fully implemented in the repo and tested, but a grep of apps/oshun/bff/src shows they are imported nowhere on the live customer answer path. They are documented here honestly as latent capability, not as running behavior.

BM25 lexical index (libs/sophia/semantic-search)#

BM25LexicalIndex in libs/sophia/semantic-search/src/bm25/bm25.ts is a genuine Okapi BM25 implementation: an inverted index with calculateBM25Idf (the log(1 + (N − df + 0.5)/(df + 0.5)) form) and length-normalized term scoring via calculateBM25TermScore. Its BM25Config exposes the standard knobs and Sophia's field weighting:

Each BM25 result carries a RetrievalLabel with method: RetrievalMethod.BM25 and a LEXICAL_MATCH decomposition step, and the index enforces a RetrievalSourceScopePolicy — the machinery the doc describes as "per-method labeling" and "source-set scoping enforcement." All of this is real. None of it is on /v1/sophia/answer. (The only BM25 references under the BFF are in the admin Studio search-ranking store, apps/oshun/bff/src/studio/, a separate surface.)

Fact-check loop (libs/sophia/verification)#

runFactCheckLoop in libs/sophia/verification/src/fact-check-loop/fact-check-loop.ts implements the doc's "claim extraction → normalization → independent retrieval → claim-to-source matching → confidence scoring → resolution" exactly, including:

• Termination states pass | partial | unsupported | contradicted, with a route of complete | contradiction-loop | unsupported-claim-loop.
• Resolution actions cite | hedge | refuse | escalate, chosen against thresholds in DEFAULT_FACT_CHECK_LOOP_CONFIG (supportThreshold 0.45, contradictionThreshold 0.55, minConfidenceToCite 0.68, minConfidenceToHedge 0.42).
• Independent retrieval enforcement — a retriever whose method equals the initial retrieval method is skipped, so corroboration genuinely comes from a different method.
• Latency budgets matching the doc: assistantInteractiveP95Ms 6 000, deepResearchP95Ms 90 000, batchPublicationP95Ms 15 min, with evaluateFactCheckLoopLatencyBudget computing a real P95.
• Source-scope enforcement — a FactCheckLoopSourceScopeViolationError is thrown if retrieved evidence escapes the declared sourceSetIds/tenantId, the hard-fail the doc calls "cross-set leakage is a hard fail."

This is not a stub. But it does not run on any live customer answer — grep -rln 'fact-check-loop' over apps/oshun/bff/src returns nothing, and the live /v1/sophia/answer composer emits one retrieved claim per passage without ever invoking it.

Contradiction loop (libs/sophia/verification)#

runContradictionLoop (same library, contradiction/contradiction-loop.ts) is likewise fully implemented: contradiction detection → claim alignment → source-quality scoring → counterclaim record creation → resolution. Its ContradictionLoopResolutionAction is prefer-source | surface-both | hedge | refuse | escalate-to-expert, and it bands source quality as high | mixed | low | contested. DEFAULT_CONTRADICTION_LOOP_CONFIG carries the real thresholds (minAlignmentSimilarity 0.18, preferSourceScoreGap 0.18, surfaceBothMinQuality 0.62, hedgeMinQuality 0.42, refuseBelowQuality 0.25, expert escalation on critical severity). It produces ContradictionLoopCounterclaimRecords with provenance, audit events, and a downstream-invalidation count — the doc's "counterclaim records," "audit," and "customer surface" contracts in code. As with the fact-check loop, it is imported nowhere under the BFF and runs on no live answer.

Reconciliation with the publication gate. The source doc's publication-gate criteria ("citation integrity 100%, contradiction backlog zero, unsupported-claim backlog zero…") presuppose backlogs computed by these loops. Because the loops do not run on /v1/sophia/answer, that gate as described is aspirational, not enforced on the live customer path. See the Spec-only section.

The @oshun/evidence-sophia adapter contract#

@oshun/evidence-sophia (libs/oshun/evidence-sophia) is the canonical evidence model — real types and builders, re-exported from one barrel: types, evidence-model, source-set, source-lifecycle, educational-claim-grounding, adapter, and canonical-adapter. The contract is versioned:

// adapter.ts:48-53
SOPHIA_EVIDENCE_ADAPTER_CONTRACT = {
  contract: 'oshun.evidence.sophia.adapter',
  version: '1.0.0',
  minimumCompatibleVersion: '1.0.0',
};

Canonical metadata declares the substrate's posture and reach:

• Consumers (adapter.ts:219-229): metis, assistant, veritas, nisaba, studio, admin, tara, arete, nyx.
• Capabilities: semantic_search, citations, evidence_packs, notebooks, source_graph, claim_verification, grounded_answers, trace_export.

SophiaEvidenceAdapter is, however, an interface — createCanonicalSophiaEvidenceAdapter delegates to an injected SophiaEvidenceApiAdapter (a @sophia/client-shaped dependency, e.g. searchCorpus, getHealth). In V1 there is no concrete @sophia/* service binding behind it on the live path; the BFF adapter routes (/v1/sophia/adapter/*) wire the registry but stand on that injected boundary and fail closed (502 evidence_adapter_unavailable) when the dependency is absent. This is a real, honest fail-loud seam, not a fabricated success.

Role-based read adapters (concrete RBAC on evidence)#

A detail the doc covers only abstractly as "Customer/Admin surfaces" is a real, shipping RBAC model. SophiaEvidenceAdapterReadRole is grounding | review | admin, and SOPHIA_EVIDENCE_ADAPTER_ROLE_CAPABILITIES (adapter.ts:138-161) defines exactly what each role may read:

The BFF route enforces this per request: requireCapability(role, capability, reply) returns 403 sophia_read_capability_unavailable for a disallowed combination, an invalid role query returns 400 invalid_sophia_read_role, and an absent role defaults to grounding (least-privileged). The registry is built per user (buildRegistry(userId)), and listNotebooks is wired to the user's real Nisaba notebooks via nisabaConsumerStateStore.listNotebooks, not an invented record.

Two distinct grounding enums#

Be careful: @oshun/evidence-sophia and the BFF answer types use two different grounding enums that must not be conflated (types.ts:50-60):

• SophiaAnswerGroundingState = grounded | partial | ungrounded | abstained | retracted_source (the per-answer posture; note retracted_source is underscored here, while the BFF SophiaGroundingState spells it retracted-source).
• SophiaGroundingStatus = grounded | partially_grounded | unsupported | conflicting (a coarser claim-grounding status used in the evidence model).

Trace exports are likewise typed: SophiaTraceExportFormat = json | csv | markdown | pdf and SophiaTraceExportKind = bibliography | evidence_table | grounded_report | review_packet (types.ts:75-80), surfaced live through GET /v1/sophia/adapter/trace-export.

Source-set lifecycle primitives#

@oshun/evidence-sophia ships real source-set lifecycle code that is richer than the doc's prose:

• validateSophiaSourceSet parses against the SourceSetSchema from @oshun/contracts/common.
• computeSophiaSourceSetHash produces a deterministic sha256: hash over a stable-stringified canonical payload, using @noble/hashes/sha2 sha256 (source-set.ts:10-11,27). The hash payload deliberately includes rights/license/retraction state per item, so a rights change re-hashes the set.
• evaluateSophiaSourceSetReadiness computes usableForGrounding and an explicit blockers list — freshness:expired, rights:blocked|expired|unknown, source-set:retracted, source-set:superseded, and per-source blockers — the freshness/rights/retraction gates the doc describes abstractly.
• materializeSophiaSourceSetArtifacts and planSophiaSourceLifecycleInvalidation (source-lifecycle.ts) model the refresh/diff/invalidation lifecycle, with a severity ladder of info | review | blocking | critical.

Spec-only in V1 (described as concrete, not yet built as a live flow)#

The following are described in the source doc as concrete subsystems. In V1 they are specification, not running customer behavior. The live BFF only retrieves over the in-process Nisaba corpus.

The six-stage ingestion pipeline#

The doc specifies one orchestrated flow per source: parse → chunk → enrich → embed → index → ingestion-quality eval, keyed for idempotency on (uri, content-hash, tenant_scope), with per-stage audit events, ancestor provenance on every chunk, and per-tenant isolation through @oshun/data-residency. This is a coherent spec; it is not the orchestrated flow serving live answers in V1.

Per-source-type SourceAdapter set#

The typed SourceAdapter interface (parse, chunk, enrich, freshnessSignal, changeFingerprint, rightsResolver, attributionRenderer) and its launch adapter set — peer-reviewed PDF (DOI resolution, preprint-vs-published detection), news HTML (readability, retraction-feed subscription), RSS/Atom, YouTube/video transcript (shot-detection alignment), audio transcript (Whisper-class, per-segment confidence), institutional LMS export (SCORM/xAPI tied to the Metis BYOM/course-import flow), IIIF manuscript (image OCR, Sefaria/CDLI cross-links for Nisaba), structured API (NASA JPL ephemeris / NOAA SWPC / IERS / IMO, feeding Nyx), and tenant BYO bundle — are spec. Each is described with per-adapter regression fixtures and Isis-style release-gate review; that governance is the target, not a wired V1 capability.

Multi-store indexing#

The doc's index fan-out — pgvector primary, Qdrant for high-cardinality fan-out, Elasticsearch for lexical, Neo4j for the concept graph, plus per-source-type relational stores — is not in the live answer path. The live /v1/sophia/answer retrieves over the in-process Nisaba public-domain corpus via searchLibrary; no Qdrant/Elasticsearch/Neo4j store participates. Architecture diagrams that show a "pgvector · Qdrant" retriever fan-out describe the spec, not V1's running retrieval.

Orchestration, checkpoints, and the publication gate#

The job envelope (declared mode/source-set/persona/strictness, mandatory checkpoints, evidence-pack target), the mandatory human checkpoints per surface (Veritas story publication, Nisaba edition release, Metis course publication, Tara teacher-script release, Sophia grounded-report publication, large-blast media), the operator workbench (live job state, plan/DAG, fact-check status, contradiction queue, ready-to-publish gate), and the publication-gate criteria are all spec-only. As noted above, the gate's "contradiction backlog zero / unsupported-claim backlog zero" criteria depend on loops that do not run on the live path, so the gate is aspirational.

Downstream invalidation cascade#

The retraction/refresh cascade — a materially changed claim firing invalidation into Veritas re-grounding, Metis lesson-validity and tutor re-grounding, Nisaba passage-edition notices, Tara explainer re-checks, and Living Scenes kept-artifact re-render banners (see Living Scenes — Concept and Customer Promise) — is modeled by the real planSophiaSourceLifecycleInvalidation primitive but is not an end-to-end live cascade in V1. Backlog tasks live at ../TODOS.md (referenced as bare §N); cross-subsystem build ordering and deps§N references resolve to ../DEPENDENCIES.md.

Evaluation surface#

The doc's evaluation suites (evidence-pack assembly completeness, claim-to-source and passage-to-source integrity on a held-out gold set, grounded correctness per domain, citation integrity / no fabricated IDs, unsupported-claim eval, hallucination red-team, trace completeness, interruption recovery, and regression-blocks-release gates) are the target spec for Sophia's own outputs.

One adjacent, live evaluation surface ships in V1 and is worth naming because it belongs to the same family of release-gate discipline: @oshun/search-discovery's buildSearchReleaseGateSummary is mounted at POST /v1/search/offline-eval (apps/oshun/bff/src/search/offline-eval-route.ts), computing real per-slice NDCG@10 / MAP@10 / recall@100 / coverage / diversity / serendipity and a pass/fail release gate for candidate rankers and recommenders. See Search, Discovery, Recommendations, and Knowledge Graph.

Edge cases and rationale, at a glance#

• Empty query is not the same as no results. Empty query → abstained (Sophia declines); a real query that retrieves nothing → ungrounded. The UI can therefore distinguish "you asked nothing" from "we have no evidence."
• A retrieval error never produces fabricated prose. searchLibrary failures collapse to zero citations, which yields ungrounded/abstained — the system refuses rather than guesses.
• An LLM can only ever enrich grounded passages, never invent citations. Citation ids are filtered against real ids; unbacked claims are dropped; the whole layer fails soft to the extractive composer.
• Domain reputation alone never grounds a claim. The 0.5 floor sits above any single domain-tier contribution (max 0.95 × 0.40 = 0.38), forcing recency, citations, or peer review to participate.
• Evidence reads are role-scoped at the request boundary, with least-privilege defaulting and per-capability 403s — RBAC on evidence, not just on routes.

Related#

• Veritas — Grounded Stories and Claims — the primary consumer of Sophia grounding for narrative claims and retraction cascades.
• Nisaba — Scholarly Study — the public-domain corpus the live answer path actually retrieves over.
• Metis — Education and Tutoring — educational-claim grounding and the LMS/BYOM source-adapter spec.
• Customer Curation, Notebooks, Collections, and Sharing — where saved evidence and notebooks live for the customer.
• Search, Discovery, Recommendations, and Knowledge Graph — the live offline-eval release gate and the BFF search ranker.
• Assistant Experience — the largest runtime consumer of grounded answers.
• Architecture, Platform Foundations, and Security — BFF routing, domain scoping, and fail-closed posture.
• Hub: ../features.md.