Sophia Domain — Technical Specifications

Knowledge engine, RAG and semantic-search domain of the Oshun monorepo.

This document specifies the parts of Sophia that exist in code today. Sophia is implemented as 27 libraries under libs/sophia/ and 4 applications under apps/sophia/. There are no services/sophia/ directories.

Two Contract Layers — Understanding the Distinction#

Sophia carries two distinct, independently-authored contract layers, and the distinction matters throughout this specification. Where a section says "Zod" it refers to @sophia/schemas; where it says "Prisma" it refers to the database schema. The two layers were written separately, use different casing conventions, and define different enum value sets.

• @sophia/schemas — the Zod validation contract layer (libs/sophia/schemas/src/). Enum members are lowercase, kebab-cased strings ('chicago-notes', 'figure'). Branded ID types prevent ID mixing at compile time.
• @sophia/database — the Prisma persistence layer (libs/sophia/database/prisma/schema.prisma, 1361 lines, 21 models). Enum members are uppercase (CHICAGO, FIGURE).

These two layers do not share enum values. A Zod 'chicago-notes' string is not the same as a Prisma CHICAGO enum member; they are used in different contexts and must not be confused.

1. Persistence Layer — Prisma Schema#

File: libs/sophia/database/prisma/schema.prisma · Generator output: libs/sophia/database/src/generated/client · Datasource: PostgreSQL via env("SOPHIA_DATABASE_URL") · Preview features: fullTextSearchPostgres.

The schema defines 21 models. The first ten model the core research/knowledge data set; eight more (Source through GroundedReport) add canonical grounding contracts for citation-backed answers and evidence packs; IngestionJob and AuditLog close the set. The table below maps each model to its database table and primary purpose.

tsvector full-text columns (Unsupported("tsvector")) exist on documents, entities, and knowledge_packs, enabling PostgreSQL full-text search alongside vector similarity search without a separate search engine.

1.1 Document (documents)#

The Document model is the primary record for every piece of knowledge in Sophia. It stores full bibliographic metadata, processing state, quality scores, and foreign keys to all related records (chunks, citations, entities).

Primary key id (cuid(), VarChar(25)). Unique constraint [ownerId, title, author].

Timestamps: createdAt, updatedAt, publishedAt?, ingestedAt?, indexedAt?, deletedAt?.

Relations: chunks (DocumentChunk[]), citations (Citation[]), entityMentions (EntityMention[]), ingestionJobs (IngestionJob[]), packDocuments (KnowledgePackDocument[]).

1.2 DocumentChunk (document_chunks)#

A DocumentChunk is one segment of a document after the chunking stage has been applied. Each chunk stores its own text, position within the document, and — crucially — a binary vector embedding that enables semantic search.

Unique constraint [documentId, index].

Relations: document, entityMentions (EntityMention[]), citationSources (Citation[] via the ChunkAsCitationSource relation).

1.3 Citation (citations)#

A Citation record links a claim in some consumer context to the specific source document (and optionally chunk) that supports it. It records both the original text being cited and the formatted citation string, plus verification state.

Relations: document, sourceChunk (DocumentChunk?).

1.4 Entity (entities) — Knowledge-Graph Node#

An Entity is a node in the knowledge graph. It represents a named real-world thing (a person, place, concept, tradition, etc.) extracted from one or more documents. The canonicalId and mergedInto fields support entity resolution — when duplicate entities are merged, both fields record the merge so the history is auditable.

Unique constraint [ownerId, type, name].

Relations: outgoingRelations/incomingRelations (Relation[]), mentions (EntityMention[]), packEntities (KnowledgePackEntity[]).

1.5 EntityMention (entity_mentions)#

An EntityMention records a specific occurrence of an entity within a document. This is what allows Sophia to say not just that an entity exists, but exactly where in which documents it appears, with surrounding context for snippet display.

1.6 Relation (relations) — Knowledge-Graph Edge#

A Relation is a directed, typed edge between two Entity records. The unique constraint [sourceId, targetId, type] prevents duplicate edges of the same type between the same pair of entities.

1.7 Knowledge Packs#

Knowledge packs are curated, versioned collections of documents and entities. The KnowledgePack model (knowledge_packs, unique [ownerId, slug]) carries name, slug, description, summary, category (KnowledgePackCategory), status (KnowledgePackStatus, default DRAFT), visibility (KnowledgePackVisibility, default PRIVATE), scope arrays (traditions, regions, topics, tags), coverage JSON, version (default "1.0.0"), previousVersionId, author, contributors JSON, count fields (documentCount, entityCount, relationCount, citationCount, chunkCount, totalTokens), usage counters (viewCount, downloadCount, queryCount, starCount), quality fields (qualityTier, qualityScore, completeness), index fields (indexId, indexStatus, lastIndexedAt), presentation fields (previewImage, color, icon), metadata, ownership, and a searchVector.

Three join models link packs to their members and track community engagement:

• KnowledgePackDocument (knowledge_pack_documents, unique [packId, documentId]) — order, notes.
• KnowledgePackEntity (knowledge_pack_entities, unique [packId, entityId]) — importance (default 1.0), notes.
• KnowledgePackStar (knowledge_pack_stars, unique [packId, userId]).

1.8 Canonical Grounding Contracts#

Eight models form the grounding contract layer that was added on top of the core research data. These models represent the concept of a grounded answer — a consumer-facing response that is fully traceable back to cited sources through a verifiable chain of evidence. All use uuid() keys, a unique slug, a primaryDomain plus a domains array, status fields, and createdAt/updatedAt.

• Source (canonical_sources) — canonical source record: type, credibilityTier, credibilityScore?, authorityScore?, bibliographic fields (publisher, doi, isbn, issn, volume, issue, series, edition), provenance JSON, versions, formattedCitations, relatedConcepts.
• Concept (canonical_concepts) — name, category, complexity, definition, traditions, relatedTerms, relationships, links.
• Notebook (notebooks) — research notebook: kind, methodology?, preferredView, itemCount, sections, items, and arrays of grounded answer / report / evidence-pack / source-set / citation-trail IDs.
• EvidencePack (evidence_packs) — subject, claims, sources, excerpts, citations, retrievalTrace, rationale, qualityMetrics?, reviewSummary, links to a notebook / provenance bundle / grounded answer / grounded report, and a consumers array.
• GroundedAnswer (grounded_answers) — consumer-facing answer: answerBody, answerHtml?, query JSON, groundingSummary, groundingState, fallback, claims, sources, citations, disclosures, followUps, evidence/provenance/notebook/report references, reviewSummary, supersededById?, retractedReason?.
• CanonicalCitation (canonical_citations) — source-of-truth binding of a grounded subject to a cited source: subject JSON, source JSON, excerptKind, stance, isDirectQuote, snippet?, label, format, location, scores, verification, rationale?, and subject-reference IDs (passageId, claimId, evidencePackId, evidencePackExcerptId, groundedAnswerId, groundedReportId, notebookId, citationTrailId).
• CitationTrail (citation_trails) — ordered derivation trail: steps, gaps, coverageScore, confidenceScore, verificationStatus, and grounded answer / report / evidence-pack references.
• SourceSet (source_sets) — curated/assembled source collection: version, sourceSetHash, scope, items, selectionCriteria, statistics?, lastCheckedAt, freshnessWindow, rights, retractionState.
• GroundedReport (grounded_reports) — long-form report: sections, keyFindings, limitations, recommendations, claims, sources, citations, disclosures, groundingSummary, fallback, authors, revisions, reviewSummary, and arrays of contributing evidence-pack / source-set / citation-trail / grounded-answer IDs.

1.9 IngestionJob and AuditLog#

IngestionJob (ingestion_jobs) tracks the state of a document as it moves through the ingestion pipeline. Every stage transition is recorded, along with retry counts, error details, and final processing statistics.

IngestionJob fields: status (IngestionJobStatus, default PENDING), progress (default 0), currentStage, sourceType, sourceUri, sourceCredentials JSON, documentId?, configuration JSON (config, extractionConfig, enrichmentConfig, chunkingConfig), priority (default 0), workerId, queueName, retryCount (default 0), maxRetries (default 3), result fields (result, chunksCreated, tokensProcessed, entitiesExtracted, relationsExtracted), error fields (errorCode, errorMessage, errorStage, errorDetails), tags, metadata, ownership, and lifecycle timestamps (startedAt, completedAt, cancelledAt). Relation: document (Document?).

AuditLog (audit_log) provides an immutable mutation history for all significant state changes in the system: action, entityType, entityId, previousState?, newState?, changes?, userId?, ipAddress?, userAgent?, requestId?, metadata, createdAt.

1.10 Prisma Enums#

All Prisma enum values are uppercase strings. The table below gives each enum's name, member count, and complete value list. These are the values used in database records and in the TypeScript string-union types re-exported from @sophia/database.

@sophia/database re-exports DocumentType, DocumentStatus, EntityType, RelationType, KnowledgePackStatus, IngestionJobStatus, and VerificationStatus as TypeScript string-union types (uppercase, matching the Prisma enums above) for type-safe use without importing the generated client.

2. Validation Contract Layer — @sophia/schemas#

Directory: libs/sophia/schemas/src/. All schemas are Zod schemas exported from index.ts. Enum members here are lowercase strings and differ in value and count from the Prisma enums in §1. This layer is the contract for data validation and serialization; the Prisma layer is the contract for storage.

2.1 Branded ID Types (ids.ts)#

Sophia uses branded ID types to prevent accidentally passing a CitationId where a DocumentId is expected. The branding is enforced at compile time by TypeScript; at runtime they are plain strings. Eleven branded types are defined.

Each type is a string brand plus a Zod schema (z.string().min(1).max(100).brand<...>()), along with a constructor function:

DocumentId, CitationId, ClaimId, KnowledgePackId, EntityId, RelationId, ChunkId, ProjectId, UserId, SourceId, IndexId.

Constructors: documentId(), citationId(), claimId(), knowledgePackId(), entityId(), relationId(), chunkId(), projectId(), userId(), sourceId(), indexId(). Also UUIDSchema (z.string().uuid()) and IdSchema (generic non-empty string).

2.2 Research Document Schemas (research-document.ts)#

ResearchDocumentSchema is the main document object at the validation layer. It is composed from several component schemas, each of which validates a distinct facet of a document.

Component schemas: LanguageInfoSchema, AuthorSchema, PublisherSchema, DateInfoSchema, RightsInfoSchema, DocumentSectionSchema, DocumentChunkSchema, SourceAttributionSchema, DocumentMetadataSchema, ProcessingInfoSchema. CRUD schemas: CreateDocumentSchema, UpdateDocumentSchema, DocumentQuerySchema. Event schemas: DocumentIngestedEventSchema, DocumentIndexedEventSchema.

ResearchDocumentSchema top-level fields: id (DocumentId), projectId, ownerId, type (DocumentType), status (DocumentStatus), metadata (DocumentMetadata), content?, plainText?, sections?, processing?, sizeBytes, mimeType?, storagePath?, assetId?, version (default 1), parentId?, tags?, isPublic (default false), isArchived (default false), createdAt, updatedAt, indexedAt?, archivedAt?.

The Zod enums in this file have different members than their Prisma counterparts. The DocumentType enum here includes media types (audio, video, image) and format-oriented types (pdf, html, markdown) that the Prisma enum does not.

2.3 Citation Schemas (citation.ts)#

The citation schema file defines both the bibliographic source model and the citation metadata model. The key enum here is CitationStyleSchema, which defines the full set of scholarly citation styles that Sophia supports.

The CitationStyleSchema Zod enum — the canonical citation-style set — has 12 members, covering Chicago and Turabian in both their notes-bibliography and author-date variants:

chicago-notes, chicago-author, mla, apa, turabian-notes, turabian-author, harvard, sbl, oxford, ieee, vancouver, ama.

(@sophia/theory defines its own CitationStyle union with the same 12 styles, naming the Chicago/Turabian author-date variants chicago-author-date and turabian-author-date.)

The other Zod enums in citation.ts are listed below. Note that the Zod SourceType here (27 members) is completely different from the Prisma SourceType (6 members) — the Zod version classifies the type of publication, while the Prisma version classifies the technical origin of the file.

Object schemas in this file: SourceAuthorSchema, SourcePublisherSchema, ReliabilityScoringSchema, SourceSchema (a complete bibliographic source — type, title, authors/editors/translators, year, publisher, journal fields, book/chapter fields, scripture fields, digital identifiers, series fields, reliability), TextPositionSchema ({start, end} offsets), InlineCitationSchema, NoteSchema, BibliographyEntrySchema, BibliographyConfigSchema, BibliographySchema, VerificationIssueSchema, CitationSourceMatchSchema, VerificationResultSchema, DocumentCitationSchema. Event schemas: CitationCreatedEventSchema, CitationVerifiedEventSchema. Constant: DEFAULT_BIBLIOGRAPHY_CONFIG (style chicago-notes, sort by author, hanging indent and double spacing on).

2.4 Claim Schemas (claim.ts)#

The claim schema file defines types for classifying and tracking individual claims within AI-generated text. This is the foundation for Sophia's citation assignment and verification workflows — before a citation can be assigned, the claim that needs citing must be identified and classified.

Zod enums in this file:

Object schemas include TemporalMarkerSchema, HedgingLanguageSchema, ExtractedClaimSchema, CitationNeededFlagSchema, ClaimSourceMatchSchema, ClaimVerificationSchema, ConfidenceFactorsSchema, StatementConfidenceSchema, CalibrationConfigSchema, UncertaintyExpressionSchema, FallbackResponseSchema, ClaimAnalysisSchema, ClaimFlaggingResultSchema.

Claim Origin Labelling#

ClaimOriginLabelSchema classifies whether a claim was retrieved verbatim from a source, synthesized from multiple sources, or generated by the model alone. This distinction drives whether a claim is considered grounded. The schema carries origin, evidenceRole, supporting sourceIds / citationIds / retrievalIds / synthesizedFromClaimIds, a confidence, a rationale, and labelling timestamps.

A superRefine enforces three invariants:

• retrieved requires at least one source, citation, or retrieval ID.
• synthesized requires source, citation, retrieval, or upstream-claim refs.
• model-only may not carry any retrieval or synthesis references.

Helper functions createClaimOriginLabel() and inferClaimOriginLabel() construct labels; the inference helper picks retrieved / synthesized / model-only from the evidence present. Default origin confidences: retrieved 0.95, synthesized 0.75, model-only 0.2. ExtractedClaimSchema, ClaimVerificationSchema, ClaimAnalysisSchema, and ClaimExtractedEventSchema all embed an originLabel. Constant: DEFAULT_CALIBRATION_CONFIG.

2.5 Knowledge Pack Schemas (knowledge-pack.ts)#

The knowledge pack Zod enums differ from their Prisma counterparts in both member count and values. Notably, the Zod KnowledgePackCategory has 11 members (including fine-grained types like philosophy and historical-period) while the Prisma version has 8. The Zod KnowledgePackVisibility omits the TEAM level that the Prisma version has.

KnowledgePackSchema is the main pack object. Component schemas cover contributors, coverage, metadata, document/entity/claim/source references, content blocks, sections, statistics, usage stats, and index config/status. CRUD schemas: CreateKnowledgePackSchema, UpdateKnowledgePackSchema, KnowledgePackQuerySchema. Event schemas: PackCreatedEventSchema, PackPublishedEventSchema, PackIndexedEventSchema.

2.6 Entity Schemas (entity.ts) — Knowledge-Graph Nodes#

Note the casing difference for EntityType between the two layers: the Zod enum uses title-cased strings ('Tradition', 'Figure'), while the Prisma enum uses uppercase (TRADITION, FIGURE).

EntitySchema is the main entity object (id, projectId, type, properties, labels?, confidence, verificationStatus, mentions?, mentionCount, sourceDocuments?, createdBy?, verifiedBy?, isMerged, mergedFrom?, timestamps). Type-specific property schemas extend BaseEntityPropertiesSchema, one per entity type: TraditionPropertiesSchema, TextPropertiesSchema, ConceptPropertiesSchema, PracticePropertiesSchema, FigurePropertiesSchema, PlacePropertiesSchema, SchoolPropertiesSchema, DoctrinePropertiesSchema, EventPropertiesSchema. EntityPropertiesSchema is the union of all nine plus the base. EntityMentionSchema records an entity occurrence. CRUD schemas: CreateEntitySchema, UpdateEntitySchema, EntityQuerySchema. Event schemas: EntityExtractedEventSchema, EntityVerifiedEventSchema, EntityMergedEventSchema.

2.7 Relation Schemas (relation.ts) — Knowledge-Graph Edges#

The RelationTypeSchema Zod enum has 57 members — significantly more than the Prisma RelationType (27 members). The extra members are domain-specific types (e.g., ALIGNS_WITH, CONTRADICTS, SACRED_TO) that are used in the Zod validation layer but do not have a corresponding Prisma enum value. The 57 members are grouped by entity-domain category in the source:

• Figure: TAUGHT, STUDIED_UNDER, WROTE, FOUNDED, LIVED_IN, BORN_IN, DIED_IN, SUCCEEDED, CONTEMPORARIES.
• Concept: ALIGNS_WITH, CONTRADICTS, DERIVES_FROM, REFINES, EXTENDS, OPPOSES, SYNTHESIZES, DEPENDS_ON.
• Practice: BELONGS_TO, REQUIRES, PRECEDES, FOLLOWS, COMBINES_WITH, VARIANT_OF.
• Text: DISCUSSES, REFERENCES, CITES, ORIGINATES_FROM, TRANSLATED_FROM, COMMENTARY_ON, PART_OF, INSPIRED_BY, RESPONDS_TO.
• Place: ASSOCIATED_WITH, SACRED_TO, LOCATED_IN, PILGRIMAGE_TO.
• School/Tradition: SCHOOL_OF, INFLUENCED, INFLUENCED_BY, MERGED_WITH, SPLIT_FROM, EVOLVED_INTO.
• General: RELATED_TO, SIMILAR_TO, EXAMPLES, EXEMPLIFIES, INSTANCE_OF, CATEGORY_OF, SYNONYM_OF, ANTONYM_OF, OCCURRED_AT, OCCURRED_DURING, PARTICIPATED_IN, RESULTED_IN, CAUSED_BY.

Other Zod enums in relation.ts:

RelationSchema is the main edge object. Traversal schemas: TraversalOptionsSchema (max depth 1–10, default 3; max nodes 1–1000, default 100), GraphPathSchema, TraversalResultSchema, ShortestPathOptionsSchema (max depth 1–20, default 10), AllPathsOptionsSchema (extends shortest-path, adds maxPaths 1–100, default 10). Statistics schemas: RelationStatsByTypeSchema, GraphStatisticsSchema. Raw-query schemas: CypherQueryParamsSchema, CypherQueryResultSchema. Metadata: RelationTypeDefinitionSchema. Event schemas: RelationDiscoveredEventSchema, RelationVerifiedEventSchema.

2.8 Sophia Error Types (errors.ts)#

Sophia defines a structured error hierarchy that allows callers to distinguish error types programmatically — for example, to decide whether to retry a transient failure or surface a permanent error to the user.

SOPHIA_ERROR_CODES is a constant map (codes prefixed SOPHIA_) grouped into collection, vector, embedding, search, ingestion, document, knowledge-graph, and index ranges. Error classes (each extending an @oshun/errors base):

• Collection: CollectionNotFoundError, CollectionAlreadyExistsError, CollectionSchemaError.
• Vector: VectorDimensionError, VectorInvalidValuesError, VectorNotFoundError.
• Embedding: EmbeddingProviderError, EmbeddingRateLimitError, EmbeddingModelNotFoundError.
• Search: SearchQueryError, SearchTimeoutError, IndexNotReadyError.
• Ingestion: IngestionParseError, UnsupportedFormatError, DocumentTooLargeError.
• Document: DocumentNotFoundError, DocumentAlreadyExistsError.
• Knowledge graph: EntityNotFoundError, RelationNotFoundError, GraphQueryError.
• Index: IndexBuildError.

Type guards: isSophiaError() (checks the SOPHIA_ code prefix), isRetryableSophiaError() (rate-limit, provider, timeout, and index-not-ready errors are retryable, plus a transient-message heuristic).

3. Applications#

Four applications live under apps/sophia/. The two backend services (search-api, knowledge-graph) run a Node http server with no external web framework; both normalize an optional /v1/ or /api/v1/ path prefix, so callers can use either form.

3.1 Search API (apps/sophia/search-api)#

Entry point: SearchAPIService (factory createSearchAPIService). Default port 3000, host localhost. The service holds documents and knowledge packs in memory at runtime, bootstrapped from corpusPath / bootstrapDocuments at startup.

Config (SearchAPIConfig): port, host, defaultTopK (default 10), maxTopK (default 100), enableCitations (default true), enableReranking (default true), vectorStoreBackend ('memory' | 'qdrant' | 'pinecone'), corpusPath, bootstrapDocuments, embeddingDimensions. createRequestHandlers(service) wraps the service's search / rag / health operations for mounting on any HTTP framework.

HTTP routes:

Search request (SearchQuery) fields: text (or query), strategy (RetrievalStrategy), topK, threshold, filters (documentIds, sources, languages, dateRange, tags, metadata), options (includeMetadata, includeHighlights, includeContext, expandQuery, rerank, deduplicate, hybridWeights), projectId, userId. RAG request (RAGRequest) fields: query, topK, strategy, systemPrompt, maxTokens, includeCitations, citationOptions (minConfidence, maxCitations, includeGraph, mustCiteThreshold).

RetrievalStrategy is the union 'dense' | 'sparse' | 'hybrid' | 'rerank' | 'multi-query'.

The search-api package also exports its internal building blocks for use by consumers that need lower-level access: HybridSearchEngine, the rerankers (CrossEncoderReranker, FeatureReranker, ListwiseReranker, RerankingPipeline), the citation stack (CitationExtractor, CitationVerifier, CitationGraph, CitationService), and the retrieval stack (QueryExpander, MultiQueryRetriever, RetrievalService, RetrievalPipelineBuilder).

3.2 Knowledge Graph (apps/sophia/knowledge-graph)#

Entry point: KnowledgeGraphService (factory createKnowledgeGraphService). Default port 3001. Config (KnowledgeGraphConfig): port, host, maxQueryLimit (default 100), maxTraversalDepth (default 5), enableResolution (default true). Backed by an in-memory MemoryGraphStore; internal services are EntityService, RelationService, GraphService, EntityResolver (when resolution is enabled), and CuratorService.

This service exposes a full REST API. The routes are grouped below by functional area.

If enableResolution is false, /resolution/* returns 400 NOT_ENABLED.

This app defines its own NodeType and RelationshipType unions, separate from @sophia/schemas, reflecting the operational subset needed at runtime:

• NodeType (9): tradition, text, concept, practice, figure, place, event, organization, custom.
• RelationshipType (29): TAUGHT, WROTE, LIVED_IN, STUDIED_UNDER, FOUNDED, ALIGNS_WITH, CONTRADICTS, DERIVES_FROM, REFINES, EXTENDS, BELONGS_TO, REQUIRES, PRECEDES, FOLLOWS, DISCUSSES, REFERENCES, CITES, ORIGINATES_FROM, TRANSLATED_FROM, ASSOCIATED_WITH, SACRED_TO, LOCATED_IN, EXEMPLIFIES, INFLUENCED_BY, RELATED_TO, PART_OF, MEMBER_OF, PARTICIPATED_IN.

Curator Subsystem#

CuratorService manages a review queue for newly extracted or flagged entities. Human curators work through the queue, approving or rejecting entities before they become authoritative in the knowledge graph.

ReviewItem carries: entityId, entityType, entityName, status (pending | approved | rejected | needs_revision), priority (low | medium | high | critical), flaggedBy, flagReason, optional assignedTo, reviewNote, reviewedBy, qualityScore, and timestamps.

Operations available: flag, get review, submit decision, assign reviewer, get queue (with status / priority / entity-type / assignee filters; sorted critical-first), get resolved history, and getStats() (totals by status and priority, average quality score, recent activity).

Graph Traversal Limits (DEFAULT_LIMITS)#

GraphService enforces hard limits on traversal operations to prevent runaway queries from exhausting memory or taking too long. The defaults below are defined in graph/graph-service.ts.

Knowledge-Graph App Errors#

GraphError is the base error class; subclasses carry specific error codes that map to HTTP status codes at the router level.

• NodeNotFoundError (NODE_NOT_FOUND) → 404
• RelationshipNotFoundError (RELATIONSHIP_NOT_FOUND) → 404
• InvalidRelationshipError (INVALID_RELATIONSHIP) → 400
• TraversalLimitError (TRAVERSAL_LIMIT_EXCEEDED) → 400

Other GraphError codes map to 400; INVALID_STATE maps to 409.

3.3 Ingestion (apps/sophia/ingestion)#

IngestionPipeline orchestrates document ingestion. It runs as a background worker (CLI or service mode) with no fixed HTTP port. Options: maxConcurrentJobs (default 5), defaultChunkSize (default 1000), defaultChunkOverlap (default 200), plus connector / extractor / enrichment / storage injection and an optional Kalika knowledge-graph integration.

The functional pipeline stages are fetch → extract → enrich → chunk, with an optional kalika-knowledge-graph sync stage afterward for math/physics documents. On success the pipeline publishes sophia.document.ingested and per-entity sophia.entity.extracted events via @sophia/event-publisher.

Pipeline lifecycle events emitted on every job and stage transition: job:created, job:started, job:progress, job:completed, job:failed, stage:started, stage:completed, stage:failed.

Type definitions in apps/sophia/ingestion/src/types.ts:

• SourceType (6): file, url, api, database, stream, archive.
• DocumentFormat (11): pdf, epub, html, markdown, plain-text, tei-xml, docx, xlsx, pptx, image, unknown.
• EntityType (9, extraction): person, organization, location, date, event, concept, work, term, custom.
• JobStatus (9): pending, fetching, extracting, enriching, chunking, indexing, completed, failed, cancelled.
• ChunkingConfig.strategy (5 here): fixed, paragraph, sentence, semantic, recursive.

Ingestion error classes: IngestionError (base) and stage-specific subclasses: FetchError (FETCH_ERROR, stage fetch, retryable), ExtractionError (EXTRACTION_ERROR, stage extract, not retryable), OCRError (OCR_ERROR, stage ocr, retryable), EnrichmentError (ENRICHMENT_ERROR, stage enrich, retryable).

3.4 Workbench (apps/sophia/workbench)#

A React SPA (Vite, React Router, Tailwind). Routes and pages: / (DashboardPage), /curation (CurationPage), /review (ReviewPage), /annotations (AnnotationsPage), /documents (DocumentsPage), /search (SearchPage), /operator (OperatorWorkbenchPage).

API service modules follow a getXService() / resetXService() singleton pattern (the reset is for test isolation): AuthService, DashboardService, DocumentsService, AnnotationsService, SearchService, ReviewService, OperatorWorkbenchService, plus resetLocalRuntimeServices().

4. Events#

All Sophia domain events are defined in libs/contracts/src/events/sophia.ts and published through @sophia/event-publisher (SophiaEventPublisher), which wraps @oshun/event-bus. SophiaEventTypes enumerates 9 event types. The key architectural point is that event default routing is pre-configured: sophia.document.ingested routes to Hathor and Bellona; sophia.entity.extracted routes to Hathor and Lilith. Other events require the caller to pass explicit targets, otherwise they are not delivered.

Default routing targets: sophia.document.ingested → ['hathor', 'bellona']; sophia.entity.extracted → ['hathor', 'lilith']; other events use no default targets unless callers pass options.targets. Publishing failures are logged and swallowed so they never break the calling flow.

The publisher is configured with a Redis URL (SophiaEventPublisherConfig: redisUrl, enabled, keyPrefix). getSophiaEventPublisher() returns a singleton; SOPHIA_EVENTS_ENABLED=false disables publishing.

The contracts package additionally exposes a DocumentType Zod enum (13 values: text, pdf, html, markdown, json, xml, code, spreadsheet, presentation, image, video, audio, other) and an IngestionStatus enum (pending, processing, indexed, failed) used in event payloads. These are distinct from the DocumentType and DocumentStatus enums in @sophia/schemas.

@sophia/schemas also defines its own typed event schemas (DocumentIngestedEventSchema, CitationCreatedEventSchema, EntityExtractedEventSchema, etc.) for internal validation; these are distinct from the cross-domain @oshun/contracts event envelopes above.

5. Vector Store Layer — @sophia/vectordb#

@sophia/vectordb provides a unified vector-store abstraction over four backends, each implementing a common VectorStore interface. The library ensures that the choice of vector database is a deployment decision, not an application-code decision.

Note: the search-api service config exposes only memory, qdrant, and pinecone as runtime backend choices; Milvus is available at the library level.

5.1 Predefined Collection Schemas#

schemas/collections.ts defines the COLLECTIONS name map and exports 10 collection schemas (ALL_SCHEMAS). Each schema pre-configures the dimensionality, index type (HNSW or IVF), and distance metric appropriate for its use case.

Each schema also specifies indexed metadata fields and index-specific parameters (HNSW: M, efConstruction, ef; IVF: nlist, nprobe). Helper functions: getSchemaByName, isValidCollectionName, getCollectionsByDimension, getCollectionsByIndexType.

6. Indexing Layer — @sophia/indexing#

6.1 Chunking Strategies#

indexing/src/chunking/strategies.ts implements 8 chunker classes, all extending BaseChunker. The ChunkingStrategy union covers all 8 names and is what the ingestion pipeline uses to select the right chunker at runtime.

Chunker classes: FixedSizeChunker, SlidingWindowChunker, ParagraphChunker, SentenceChunker, RecursiveChunker, MarkdownChunker, CodeChunker, SemanticChunker.

ChunkingStrategy union: 'fixed' | 'sliding-window' | 'paragraph' | 'sentence' | 'semantic' | 'markdown' | 'code' | 'recursive'.

6.2 Embedding Providers#

indexing/src/embedding/providers.ts defines 3 provider classes extending BaseEmbeddingProvider: OpenAIEmbeddingProvider (provider: 'openai'), CohereEmbeddingProvider (provider: 'cohere'), LocalEmbeddingProvider (provider: 'local').

The indexing/embedding/ subtree also provides an embedding cache and service. Note that libs/sophia/embeddings/ is a separate library that provides a multimodal/audio/text embedding subsystem distinct from @sophia/indexing's document-chunking embedding providers.

7. Configuration and Environment Variables#

The table below lists all environment variables consumed by Sophia's applications and libraries. Services bind to localhost and default ports unless overridden.

The OpenAI and Cohere embedding providers read their API keys from the respective provider environment variables (e.g., OPENAI_API_KEY, COHERE_API_KEY) and throw an error at invocation time if a key is missing.

8. V2 Release-Candidate Ingestion Contract#

@sophia/ingestion is the source-of-truth ingestion pipeline for all production Sophia usage. The V2 release-candidate bridge described here is a separate package that wraps it for a specific use case.

@v2/sophia-release-candidate-ingestion (apps/v2/sophia-release-candidate-ingestion/) is a V2 release-candidate cook bridge over @sophia/ingestion for frame-data and patch-note sources. It exports the cook hook constant V2_SOPHIA_RELEASE_CANDIDATE_COOK_HOOK = 'SophiaReleaseCandidateIngestion' and runs before each release-candidate cook.

The bridge ingests a frame-data spreadsheet and a patch-notes corpus, emits Sophia parse/chunk/enrich/embed/index/quality-eval events for those documents, and writes an ingestion manifest as a cook artifact. The manifest is off rollback authority: it may feed citations, companion-app RAG, wiki pages, and human release review, but deterministic combat and rollback simulation must not depend on it.

9. Cross-Domain Integration#

Sophia integrates with other domains through a small, well-defined set of channels. It depends on shared infrastructure libraries for event publishing and logging; consumer domains subscribe to its events or call its APIs.

• Hathor consumes sophia.document.ingested and sophia.entity.extracted via the @hathor/sophia-integration library, which provides research grounding, lore validation, and a citation service.
• Bellona is a default routing target for sophia.document.ingested.
• Lilith is a default routing target for sophia.entity.extracted.

Sophia's shared library dependencies:

• @oshun/event-bus — Kafka-backed event publishing
• @oshun/contracts — event schemas and SophiaEventTypes definitions
• @oshun/logging — Pino-based structured logging
• @oshun/errors — error base classes used by @sophia/schemas

10. Acceptance Criteria#

A Sophia change is acceptance-complete when all of the following hold:

1. @sophia/database Prisma schema migrations apply cleanly and the generated client builds.
2. @sophia/schemas Zod schemas compile and exported branded ID types remain type-safe (no ID cross-assignment).
3. search-api serves the routes of §3.1 and knowledge-graph serves the routes of §3.2; both handle the /v1/ and /api/v1/ path prefixes.
4. The ingestion pipeline emits the §4 events on success and the pipeline stage events on every stage transition.
5. Event payloads validate against the @oshun/contracts Sophia event schemas.
6. Enum value sets in code and in this document match exactly — including the 12-member CitationStyleSchema, the 57-member Zod RelationTypeSchema, and the distinct uppercase Prisma enums.
7. The vector-store abstraction exposes all 10 predefined collection schemas and all four backends implement the common VectorStore interface.
8. Tests pass under Vitest for the touched libraries and applications.