# Sophia Domain — Technical Specifications > Knowledge engine, RAG and semantic-search domain of the Oshun monorepo. This document specifies the parts of Sophia that exist in code today. Sophia is implemented as **27 libraries** under `libs/sophia/` and **4 applications** under `apps/sophia/`. There are no `services/sophia/` directories. ## Two Contract Layers — Understanding the Distinction Sophia carries **two distinct, independently-authored contract layers**, and the distinction matters throughout this specification. Where a section says "Zod" it refers to `@sophia/schemas`; where it says "Prisma" it refers to the database schema. The two layers were written separately, use different casing conventions, and define different enum value sets. - **`@sophia/schemas`** — the Zod validation contract layer (`libs/sophia/schemas/src/`). Enum members are lowercase, kebab-cased strings (`'chicago-notes'`, `'figure'`). Branded ID types prevent ID mixing at compile time. - **`@sophia/database`** — the Prisma persistence layer (`libs/sophia/database/prisma/schema.prisma`, 1361 lines, 21 models). Enum members are uppercase (`CHICAGO`, `FIGURE`). These two layers do **not** share enum values. A Zod `'chicago-notes'` string is not the same as a Prisma `CHICAGO` enum member; they are used in different contexts and must not be confused. --- ## 1. Persistence Layer — Prisma Schema **File:** `libs/sophia/database/prisma/schema.prisma` · **Generator output:** `libs/sophia/database/src/generated/client` · **Datasource:** PostgreSQL via `env("SOPHIA_DATABASE_URL")` · **Preview features:** `fullTextSearchPostgres`. The schema defines **21 models**. The first ten model the core research/knowledge data set; eight more (`Source` through `GroundedReport`) add canonical grounding contracts for citation-backed answers and evidence packs; `IngestionJob` and `AuditLog` close the set. The table below maps each model to its database table and primary purpose. | Model | Table | Purpose | | ----------------------- | -------------------------- | ----------------------------------------------------- | | `Document` | `documents` | Research document with bibliographic metadata | | `DocumentChunk` | `document_chunks` | Indexed text chunk of a document | | `Citation` | `citations` | Citation record linking a claim to a source | | `Entity` | `entities` | Knowledge-graph node | | `EntityMention` | `entity_mentions` | Occurrence of an entity in a document | | `Relation` | `relations` | Knowledge-graph edge | | `KnowledgePack` | `knowledge_packs` | Curated collection of documents and entities | | `KnowledgePackDocument` | `knowledge_pack_documents` | Document membership in a pack | | `KnowledgePackEntity` | `knowledge_pack_entities` | Entity membership in a pack | | `KnowledgePackStar` | `knowledge_pack_stars` | Per-user star/favourite of a pack | | `Source` | `canonical_sources` | Canonical grounding source | | `Concept` | `canonical_concepts` | Canonical grounding concept | | `Notebook` | `notebooks` | Research notebook of grounded items | | `EvidencePack` | `evidence_packs` | Bundled claims, sources, excerpts and retrieval trace | | `GroundedAnswer` | `grounded_answers` | Consumer-facing grounded answer | | `CanonicalCitation` | `canonical_citations` | Source-of-truth subject-to-source citation binding | | `CitationTrail` | `citation_trails` | Ordered derivation trail of canonical citations | | `SourceSet` | `source_sets` | Curated/assembled collection of sources | | `GroundedReport` | `grounded_reports` | Long-form, multi-section grounded report | | `IngestionJob` | `ingestion_jobs` | Document ingestion pipeline job | | `AuditLog` | `audit_log` | Mutation audit trail | `tsvector` full-text columns (`Unsupported("tsvector")`) exist on `documents`, `entities`, and `knowledge_packs`, enabling PostgreSQL full-text search alongside vector similarity search without a separate search engine. ### 1.1 Document (`documents`) The `Document` model is the primary record for every piece of knowledge in Sophia. It stores full bibliographic metadata, processing state, quality scores, and foreign keys to all related records (chunks, citations, entities). Primary key `id` (`cuid()`, `VarChar(25)`). Unique constraint `[ownerId, title, author]`. | Field | Type | Default | Notes | | ------------------ | ---------------- | --------- | ---------------------------- | | `title` | `VarChar(500)` | — | Document title | | `originalTitle` | `VarChar(500)?` | — | Title in original language | | `type` | `DocumentType` | `OTHER` | Prisma enum (see §1.10) | | `genre` | `DocumentGenre?` | — | Prisma enum (see §1.10) | | `status` | `DocumentStatus` | `DRAFT` | Pipeline status (see §1.10) | | `author` | `VarChar(500)?` | — | Primary author | | `authors` | `Json` | `[]` | All authors `[{name, role}]` | | `editor` | `VarChar(500)?` | — | Editor | | `translator` | `VarChar(500)?` | — | Translator | | `publisher` | `VarChar(255)?` | — | Publisher | | `publicationPlace` | `VarChar(255)?` | — | Place of publication | | `publicationYear` | `Int?` | — | Publication year | | `edition` | `VarChar(100)?` | — | Edition | | `volume` | `VarChar(50)?` | — | Volume | | `issue` | `VarChar(50)?` | — | Issue | | `pages` | `VarChar(50)?` | — | Page range | | `isbn` / `issn` | `VarChar?` | — | Standard identifiers | | `doi` | `VarChar(100)?` | — | DOI | | `url` | `String?` | — | URL | | `externalIds` | `Json` | `{}` | Other external identifiers | | `language` | `VarChar(20)` | `"en"` | Primary language | | `languages` | `String[]` | `[]` | All languages | | `originRegion` | `VarChar(100)?` | — | Region of origin | | `abstract` | `Text?` | — | Abstract | | `summary` | `Text?` | — | Generated summary | | `keyPoints` | `Json` | `[]` | Extracted key points | | `keywords` | `String[]` | `[]` | Keywords | | `subjects` | `String[]` | `[]` | Subject tags | | `tags` | `String[]` | `[]` | Free-form tags | | `tradition` | `VarChar(200)?` | — | Primary tradition | | `traditions` | `String[]` | `[]` | All traditions | | `category` | `VarChar(100)?` | — | Category | | `compositionYear` | `Int?` | — | Composition year | | `compositionEra` | `VarChar(20)?` | — | Composition era | | `dateRange` | `Json?` | — | `{from, to}` | | `rightsStatus` | `RightsStatus` | `UNKNOWN` | Copyright status (see §1.10) | | `rightsHolder` | `VarChar(255)?` | — | Rights holder | | `license` | `VarChar(100)?` | — | License | | `sourceUri` | `String?` | — | Source file URI | | `sourceType` | `SourceType?` | — | Source type (see §1.10) | | `fileFormat` | `VarChar(50)?` | — | File format | | `fileSize` | `BigInt?` | — | File size | | `checksum` | `VarChar(128)?` | — | Content checksum | | `mimeType` | `VarChar(100)?` | — | MIME type | | `pageCount` | `Int?` | — | Page count | | `wordCount` | `Int?` | — | Word count | | `tokenCount` | `Int?` | — | Token count | | `chunkCount` | `Int` | `0` | Indexed chunk count | | `processingTimeMs` | `Int?` | — | Processing duration | | `processingError` | `Text?` | — | Processing error message | | `qualityScore` | `Float?` | — | Quality score | | `completeness` | `Float?` | — | Metadata completeness score | | `accuracy` | `Float?` | — | Accuracy score | | `metadata` | `Json` | `{}` | Free-form metadata | | `ownerId` | `VarChar(50)` | — | Owner user ID | | `organizationId` | `VarChar(50)?` | — | Organization | | `projectId` | `VarChar(50)?` | — | Project reference | | `searchVector` | `tsvector?` | — | Full-text search vector | Timestamps: `createdAt`, `updatedAt`, `publishedAt?`, `ingestedAt?`, `indexedAt?`, `deletedAt?`. Relations: `chunks` (`DocumentChunk[]`), `citations` (`Citation[]`), `entityMentions` (`EntityMention[]`), `ingestionJobs` (`IngestionJob[]`), `packDocuments` (`KnowledgePackDocument[]`). ### 1.2 DocumentChunk (`document_chunks`) A `DocumentChunk` is one segment of a document after the chunking stage has been applied. Each chunk stores its own text, position within the document, and — crucially — a binary vector embedding that enables semantic search. Unique constraint `[documentId, index]`. | Field | Type | Default | Notes | | --------------------------------- | --------------- | ----------- | ---------------------------------------------- | | `documentId` | `VarChar(25)` | — | FK to `Document` (cascade delete) | | `index` | `Int` | — | Chunk index within document | | `type` | `ChunkType` | `PARAGRAPH` | Prisma enum (see §1.10) | | `text` | `Text` | — | Chunk text | | `startOffset` / `endOffset` | `Int` | — | Character offsets in document | | `pageNumber` | `Int?` | — | Source page | | `sectionTitle` | `VarChar(255)?` | — | Section title | | `sectionIndex` / `paragraphIndex` | `Int?` | — | Structural indices | | `tokenCount` / `charCount` | `Int?` | — | Size metrics | | `embeddingModel` | `VarChar(100)?` | — | Embedding model used | | `embeddingDimension` | `Int?` | — | Embedding vector dimension | | `embeddingVector` | `Bytes?` | — | Binary embedding vector | | `embeddingGeneratedAt` | `DateTime?` | — | Embedding timestamp | | `overlapBefore` / `overlapAfter` | `Int` | `0` | Overlap characters | | `previousChunkId` / `nextChunkId` | `VarChar(25)?` | — | Sibling chunk links | | `context` | `Json?` | — | `{before, after, documentTitle, chapterTitle}` | | `language` | `VarChar(20)?` | — | Chunk language | | `metadata` | `Json` | `{}` | Free-form metadata | | `qualityScore` | `Float?` | — | Quality score | Relations: `document`, `entityMentions` (`EntityMention[]`), `citationSources` (`Citation[]` via the `ChunkAsCitationSource` relation). ### 1.3 Citation (`citations`) A `Citation` record links a claim in some consumer context to the specific source document (and optionally chunk) that supports it. It records both the original text being cited and the formatted citation string, plus verification state. | Field | Type | Default | Notes | | -------------------------------------------- | ---------------------------- | ------------ | ------------------------------ | | `documentId` | `VarChar(25)` | — | FK to `Document` (cascade) | | `chunkId` | `VarChar(25)?` | — | FK to `DocumentChunk` | | `citedText` | `Text` | — | The cited text | | `citationStyle` | `CitationStyle` | `CHICAGO` | Prisma enum (see §1.10) | | `formattedCitation` | `Text?` | — | Pre-formatted citation | | `startOffset` / `endOffset` / `pageNumber` | `Int?` | — | Position in source | | `sourceTitle` / `sourceAuthor` / `sourceUrl` | `VarChar?`/`String?` | — | Source info | | `sourceYear` | `Int?` | — | Source year | | `sourceIdentifier` | `VarChar(200)?` | — | Source identifier | | `confidence` | `Float` | `0` | Confidence score (0–1) | | `verificationStatus` | `VerificationStatus` | `UNVERIFIED` | Verification state (see §1.10) | | `verifiedAt` / `verifiedBy` | `DateTime?`/`VarChar(50)?` | — | Verification audit | | `claimText` / `claimType` | `Text?`/`VarChar(50)?` | — | Associated claim | | `isInline` | `Boolean` | `true` | Inline vs. bibliography | | `bibliographyOrder` | `Int?` | — | Bibliography ordering | | `noteType` / `noteMarker` / `noteContent` | `VarChar?`/`Text?` | — | Footnote/endnote fields | | `metadata` | `Json` | `{}` | Free-form metadata | | `ownerId` / `projectId` | `VarChar(50)`/`VarChar(50)?` | — | Ownership | Relations: `document`, `sourceChunk` (`DocumentChunk?`). ### 1.4 Entity (`entities`) — Knowledge-Graph Node An `Entity` is a node in the knowledge graph. It represents a named real-world thing (a person, place, concept, tradition, etc.) extracted from one or more documents. The `canonicalId` and `mergedInto` fields support entity resolution — when duplicate entities are merged, both fields record the merge so the history is auditable. Unique constraint `[ownerId, type, name]`. | Field | Type | Default | Notes | | --------------------------------------------------- | -------------------------- | ------------ | ------------------------------ | | `type` | `EntityType` | — | Prisma enum, 15 values (§1.10) | | `name` | `VarChar(500)` | — | Entity name | | `originalName` | `VarChar(500)?` | — | Name in original language | | `aliases` | `String[]` | `[]` | Alternative names | | `description` / `summary` | `Text?` | — | Description and summary | | `language` | `VarChar(20)` | `"en"` | Entity language | | `properties` | `Json` | `{}` | Type-specific properties | | `tradition` / `traditions` / `category` | `VarChar?`/`String[]` | — | Categorization | | `tags` / `labels` | `String[]` | `[]` | Tags and labels | | `confidence` | `EntityConfidence` | `PROBABLE` | Prisma enum (see §1.10) | | `verificationStatus` | `VerificationStatus` | `UNVERIFIED` | Verification state | | `verifiedAt` / `verifiedBy` | `DateTime?`/`VarChar(50)?` | — | Verification audit | | `wikipediaUrl` | `String?` | — | Wikipedia URL | | `externalIds` | `Json` | `{}` | External identifiers | | `mentionCount` / `relationCount` / `referenceCount` | `Int` | `0` | Statistics | | `canonicalId` / `mergedInto` | `VarChar(25)?` | — | Entity resolution / merge | | `mergedAt` | `DateTime?` | — | Merge timestamp | | `metadata` | `Json` | `{}` | Free-form metadata | | `ownerId` / `organizationId` / `projectId` | `VarChar` | — | Ownership | | `searchVector` | `tsvector?` | — | Full-text search vector | Relations: `outgoingRelations`/`incomingRelations` (`Relation[]`), `mentions` (`EntityMention[]`), `packEntities` (`KnowledgePackEntity[]`). ### 1.5 EntityMention (`entity_mentions`) An `EntityMention` records a specific occurrence of an entity within a document. This is what allows Sophia to say not just that an entity exists, but exactly where in which documents it appears, with surrounding context for snippet display. | Field | Type | Default | Notes | | -------------------------------- | --------------- | ------- | --------------------- | | `entityId` / `documentId` | `VarChar(25)` | — | FK (cascade delete) | | `chunkId` | `VarChar(25)?` | — | FK to `DocumentChunk` | | `startOffset` / `endOffset` | `Int` | — | Mention position | | `text` | `VarChar(500)` | — | Mention text | | `contextBefore` / `contextAfter` | `VarChar(200)?` | — | Surrounding context | | `confidence` | `Float` | `0` | Extraction confidence | | `verified` | `Boolean` | `false` | Verified flag | ### 1.6 Relation (`relations`) — Knowledge-Graph Edge A `Relation` is a directed, typed edge between two `Entity` records. The unique constraint `[sourceId, targetId, type]` prevents duplicate edges of the same type between the same pair of entities. | Field | Type | Default | Notes | | ------------------------------------------------ | ---------------------------- | ------------ | ------------------------------ | | `type` | `RelationType` | — | Prisma enum, 27 values (§1.10) | | `customType` | `VarChar(100)?` | — | Custom relation type label | | `sourceId` / `targetId` | `VarChar(25)` | — | FK to `Entity` (cascade) | | `weight` | `Float` | `1.0` | Edge weight | | `bidirectional` | `Boolean` | `false` | Bidirectional flag | | `properties` | `Json` | `{}` | Type-specific properties | | `confidence` | `Float` | `0` | Confidence score | | `verificationStatus` | `VerificationStatus` | `UNVERIFIED` | Verification state | | `verifiedAt` / `verifiedBy` | `DateTime?`/`VarChar(50)?` | — | Verification audit | | `evidenceDocumentId` / `evidenceChunkId` | `VarChar(25)?` | — | Evidence source | | `evidenceText` / `evidenceType` | `Text?`/`VarChar(50)?` | — | Evidence detail | | `context` / `temporalContext` / `spatialContext` | `Text?`/`VarChar?` | — | Context fields | | `labels` | `String[]` | `[]` | Labels | | `metadata` | `Json` | `{}` | Free-form metadata | | `ownerId` / `projectId` | `VarChar(50)`/`VarChar(50)?` | — | Ownership | ### 1.7 Knowledge Packs Knowledge packs are curated, versioned collections of documents and entities. The `KnowledgePack` model (`knowledge_packs`, unique `[ownerId, slug]`) carries `name`, `slug`, `description`, `summary`, `category` (`KnowledgePackCategory`), `status` (`KnowledgePackStatus`, default `DRAFT`), `visibility` (`KnowledgePackVisibility`, default `PRIVATE`), scope arrays (`traditions`, `regions`, `topics`, `tags`), `coverage` JSON, `version` (default `"1.0.0"`), `previousVersionId`, `author`, `contributors` JSON, count fields (`documentCount`, `entityCount`, `relationCount`, `citationCount`, `chunkCount`, `totalTokens`), usage counters (`viewCount`, `downloadCount`, `queryCount`, `starCount`), quality fields (`qualityTier`, `qualityScore`, `completeness`), index fields (`indexId`, `indexStatus`, `lastIndexedAt`), presentation fields (`previewImage`, `color`, `icon`), `metadata`, ownership, and a `searchVector`. Three join models link packs to their members and track community engagement: - `KnowledgePackDocument` (`knowledge_pack_documents`, unique `[packId, documentId]`) — `order`, `notes`. - `KnowledgePackEntity` (`knowledge_pack_entities`, unique `[packId, entityId]`) — `importance` (default `1.0`), `notes`. - `KnowledgePackStar` (`knowledge_pack_stars`, unique `[packId, userId]`). ### 1.8 Canonical Grounding Contracts Eight models form the grounding contract layer that was added on top of the core research data. These models represent the concept of a _grounded answer_ — a consumer-facing response that is fully traceable back to cited sources through a verifiable chain of evidence. All use `uuid()` keys, a unique `slug`, a `primaryDomain` plus a `domains` array, status fields, and `createdAt`/`updatedAt`. - **`Source`** (`canonical_sources`) — canonical source record: `type`, `credibilityTier`, `credibilityScore?`, `authorityScore?`, bibliographic fields (`publisher`, `doi`, `isbn`, `issn`, `volume`, `issue`, `series`, `edition`), `provenance` JSON, `versions`, `formattedCitations`, `relatedConcepts`. - **`Concept`** (`canonical_concepts`) — `name`, `category`, `complexity`, `definition`, `traditions`, `relatedTerms`, `relationships`, `links`. - **`Notebook`** (`notebooks`) — research notebook: `kind`, `methodology?`, `preferredView`, `itemCount`, `sections`, `items`, and arrays of grounded answer / report / evidence-pack / source-set / citation-trail IDs. - **`EvidencePack`** (`evidence_packs`) — `subject`, `claims`, `sources`, `excerpts`, `citations`, `retrievalTrace`, `rationale`, `qualityMetrics?`, `reviewSummary`, links to a notebook / provenance bundle / grounded answer / grounded report, and a `consumers` array. - **`GroundedAnswer`** (`grounded_answers`) — consumer-facing answer: `answerBody`, `answerHtml?`, `query` JSON, `groundingSummary`, `groundingState`, `fallback`, `claims`, `sources`, `citations`, `disclosures`, `followUps`, evidence/provenance/notebook/report references, `reviewSummary`, `supersededById?`, `retractedReason?`. - **`CanonicalCitation`** (`canonical_citations`) — source-of-truth binding of a grounded subject to a cited source: `subject` JSON, `source` JSON, `excerptKind`, `stance`, `isDirectQuote`, `snippet?`, `label`, `format`, `location`, `scores`, `verification`, `rationale?`, and subject-reference IDs (`passageId`, `claimId`, `evidencePackId`, `evidencePackExcerptId`, `groundedAnswerId`, `groundedReportId`, `notebookId`, `citationTrailId`). - **`CitationTrail`** (`citation_trails`) — ordered derivation trail: `steps`, `gaps`, `coverageScore`, `confidenceScore`, `verificationStatus`, and grounded answer / report / evidence-pack references. - **`SourceSet`** (`source_sets`) — curated/assembled source collection: `version`, `sourceSetHash`, `scope`, `items`, `selectionCriteria`, `statistics?`, `lastCheckedAt`, `freshnessWindow`, `rights`, `retractionState`. - **`GroundedReport`** (`grounded_reports`) — long-form report: `sections`, `keyFindings`, `limitations`, `recommendations`, `claims`, `sources`, `citations`, `disclosures`, `groundingSummary`, `fallback`, `authors`, `revisions`, `reviewSummary`, and arrays of contributing evidence-pack / source-set / citation-trail / grounded-answer IDs. ### 1.9 IngestionJob and AuditLog `IngestionJob` (`ingestion_jobs`) tracks the state of a document as it moves through the ingestion pipeline. Every stage transition is recorded, along with retry counts, error details, and final processing statistics. `IngestionJob` fields: `status` (`IngestionJobStatus`, default `PENDING`), `progress` (default `0`), `currentStage`, `sourceType`, `sourceUri`, `sourceCredentials` JSON, `documentId?`, configuration JSON (`config`, `extractionConfig`, `enrichmentConfig`, `chunkingConfig`), `priority` (default `0`), `workerId`, `queueName`, `retryCount` (default `0`), `maxRetries` (default `3`), result fields (`result`, `chunksCreated`, `tokensProcessed`, `entitiesExtracted`, `relationsExtracted`), error fields (`errorCode`, `errorMessage`, `errorStage`, `errorDetails`), `tags`, `metadata`, ownership, and lifecycle timestamps (`startedAt`, `completedAt`, `cancelledAt`). Relation: `document` (`Document?`). `AuditLog` (`audit_log`) provides an immutable mutation history for all significant state changes in the system: `action`, `entityType`, `entityId`, `previousState?`, `newState?`, `changes?`, `userId?`, `ipAddress?`, `userAgent?`, `requestId?`, `metadata`, `createdAt`. ### 1.10 Prisma Enums All Prisma enum values are uppercase strings. The table below gives each enum's name, member count, and complete value list. These are the values used in database records and in the TypeScript string-union types re-exported from `@sophia/database`. | Enum | Members | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DocumentType` (14) | `BOOK`, `ARTICLE`, `THESIS`, `MANUSCRIPT`, `SCRIPTURE`, `COMMENTARY`, `ESSAY`, `LETTER`, `SPEECH`, `TRANSLATION`, `ANTHOLOGY`, `ENCYCLOPEDIA`, `DICTIONARY`, `OTHER` | | `DocumentStatus` (10) | `DRAFT`, `INGESTING`, `EXTRACTING`, `ENRICHING`, `CHUNKING`, `INDEXING`, `PUBLISHED`, `ARCHIVED`, `FAILED`, `DELETED` | | `DocumentGenre` (14) | `RELIGIOUS`, `PHILOSOPHICAL`, `HISTORICAL`, `SCHOLARLY`, `PRACTICAL`, `DEVOTIONAL`, `COMMENTARY`, `NARRATIVE`, `POETRY`, `LITURGICAL`, `LEGAL`, `BIOGRAPHICAL`, `INSTRUCTIONAL`, `OTHER` | | `RightsStatus` (4) | `PUBLIC_DOMAIN`, `CREATIVE_COMMONS`, `RESTRICTED`, `UNKNOWN` | | `ChunkType` (6) | `PARAGRAPH`, `SENTENCE`, `SECTION`, `PAGE`, `SEMANTIC`, `CUSTOM` | | `CitationStyle` (6) | `CHICAGO`, `MLA`, `APA`, `HARVARD`, `TURABIAN`, `CUSTOM` | | `VerificationStatus` (5) | `VERIFIED`, `REVIEWED`, `UNVERIFIED`, `DISPUTED`, `PENDING` | | `EntityType` (15) | `TRADITION`, `TEXT`, `CONCEPT`, `PRACTICE`, `FIGURE`, `PLACE`, `SCHOOL`, `DOCTRINE`, `EVENT`, `TERM`, `SYMBOL`, `RITUAL`, `ARTIFACT`, `ORGANIZATION`, `CUSTOM` | | `EntityConfidence` (5) | `CERTAIN`, `PROBABLE`, `POSSIBLE`, `UNCERTAIN`, `INFERRED` | | `RelationType` (27) | `TAUGHT`, `STUDIED_WITH`, `INFLUENCED`, `WROTE`, `EDITED`, `TRANSLATED`, `FOUNDED`, `MEMBER_OF`, `BELONGS_TO`, `LIVED_IN`, `BORN_IN`, `DIED_IN`, `LOCATED_IN`, `PRECEDED`, `SUCCEEDED`, `CONTEMPORARY_WITH`, `REFERENCES`, `QUOTES`, `CITES`, `RELATED_TO`, `SIMILAR_TO`, `OPPOSITE_OF`, `DERIVED_FROM`, `PART_OF`, `PRACTICES`, `PRESCRIBES`, `CUSTOM` | | `RelationDirection` (3) | `OUTGOING`, `INCOMING`, `BOTH` | | `KnowledgePackStatus` (5) | `DRAFT`, `REVIEW`, `PUBLISHED`, `ARCHIVED`, `DEPRECATED` | | `KnowledgePackVisibility` (4) | `PRIVATE`, `TEAM`, `ORGANIZATION`, `PUBLIC` | | `KnowledgePackCategory` (8) | `TRADITION`, `PERIOD`, `TOPIC`, `FIGURE`, `TEXT`, `REGION`, `PRACTICE`, `CUSTOM` | | `IngestionJobStatus` (10) | `PENDING`, `QUEUED`, `FETCHING`, `EXTRACTING`, `ENRICHING`, `CHUNKING`, `INDEXING`, `COMPLETED`, `FAILED`, `CANCELLED` | | `SourceType` (6) | `FILE`, `URL`, `API`, `DATABASE`, `STREAM`, `ARCHIVE` | `@sophia/database` re-exports `DocumentType`, `DocumentStatus`, `EntityType`, `RelationType`, `KnowledgePackStatus`, `IngestionJobStatus`, and `VerificationStatus` as TypeScript string-union types (uppercase, matching the Prisma enums above) for type-safe use without importing the generated client. --- ## 2. Validation Contract Layer — `@sophia/schemas` **Directory:** `libs/sophia/schemas/src/`. All schemas are Zod schemas exported from `index.ts`. Enum members here are lowercase strings and **differ in value and count** from the Prisma enums in §1. This layer is the contract for data _validation and serialization_; the Prisma layer is the contract for _storage_. ### 2.1 Branded ID Types (`ids.ts`) Sophia uses branded ID types to prevent accidentally passing a `CitationId` where a `DocumentId` is expected. The branding is enforced at compile time by TypeScript; at runtime they are plain strings. Eleven branded types are defined. Each type is a `string` brand plus a Zod schema (`z.string().min(1).max(100).brand<...>()`), along with a constructor function: `DocumentId`, `CitationId`, `ClaimId`, `KnowledgePackId`, `EntityId`, `RelationId`, `ChunkId`, `ProjectId`, `UserId`, `SourceId`, `IndexId`. Constructors: `documentId()`, `citationId()`, `claimId()`, `knowledgePackId()`, `entityId()`, `relationId()`, `chunkId()`, `projectId()`, `userId()`, `sourceId()`, `indexId()`. Also `UUIDSchema` (`z.string().uuid()`) and `IdSchema` (generic non-empty string). ### 2.2 Research Document Schemas (`research-document.ts`) `ResearchDocumentSchema` is the main document object at the validation layer. It is composed from several component schemas, each of which validates a distinct facet of a document. Component schemas: `LanguageInfoSchema`, `AuthorSchema`, `PublisherSchema`, `DateInfoSchema`, `RightsInfoSchema`, `DocumentSectionSchema`, `DocumentChunkSchema`, `SourceAttributionSchema`, `DocumentMetadataSchema`, `ProcessingInfoSchema`. CRUD schemas: `CreateDocumentSchema`, `UpdateDocumentSchema`, `DocumentQuerySchema`. Event schemas: `DocumentIngestedEventSchema`, `DocumentIndexedEventSchema`. `ResearchDocumentSchema` top-level fields: `id` (`DocumentId`), `projectId`, `ownerId`, `type` (`DocumentType`), `status` (`DocumentStatus`), `metadata` (`DocumentMetadata`), `content?`, `plainText?`, `sections?`, `processing?`, `sizeBytes`, `mimeType?`, `storagePath?`, `assetId?`, `version` (default `1`), `parentId?`, `tags?`, `isPublic` (default `false`), `isArchived` (default `false`), `createdAt`, `updatedAt`, `indexedAt?`, `archivedAt?`. The Zod enums in this file have different members than their Prisma counterparts. The `DocumentType` enum here includes media types (audio, video, image) and format-oriented types (pdf, html, markdown) that the Prisma enum does not. | Zod enum (`@sophia/schemas`) | Members | | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DocumentType` (20) | `text`, `pdf`, `html`, `markdown`, `json`, `xml`, `tei-xml`, `epub`, `docx`, `rtf`, `code`, `spreadsheet`, `presentation`, `image`, `audio`, `video`, `scripture`, `ancient-text`, `commentary`, `other` | | `DocumentStatus` (9) | `pending`, `ingesting`, `processing`, `chunking`, `embedding`, `indexing`, `indexed`, `failed`, `archived` | | `DocumentGenre` (14) | `academic`, `religious`, `philosophical`, `scientific`, `historical`, `literary`, `legal`, `technical`, `reference`, `commentary`, `translation`, `primary-source`, `secondary-source`, `other` | | `AuthorRole` (8) | `author`, `editor`, `translator`, `compiler`, `contributor`, `commentator`, `annotator`, `scribe` | | `RightsStatus` (7) | `public-domain`, `copyrighted`, `creative-commons`, `open-access`, `restricted`, `fair-use`, `unknown` | ### 2.3 Citation Schemas (`citation.ts`) The citation schema file defines both the bibliographic source model and the citation metadata model. The key enum here is `CitationStyleSchema`, which defines the full set of scholarly citation styles that Sophia supports. The `CitationStyleSchema` Zod enum — the canonical citation-style set — has **12 members**, covering Chicago and Turabian in both their notes-bibliography and author-date variants: `chicago-notes`, `chicago-author`, `mla`, `apa`, `turabian-notes`, `turabian-author`, `harvard`, `sbl`, `oxford`, `ieee`, `vancouver`, `ama`. (`@sophia/theory` defines its own `CitationStyle` union with the same 12 styles, naming the Chicago/Turabian author-date variants `chicago-author-date` and `turabian-author-date`.) The other Zod enums in `citation.ts` are listed below. Note that the Zod `SourceType` here (27 members) is completely different from the Prisma `SourceType` (6 members) — the Zod version classifies the _type of publication_, while the Prisma version classifies the _technical origin of the file_. | Zod enum | Members | | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `SourceType` (27) | `book`, `journal-article`, `chapter`, `edited-volume`, `dissertation`, `thesis`, `conference-paper`, `website`, `webpage`, `manuscript`, `scripture`, `ancient-text`, `commentary`, `encyclopedia`, `dictionary`, `review`, `letter`, `interview`, `lecture`, `podcast`, `video`, `film`, `report`, `legal-document`, `patent`, `unpublished`, `other` | | `SourceAuthorRole` (9) | `author`, `editor`, `translator`, `compiler`, `contributor`, `director`, `performer`, `interviewer`, `interviewee` | | `NoteType` (2) | `footnote`, `endnote` | | `VerificationStatus` (8) | `verified`, `partially-verified`, `unverified`, `contradicted`, `misattributed`, `misquoted`, `context-missing`, `pending` | | `VerificationIssueType` (10) | `misattribution`, `misquote`, `overstatement`, `understatement`, `missing-context`, `outdated`, `translation-error`, `page-error`, `source-not-found`, `access-denied` | Object schemas in this file: `SourceAuthorSchema`, `SourcePublisherSchema`, `ReliabilityScoringSchema`, `SourceSchema` (a complete bibliographic source — type, title, authors/editors/translators, year, publisher, journal fields, book/chapter fields, scripture fields, digital identifiers, series fields, reliability), `TextPositionSchema` (`{start, end}` offsets), `InlineCitationSchema`, `NoteSchema`, `BibliographyEntrySchema`, `BibliographyConfigSchema`, `BibliographySchema`, `VerificationIssueSchema`, `CitationSourceMatchSchema`, `VerificationResultSchema`, `DocumentCitationSchema`. Event schemas: `CitationCreatedEventSchema`, `CitationVerifiedEventSchema`. Constant: `DEFAULT_BIBLIOGRAPHY_CONFIG` (style `chicago-notes`, sort by `author`, hanging indent and double spacing on). ### 2.4 Claim Schemas (`claim.ts`) The claim schema file defines types for classifying and tracking individual claims within AI-generated text. This is the foundation for Sophia's citation assignment and verification workflows — before a citation can be assigned, the claim that needs citing must be identified and classified. Zod enums in this file: | Zod enum | Members | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ClaimType` (12) | `factual`, `historical`, `interpretive`, `definitional`, `attribution`, `statistical`, `comparative`, `causal`, `opinion`, `uncertain`, `quote`, `paraphrase` | | `CitationNeededReason` (12) | `factual-claim`, `statistical-claim`, `direct-quote`, `attributed-claim`, `controversial`, `historical-claim`, `scientific-claim`, `interpretation`, `translation`, `definition`, `comparison`, `custom` | | `CitationSeverity` (3) | `required`, `recommended`, `optional` | | `ClaimOrigin` (3) | `retrieved`, `synthesized`, `model-only` | | `ClaimOriginEvidenceRole` (4) | `direct-source`, `combined-sources`, `synthesis-input`, `none` | | `ConfidenceLevel` (6) | `very-high`, `high`, `moderate`, `low`, `very-low`, `unknown` | | `KnowledgeBoundary` (8) | `no-information`, `insufficient-data`, `beyond-expertise`, `disputed`, `temporal-limit`, `context-dependent`, `speculation`, `personal-question` | Object schemas include `TemporalMarkerSchema`, `HedgingLanguageSchema`, `ExtractedClaimSchema`, `CitationNeededFlagSchema`, `ClaimSourceMatchSchema`, `ClaimVerificationSchema`, `ConfidenceFactorsSchema`, `StatementConfidenceSchema`, `CalibrationConfigSchema`, `UncertaintyExpressionSchema`, `FallbackResponseSchema`, `ClaimAnalysisSchema`, `ClaimFlaggingResultSchema`. #### Claim Origin Labelling `ClaimOriginLabelSchema` classifies whether a claim was retrieved verbatim from a source, synthesized from multiple sources, or generated by the model alone. This distinction drives whether a claim is considered grounded. The schema carries `origin`, `evidenceRole`, supporting `sourceIds` / `citationIds` / `retrievalIds` / `synthesizedFromClaimIds`, a `confidence`, a `rationale`, and labelling timestamps. A `superRefine` enforces three invariants: - `retrieved` requires at least one source, citation, or retrieval ID. - `synthesized` requires source, citation, retrieval, or upstream-claim refs. - `model-only` may **not** carry any retrieval or synthesis references. Helper functions `createClaimOriginLabel()` and `inferClaimOriginLabel()` construct labels; the inference helper picks `retrieved` / `synthesized` / `model-only` from the evidence present. Default origin confidences: `retrieved` `0.95`, `synthesized` `0.75`, `model-only` `0.2`. `ExtractedClaimSchema`, `ClaimVerificationSchema`, `ClaimAnalysisSchema`, and `ClaimExtractedEventSchema` all embed an `originLabel`. Constant: `DEFAULT_CALIBRATION_CONFIG`. ### 2.5 Knowledge Pack Schemas (`knowledge-pack.ts`) The knowledge pack Zod enums differ from their Prisma counterparts in both member count and values. Notably, the Zod `KnowledgePackCategory` has 11 members (including fine-grained types like `philosophy` and `historical-period`) while the Prisma version has 8. The Zod `KnowledgePackVisibility` omits the `TEAM` level that the Prisma version has. | Zod enum | Members | | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `KnowledgePackCategory` (11) | `tradition`, `philosophy`, `historical-period`, `geographic-region`, `topic`, `discipline`, `practice`, `text-collection`, `figure-study`, `comparative`, `custom` | | `KnowledgePackStatus` (5) | `draft`, `review`, `published`, `archived`, `deprecated` | | `KnowledgePackVisibility` (3) | `private`, `organization`, `public` | | `ContentQualityTier` (5) | `verified`, `reviewed`, `curated`, `contributed`, `automated` | `KnowledgePackSchema` is the main pack object. Component schemas cover contributors, coverage, metadata, document/entity/claim/source references, content blocks, sections, statistics, usage stats, and index config/status. CRUD schemas: `CreateKnowledgePackSchema`, `UpdateKnowledgePackSchema`, `KnowledgePackQuerySchema`. Event schemas: `PackCreatedEventSchema`, `PackPublishedEventSchema`, `PackIndexedEventSchema`. ### 2.6 Entity Schemas (`entity.ts`) — Knowledge-Graph Nodes Note the casing difference for `EntityType` between the two layers: the Zod enum uses title-cased strings (`'Tradition'`, `'Figure'`), while the Prisma enum uses uppercase (`TRADITION`, `FIGURE`). | Zod enum | Members | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EntityType` (15) | `Tradition`, `Text`, `Concept`, `Practice`, `Figure`, `Place`, `School`, `Doctrine`, `Event`, `Term`, `Symbol`, `Ritual`, `Artifact`, `Organization`, `Custom` | | `EntityConfidence` (5) | `certain`, `probable`, `possible`, `uncertain`, `inferred` | | `EntityVerificationStatus` (5) | `verified`, `reviewed`, `unverified`, `disputed`, `deprecated` | `EntitySchema` is the main entity object (`id`, `projectId`, `type`, `properties`, `labels?`, `confidence`, `verificationStatus`, `mentions?`, `mentionCount`, `sourceDocuments?`, `createdBy?`, `verifiedBy?`, `isMerged`, `mergedFrom?`, timestamps). Type-specific property schemas extend `BaseEntityPropertiesSchema`, one per entity type: `TraditionPropertiesSchema`, `TextPropertiesSchema`, `ConceptPropertiesSchema`, `PracticePropertiesSchema`, `FigurePropertiesSchema`, `PlacePropertiesSchema`, `SchoolPropertiesSchema`, `DoctrinePropertiesSchema`, `EventPropertiesSchema`. `EntityPropertiesSchema` is the union of all nine plus the base. `EntityMentionSchema` records an entity occurrence. CRUD schemas: `CreateEntitySchema`, `UpdateEntitySchema`, `EntityQuerySchema`. Event schemas: `EntityExtractedEventSchema`, `EntityVerifiedEventSchema`, `EntityMergedEventSchema`. ### 2.7 Relation Schemas (`relation.ts`) — Knowledge-Graph Edges The `RelationTypeSchema` Zod enum has **57 members** — significantly more than the Prisma `RelationType` (27 members). The extra members are domain-specific types (e.g., `ALIGNS_WITH`, `CONTRADICTS`, `SACRED_TO`) that are used in the Zod validation layer but do not have a corresponding Prisma enum value. The 57 members are grouped by entity-domain category in the source: - Figure: `TAUGHT`, `STUDIED_UNDER`, `WROTE`, `FOUNDED`, `LIVED_IN`, `BORN_IN`, `DIED_IN`, `SUCCEEDED`, `CONTEMPORARIES`. - Concept: `ALIGNS_WITH`, `CONTRADICTS`, `DERIVES_FROM`, `REFINES`, `EXTENDS`, `OPPOSES`, `SYNTHESIZES`, `DEPENDS_ON`. - Practice: `BELONGS_TO`, `REQUIRES`, `PRECEDES`, `FOLLOWS`, `COMBINES_WITH`, `VARIANT_OF`. - Text: `DISCUSSES`, `REFERENCES`, `CITES`, `ORIGINATES_FROM`, `TRANSLATED_FROM`, `COMMENTARY_ON`, `PART_OF`, `INSPIRED_BY`, `RESPONDS_TO`. - Place: `ASSOCIATED_WITH`, `SACRED_TO`, `LOCATED_IN`, `PILGRIMAGE_TO`. - School/Tradition: `SCHOOL_OF`, `INFLUENCED`, `INFLUENCED_BY`, `MERGED_WITH`, `SPLIT_FROM`, `EVOLVED_INTO`. - General: `RELATED_TO`, `SIMILAR_TO`, `EXAMPLES`, `EXEMPLIFIES`, `INSTANCE_OF`, `CATEGORY_OF`, `SYNONYM_OF`, `ANTONYM_OF`, `OCCURRED_AT`, `OCCURRED_DURING`, `PARTICIPATED_IN`, `RESULTED_IN`, `CAUSED_BY`. Other Zod enums in `relation.ts`: | Zod enum | Members | | -------------------------- | ---------------------------------------------------------------------- | | `RelationDirection` (3) | `outgoing`, `incoming`, `both` | | `RelationConfidence` (6) | `certain`, `probable`, `possible`, `uncertain`, `inferred`, `disputed` | | `RelationEvidenceType` (6) | `explicit`, `implicit`, `scholarly`, `traditional`, `inferred`, `user` | `RelationSchema` is the main edge object. Traversal schemas: `TraversalOptionsSchema` (max depth 1–10, default 3; max nodes 1–1000, default 100), `GraphPathSchema`, `TraversalResultSchema`, `ShortestPathOptionsSchema` (max depth 1–20, default 10), `AllPathsOptionsSchema` (extends shortest-path, adds `maxPaths` 1–100, default 10). Statistics schemas: `RelationStatsByTypeSchema`, `GraphStatisticsSchema`. Raw-query schemas: `CypherQueryParamsSchema`, `CypherQueryResultSchema`. Metadata: `RelationTypeDefinitionSchema`. Event schemas: `RelationDiscoveredEventSchema`, `RelationVerifiedEventSchema`. ### 2.8 Sophia Error Types (`errors.ts`) Sophia defines a structured error hierarchy that allows callers to distinguish error types programmatically — for example, to decide whether to retry a transient failure or surface a permanent error to the user. `SOPHIA_ERROR_CODES` is a constant map (codes prefixed `SOPHIA_`) grouped into collection, vector, embedding, search, ingestion, document, knowledge-graph, and index ranges. Error classes (each extending an `@oshun/errors` base): - Collection: `CollectionNotFoundError`, `CollectionAlreadyExistsError`, `CollectionSchemaError`. - Vector: `VectorDimensionError`, `VectorInvalidValuesError`, `VectorNotFoundError`. - Embedding: `EmbeddingProviderError`, `EmbeddingRateLimitError`, `EmbeddingModelNotFoundError`. - Search: `SearchQueryError`, `SearchTimeoutError`, `IndexNotReadyError`. - Ingestion: `IngestionParseError`, `UnsupportedFormatError`, `DocumentTooLargeError`. - Document: `DocumentNotFoundError`, `DocumentAlreadyExistsError`. - Knowledge graph: `EntityNotFoundError`, `RelationNotFoundError`, `GraphQueryError`. - Index: `IndexBuildError`. Type guards: `isSophiaError()` (checks the `SOPHIA_` code prefix), `isRetryableSophiaError()` (rate-limit, provider, timeout, and index-not-ready errors are retryable, plus a transient-message heuristic). --- ## 3. Applications Four applications live under `apps/sophia/`. The two backend services (`search-api`, `knowledge-graph`) run a Node `http` server with no external web framework; both normalize an optional `/v1/` or `/api/v1/` path prefix, so callers can use either form. ### 3.1 Search API (`apps/sophia/search-api`) Entry point: `SearchAPIService` (factory `createSearchAPIService`). Default port **3000**, host `localhost`. The service holds documents and knowledge packs in memory at runtime, bootstrapped from `corpusPath` / `bootstrapDocuments` at startup. Config (`SearchAPIConfig`): `port`, `host`, `defaultTopK` (default 10), `maxTopK` (default 100), `enableCitations` (default true), `enableReranking` (default true), `vectorStoreBackend` (`'memory' | 'qdrant' | 'pinecone'`), `corpusPath`, `bootstrapDocuments`, `embeddingDimensions`. `createRequestHandlers(service)` wraps the service's `search` / `rag` / `health` operations for mounting on any HTTP framework. HTTP routes: | Method | Path | Description | | ------ | ----------------------- | ---------------------------------------------------------- | | GET | `/health` | Service health (`status`, `uptime`) | | POST | `/search` | Search with the requested retrieval strategy | | POST | `/search/hybrid` | Search forced to the `hybrid` strategy | | POST | `/rag`, `/ask` | RAG: retrieve, build grounded answer, cite | | GET | `/documents` | List indexed documents (limit/offset/format/tag/tradition) | | GET | `/documents/:id` | Get one document summary | | GET | `/citations` | List stored citations | | POST | `/citations` | Create a citation | | GET | `/citations/:id` | Get one citation | | POST | `/citations/:id/verify` | Verify a citation against its source | | POST | `/citations/sources` | Find candidate citation sources for a claim | | GET | `/knowledge-packs` | List knowledge packs (tag/search filters) | | POST | `/knowledge-packs` | Create a knowledge pack | | GET | `/knowledge-packs/:id` | Get one knowledge pack | | PATCH | `/knowledge-packs/:id` | Update a knowledge pack | | DELETE | `/knowledge-packs/:id` | Delete a knowledge pack | Search request (`SearchQuery`) fields: `text` (or `query`), `strategy` (`RetrievalStrategy`), `topK`, `threshold`, `filters` (`documentIds`, `sources`, `languages`, `dateRange`, `tags`, `metadata`), `options` (`includeMetadata`, `includeHighlights`, `includeContext`, `expandQuery`, `rerank`, `deduplicate`, `hybridWeights`), `projectId`, `userId`. RAG request (`RAGRequest`) fields: `query`, `topK`, `strategy`, `systemPrompt`, `maxTokens`, `includeCitations`, `citationOptions` (`minConfidence`, `maxCitations`, `includeGraph`, `mustCiteThreshold`). `RetrievalStrategy` is the union `'dense' | 'sparse' | 'hybrid' | 'rerank' | 'multi-query'`. The `search-api` package also exports its internal building blocks for use by consumers that need lower-level access: `HybridSearchEngine`, the rerankers (`CrossEncoderReranker`, `FeatureReranker`, `ListwiseReranker`, `RerankingPipeline`), the citation stack (`CitationExtractor`, `CitationVerifier`, `CitationGraph`, `CitationService`), and the retrieval stack (`QueryExpander`, `MultiQueryRetriever`, `RetrievalService`, `RetrievalPipelineBuilder`). ### 3.2 Knowledge Graph (`apps/sophia/knowledge-graph`) Entry point: `KnowledgeGraphService` (factory `createKnowledgeGraphService`). Default port **3001**. Config (`KnowledgeGraphConfig`): `port`, `host`, `maxQueryLimit` (default 100), `maxTraversalDepth` (default 5), `enableResolution` (default true). Backed by an in-memory `MemoryGraphStore`; internal services are `EntityService`, `RelationService`, `GraphService`, `EntityResolver` (when resolution is enabled), and `CuratorService`. This service exposes a full REST API. The routes are grouped below by functional area. | Group | Routes | | ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Health | `GET /health` | | Entities | `POST /entities`, `GET /entities`, `POST /entities/batch`, `POST /entities/batch-delete`, `GET/PATCH/DELETE /entities/:id`, `GET /entities/:id/relationships` | | Relationships | `POST /relationships`, `GET /relationships`, `GET/PATCH/DELETE /relationships/:id` | | Graph | `POST /graph/traverse`, `POST /graph/neighbors`, `POST /graph/shortest-path`, `POST /graph/all-paths`, `POST /graph/subgraph`, `GET /graph/statistics`, `GET /graph/most-connected` | | Resolution | `POST /resolution/resolve`, `POST /resolution/merge`, `POST /resolution/find-duplicates`, `POST /resolution/suggest-links/:id` | | Curator | `GET /curator/queue`, `GET /curator/stats`, `GET /curator/history`, `POST /curator/flag/:entityId`, `GET /curator/review/:id`, `POST /curator/review/:id`, `POST /curator/assign/:id`, `POST /curator/approve/:entityId`, `POST /curator/reject/:entityId` | If `enableResolution` is false, `/resolution/*` returns `400 NOT_ENABLED`. This app defines its own `NodeType` and `RelationshipType` unions, separate from `@sophia/schemas`, reflecting the operational subset needed at runtime: - `NodeType` (9): `tradition`, `text`, `concept`, `practice`, `figure`, `place`, `event`, `organization`, `custom`. - `RelationshipType` (29): `TAUGHT`, `WROTE`, `LIVED_IN`, `STUDIED_UNDER`, `FOUNDED`, `ALIGNS_WITH`, `CONTRADICTS`, `DERIVES_FROM`, `REFINES`, `EXTENDS`, `BELONGS_TO`, `REQUIRES`, `PRECEDES`, `FOLLOWS`, `DISCUSSES`, `REFERENCES`, `CITES`, `ORIGINATES_FROM`, `TRANSLATED_FROM`, `ASSOCIATED_WITH`, `SACRED_TO`, `LOCATED_IN`, `EXEMPLIFIES`, `INFLUENCED_BY`, `RELATED_TO`, `PART_OF`, `MEMBER_OF`, `PARTICIPATED_IN`. #### Curator Subsystem `CuratorService` manages a review queue for newly extracted or flagged entities. Human curators work through the queue, approving or rejecting entities before they become authoritative in the knowledge graph. `ReviewItem` carries: `entityId`, `entityType`, `entityName`, `status` (`pending | approved | rejected | needs_revision`), `priority` (`low | medium | high | critical`), `flaggedBy`, `flagReason`, optional `assignedTo`, `reviewNote`, `reviewedBy`, `qualityScore`, and timestamps. Operations available: flag, get review, submit decision, assign reviewer, get queue (with status / priority / entity-type / assignee filters; sorted critical-first), get resolved history, and `getStats()` (totals by status and priority, average quality score, recent activity). #### Graph Traversal Limits (`DEFAULT_LIMITS`) `GraphService` enforces hard limits on traversal operations to prevent runaway queries from exhausting memory or taking too long. The defaults below are defined in `graph/graph-service.ts`. | Limit | Default | | ---------------------- | ------- | | `MAX_HOPS` | 5 | | `MAX_PATH_DEPTH` | 10 | | `MAX_RESULTS` | 100 | | `MAX_NEIGHBORS` | 50 | | `MAX_TRAVERSAL_NODES` | 1000 | | `MAX_PATHS` | 50 | | `TRAVERSAL_TIMEOUT_MS` | 30000 | #### Knowledge-Graph App Errors `GraphError` is the base error class; subclasses carry specific error codes that map to HTTP status codes at the router level. - `NodeNotFoundError` (`NODE_NOT_FOUND`) → 404 - `RelationshipNotFoundError` (`RELATIONSHIP_NOT_FOUND`) → 404 - `InvalidRelationshipError` (`INVALID_RELATIONSHIP`) → 400 - `TraversalLimitError` (`TRAVERSAL_LIMIT_EXCEEDED`) → 400 Other `GraphError` codes map to 400; `INVALID_STATE` maps to 409. ### 3.3 Ingestion (`apps/sophia/ingestion`) `IngestionPipeline` orchestrates document ingestion. It runs as a background worker (CLI or service mode) with no fixed HTTP port. Options: `maxConcurrentJobs` (default 5), `defaultChunkSize` (default 1000), `defaultChunkOverlap` (default 200), plus connector / extractor / enrichment / storage injection and an optional Kalika knowledge-graph integration. The functional pipeline stages are **fetch → extract → enrich → chunk**, with an optional **kalika-knowledge-graph** sync stage afterward for math/physics documents. On success the pipeline publishes `sophia.document.ingested` and per-entity `sophia.entity.extracted` events via `@sophia/event-publisher`. Pipeline lifecycle events emitted on every job and stage transition: `job:created`, `job:started`, `job:progress`, `job:completed`, `job:failed`, `stage:started`, `stage:completed`, `stage:failed`. Type definitions in `apps/sophia/ingestion/src/types.ts`: - `SourceType` (6): `file`, `url`, `api`, `database`, `stream`, `archive`. - `DocumentFormat` (11): `pdf`, `epub`, `html`, `markdown`, `plain-text`, `tei-xml`, `docx`, `xlsx`, `pptx`, `image`, `unknown`. - `EntityType` (9, extraction): `person`, `organization`, `location`, `date`, `event`, `concept`, `work`, `term`, `custom`. - `JobStatus` (9): `pending`, `fetching`, `extracting`, `enriching`, `chunking`, `indexing`, `completed`, `failed`, `cancelled`. - `ChunkingConfig.strategy` (5 here): `fixed`, `paragraph`, `sentence`, `semantic`, `recursive`. Ingestion error classes: `IngestionError` (base) and stage-specific subclasses: `FetchError` (`FETCH_ERROR`, stage `fetch`, retryable), `ExtractionError` (`EXTRACTION_ERROR`, stage `extract`, not retryable), `OCRError` (`OCR_ERROR`, stage `ocr`, retryable), `EnrichmentError` (`ENRICHMENT_ERROR`, stage `enrich`, retryable). ### 3.4 Workbench (`apps/sophia/workbench`) A React SPA (Vite, React Router, Tailwind). Routes and pages: `/` (`DashboardPage`), `/curation` (`CurationPage`), `/review` (`ReviewPage`), `/annotations` (`AnnotationsPage`), `/documents` (`DocumentsPage`), `/search` (`SearchPage`), `/operator` (`OperatorWorkbenchPage`). API service modules follow a `getXService()` / `resetXService()` singleton pattern (the reset is for test isolation): `AuthService`, `DashboardService`, `DocumentsService`, `AnnotationsService`, `SearchService`, `ReviewService`, `OperatorWorkbenchService`, plus `resetLocalRuntimeServices()`. --- ## 4. Events All Sophia domain events are defined in `libs/contracts/src/events/sophia.ts` and published through `@sophia/event-publisher` (`SophiaEventPublisher`), which wraps `@oshun/event-bus`. `SophiaEventTypes` enumerates **9 event types**. The key architectural point is that event default routing is pre-configured: `sophia.document.ingested` routes to Hathor and Bellona; `sophia.entity.extracted` routes to Hathor and Lilith. Other events require the caller to pass explicit `targets`, otherwise they are not delivered. | Event type | Publisher method | Payload (key fields) | | ---------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `sophia.document.ingested` | `publishDocumentIngested` | `documentId`, `projectId`, `userId`, `type`, `title`, `sizeBytes`, `contentHash`, `chunkCount`, `tokenCount`, `processingTimeMs`, optional `sourceUrl`/`sourceAssetId`/`language` | | `sophia.document.updated` | `publishDocumentUpdated` | `documentId`, `projectId`, `previousVersion`, `newVersion`, `changedChunks`, `reindexed` | | `sophia.document.deleted` | `publishDocumentDeleted` | `documentId`, `projectId`, `removedChunks` | | `sophia.index.updated` | `publishIndexUpdated` | `indexId`, `projectId`, `documentCount`, `chunkCount`, `totalTokens`, `embeddingModel`, `embeddingDimensions`, `updateType` (`full`/`incremental`), `processingTimeMs` | | `sophia.index.rebuilt` | `publishIndexRebuilt` | `indexId`, `projectId`, `previousDocumentCount`, `newDocumentCount`, `reason` (`migration`/`corruption`/`model_change`/`manual`), `processingTimeMs` | | `sophia.search.performed` | `publishSearchPerformed` | `searchId`, `projectId`, `userId`, `query`, `queryType` (`semantic`/`keyword`/`hybrid`), `resultCount`, optional `topScore`, `latencyMs` | | `sophia.entity.extracted` | `publishEntityExtracted` | `entityId`, `projectId`, `documentId`, `type`, `name`, `confidence`, `mentions[]`, optional `description`/`attributes` | | `sophia.relation.discovered` | `publishRelationDiscovered` | `relationId`, `projectId`, `sourceEntityId`, `targetEntityId`, `type`, `confidence`, optional `documentId`/`evidence`/`attributes` | | `sophia.citation.created` | `publishCitationCreated` | `citationId`, `projectId`, `documentId`, `chunkId`, `referencedBy` (`{type, id}`), `excerpt`, `relevanceScore` | Default routing targets: `sophia.document.ingested` → `['hathor', 'bellona']`; `sophia.entity.extracted` → `['hathor', 'lilith']`; other events use no default targets unless callers pass `options.targets`. Publishing failures are logged and swallowed so they never break the calling flow. The publisher is configured with a Redis URL (`SophiaEventPublisherConfig`: `redisUrl`, `enabled`, `keyPrefix`). `getSophiaEventPublisher()` returns a singleton; `SOPHIA_EVENTS_ENABLED=false` disables publishing. The contracts package additionally exposes a `DocumentType` Zod enum (13 values: `text`, `pdf`, `html`, `markdown`, `json`, `xml`, `code`, `spreadsheet`, `presentation`, `image`, `video`, `audio`, `other`) and an `IngestionStatus` enum (`pending`, `processing`, `indexed`, `failed`) used in event payloads. These are distinct from the `DocumentType` and `DocumentStatus` enums in `@sophia/schemas`. `@sophia/schemas` also defines its own typed event schemas (`DocumentIngestedEventSchema`, `CitationCreatedEventSchema`, `EntityExtractedEventSchema`, etc.) for internal validation; these are distinct from the cross-domain `@oshun/contracts` event envelopes above. --- ## 5. Vector Store Layer — `@sophia/vectordb` `@sophia/vectordb` provides a unified vector-store abstraction over four backends, each implementing a common `VectorStore` interface. The library ensures that the choice of vector database is a deployment decision, not an application-code decision. | Backend | Store class | SDK | | -------- | --------------------- | ----------------------------- | | Memory | `MemoryVectorStore` | Built-in | | Qdrant | `QdrantVectorStore` | `@qdrant/js-client-rest` | | Pinecone | `PineconeVectorStore` | `@pinecone-database/pinecone` | | Milvus | `MilvusVectorStore` | `@zilliz/milvus2-sdk-node` | Note: the `search-api` service config exposes only `memory`, `qdrant`, and `pinecone` as runtime backend choices; Milvus is available at the library level. ### 5.1 Predefined Collection Schemas `schemas/collections.ts` defines the `COLLECTIONS` name map and exports **10 collection schemas** (`ALL_SCHEMAS`). Each schema pre-configures the dimensionality, index type (HNSW or IVF), and distance metric appropriate for its use case. | Constant | Collection name | Dim | Index | Metric | | ----------------------------------- | ---------------------------- | --- | ---------- | ------------- | | `WISDOM_EMBEDDINGS_SCHEMA` | `wisdom_embeddings` | 768 | `hnsw` | `cosine` | | `CONTENT_EMBEDDINGS_SCHEMA` | `content_embeddings` | 768 | `hnsw` | `cosine` | | `USER_PREFERENCE_EMBEDDINGS_SCHEMA` | `user_preference_embeddings` | 384 | `hnsw` | `cosine` | | `CONVERSATION_EMBEDDINGS_SCHEMA` | `conversation_embeddings` | 768 | `ivf_flat` | `cosine` | | `SEMANTIC_SEARCH_SCHEMA` | `semantic_search` | 768 | `ivf_flat` | `dot_product` | | `PERSONA_EMBEDDINGS_SCHEMA` | `persona_embeddings` | 384 | `hnsw` | `euclidean` | | `KNOWLEDGE_CHUNKS_SCHEMA` | `knowledge_chunks` | 768 | `hnsw` | `cosine` | | `RESEARCH_DOCUMENTS_SCHEMA` | `research_documents` | 768 | `hnsw` | `cosine` | | `CITATIONS_SCHEMA` | `citation_embeddings` | 768 | `hnsw` | `cosine` | | `ENTITIES_SCHEMA` | `entity_embeddings` | 768 | `hnsw` | `cosine` | Each schema also specifies indexed metadata fields and index-specific parameters (HNSW: `M`, `efConstruction`, `ef`; IVF: `nlist`, `nprobe`). Helper functions: `getSchemaByName`, `isValidCollectionName`, `getCollectionsByDimension`, `getCollectionsByIndexType`. --- ## 6. Indexing Layer — `@sophia/indexing` ### 6.1 Chunking Strategies `indexing/src/chunking/strategies.ts` implements **8 chunker classes**, all extending `BaseChunker`. The `ChunkingStrategy` union covers all 8 names and is what the ingestion pipeline uses to select the right chunker at runtime. Chunker classes: `FixedSizeChunker`, `SlidingWindowChunker`, `ParagraphChunker`, `SentenceChunker`, `RecursiveChunker`, `MarkdownChunker`, `CodeChunker`, `SemanticChunker`. `ChunkingStrategy` union: `'fixed' | 'sliding-window' | 'paragraph' | 'sentence' | 'semantic' | 'markdown' | 'code' | 'recursive'`. ### 6.2 Embedding Providers `indexing/src/embedding/providers.ts` defines **3 provider classes** extending `BaseEmbeddingProvider`: `OpenAIEmbeddingProvider` (`provider: 'openai'`), `CohereEmbeddingProvider` (`provider: 'cohere'`), `LocalEmbeddingProvider` (`provider: 'local'`). The `indexing/embedding/` subtree also provides an embedding cache and service. Note that `libs/sophia/embeddings/` is a separate library that provides a multimodal/audio/text embedding subsystem distinct from `@sophia/indexing`'s document-chunking embedding providers. --- ## 7. Configuration and Environment Variables The table below lists all environment variables consumed by Sophia's applications and libraries. Services bind to `localhost` and default ports unless overridden. | Variable | Used by | Notes | | ------------------------ | ------------------------------- | -------------------------------------------------------- | | `SOPHIA_DATABASE_URL` | `@sophia/database` | PostgreSQL connection (Prisma datasource) | | `REDIS_URL` | `@sophia/event-publisher` | Event bus Redis URL; default `redis://localhost:6379` | | `SOPHIA_EVENTS_ENABLED` | `@sophia/event-publisher` | `false` disables event publishing | | `PORT` / `HOST` | `search-api`, `knowledge-graph` | Service bind address (defaults 3000 / 3001, `localhost`) | | `SEARCH_API_CORPUS_PATH` | `search-api` | Optional startup corpus path | | `MAX_QUERY_LIMIT` | `knowledge-graph` | Default 100 | | `MAX_TRAVERSAL_DEPTH` | `knowledge-graph` | Default 5 | | `ENABLE_RESOLUTION` | `knowledge-graph` | `false` disables entity resolution | The OpenAI and Cohere embedding providers read their API keys from the respective provider environment variables (e.g., `OPENAI_API_KEY`, `COHERE_API_KEY`) and throw an error at invocation time if a key is missing. --- ## 8. V2 Release-Candidate Ingestion Contract `@sophia/ingestion` is the source-of-truth ingestion pipeline for all production Sophia usage. The V2 release-candidate bridge described here is a separate package that wraps it for a specific use case. `@v2/sophia-release-candidate-ingestion` (`apps/v2/sophia-release-candidate-ingestion/`) is a V2 release-candidate cook bridge over `@sophia/ingestion` for frame-data and patch-note sources. It exports the cook hook constant `V2_SOPHIA_RELEASE_CANDIDATE_COOK_HOOK = 'SophiaReleaseCandidateIngestion'` and runs before each release-candidate cook. The bridge ingests a frame-data spreadsheet and a patch-notes corpus, emits Sophia parse/chunk/enrich/embed/index/quality-eval events for those documents, and writes an ingestion manifest as a cook artifact. The manifest is off rollback authority: it may feed citations, companion-app RAG, wiki pages, and human release review, but deterministic combat and rollback simulation must not depend on it. --- ## 9. Cross-Domain Integration Sophia integrates with other domains through a small, well-defined set of channels. It depends on shared infrastructure libraries for event publishing and logging; consumer domains subscribe to its events or call its APIs. - **Hathor** consumes `sophia.document.ingested` and `sophia.entity.extracted` via the `@hathor/sophia-integration` library, which provides research grounding, lore validation, and a citation service. - **Bellona** is a default routing target for `sophia.document.ingested`. - **Lilith** is a default routing target for `sophia.entity.extracted`. Sophia's shared library dependencies: - `@oshun/event-bus` — Kafka-backed event publishing - `@oshun/contracts` — event schemas and `SophiaEventTypes` definitions - `@oshun/logging` — Pino-based structured logging - `@oshun/errors` — error base classes used by `@sophia/schemas` --- ## 10. Acceptance Criteria A Sophia change is acceptance-complete when all of the following hold: 1. `@sophia/database` Prisma schema migrations apply cleanly and the generated client builds. 2. `@sophia/schemas` Zod schemas compile and exported branded ID types remain type-safe (no ID cross-assignment). 3. `search-api` serves the routes of §3.1 and `knowledge-graph` serves the routes of §3.2; both handle the `/v1/` and `/api/v1/` path prefixes. 4. The ingestion pipeline emits the §4 events on success and the pipeline stage events on every stage transition. 5. Event payloads validate against the `@oshun/contracts` Sophia event schemas. 6. Enum value sets in code and in this document match exactly — including the 12-member `CitationStyleSchema`, the 57-member Zod `RelationTypeSchema`, and the distinct uppercase Prisma enums. 7. The vector-store abstraction exposes all 10 predefined collection schemas and all four backends implement the common `VectorStore` interface. 8. Tests pass under Vitest for the touched libraries and applications.