The
apps/sophia/area: four runnable Nx applications that turn raw source documents into a searchable, curated research knowledge base — an ingestion service, a knowledge-graph service, a hybrid search/RAG API, and the React workbench operators use to curate it all.
What this area is#
Sophia is Oshun's research / knowledge-management domain, and apps/sophia/
holds its deployable applications (as opposed to libs/sophia/, which holds
the shared building blocks these apps compose — @sophia/embeddings,
@sophia/vectordb, @sophia/indexing, @sophia/event-publisher,
@sophia/client). All four projects carry the scope:sophia / type:app Nx
tags, and each is independently versioned with its own package.json,
tsconfig, and Vitest config.
The four apps form a pipeline. @sophia/ingestion-app is the front door: it
fetches documents from files, URLs, APIs, archives, and academic sources,
extracts and OCRs their text, enriches them with metadata/entities/summaries,
and chunks them for indexing. @sophia/knowledge-graph-app holds the
entity/relationship graph that ingestion can feed (entities → nodes, with
resolution and deduplication). @sophia/search-api is the retrieval surface — a
hybrid dense+sparse search and RAG service over the indexed corpus.
@sophia/workbench is the operator UI on top of all of it: curation, review,
annotation, document management, search, and an operator remediation console.
Three of the four (ingestion-app, knowledge-graph-app, search-api) are
Node services built with @nx/js:tsc whose main.ts exposes a standalone
node:http REST server while their index.ts re-exports the same machinery as
an importable library. The fourth (workbench) is a Vite-built React SPA
(platform:web) with its own Playwright e2e target. The domain that runs
through every node is research/knowledge work — note the knowledge-graph's node
types (tradition, text, concept, practice, figure, place, event,
organization) and the academic ArXiv/PubMed connectors in ingestion.
How it fits the wider system#
These apps depend downward on the libs/sophia/ packages and on platform-shared
libs (@oshun/logging, and, in the workbench, @oshun/http-client,
@oshun/auth-primitives, @oshun/config). The ingestion pipeline emits
cross-domain events through @sophia/event-publisher
(SophiaDocumentIngestedPayload, SophiaEntityExtractedPayload), so other
Oshun domains can react to new documents and extracted entities without coupling
to Sophia's internals. The workbench is a pure client: its ApiClient
(apps/sophia/workbench/src/api/client.ts) speaks REST to these backend
services (/api/documents, /api/search, /api/annotations, /api/reviews,
/api/operator/workbench/...) over fetch with bearer-token auth, and falls
back to an in-memory local runtime when no backend is wired. The "used by" edges
on each node below show the concrete consumers.
Entity catalog (4)#
The 4 tracked Nx projects in sophia, each a code-linked entity node — package, type, source path, declared targets, and its internal dependency graph (depends-on / used-by, resolved from the package manifests, §6/§8), read from the project graph. Grouped by architectural layer; walk the dependency links to travel the system. 4 of these carry an authored deep-dive (what / why / how it fits); the rest are generated scaffolds awaiting one.
unclassified (4)#
Document ingestion service with connectors, OCR, and enrichment pipelines for the Sophia domain
The document-ingestion service (apps/sophia/ingestion). Its core is
IngestionPipeline (src/pipeline/ingestion-pipeline.ts), which orchestrates a
four-stage job — fetch → extract → enrich → chunk — emitting typed
PipelineEvents and publishing document.ingested / entity.extracted
cross-domain events via @sophia/event-publisher. It is genuinely feature-rich:
source connectors for file/URL/API/archive plus academic ArXiv and PubMed
(src/connectors/), Tesseract.js OCR (src/ocr/), rule-based enrichment
(title/author/date/language/entity extraction and extractive summarization,
src/enrichment/), five real chunking strategies implemented in full — fixed
(with overlap), paragraph, sentence, Jaccard-keyword semantic, and
hierarchical recursive — and a Kalika knowledge-graph sync integration
(src/integrations/). src/main.ts wraps the pipeline in an
IngestionAPIService (node:http) with routes for ingest/upload (incl.
hand-rolled multipart parsing), jobs, documents, and chunks, backed by
FileSystemDocumentStorage/FileSystemChunkStorage. Fully implemented, not a
scaffold.
FileSystemDocumentStorage116FileSystemChunkStorage116createDocumentId119createSourceId119createJobId119createChunkId119IngestionError122FetchError122ExtractionError122OCRError122EnrichmentError122FileConnector128URLConnector128APIConnector128 +55 moreThe knowledge-graph microservice (apps/sophia/knowledge-graph). It exposes
four real services from src/index.ts: EntityService (node CRUD over a
MemoryGraphStore), RelationService (typed relationships), GraphService
(traversal, path-finding, subgraphs), and EntityResolver
(src/resolution/entity-resolver.ts). The resolver is the substantive piece —
real entity-resolution algorithms: Levenshtein-based stringSimilarity, Jaccard
alias/tag overlap, substring containsSimilarity, weighted scoring with
confidence levels, duplicate-group discovery (findDuplicates), link suggestion
(suggestLinks), and a property-merging merge that rewires relationships and
supports prefer_primary/prefer_secondary/combine strategies plus per-field
conflict resolution. src/main.ts serves it as a REST API (default port 3001)
including a curator-administration surface. Honest caveat: the backing store is
in-memory (MemoryGraphStore), so persistence is process-lifetime; the resolver
and graph logic themselves are complete and domain-specific.
RelationshipId12TraversalId12ResolutionId12createNodeId12createRelationshipId12createTraversalId12createResolutionId12BaseNodeProperties12TraditionProperties12TextProperties12ConceptProperties12PracticeProperties12FigureProperties12PlaceProperties12 +47 moreSearch and RAG API service for the Sophia domain with hybrid search, reranking, and citation endpoints
The search and RAG service (apps/sophia/search-api). HybridSearchEngine
(src/search/hybrid-search.ts) combines dense (vector) and sparse (keyword)
retrieval with two fusion methods — Reciprocal Rank Fusion (default, rrfK=60)
and weighted linear fusion with min-max normalization — plus dense-only and
sparse-only modes, threshold filtering, and document deduplication. The
retrieval is real, not faked: src/search/runtime.ts wires a
TransformerEmbeddingProvider backed by @sophia/embeddings (MiniLM-L6-v2,
384-d, in-process ONNX) and a hand-written BM25TextIndex implementing the
BM25+ scoring formula (k1, b, and a delta lower-bound), over a
@sophia/vectordb store selectable as memory/Qdrant/Pinecone from env. Around
the core sit a reranking stack (cross-encoder/feature/listwise pipelines,
src/ranking/reranker.ts), citation extraction/verification/graph
(src/citations/), and query-expansion / multi-query retrieval
(src/retrieval/). src/main.ts exposes it as an HTTP service. Fully
implemented.
ResultId12ChunkId12DocumentId12CitationId12createQueryId12createResultId12createChunkId12createDocumentId12createCitationId12SearchQuery12SearchFilters12SearchOptions12ResultMetadata12TextHighlight12 +59 moreSophia Research Workbench - Curation, review, and annotation tools for knowledge management
The Sophia Research Workbench (apps/sophia/workbench) — a React + Vite SPA and
the only platform:web node in the area, with its own Playwright e2e suite
(e2e/). src/App.tsx defines seven routed pages: Dashboard, Curation, Review,
Annotations, Documents, Search, and an Operator workbench. Each page calls a
service in src/api/services/, which dispatches to either the REST ApiClient
(src/api/client.ts, a real fetch-based client with JWT bearer auth, timeout
handling, and typed error parsing) or an in-memory local runtime
(src/api/services/local-runtime.ts, ~1,200 lines of seeded
documents/annotations/reviews/operator state) selected by
isLocalRuntimeEnabled(). That local runtime is an honest dev/demo fallback so
the UI runs without live backends — not a fake stand-in for the app under test —
and the production path (RealOperatorWorkbenchService, etc.) goes to the real
services. Includes auth context, keyboard navigation, Tailwind styling, and
component/page unit tests. A complete frontend application.