This document outlines the strategy for converging Lilith's knowledge services into the unified Sophia platform.
1. Audit Summary#
1.1 Lilith Services Audited#
spiritual-embeddings Service#
Location: lilith/services/spiritual-embeddings/
Purpose: Text embedding generation for semantic search and content analysis
Components:
src/app.ts- Main Fastify application with deterministic hash-based embedding endpointsrc/providers/openai.ts- OpenAI embeddings provider (text-embedding-3-small/large)src/providers/huggingface.ts- HuggingFace embeddings provider (sentence-transformers)src/providers/types.ts- Shared interfaces for embedding providers
Key Features:
- Multi-provider support (OpenAI, HuggingFace, hash-based fallback)
- Batch embedding generation
- Cosine similarity calculation
- Vector normalization utilities
Migration Path: These capabilities should be consolidated into @sophia/embeddings service.
knowledge-access Service#
Location: lilith/services/knowledge-access/
Purpose: Content bundle management with access control and NFT-based access passes
Components:
src/service.ts- Core service with dependency injection patternsrc/app.ts- Full Fastify REST API with CRUD endpoints
Key Features:
- Content bundle management (CRUD operations)
- Access policy management (tier-based, time-limited, token-gated)
- Access pass minting and revocation
- Seeker (user) access management
- Storage record tracking (IPFS URIs)
- Event logging and statistics
Data Models:
ContentBundle- Course, meditation, scripture, teaching, practice contentAccessPolicy- Tier, access type, pricing, seat limitsAccessPass- User's access grant with metadataAccessCheck- Access verification logsStorageRecord- IPFS/distributed storage tracking
Migration Path: Access control features should integrate with Sophia's KnowledgePack model.
knowledge Service (Graph Store)#
Location: lilith/services/knowledge/
Purpose: Knowledge graph storage with Neo4j/Milvus/PostgreSQL backends
Components:
src/storage/- Multi-backend storage abstraction- Document store (PostgreSQL/memory)
- Graph store (Neo4j/memory)
- Vector store (Milvus/memory)
- Time-series store (TimescaleDB/memory)
- Relational store (memory)
src/seeds/knowledge-graph-seed.ts- Comprehensive seed data
Key Features:
- Node types: Tradition, Text, Concept, Practice, Figure, Place, School, Doctrine, Event
- Relationship types: 25+ relationship types for knowledge graph edges
- Graph traversal, path finding, Cypher query support
- Batch operations for bulk import
Seed Data:
- 10 traditions (Buddhism, Hinduism, Taoism, Stoicism, Sufism, etc.)
- 25+ concepts across traditions
- 17 historical figures
- 19 practices
- 16 sacred texts
- 10 sacred places
- Cross-tradition concept alignments
Migration Path: Core graph capabilities consolidate into @sophia/database and new @sophia/graph service.
ai Service - Knowledge Base#
Location: lilith/services/ai/src/knowledge-base.ts
Purpose: In-memory knowledge repository for teachings, practices, and concepts
Components:
KnowledgeBaseclass with indexed storage- Search with tradition, category, keyword, and difficulty filtering
- Teaching progression system (beginner -> advanced)
Migration Path: Merge into Sophia's document and entity models.
ai Service - Embedding Router#
Location: lilith/services/ai/src/providers/embedding-router.ts
Purpose: Intelligent embedding model routing and cost optimization
Features:
- Model profiles (Qwen3, OpenAI, Google Gemini)
- Use-case routing (scholarly, query, cross-tradition, multimodal)
- Dimension configuration and reduction
- Batch processing optimization
- Cost estimation and comparison
- Cache TTL suggestions
- Fallback chain generation
Migration Path: Consolidate into @sophia/embeddings service.
2. Convergence Architecture#
2.1 Target State#
Sophia Platform
├── @sophia/database # PostgreSQL schema (Prisma)
│ ├── Document model # From knowledge-access ContentBundle
│ ├── Entity model # From knowledge-graph nodes
│ ├── Relation model # From knowledge-graph relationships
│ ├── KnowledgePack # From knowledge-access bundles + policies
│ └── IngestionJob # Document processing pipeline
│
├── @sophia/client # TypeScript SDK for Sophia APIs
│ ├── SearchClient # Semantic search operations
│ ├── IngestionClient # Document upload and processing
│ └── KnowledgeGraphClient # Graph traversal and queries
│
├── @sophia/embeddings # NEW: Embedding service
│ ├── EmbeddingRouter # From ai/embedding-router
│ ├── OpenAIProvider # From spiritual-embeddings
│ ├── HuggingFaceProvider # From spiritual-embeddings
│ └── QwenProvider # New addition
│
├── @sophia/graph # NEW: Knowledge graph service
│ ├── GraphStore # From knowledge/storage
│ ├── Neo4jAdapter # From knowledge/neo4j
│ └── TraversalEngine # Path finding, neighbors
│
└── @sophia/search # NEW: Search service
├── VectorStore # From knowledge/milvus
├── HybridSearch # Dense + sparse retrieval
└── RAGPipeline # Retrieval-augmented generation
2.2 Data Model Mapping#
| Lilith Model | Sophia Model | Notes |
|---|---|---|
| ContentBundle | Document + KnowledgePack | Split into document metadata and pack |
| AccessPolicy | KnowledgePack fields | Embedded in pack configuration |
| AccessPass | KnowledgePackStar + custom | User access grants |
| GraphNode (Tradition) | Entity (type=TRADITION) | Direct mapping |
| GraphNode (Text) | Entity (type=TEXT) | Direct mapping |
| GraphNode (Concept) | Entity (type=CONCEPT) | Direct mapping |
| GraphNode (Practice) | Entity (type=PRACTICE) | Direct mapping |
| GraphNode (Figure) | Entity (type=FIGURE) | Direct mapping |
| GraphNode (Place) | Entity (type=PLACE) | Direct mapping |
| GraphRelationship | Relation | Direct mapping with type conversion |
| Teaching | Document + DocumentChunk | Split content and metadata |
| Practice (knowledge-base) | Entity (type=PRACTICE) | Merged |
| Concept (knowledge-base) | Entity (type=CONCEPT) | Merged |
2.3 Relationship Type Mapping#
| Lilith RelationType | Sophia RelationType | Notes |
|---|---|---|
| TAUGHT | TAUGHT | Direct |
| WROTE | WROTE | Direct |
| LIVED_IN | LIVED_IN | Direct |
| STUDIED_UNDER | STUDIED_WITH | Rename |
| FOUNDED | FOUNDED | Direct |
| ALIGNS_WITH | SIMILAR_TO | Semantic equivalent |
| CONTRADICTS | OPPOSITE_OF | Semantic equivalent |
| DERIVES_FROM | DERIVED_FROM | Direct |
| BELONGS_TO | BELONGS_TO | Direct |
| REFERENCES | REFERENCES | Direct |
| CITES | CITES | Direct |
| ORIGINATES_FROM | DERIVED_FROM | Semantic equivalent |
| RELATED_TO | RELATED_TO | Direct |
| INFLUENCED_BY | INFLUENCED | Reversed direction |
| SACRED_TO | PART_OF | Context-specific |
3. Migration Strategy#
3.1 Phase 1: Schema Alignment (Week 1)#
- Create Sophia database schema with Entity and Relation models
- Add missing entity types to Sophia schema if needed
- Add missing relation types to Sophia schema
- Create Prisma migrations
3.2 Phase 2: Data Migration Scripts (Week 2)#
- Create migration script for Lilith knowledge-graph seed data
- Create migration script for existing content bundles
- Create migration script for access policies and passes
- Validate data integrity after migration
3.3 Phase 3: Service Implementation (Week 3-4)#
- Implement @sophia/embeddings service
- Implement @sophia/graph service
- Implement @sophia/search service
- Update @sophia/client with new service clients
3.4 Phase 4: Integration (Week 5)#
- Update Lilith services to use Sophia APIs
- Create adapter layer for backward compatibility
- Integration testing
- Performance benchmarking
3.5 Phase 5: Deprecation (Week 6+)#
- Mark Lilith knowledge services as deprecated
- Create migration guide for consumers
- Monitor usage and decommission
4. Migration Scripts#
4.1 Knowledge Graph Seed Migration#
// libs/sophia/database/prisma/migrate-lilith-knowledge.ts
import { getSophiaClient } from '../src/client.js';
import { TRADITIONS, CONCEPTS, FIGURES, PRACTICES, TEXTS, PLACES } from
'../../../../lilith/services/knowledge/src/seeds/knowledge-graph-seed.js';
const ENTITY_TYPE_MAP = {
Tradition: 'TRADITION',
Concept: 'CONCEPT',
Figure: 'FIGURE',
Practice: 'PRACTICE',
Text: 'TEXT',
Place: 'PLACE',
} as const;
const RELATION_TYPE_MAP = {
'TAUGHT': 'TAUGHT',
'WROTE': 'WROTE',
'LIVED_IN': 'LIVED_IN',
'STUDIED_UNDER': 'STUDIED_WITH',
'FOUNDED': 'FOUNDED',
'ALIGNS_WITH': 'SIMILAR_TO',
'BELONGS_TO': 'BELONGS_TO',
'REFERENCES': 'REFERENCES',
'CITES': 'CITES',
'ORIGINATES_FROM': 'DERIVED_FROM',
'RELATED_TO': 'RELATED_TO',
'SACRED_TO': 'PART_OF',
} as const;
async function migrateKnowledgeGraph() {
const prisma = getSophiaClient();
const entityMap = new Map<string, string>();
// Migrate traditions
for (const tradition of TRADITIONS) {
const entity = await prisma.entity.create({
data: {
type: 'TRADITION',
name: tradition.name,
description: tradition.description,
language: 'en',
tradition: tradition.name,
properties: {
origin_region: tradition.origin_region,
time_period: tradition.time_period,
key_principles: tradition.key_principles,
category: tradition.category,
},
confidence: 'CERTAIN',
verificationStatus: 'VERIFIED',
ownerId: 'system',
},
});
entityMap.set(`Tradition:${tradition.name}`, entity.id);
}
// Migrate concepts, figures, practices, texts, places...
// (Similar pattern for each entity type)
// Create relationships based on seed data alignments
// ...
console.log(`Migrated ${entityMap.size} entities`);
}
4.2 Content Bundle Migration#
// libs/sophia/database/prisma/migrate-content-bundles.ts
import { getSophiaClient } from '../src/client.js';
interface LilithContentBundle {
bundleId: string;
contentKey: string;
title: string;
summary: string;
category: string;
tradition?: string;
lessons: string[];
media: string[];
difficulty: string;
estimatedHours: number;
}
async function migrateContentBundles(bundles: LilithContentBundle[]) {
const prisma = getSophiaClient();
for (const bundle of bundles) {
// Create document for the bundle
const document = await prisma.document.create({
data: {
title: bundle.title,
type: mapCategoryToDocType(bundle.category),
status: 'PUBLISHED',
language: 'en',
abstract: bundle.summary,
keywords: [bundle.category, bundle.tradition].filter(Boolean) as string[],
metadata: {
lilithBundleId: bundle.bundleId,
contentKey: bundle.contentKey,
lessons: bundle.lessons,
media: bundle.media,
difficulty: bundle.difficulty,
estimatedHours: bundle.estimatedHours,
},
ownerId: 'system',
},
});
// Create knowledge pack
await prisma.knowledgePack.create({
data: {
name: bundle.title,
slug: slugify(bundle.contentKey),
description: bundle.summary,
category: 'TOPIC',
status: 'PUBLISHED',
visibility: 'PUBLIC',
traditions: bundle.tradition ? [bundle.tradition] : [],
topics: [bundle.category],
author: 'Lilith Migration',
ownerId: 'system',
publishedAt: new Date(),
documents: {
create: {
documentId: document.id,
order: 0,
},
},
},
});
}
}
function mapCategoryToDocType(category: string): string {
const map: Record<string, string> = {
course: 'ANTHOLOGY',
meditation: 'MANUSCRIPT',
scripture: 'SCRIPTURE',
teaching: 'ESSAY',
practice: 'MANUSCRIPT',
};
return map[category] || 'OTHER';
}
5. Service Updates#
5.1 Lilith Service Adapter Pattern#
To maintain backward compatibility during migration, create adapter services:
// lilith/services/knowledge-access-adapter/src/app.ts
import { createSophiaClient, type SophiaClient } from '@sophia/client';
export function createKnowledgeAccessAdapter(sophiaClient: SophiaClient) {
return {
async fetchContentBundle(contentKey: string) {
// Translate to Sophia API call
const result = await sophiaClient.search
.query()
.filter('contentKey', contentKey)
.type('knowledge-pack')
.execute();
return translateToLilithBundle(result.results[0]);
},
async checkAccess(seekerAddress: string, bundleId: string) {
// Translate access check to Sophia
const pack = await sophiaClient.graph.getEntity(bundleId);
// Check user's starred/purchased status
return { hasAccess: true, reason: 'Migrated to Sophia' };
},
};
}
5.2 Embedding Service Consolidation#
// libs/sophia/embeddings/src/router.ts
export class SophiaEmbeddingRouter {
// Consolidate Lilith's embedding-router.ts and spiritual-embeddings providers
private providers: Map<string, EmbeddingProvider> = new Map();
private useCaseRoutes: Map<string, string> = new Map();
constructor(config: EmbeddingRouterConfig) {
// Register default providers
this.registerProvider('openai', new OpenAIEmbeddingProvider(config.openai));
this.registerProvider('huggingface', new HuggingFaceEmbeddingProvider(config.huggingface));
this.registerProvider('qwen', new QwenEmbeddingProvider(config.qwen));
// Set up use case routing
this.useCaseRoutes.set('scholarly-content', 'qwen');
this.useCaseRoutes.set('user-query', 'openai');
this.useCaseRoutes.set('cross-tradition', 'qwen');
// ... more routes
}
async embed(text: string, options: EmbedOptions): Promise<EmbeddingResult> {
const provider = this.selectProvider(options);
return provider.embed(text, options.dimensions);
}
}
6. Testing Strategy#
6.1 Migration Validation Tests#
describe('Knowledge Graph Migration', () => {
it('should migrate all traditions', async () => {
const traditions = await prisma.entity.findMany({
where: { type: 'TRADITION' },
});
expect(traditions.length).toBe(TRADITIONS.length);
});
it('should preserve tradition properties', async () => {
const buddhism = await prisma.entity.findFirst({
where: { name: 'Buddhism', type: 'TRADITION' },
});
expect(buddhism?.properties).toMatchObject({
origin_region: expect.any(String),
key_principles: expect.any(Array),
});
});
it('should create cross-tradition relationships', async () => {
const alignments = await prisma.relation.findMany({
where: { type: 'SIMILAR_TO' },
});
expect(alignments.length).toBeGreaterThan(0);
});
});
6.2 API Compatibility Tests#
describe('Lilith API Compatibility', () => {
it('should return same structure for content bundles', async () => {
const lilithResponse = await lilithApi.getBundles();
const sophiaResponse = await sophiaAdapter.getBundles();
expect(sophiaResponse).toMatchStructure(lilithResponse);
});
});
7. Rollback Plan#
If migration issues are discovered:
- Data Rollback: Keep Lilith database intact during migration; only deprecate after validation
- API Rollback: Adapter layer allows quick switch back to Lilith services
- Feature Flags: Use feature flags to gradually enable Sophia APIs
- Monitoring: Track error rates, latency, and data consistency metrics
8. Success Criteria#
- All Lilith knowledge graph data migrated to Sophia
- All Lilith content bundles migrated to Sophia documents/packs
- Sophia APIs provide equivalent or better functionality
- Lilith services successfully consume Sophia APIs via adapters
- No data loss or corruption during migration
- Performance meets or exceeds Lilith baselines
- All tests pass (unit, integration, migration validation)
9. Timeline#
| Week | Milestone |
|---|---|
| 1 | Schema alignment and Prisma migrations |
| 2 | Migration scripts development and testing |
| 3 | @sophia/embeddings service implementation |
| 4 | @sophia/graph and @sophia/search services |
| 5 | Lilith adapter integration and testing |
| 6 | Production migration and monitoring |
| 7+ | Deprecation of Lilith services |
10. Open Questions#
-
Access Control Migration: How to handle existing NFT-based access passes?
- Option A: Create new Sophia access tokens
- Option B: Maintain dual-system during transition
-
Real-time Subscriptions: Lilith uses SSE for real-time updates. Does Sophia need this?
-
Storage URIs: Lilith stores IPFS URIs. Should Sophia maintain these or migrate content?
-
User Data: How to migrate seeker addresses and pass histories?