Eve library

Sophia Knowledge Stack Convergence Plan

Other

Revision 1

On this page

This document outlines the strategy for converging Lilith's knowledge services into the unified Sophia platform.

1. Audit Summary#

1.1 Lilith Services Audited#

spiritual-embeddings Service#

Location: lilith/services/spiritual-embeddings/ Purpose: Text embedding generation for semantic search and content analysis

Components:

  • src/app.ts - Main Fastify application with deterministic hash-based embedding endpoint
  • src/providers/openai.ts - OpenAI embeddings provider (text-embedding-3-small/large)
  • src/providers/huggingface.ts - HuggingFace embeddings provider (sentence-transformers)
  • src/providers/types.ts - Shared interfaces for embedding providers

Key Features:

  • Multi-provider support (OpenAI, HuggingFace, hash-based fallback)
  • Batch embedding generation
  • Cosine similarity calculation
  • Vector normalization utilities

Migration Path: These capabilities should be consolidated into @sophia/embeddings service.


knowledge-access Service#

Location: lilith/services/knowledge-access/ Purpose: Content bundle management with access control and NFT-based access passes

Components:

  • src/service.ts - Core service with dependency injection pattern
  • src/app.ts - Full Fastify REST API with CRUD endpoints

Key Features:

  • Content bundle management (CRUD operations)
  • Access policy management (tier-based, time-limited, token-gated)
  • Access pass minting and revocation
  • Seeker (user) access management
  • Storage record tracking (IPFS URIs)
  • Event logging and statistics

Data Models:

  • ContentBundle - Course, meditation, scripture, teaching, practice content
  • AccessPolicy - Tier, access type, pricing, seat limits
  • AccessPass - User's access grant with metadata
  • AccessCheck - Access verification logs
  • StorageRecord - IPFS/distributed storage tracking

Migration Path: Access control features should integrate with Sophia's KnowledgePack model.


knowledge Service (Graph Store)#

Location: lilith/services/knowledge/ Purpose: Knowledge graph storage with Neo4j/Milvus/PostgreSQL backends

Components:

  • src/storage/ - Multi-backend storage abstraction
    • Document store (PostgreSQL/memory)
    • Graph store (Neo4j/memory)
    • Vector store (Milvus/memory)
    • Time-series store (TimescaleDB/memory)
    • Relational store (memory)
  • src/seeds/knowledge-graph-seed.ts - Comprehensive seed data

Key Features:

  • Node types: Tradition, Text, Concept, Practice, Figure, Place, School, Doctrine, Event
  • Relationship types: 25+ relationship types for knowledge graph edges
  • Graph traversal, path finding, Cypher query support
  • Batch operations for bulk import

Seed Data:

  • 10 traditions (Buddhism, Hinduism, Taoism, Stoicism, Sufism, etc.)
  • 25+ concepts across traditions
  • 17 historical figures
  • 19 practices
  • 16 sacred texts
  • 10 sacred places
  • Cross-tradition concept alignments

Migration Path: Core graph capabilities consolidate into @sophia/database and new @sophia/graph service.


ai Service - Knowledge Base#

Location: lilith/services/ai/src/knowledge-base.ts Purpose: In-memory knowledge repository for teachings, practices, and concepts

Components:

  • KnowledgeBase class with indexed storage
  • Search with tradition, category, keyword, and difficulty filtering
  • Teaching progression system (beginner -> advanced)

Migration Path: Merge into Sophia's document and entity models.


ai Service - Embedding Router#

Location: lilith/services/ai/src/providers/embedding-router.ts Purpose: Intelligent embedding model routing and cost optimization

Features:

  • Model profiles (Qwen3, OpenAI, Google Gemini)
  • Use-case routing (scholarly, query, cross-tradition, multimodal)
  • Dimension configuration and reduction
  • Batch processing optimization
  • Cost estimation and comparison
  • Cache TTL suggestions
  • Fallback chain generation

Migration Path: Consolidate into @sophia/embeddings service.


2. Convergence Architecture#

2.1 Target State#

text
Sophia Platform
├── @sophia/database        # PostgreSQL schema (Prisma)
│   ├── Document model      # From knowledge-access ContentBundle
│   ├── Entity model        # From knowledge-graph nodes
│   ├── Relation model      # From knowledge-graph relationships
│   ├── KnowledgePack       # From knowledge-access bundles + policies
│   └── IngestionJob        # Document processing pipeline
│
├── @sophia/client          # TypeScript SDK for Sophia APIs
│   ├── SearchClient        # Semantic search operations
│   ├── IngestionClient     # Document upload and processing
│   └── KnowledgeGraphClient # Graph traversal and queries
│
├── @sophia/embeddings      # NEW: Embedding service
│   ├── EmbeddingRouter     # From ai/embedding-router
│   ├── OpenAIProvider      # From spiritual-embeddings
│   ├── HuggingFaceProvider # From spiritual-embeddings
│   └── QwenProvider        # New addition
│
├── @sophia/graph           # NEW: Knowledge graph service
│   ├── GraphStore          # From knowledge/storage
│   ├── Neo4jAdapter        # From knowledge/neo4j
│   └── TraversalEngine     # Path finding, neighbors
│
└── @sophia/search          # NEW: Search service
    ├── VectorStore         # From knowledge/milvus
    ├── HybridSearch        # Dense + sparse retrieval
    └── RAGPipeline         # Retrieval-augmented generation

2.2 Data Model Mapping#

Lilith Model Sophia Model Notes
ContentBundle Document + KnowledgePack Split into document metadata and pack
AccessPolicy KnowledgePack fields Embedded in pack configuration
AccessPass KnowledgePackStar + custom User access grants
GraphNode (Tradition) Entity (type=TRADITION) Direct mapping
GraphNode (Text) Entity (type=TEXT) Direct mapping
GraphNode (Concept) Entity (type=CONCEPT) Direct mapping
GraphNode (Practice) Entity (type=PRACTICE) Direct mapping
GraphNode (Figure) Entity (type=FIGURE) Direct mapping
GraphNode (Place) Entity (type=PLACE) Direct mapping
GraphRelationship Relation Direct mapping with type conversion
Teaching Document + DocumentChunk Split content and metadata
Practice (knowledge-base) Entity (type=PRACTICE) Merged
Concept (knowledge-base) Entity (type=CONCEPT) Merged

2.3 Relationship Type Mapping#

Lilith RelationType Sophia RelationType Notes
TAUGHT TAUGHT Direct
WROTE WROTE Direct
LIVED_IN LIVED_IN Direct
STUDIED_UNDER STUDIED_WITH Rename
FOUNDED FOUNDED Direct
ALIGNS_WITH SIMILAR_TO Semantic equivalent
CONTRADICTS OPPOSITE_OF Semantic equivalent
DERIVES_FROM DERIVED_FROM Direct
BELONGS_TO BELONGS_TO Direct
REFERENCES REFERENCES Direct
CITES CITES Direct
ORIGINATES_FROM DERIVED_FROM Semantic equivalent
RELATED_TO RELATED_TO Direct
INFLUENCED_BY INFLUENCED Reversed direction
SACRED_TO PART_OF Context-specific

3. Migration Strategy#

3.1 Phase 1: Schema Alignment (Week 1)#

  • Create Sophia database schema with Entity and Relation models
  • Add missing entity types to Sophia schema if needed
  • Add missing relation types to Sophia schema
  • Create Prisma migrations

3.2 Phase 2: Data Migration Scripts (Week 2)#

  • Create migration script for Lilith knowledge-graph seed data
  • Create migration script for existing content bundles
  • Create migration script for access policies and passes
  • Validate data integrity after migration

3.3 Phase 3: Service Implementation (Week 3-4)#

  • Implement @sophia/embeddings service
  • Implement @sophia/graph service
  • Implement @sophia/search service
  • Update @sophia/client with new service clients

3.4 Phase 4: Integration (Week 5)#

  • Update Lilith services to use Sophia APIs
  • Create adapter layer for backward compatibility
  • Integration testing
  • Performance benchmarking

3.5 Phase 5: Deprecation (Week 6+)#

  • Mark Lilith knowledge services as deprecated
  • Create migration guide for consumers
  • Monitor usage and decommission

4. Migration Scripts#

4.1 Knowledge Graph Seed Migration#

typescript
// libs/sophia/database/prisma/migrate-lilith-knowledge.ts

import { getSophiaClient } from '../src/client.js';
import { TRADITIONS, CONCEPTS, FIGURES, PRACTICES, TEXTS, PLACES } from
  '../../../../lilith/services/knowledge/src/seeds/knowledge-graph-seed.js';

const ENTITY_TYPE_MAP = {
  Tradition: 'TRADITION',
  Concept: 'CONCEPT',
  Figure: 'FIGURE',
  Practice: 'PRACTICE',
  Text: 'TEXT',
  Place: 'PLACE',
} as const;

const RELATION_TYPE_MAP = {
  'TAUGHT': 'TAUGHT',
  'WROTE': 'WROTE',
  'LIVED_IN': 'LIVED_IN',
  'STUDIED_UNDER': 'STUDIED_WITH',
  'FOUNDED': 'FOUNDED',
  'ALIGNS_WITH': 'SIMILAR_TO',
  'BELONGS_TO': 'BELONGS_TO',
  'REFERENCES': 'REFERENCES',
  'CITES': 'CITES',
  'ORIGINATES_FROM': 'DERIVED_FROM',
  'RELATED_TO': 'RELATED_TO',
  'SACRED_TO': 'PART_OF',
} as const;

async function migrateKnowledgeGraph() {
  const prisma = getSophiaClient();
  const entityMap = new Map<string, string>();

  // Migrate traditions
  for (const tradition of TRADITIONS) {
    const entity = await prisma.entity.create({
      data: {
        type: 'TRADITION',
        name: tradition.name,
        description: tradition.description,
        language: 'en',
        tradition: tradition.name,
        properties: {
          origin_region: tradition.origin_region,
          time_period: tradition.time_period,
          key_principles: tradition.key_principles,
          category: tradition.category,
        },
        confidence: 'CERTAIN',
        verificationStatus: 'VERIFIED',
        ownerId: 'system',
      },
    });
    entityMap.set(`Tradition:${tradition.name}`, entity.id);
  }

  // Migrate concepts, figures, practices, texts, places...
  // (Similar pattern for each entity type)

  // Create relationships based on seed data alignments
  // ...

  console.log(`Migrated ${entityMap.size} entities`);
}

4.2 Content Bundle Migration#

typescript
// libs/sophia/database/prisma/migrate-content-bundles.ts

import { getSophiaClient } from '../src/client.js';

interface LilithContentBundle {
  bundleId: string;
  contentKey: string;
  title: string;
  summary: string;
  category: string;
  tradition?: string;
  lessons: string[];
  media: string[];
  difficulty: string;
  estimatedHours: number;
}

async function migrateContentBundles(bundles: LilithContentBundle[]) {
  const prisma = getSophiaClient();

  for (const bundle of bundles) {
    // Create document for the bundle
    const document = await prisma.document.create({
      data: {
        title: bundle.title,
        type: mapCategoryToDocType(bundle.category),
        status: 'PUBLISHED',
        language: 'en',
        abstract: bundle.summary,
        keywords: [bundle.category, bundle.tradition].filter(Boolean) as string[],
        metadata: {
          lilithBundleId: bundle.bundleId,
          contentKey: bundle.contentKey,
          lessons: bundle.lessons,
          media: bundle.media,
          difficulty: bundle.difficulty,
          estimatedHours: bundle.estimatedHours,
        },
        ownerId: 'system',
      },
    });

    // Create knowledge pack
    await prisma.knowledgePack.create({
      data: {
        name: bundle.title,
        slug: slugify(bundle.contentKey),
        description: bundle.summary,
        category: 'TOPIC',
        status: 'PUBLISHED',
        visibility: 'PUBLIC',
        traditions: bundle.tradition ? [bundle.tradition] : [],
        topics: [bundle.category],
        author: 'Lilith Migration',
        ownerId: 'system',
        publishedAt: new Date(),
        documents: {
          create: {
            documentId: document.id,
            order: 0,
          },
        },
      },
    });
  }
}

function mapCategoryToDocType(category: string): string {
  const map: Record<string, string> = {
    course: 'ANTHOLOGY',
    meditation: 'MANUSCRIPT',
    scripture: 'SCRIPTURE',
    teaching: 'ESSAY',
    practice: 'MANUSCRIPT',
  };
  return map[category] || 'OTHER';
}

5. Service Updates#

5.1 Lilith Service Adapter Pattern#

To maintain backward compatibility during migration, create adapter services:

typescript
// lilith/services/knowledge-access-adapter/src/app.ts

import { createSophiaClient, type SophiaClient } from '@sophia/client';

export function createKnowledgeAccessAdapter(sophiaClient: SophiaClient) {
  return {
    async fetchContentBundle(contentKey: string) {
      // Translate to Sophia API call
      const result = await sophiaClient.search
        .query()
        .filter('contentKey', contentKey)
        .type('knowledge-pack')
        .execute();

      return translateToLilithBundle(result.results[0]);
    },

    async checkAccess(seekerAddress: string, bundleId: string) {
      // Translate access check to Sophia
      const pack = await sophiaClient.graph.getEntity(bundleId);
      // Check user's starred/purchased status
      return { hasAccess: true, reason: 'Migrated to Sophia' };
    },
  };
}

5.2 Embedding Service Consolidation#

typescript
// libs/sophia/embeddings/src/router.ts

export class SophiaEmbeddingRouter {
  // Consolidate Lilith's embedding-router.ts and spiritual-embeddings providers

  private providers: Map<string, EmbeddingProvider> = new Map();
  private useCaseRoutes: Map<string, string> = new Map();

  constructor(config: EmbeddingRouterConfig) {
    // Register default providers
    this.registerProvider('openai', new OpenAIEmbeddingProvider(config.openai));
    this.registerProvider('huggingface', new HuggingFaceEmbeddingProvider(config.huggingface));
    this.registerProvider('qwen', new QwenEmbeddingProvider(config.qwen));

    // Set up use case routing
    this.useCaseRoutes.set('scholarly-content', 'qwen');
    this.useCaseRoutes.set('user-query', 'openai');
    this.useCaseRoutes.set('cross-tradition', 'qwen');
    // ... more routes
  }

  async embed(text: string, options: EmbedOptions): Promise<EmbeddingResult> {
    const provider = this.selectProvider(options);
    return provider.embed(text, options.dimensions);
  }
}

6. Testing Strategy#

6.1 Migration Validation Tests#

typescript
describe('Knowledge Graph Migration', () => {
  it('should migrate all traditions', async () => {
    const traditions = await prisma.entity.findMany({
      where: { type: 'TRADITION' },
    });
    expect(traditions.length).toBe(TRADITIONS.length);
  });

  it('should preserve tradition properties', async () => {
    const buddhism = await prisma.entity.findFirst({
      where: { name: 'Buddhism', type: 'TRADITION' },
    });
    expect(buddhism?.properties).toMatchObject({
      origin_region: expect.any(String),
      key_principles: expect.any(Array),
    });
  });

  it('should create cross-tradition relationships', async () => {
    const alignments = await prisma.relation.findMany({
      where: { type: 'SIMILAR_TO' },
    });
    expect(alignments.length).toBeGreaterThan(0);
  });
});

6.2 API Compatibility Tests#

typescript
describe('Lilith API Compatibility', () => {
  it('should return same structure for content bundles', async () => {
    const lilithResponse = await lilithApi.getBundles();
    const sophiaResponse = await sophiaAdapter.getBundles();

    expect(sophiaResponse).toMatchStructure(lilithResponse);
  });
});

7. Rollback Plan#

If migration issues are discovered:

  1. Data Rollback: Keep Lilith database intact during migration; only deprecate after validation
  2. API Rollback: Adapter layer allows quick switch back to Lilith services
  3. Feature Flags: Use feature flags to gradually enable Sophia APIs
  4. Monitoring: Track error rates, latency, and data consistency metrics

8. Success Criteria#

  • All Lilith knowledge graph data migrated to Sophia
  • All Lilith content bundles migrated to Sophia documents/packs
  • Sophia APIs provide equivalent or better functionality
  • Lilith services successfully consume Sophia APIs via adapters
  • No data loss or corruption during migration
  • Performance meets or exceeds Lilith baselines
  • All tests pass (unit, integration, migration validation)

9. Timeline#

Week Milestone
1 Schema alignment and Prisma migrations
2 Migration scripts development and testing
3 @sophia/embeddings service implementation
4 @sophia/graph and @sophia/search services
5 Lilith adapter integration and testing
6 Production migration and monitoring
7+ Deprecation of Lilith services

10. Open Questions#

  1. Access Control Migration: How to handle existing NFT-based access passes?

    • Option A: Create new Sophia access tokens
    • Option B: Maintain dual-system during transition
  2. Real-time Subscriptions: Lilith uses SSE for real-time updates. Does Sophia need this?

  3. Storage URIs: Lilith stores IPFS URIs. Should Sophia maintain these or migrate content?

  4. User Data: How to migrate seeker addresses and pass histories?