# Sophia Knowledge Stack Convergence Plan

This document outlines the strategy for converging Lilith's knowledge services into the unified Sophia platform.

## 1. Audit Summary

### 1.1 Lilith Services Audited

#### spiritual-embeddings Service
**Location**: `lilith/services/spiritual-embeddings/`
**Purpose**: Text embedding generation for semantic search and content analysis

**Components**:
- `src/app.ts` - Main Fastify application with deterministic hash-based embedding endpoint
- `src/providers/openai.ts` - OpenAI embeddings provider (text-embedding-3-small/large)
- `src/providers/huggingface.ts` - HuggingFace embeddings provider (sentence-transformers)
- `src/providers/types.ts` - Shared interfaces for embedding providers

**Key Features**:
- Multi-provider support (OpenAI, HuggingFace, hash-based fallback)
- Batch embedding generation
- Cosine similarity calculation
- Vector normalization utilities

**Migration Path**: These capabilities should be consolidated into `@sophia/embeddings` service.

---

#### knowledge-access Service
**Location**: `lilith/services/knowledge-access/`
**Purpose**: Content bundle management with access control and NFT-based access passes

**Components**:
- `src/service.ts` - Core service with dependency injection pattern
- `src/app.ts` - Full Fastify REST API with CRUD endpoints

**Key Features**:
- Content bundle management (CRUD operations)
- Access policy management (tier-based, time-limited, token-gated)
- Access pass minting and revocation
- Seeker (user) access management
- Storage record tracking (IPFS URIs)
- Event logging and statistics

**Data Models**:
- `ContentBundle` - Course, meditation, scripture, teaching, practice content
- `AccessPolicy` - Tier, access type, pricing, seat limits
- `AccessPass` - User's access grant with metadata
- `AccessCheck` - Access verification logs
- `StorageRecord` - IPFS/distributed storage tracking

**Migration Path**: Access control features should integrate with Sophia's KnowledgePack model.

---

#### knowledge Service (Graph Store)
**Location**: `lilith/services/knowledge/`
**Purpose**: Knowledge graph storage with Neo4j/Milvus/PostgreSQL backends

**Components**:
- `src/storage/` - Multi-backend storage abstraction
  - Document store (PostgreSQL/memory)
  - Graph store (Neo4j/memory)
  - Vector store (Milvus/memory)
  - Time-series store (TimescaleDB/memory)
  - Relational store (memory)
- `src/seeds/knowledge-graph-seed.ts` - Comprehensive seed data

**Key Features**:
- Node types: Tradition, Text, Concept, Practice, Figure, Place, School, Doctrine, Event
- Relationship types: 25+ relationship types for knowledge graph edges
- Graph traversal, path finding, Cypher query support
- Batch operations for bulk import

**Seed Data**:
- 10 traditions (Buddhism, Hinduism, Taoism, Stoicism, Sufism, etc.)
- 25+ concepts across traditions
- 17 historical figures
- 19 practices
- 16 sacred texts
- 10 sacred places
- Cross-tradition concept alignments

**Migration Path**: Core graph capabilities consolidate into `@sophia/database` and new `@sophia/graph` service.

---

#### ai Service - Knowledge Base
**Location**: `lilith/services/ai/src/knowledge-base.ts`
**Purpose**: In-memory knowledge repository for teachings, practices, and concepts

**Components**:
- `KnowledgeBase` class with indexed storage
- Search with tradition, category, keyword, and difficulty filtering
- Teaching progression system (beginner -> advanced)

**Migration Path**: Merge into Sophia's document and entity models.

---

#### ai Service - Embedding Router
**Location**: `lilith/services/ai/src/providers/embedding-router.ts`
**Purpose**: Intelligent embedding model routing and cost optimization

**Features**:
- Model profiles (Qwen3, OpenAI, Google Gemini)
- Use-case routing (scholarly, query, cross-tradition, multimodal)
- Dimension configuration and reduction
- Batch processing optimization
- Cost estimation and comparison
- Cache TTL suggestions
- Fallback chain generation

**Migration Path**: Consolidate into `@sophia/embeddings` service.

---

## 2. Convergence Architecture

### 2.1 Target State

```
Sophia Platform
├── @sophia/database        # PostgreSQL schema (Prisma)
│   ├── Document model      # From knowledge-access ContentBundle
│   ├── Entity model        # From knowledge-graph nodes
│   ├── Relation model      # From knowledge-graph relationships
│   ├── KnowledgePack       # From knowledge-access bundles + policies
│   └── IngestionJob        # Document processing pipeline
│
├── @sophia/client          # TypeScript SDK for Sophia APIs
│   ├── SearchClient        # Semantic search operations
│   ├── IngestionClient     # Document upload and processing
│   └── KnowledgeGraphClient # Graph traversal and queries
│
├── @sophia/embeddings      # NEW: Embedding service
│   ├── EmbeddingRouter     # From ai/embedding-router
│   ├── OpenAIProvider      # From spiritual-embeddings
│   ├── HuggingFaceProvider # From spiritual-embeddings
│   └── QwenProvider        # New addition
│
├── @sophia/graph           # NEW: Knowledge graph service
│   ├── GraphStore          # From knowledge/storage
│   ├── Neo4jAdapter        # From knowledge/neo4j
│   └── TraversalEngine     # Path finding, neighbors
│
└── @sophia/search          # NEW: Search service
    ├── VectorStore         # From knowledge/milvus
    ├── HybridSearch        # Dense + sparse retrieval
    └── RAGPipeline         # Retrieval-augmented generation
```

### 2.2 Data Model Mapping

| Lilith Model | Sophia Model | Notes |
|-------------|--------------|-------|
| ContentBundle | Document + KnowledgePack | Split into document metadata and pack |
| AccessPolicy | KnowledgePack fields | Embedded in pack configuration |
| AccessPass | KnowledgePackStar + custom | User access grants |
| GraphNode (Tradition) | Entity (type=TRADITION) | Direct mapping |
| GraphNode (Text) | Entity (type=TEXT) | Direct mapping |
| GraphNode (Concept) | Entity (type=CONCEPT) | Direct mapping |
| GraphNode (Practice) | Entity (type=PRACTICE) | Direct mapping |
| GraphNode (Figure) | Entity (type=FIGURE) | Direct mapping |
| GraphNode (Place) | Entity (type=PLACE) | Direct mapping |
| GraphRelationship | Relation | Direct mapping with type conversion |
| Teaching | Document + DocumentChunk | Split content and metadata |
| Practice (knowledge-base) | Entity (type=PRACTICE) | Merged |
| Concept (knowledge-base) | Entity (type=CONCEPT) | Merged |

### 2.3 Relationship Type Mapping

| Lilith RelationType | Sophia RelationType | Notes |
|--------------------|---------------------|-------|
| TAUGHT | TAUGHT | Direct |
| WROTE | WROTE | Direct |
| LIVED_IN | LIVED_IN | Direct |
| STUDIED_UNDER | STUDIED_WITH | Rename |
| FOUNDED | FOUNDED | Direct |
| ALIGNS_WITH | SIMILAR_TO | Semantic equivalent |
| CONTRADICTS | OPPOSITE_OF | Semantic equivalent |
| DERIVES_FROM | DERIVED_FROM | Direct |
| BELONGS_TO | BELONGS_TO | Direct |
| REFERENCES | REFERENCES | Direct |
| CITES | CITES | Direct |
| ORIGINATES_FROM | DERIVED_FROM | Semantic equivalent |
| RELATED_TO | RELATED_TO | Direct |
| INFLUENCED_BY | INFLUENCED | Reversed direction |
| SACRED_TO | PART_OF | Context-specific |

---

## 3. Migration Strategy

### 3.1 Phase 1: Schema Alignment (Week 1)
- [x] Create Sophia database schema with Entity and Relation models
- [ ] Add missing entity types to Sophia schema if needed
- [ ] Add missing relation types to Sophia schema
- [ ] Create Prisma migrations

### 3.2 Phase 2: Data Migration Scripts (Week 2)
- [ ] Create migration script for Lilith knowledge-graph seed data
- [ ] Create migration script for existing content bundles
- [ ] Create migration script for access policies and passes
- [ ] Validate data integrity after migration

### 3.3 Phase 3: Service Implementation (Week 3-4)
- [ ] Implement @sophia/embeddings service
- [ ] Implement @sophia/graph service
- [ ] Implement @sophia/search service
- [ ] Update @sophia/client with new service clients

### 3.4 Phase 4: Integration (Week 5)
- [ ] Update Lilith services to use Sophia APIs
- [ ] Create adapter layer for backward compatibility
- [ ] Integration testing
- [ ] Performance benchmarking

### 3.5 Phase 5: Deprecation (Week 6+)
- [ ] Mark Lilith knowledge services as deprecated
- [ ] Create migration guide for consumers
- [ ] Monitor usage and decommission

---

## 4. Migration Scripts

### 4.1 Knowledge Graph Seed Migration

```typescript
// libs/sophia/database/prisma/migrate-lilith-knowledge.ts

import { getSophiaClient } from '../src/client.js';
import { TRADITIONS, CONCEPTS, FIGURES, PRACTICES, TEXTS, PLACES } from
  '../../../../lilith/services/knowledge/src/seeds/knowledge-graph-seed.js';

const ENTITY_TYPE_MAP = {
  Tradition: 'TRADITION',
  Concept: 'CONCEPT',
  Figure: 'FIGURE',
  Practice: 'PRACTICE',
  Text: 'TEXT',
  Place: 'PLACE',
} as const;

const RELATION_TYPE_MAP = {
  'TAUGHT': 'TAUGHT',
  'WROTE': 'WROTE',
  'LIVED_IN': 'LIVED_IN',
  'STUDIED_UNDER': 'STUDIED_WITH',
  'FOUNDED': 'FOUNDED',
  'ALIGNS_WITH': 'SIMILAR_TO',
  'BELONGS_TO': 'BELONGS_TO',
  'REFERENCES': 'REFERENCES',
  'CITES': 'CITES',
  'ORIGINATES_FROM': 'DERIVED_FROM',
  'RELATED_TO': 'RELATED_TO',
  'SACRED_TO': 'PART_OF',
} as const;

async function migrateKnowledgeGraph() {
  const prisma = getSophiaClient();
  const entityMap = new Map<string, string>();

  // Migrate traditions
  for (const tradition of TRADITIONS) {
    const entity = await prisma.entity.create({
      data: {
        type: 'TRADITION',
        name: tradition.name,
        description: tradition.description,
        language: 'en',
        tradition: tradition.name,
        properties: {
          origin_region: tradition.origin_region,
          time_period: tradition.time_period,
          key_principles: tradition.key_principles,
          category: tradition.category,
        },
        confidence: 'CERTAIN',
        verificationStatus: 'VERIFIED',
        ownerId: 'system',
      },
    });
    entityMap.set(`Tradition:${tradition.name}`, entity.id);
  }

  // Migrate concepts, figures, practices, texts, places...
  // (Similar pattern for each entity type)

  // Create relationships based on seed data alignments
  // ...

  console.log(`Migrated ${entityMap.size} entities`);
}
```

### 4.2 Content Bundle Migration

```typescript
// libs/sophia/database/prisma/migrate-content-bundles.ts

import { getSophiaClient } from '../src/client.js';

interface LilithContentBundle {
  bundleId: string;
  contentKey: string;
  title: string;
  summary: string;
  category: string;
  tradition?: string;
  lessons: string[];
  media: string[];
  difficulty: string;
  estimatedHours: number;
}

async function migrateContentBundles(bundles: LilithContentBundle[]) {
  const prisma = getSophiaClient();

  for (const bundle of bundles) {
    // Create document for the bundle
    const document = await prisma.document.create({
      data: {
        title: bundle.title,
        type: mapCategoryToDocType(bundle.category),
        status: 'PUBLISHED',
        language: 'en',
        abstract: bundle.summary,
        keywords: [bundle.category, bundle.tradition].filter(Boolean) as string[],
        metadata: {
          lilithBundleId: bundle.bundleId,
          contentKey: bundle.contentKey,
          lessons: bundle.lessons,
          media: bundle.media,
          difficulty: bundle.difficulty,
          estimatedHours: bundle.estimatedHours,
        },
        ownerId: 'system',
      },
    });

    // Create knowledge pack
    await prisma.knowledgePack.create({
      data: {
        name: bundle.title,
        slug: slugify(bundle.contentKey),
        description: bundle.summary,
        category: 'TOPIC',
        status: 'PUBLISHED',
        visibility: 'PUBLIC',
        traditions: bundle.tradition ? [bundle.tradition] : [],
        topics: [bundle.category],
        author: 'Lilith Migration',
        ownerId: 'system',
        publishedAt: new Date(),
        documents: {
          create: {
            documentId: document.id,
            order: 0,
          },
        },
      },
    });
  }
}

function mapCategoryToDocType(category: string): string {
  const map: Record<string, string> = {
    course: 'ANTHOLOGY',
    meditation: 'MANUSCRIPT',
    scripture: 'SCRIPTURE',
    teaching: 'ESSAY',
    practice: 'MANUSCRIPT',
  };
  return map[category] || 'OTHER';
}
```

---

## 5. Service Updates

### 5.1 Lilith Service Adapter Pattern

To maintain backward compatibility during migration, create adapter services:

```typescript
// lilith/services/knowledge-access-adapter/src/app.ts

import { createSophiaClient, type SophiaClient } from '@sophia/client';

export function createKnowledgeAccessAdapter(sophiaClient: SophiaClient) {
  return {
    async fetchContentBundle(contentKey: string) {
      // Translate to Sophia API call
      const result = await sophiaClient.search
        .query()
        .filter('contentKey', contentKey)
        .type('knowledge-pack')
        .execute();

      return translateToLilithBundle(result.results[0]);
    },

    async checkAccess(seekerAddress: string, bundleId: string) {
      // Translate access check to Sophia
      const pack = await sophiaClient.graph.getEntity(bundleId);
      // Check user's starred/purchased status
      return { hasAccess: true, reason: 'Migrated to Sophia' };
    },
  };
}
```

### 5.2 Embedding Service Consolidation

```typescript
// libs/sophia/embeddings/src/router.ts

export class SophiaEmbeddingRouter {
  // Consolidate Lilith's embedding-router.ts and spiritual-embeddings providers

  private providers: Map<string, EmbeddingProvider> = new Map();
  private useCaseRoutes: Map<string, string> = new Map();

  constructor(config: EmbeddingRouterConfig) {
    // Register default providers
    this.registerProvider('openai', new OpenAIEmbeddingProvider(config.openai));
    this.registerProvider('huggingface', new HuggingFaceEmbeddingProvider(config.huggingface));
    this.registerProvider('qwen', new QwenEmbeddingProvider(config.qwen));

    // Set up use case routing
    this.useCaseRoutes.set('scholarly-content', 'qwen');
    this.useCaseRoutes.set('user-query', 'openai');
    this.useCaseRoutes.set('cross-tradition', 'qwen');
    // ... more routes
  }

  async embed(text: string, options: EmbedOptions): Promise<EmbeddingResult> {
    const provider = this.selectProvider(options);
    return provider.embed(text, options.dimensions);
  }
}
```

---

## 6. Testing Strategy

### 6.1 Migration Validation Tests

```typescript
describe('Knowledge Graph Migration', () => {
  it('should migrate all traditions', async () => {
    const traditions = await prisma.entity.findMany({
      where: { type: 'TRADITION' },
    });
    expect(traditions.length).toBe(TRADITIONS.length);
  });

  it('should preserve tradition properties', async () => {
    const buddhism = await prisma.entity.findFirst({
      where: { name: 'Buddhism', type: 'TRADITION' },
    });
    expect(buddhism?.properties).toMatchObject({
      origin_region: expect.any(String),
      key_principles: expect.any(Array),
    });
  });

  it('should create cross-tradition relationships', async () => {
    const alignments = await prisma.relation.findMany({
      where: { type: 'SIMILAR_TO' },
    });
    expect(alignments.length).toBeGreaterThan(0);
  });
});
```

### 6.2 API Compatibility Tests

```typescript
describe('Lilith API Compatibility', () => {
  it('should return same structure for content bundles', async () => {
    const lilithResponse = await lilithApi.getBundles();
    const sophiaResponse = await sophiaAdapter.getBundles();

    expect(sophiaResponse).toMatchStructure(lilithResponse);
  });
});
```

---

## 7. Rollback Plan

If migration issues are discovered:

1. **Data Rollback**: Keep Lilith database intact during migration; only deprecate after validation
2. **API Rollback**: Adapter layer allows quick switch back to Lilith services
3. **Feature Flags**: Use feature flags to gradually enable Sophia APIs
4. **Monitoring**: Track error rates, latency, and data consistency metrics

---

## 8. Success Criteria

- [ ] All Lilith knowledge graph data migrated to Sophia
- [ ] All Lilith content bundles migrated to Sophia documents/packs
- [ ] Sophia APIs provide equivalent or better functionality
- [ ] Lilith services successfully consume Sophia APIs via adapters
- [ ] No data loss or corruption during migration
- [ ] Performance meets or exceeds Lilith baselines
- [ ] All tests pass (unit, integration, migration validation)

---

## 9. Timeline

| Week | Milestone |
|------|-----------|
| 1 | Schema alignment and Prisma migrations |
| 2 | Migration scripts development and testing |
| 3 | @sophia/embeddings service implementation |
| 4 | @sophia/graph and @sophia/search services |
| 5 | Lilith adapter integration and testing |
| 6 | Production migration and monitoring |
| 7+ | Deprecation of Lilith services |

---

## 10. Open Questions

1. **Access Control Migration**: How to handle existing NFT-based access passes?
   - Option A: Create new Sophia access tokens
   - Option B: Maintain dual-system during transition

2. **Real-time Subscriptions**: Lilith uses SSE for real-time updates. Does Sophia need this?

3. **Storage URIs**: Lilith stores IPFS URIs. Should Sophia maintain these or migrate content?

4. **User Data**: How to migrate seeker addresses and pass histories?
