# System Architecture

This document provides comprehensive documentation of the Lilith platform system
architecture.

## Overview

The Lilith platform is a comprehensive metaverse and AI learning platform built
on a microservices architecture. It supports multi-modal content delivery (text,
voice, video, 3D), advanced AI orchestration, and immersive consciousness
exploration experiences.

**Architecture Type**: Microservices with API Gateway/BFF **Total Services**: 83
microservices **Deployment Model**: Kubernetes with GitOps (ArgoCD) **Primary
Stack**: Node.js/Fastify, TypeScript, PostgreSQL, Redis

---

## System Context (C4 Level 1)

```
┌─────────────────────────────────────────────────────────────────────┐
│                          EXTERNAL SYSTEMS                            │
├─────────────┬───────────────┬──────────────┬────────────────────────┤
│ OAuth       │ Payment       │ AI Providers │ Content Delivery       │
│ Providers   │ Gateways      │ (LLM, TTS)   │ Networks (CDN)         │
│ (Google,    │ (Stripe,      │ (OpenAI,     │ (CloudFront,           │
│ Apple)      │ Crypto)       │ Anthropic)   │ Cloudflare)            │
└──────┬──────┴───────┬───────┴──────┬───────┴────────────┬───────────┘
       │              │              │                    │
       │              │              │                    │
       ▼              ▼              ▼                    ▼
┌─────────────────────────────────────────────────────────────────────┐
│                                                                      │
│                        LILITH PLATFORM                               │
│                                                                      │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │                       API GATEWAY (BFF)                        │ │
│  │               GraphQL + REST + WebSocket + SSE                  │ │
│  └─────────────────────────┬──────────────────────────────────────┘ │
│                            │                                         │
│  ┌─────────┬─────────┬─────┴─────┬─────────┬─────────┬───────────┐  │
│  │  Auth   │Conversa-│  Content  │  Media  │   AI    │   RAG     │  │
│  │ Service │  tion   │  Service  │ Service │ Service │  Service  │  │
│  └────┬────┴────┬────┴────┬──────┴────┬────┴────┬────┴────┬──────┘  │
│       │         │         │           │         │         │          │
│  ┌────┴─────────┴─────────┴───────────┴─────────┴─────────┴────────┐│
│  │                     DATA LAYER                                   ││
│  │  PostgreSQL │ Redis │ Milvus │ Neo4j │ Elasticsearch │ MinIO   ││
│  └──────────────────────────────────────────────────────────────────┘│
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
       ▲              ▲              ▲                    ▲
       │              │              │                    │
┌──────┴──────┬───────┴──────┬───────┴──────┬─────────────┴───────────┐
│ Web PWA     │ Mobile App   │ VR/AR        │ Partner                  │
│ (Next.js)   │ (React       │ (WebXR)      │ Integrations             │
│             │ Native)      │              │                          │
└─────────────┴──────────────┴──────────────┴──────────────────────────┘
                          CLIENT APPLICATIONS
```

---

## Container Diagram (C4 Level 2)

### Core Platform Services

```
┌──────────────────────────────────────────────────────────────────────┐
│                           API LAYER                                   │
│  ┌─────────────────────────────────────────────────────────────────┐ │
│  │                    BFF Service (Port 3000)                       │ │
│  │  - Request aggregation    - GraphQL federation (Mercurius)      │ │
│  │  - Response shaping       - SSE streaming                        │ │
│  │  - Client versioning      - WebSocket support                    │ │
│  └─────────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────────┘
                                   │
           ┌───────────────────────┼───────────────────────┐
           │                       │                       │
           ▼                       ▼                       ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│   Auth Service      │ │ Conversation Service│ │   Content Service   │
│   (Port 3003)       │ │    (Port 3001)      │ │    (Port 3004)      │
│                     │ │                     │ │                     │
│ - JWT tokens        │ │ - Thread/message    │ │ - Curriculum mgmt   │
│ - OAuth2/WebAuthn   │ │ - SSE streaming     │ │ - Course versioning │
│ - MFA/RBAC          │ │ - Context windows   │ │ - Lecture pipeline  │
│ - Session mgmt      │ │ - WebSocket chat    │ │ - Multi-language    │
└──────────┬──────────┘ └──────────┬──────────┘ └──────────┬──────────┘
           │                       │                       │
           ▼                       ▼                       ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│   Media Service     │ │    AI Service       │ │    RAG Service      │
│   (Port 3005)       │ │   (Port 3002)       │ │    (Port 3008)      │
│                     │ │                     │ │                     │
│ - S3 storage        │ │ - Multi-agent       │ │ - Hybrid search     │
│ - HLS/DASH          │ │ - LLM abstraction   │ │ - Embeddings        │
│ - Signed URLs       │ │ - Provider routing  │ │ - Citation tracking │
│ - CDN integration   │ │ - Budget caps       │ │ - Context packs     │
└──────────┬──────────┘ └──────────┬──────────┘ └──────────┬──────────┘
           │                       │                       │
           ▼                       ▼                       ▼
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Notification Service│ │ Moderation Service  │ │  Knowledge Service  │
│   (Port 3006)       │ │   (Port 3007)       │ │    (Port 3012)      │
│                     │ │                     │ │                     │
│ - Push (APNS/FCM)   │ │ - Content filtering │ │ - Knowledge graph   │
│ - Email dispatch    │ │ - Safety checks     │ │ - Entity linking    │
│ - Template i18n     │ │ - Policy enforcement│ │ - Ontology mgmt     │
│ - Quiet hours       │ │ - Age-gating        │ │ - Topic tagging     │
└─────────────────────┘ └─────────────────────┘ └─────────────────────┘
```

### Speech & Real-Time Services

```
┌─────────────────────────────────────────────────────────────────────┐
│                      SPEECH & REAL-TIME                              │
├─────────────────────┬─────────────────────┬─────────────────────────┤
│   TTS Service       │   STT Service       │   WebRTC Service        │
│   (Port 3009)       │   (Port 3010)       │   (SFU Gateway)         │
│                     │                     │                         │
│ - Chatterbox GPU    │ - Whisper models    │ - Audio/video streams   │
│ - SSML support      │ - VAD/diarization   │ - TURN fallback         │
│ - Voice cloning     │ - Streaming partial │ - Spatial audio         │
│ - 500ms first-byte  │ - Domain boosting   │ - Group sessions        │
└─────────────────────┴─────────────────────┴─────────────────────────┘
```

### Data Layer

```
┌─────────────────────────────────────────────────────────────────────┐
│                          DATA LAYER                                  │
├─────────────┬─────────────┬─────────────┬─────────────┬─────────────┤
│ PostgreSQL  │   Redis     │   Milvus    │   Neo4j     │Elasticsearch│
│ (Port 5432) │ (Port 6379) │(Port 19530) │ (Port 7687) │ (Port 9200) │
│             │             │             │             │             │
│ Users       │ Sessions    │ Embeddings  │ Knowledge   │ Full-text   │
│ Content     │ Cache       │ Vector      │ Graph       │ Search      │
│ Courses     │ Rate limits │ Search      │ Entities    │ Indexing    │
│ Messages    │ Pub/Sub     │ RAG         │ Relations   │ Analytics   │
└─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘
                              │
                    ┌─────────┴─────────┐
                    │     MinIO         │
                    │   (Port 9000)     │
                    │                   │
                    │ S3-compatible     │
                    │ Object Storage    │
                    │ Media Assets      │
                    └───────────────────┘
```

---

## Service Catalog

### Tier 1: Core Platform (12 Services)

| Service      | Port | Responsibility                          |
| ------------ | ---- | --------------------------------------- |
| BFF          | 3000 | API aggregation, GraphQL federation     |
| Auth         | 3003 | Authentication, authorization, sessions |
| Conversation | 3001 | Chat threads, messages, streaming       |
| Content      | 3004 | Courses, curricula, versioning          |
| Media        | 3005 | Asset storage, CDN, streaming           |
| Notification | 3006 | Push, email, in-app notifications       |
| Moderation   | 3007 | Content safety, policy enforcement      |
| AI           | 3002 | LLM orchestration, multi-agent system   |
| RAG          | 3008 | Retrieval, embeddings, citations        |
| Knowledge    | 3012 | Knowledge graph, entity management      |
| Ingestion    | 3011 | Content pipeline, OCR, enrichment       |
| TTS          | 3009 | Text-to-speech synthesis                |

### Tier 2: Extended Services (35+ Services)

**Spiritual & Consciousness**:

- `spiritual-embeddings` - Wisdom tradition embeddings
- `spiritual-guidance` - Personalized guidance engine
- `meditation-core` - Core meditation management
- `meditation-generation` - Dynamic meditation creation
- `group-meditation` - Synchronized group sessions

**Community & Social**:

- `community` - Community features
- `forum` - Discussion forums
- `sharing` - Content sharing
- `social-features` - Social interactions

**Content & Curation**:

- `daily-content` - Personalized daily selections
- `user-preferences` - Preference management
- `notes` - Scholar annotations
- `review` - Content review workflow
- `curator-admin` - Curation workbench

**Blockchain & Web3** (8 services):

- `blockchain` - Network interaction
- `payment-orchestrator` - Payment processing
- `token-verification` - Token validation
- `transaction-manager` - Transaction processing
- `settlement` - Settlements & reconciliation
- `defi-integration` - DeFi protocols
- `cross-chain-bridge` - Multi-chain support
- `staking-mechanism` - Staking functionality

**Compliance & Safety** (6 services):

- `data-governance` - GDPR/CCPA compliance
- `anonymization` - Privacy pipelines
- `consent-management` - Consent tracking
- `content-licensing` - License validation
- `rights-management` - IP & creator rights
- `safety-automation` - Automated safety

---

## Data Flow Patterns

### Chat Request Flow

```
┌────────┐   HTTP/WS   ┌─────┐   REST    ┌─────────────┐
│ Client │ ──────────► │ BFF │ ────────► │Conversation │
└────────┘             └──┬──┘           │   Service   │
                          │              └──────┬──────┘
                          │                     │
                          ▼                     ▼
                    ┌─────────┐          ┌─────────────┐
                    │   RAG   │ ◄────────│ AI Service  │
                    │ Service │          └──────┬──────┘
                    └─────────┘                 │
                                               ▼
                                        ┌─────────────┐
                                        │ Moderation  │
                                        │   Service   │
                                        └──────┬──────┘
                                               │
                                               ▼
                                        ┌─────────────┐
                                        │ TTS Service │
                                        │ (optional)  │
                                        └─────────────┘
                                               │
◄──────────────────────────────────────────────┘
    SSE Tokens + Citations + Audio URL
```

### Lecture Generation Flow

```
┌────────┐          ┌─────────┐          ┌─────────────┐
│ Client │ ────────►│ Content │ ────────►│ Scholar     │
└────────┘          │ Service │          │ Agent       │
                    └────┬────┘          └──────┬──────┘
                         │                      │
                         │                      ▼
                         │               ┌─────────────┐
                         │               │ RAG Service │
                         │               └──────┬──────┘
                         │                      │
                         │                      ▼
                         │               ┌─────────────┐
                         │               │  Curator    │
                         │               │   Agent     │
                         │               └──────┬──────┘
                         │                      │
                         │                      ▼
                         │               ┌─────────────┐
                         │               │ Moderation  │
                         │               └──────┬──────┘
                         │                      │
                         │                      ▼
                         │               ┌─────────────┐
                         │               │ Narrator    │
                         │               │   Agent     │
                         │               └──────┬──────┘
                         │                      │
                         │                      ▼
                         │               ┌─────────────┐
                         │◄──────────────│ TTS Service │
                         │               └─────────────┘
                         ▼
              ┌────────────────────┐
              │ Response:          │
              │ - lecture_id       │
              │ - stream_url       │
              │ - transcript_url   │
              │ - citations[]      │
              └────────────────────┘
```

### Knowledge Ingestion Flow

```
┌────────────┐     ┌───────────┐     ┌───────────────┐
│ External   │────►│ Ingestion │────►│ De-dup        │
│ Source     │     │ Service   │     │ OCR, Parse    │
└────────────┘     └─────┬─────┘     └───────┬───────┘
                         │                   │
                         ▼                   ▼
                  ┌─────────────┐     ┌───────────────┐
                  │ NLP         │────►│ Knowledge     │
                  │ Enrichment  │     │ Graph Service │
                  └──────┬──────┘     └───────┬───────┘
                         │                    │
                         ▼                    ▼
                  ┌─────────────┐     ┌───────────────┐
                  │ RAG Service │     │ Human Review  │
                  │ (indexing)  │     │ (optional)    │
                  └─────────────┘     └───────────────┘
```

---

## Communication Patterns

### Synchronous Communication

| Pattern   | Use Case                     | Example            |
| --------- | ---------------------------- | ------------------ |
| HTTP/REST | Service-to-service calls     | BFF → Auth Service |
| GraphQL   | Complex queries, aggregation | Client → BFF       |
| gRPC      | High-performance internal    | (future)           |

### Asynchronous Communication

| Pattern       | Use Case                | Example              |
| ------------- | ----------------------- | -------------------- |
| SSE           | Streaming responses     | Chat token streaming |
| WebSocket     | Bidirectional real-time | Voice chat           |
| WebRTC        | Audio/video             | Group meditation     |
| Redis Pub/Sub | Event broadcasting      | Notifications        |
| Message Queue | Async processing        | TTS generation       |

---

## Database Architecture

### PostgreSQL Schema (Primary)

```sql
-- Core Tables
users (user_id, email_hash, created_at, locale, consent_flags)
profiles (user_id, display_name, timezone, preferred_voices)
personas (id, key, title, style_config, policy_tags)
content (id, type, title, lang, body_ref, metadata, status, version)
courses (id, title, syllabus)
conversations (id, user_id, persona_key, created_at)
messages (id, conv_id, role, text_ref, citations, metrics)
media_assets (id, uri, kind, lang, duration, hash, drm_key_id)
events (id, user_id, type, payload, timestamp)
```

### Redis Key Patterns

| Pattern           | Purpose                      |
| ----------------- | ---------------------------- |
| `user:*:session`  | JWT session data             |
| `chat:*:context`  | Conversation context windows |
| `cache:lecture:*` | Pre-generated TTS cache      |
| `rate_limit:*`    | API rate limiting            |
| `locks:*`         | Distributed locks            |
| `metrics:*`       | Real-time metrics            |

### Milvus (Vector Database)

```json
{
  "collection": "wisdom_chunks",
  "schema": {
    "doc_id": "string",
    "chunk_id": "string",
    "lang": "string",
    "tradition": ["string"],
    "entities": ["string"],
    "year_range": [int, int],
    "license": "string",
    "text": "string",
    "embedding": "float[1536]"
  }
}
```

### Neo4j (Knowledge Graph)

**Node Types**:

- `Tradition`, `Text`, `Concept`, `Practice`, `Figure`, `Place`

**Relationship Types**:

- `TAUGHT`, `WROTE`, `ALIGNS_WITH`, `BELONGS_TO`, `ASSOCIATED_WITH`

---

## Deployment Architecture

### Kubernetes Topology

```
┌─────────────────────────────────────────────────────────────────────┐
│                        KUBERNETES CLUSTER                            │
│                                                                      │
│  ┌───────────────────────────────────────────────────────────────┐  │
│  │                      INGRESS LAYER                             │  │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────────────────┐│  │
│  │  │ ALB/nginx   │  │    WAF      │  │   SSL/TLS (ACM)         ││  │
│  │  │ Ingress     │  │             │  │                         ││  │
│  │  └──────┬──────┘  └─────────────┘  └─────────────────────────┘│  │
│  └─────────┼─────────────────────────────────────────────────────┘  │
│            │                                                         │
│  ┌─────────┼─────────────────────────────────────────────────────┐  │
│  │         ▼         SERVICE MESH (Optional - Istio/Linkerd)      │  │
│  │  ┌─────────────────────────────────────────────────────────┐  │  │
│  │  │                    APPLICATION LAYER                     │  │  │
│  │  │  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐       │  │  │
│  │  │  │ BFF │ │Auth │ │Conv │ │AI   │ │RAG  │ │TTS  │  ...  │  │  │
│  │  │  │     │ │     │ │     │ │     │ │     │ │     │       │  │  │
│  │  │  └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘ └──┬──┘       │  │  │
│  │  │     │       │       │       │       │       │          │  │  │
│  │  │     │       │       │       │       │       │          │  │  │
│  │  └─────┼───────┼───────┼───────┼───────┼───────┼──────────┘  │  │
│  └────────┼───────┼───────┼───────┼───────┼───────┼─────────────┘  │
│           │       │       │       │       │       │                 │
│  ┌────────┼───────┼───────┼───────┼───────┼───────┼─────────────┐  │
│  │        ▼       ▼       ▼       ▼       ▼       ▼              │  │
│  │                       DATA LAYER                              │  │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐         │  │
│  │  │PostgreSQL│ │  Redis   │ │  Milvus  │ │  Neo4j   │         │  │
│  │  │(StatefulS│ │(StatefulS│ │(StatefulS│ │(StatefulS│         │  │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘         │  │
│  └──────────────────────────────────────────────────────────────┘  │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐  │
│  │                    OBSERVABILITY LAYER                        │  │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐         │  │
│  │  │Prometheus│ │ Grafana  │ │  Jaeger  │ │AlertMgr  │         │  │
│  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘         │  │
│  └──────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘
```

### GitOps with ArgoCD

```
┌─────────────────────────────────────────────────────────────────────┐
│                         GITOPS FLOW                                  │
│                                                                      │
│  ┌────────────┐    push    ┌────────────┐   sync   ┌────────────┐  │
│  │ Developer  │ ─────────► │   GitHub   │ ◄──────► │  ArgoCD    │  │
│  │            │            │            │          │            │  │
│  └────────────┘            └────────────┘          └─────┬──────┘  │
│                                                          │         │
│  ┌───────────────────────────────────────────────────────┼───────┐ │
│  │                                                       ▼       │ │
│  │                      KUBERNETES CLUSTER                       │ │
│  │                                                               │ │
│  │  app-of-apps.yaml                                             │ │
│  │       │                                                       │ │
│  │       ├──► apps/dev/*.yaml                                    │ │
│  │       ├──► apps/staging/*.yaml                                │ │
│  │       └──► apps/prod/*.yaml                                   │ │
│  │                                                               │ │
│  └───────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
```

### Helm Chart Structure

```
deploy/helm/lilith-service/
├── Chart.yaml
├── values.yaml           # Base configuration
├── values-dev.yaml       # Development overrides
├── values-staging.yaml   # Staging overrides
├── values-production.yaml# Production overrides
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    ├── hpa.yaml          # Horizontal Pod Autoscaler
    ├── pdb.yaml          # Pod Disruption Budget
    └── networkpolicy.yaml
```

---

## Security Architecture

### Authentication Flow

```
┌────────┐   1. Login    ┌─────────┐   2. Validate   ┌──────────┐
│ Client │ ────────────► │   BFF   │ ───────────────►│  Auth    │
└────────┘               └────┬────┘                 │ Service  │
     ▲                        │                      └────┬─────┘
     │                        │                           │
     │   4. JWT Token         │   3. Generate JWT         │
     └────────────────────────┴───────────────────────────┘
```

### Authorization (RBAC)

| Role    | Permissions                          |
| ------- | ------------------------------------ |
| User    | Read content, chat, manage profile   |
| Premium | + Premium content, advanced features |
| Curator | + Content management, review         |
| Admin   | + User management, system config     |

### Data Protection

| Layer      | Protection                     |
| ---------- | ------------------------------ |
| In Transit | TLS 1.2+, mTLS (mesh)          |
| At Rest    | AES-256, envelope encryption   |
| Secrets    | Vault with dynamic credentials |
| PII        | Pseudonymization, hashing      |

---

## Observability Stack

### Metrics (Prometheus)

```yaml
Service Metrics:
  - http_request_duration_seconds (p50, p95, p99)
  - http_requests_total (by status, method)
  - cache_hit_ratio
  - queue_depth

Application Metrics:
  - rag_hit_rate
  - tts_generation_time
  - stt_word_error_rate
  - chat_response_latency

Infrastructure Metrics:
  - container_cpu_usage
  - container_memory_usage
  - node_disk_io
```

### Tracing (Jaeger)

All requests include:

- `X-Trace-ID`: Correlation ID
- `X-Span-ID`: Operation ID
- Service name, operation, duration
- Tags for metadata

### Logging (Structured)

```json
{
  "level": "info",
  "service": "bff",
  "pod": "bff-7d8f9-x2k4l",
  "trace_id": "abc123",
  "message": "Request completed",
  "duration_ms": 245,
  "status_code": 200
}
```

---

## Performance Targets

| Component | Metric               | Target     |
| --------- | -------------------- | ---------- |
| Chat      | p50 latency          | <2s        |
| Chat      | p95 latency          | <4s        |
| TTS       | First byte (short)   | <600ms     |
| TTS       | First byte (lecture) | <2s        |
| Web PWA   | Cold start           | <2.5s (4G) |
| Web PWA   | TTI                  | <4s        |
| Metaverse | FPS                  | 60         |

---

## Scaling Configuration

### Horizontal Pod Autoscaler

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
```

### Resource Defaults

```yaml
resources:
  requests:
    memory: '256Mi'
    cpu: '100m'
  limits:
    memory: '1Gi'
    cpu: '500m'
```

---

## Technology Stack

| Layer         | Technology                       |
| ------------- | -------------------------------- |
| Runtime       | Node.js 20+                      |
| Framework     | Fastify 4.x                      |
| Language      | TypeScript 6.x (ESM)             |
| API           | REST, GraphQL, WebSocket, SSE    |
| Databases     | PostgreSQL, Redis, Milvus, Neo4j |
| Search        | Elasticsearch 8.x                |
| Storage       | MinIO (S3-compatible)            |
| Orchestration | Kubernetes 1.27+                 |
| IaC           | Terraform, Helm 3.x              |
| GitOps        | ArgoCD 2.x                       |
| Observability | Prometheus, Grafana, Jaeger      |
| Testing       | Jest, Playwright                 |
| CI/CD         | GitHub Actions                   |

---

## Related Documentation

- [API Reference](../reference/typescript-sdk.md) - SDK documentation
- [GraphQL API](../reference/graphql-api.md) - GraphQL schema
- [Authentication](../reference/authentication.md) - Auth flows
- [Error Codes](../reference/error-codes.md) - Error handling
