# Lilith Domain - Developer Guide Lilith is the largest domain in the Oshun monorepo. It is a consciousness and spiritual wellness platform comprising a Backend-for-Frontend (BFF) aggregation layer, dozens of microservices, client applications (web, mobile, desktop), and shared libraries for configuration, logging, service communication, and testing. ## Architecture Overview ### Applications (`apps/lilith/`) **Core Gateway:** - `bff/` - Backend-for-Frontend (Fastify). The central aggregation gateway that proxies and composes responses from downstream services. Includes GraphQL, REST, and SSE streaming endpoints. **Microservices (selected):** | Service | Description | | ----------------------------- | -------------------------------------------- | | `svc-auth` | Authentication and authorization | | `svc-auth-orchestrator` | Multi-provider auth coordination | | `svc-ai` | AI/LLM integration and reasoning | | `svc-content` | Content management | | `svc-catalog` | Content catalog and discovery | | `svc-conversation` | Conversational AI interactions | | `svc-curricula` | Learning curricula management | | `svc-daily-content` | Daily content scheduling and delivery | | `svc-spiritual-guidance` | AI spiritual guidance engine | | `svc-meditation-core` | Core meditation session management | | `svc-meditation-experience` | Meditation experience orchestration | | `svc-meditation-generation` | AI meditation content generation | | `svc-yoga-practice` | Yoga practice tracking | | `svc-breathwork` | Breathwork session management | | `svc-group-meditation` | Group meditation coordination | | `svc-journal` | Journaling service | | `svc-notes` | Notes management | | `svc-review` | Content review workflow | | `svc-media` | Media file management | | `svc-tts` | Text-to-speech synthesis | | `svc-stt` | Speech-to-text transcription | | `svc-voice-pipeline` | Voice processing pipeline | | `svc-notification` | Push/email notification delivery | | `svc-analytics` | Analytics collection and reporting | | `svc-moderation` | Content moderation | | `svc-sync` | Data synchronization | | `svc-offline` | Offline capability management | | `svc-webrtc` | WebRTC signaling and real-time communication | | `svc-real-time-sync` | Real-time state synchronization | | `svc-progress-sync` | Progress tracking synchronization | | `svc-user-preferences` | User preference management | | `svc-partner-api` | Partner API access | | `svc-observability` | Service observability endpoints | | `svc-error-handler` | Centralized error handling | | `svc-language-detection` | Language detection | | `svc-community-translation` | Community translation management | | `svc-anonymization` | Data anonymization | | `svc-consent-management` | User consent tracking | | `svc-data-governance` | Data governance enforcement | | `svc-safety-automation` | Automated safety checks | | `svc-content-licensing` | Content licensing management | | `svc-content-verification` | Content verification | | `svc-rights-management` | Digital rights management | | `svc-blockchain` | Blockchain integration | | `svc-native-token` | Native token operations | | `svc-token-core` | Token core logic | | `svc-token-access` | Token-based access control | | `svc-token-verification` | Token verification | | `svc-staking-mechanism` | Token staking | | `svc-dao-governance` | DAO governance | | `svc-settlement` | Financial settlement | | `svc-fiat-ramp` | Fiat on/off ramp | | `svc-payment-orchestrator` | Payment workflow orchestration | | `svc-tiered-subscription` | Subscription tier management | | `svc-micro-transaction` | Micro-transaction processing | | `svc-creator-royalty` | Creator royalty distribution | | `svc-teacher-blessing` | Teacher endorsement workflow | | `svc-metaverse` | Metaverse integration | | `svc-avatar-cosmetic` | Avatar cosmetic management | | `svc-biometric` | Biometric data processing | | `svc-veilborn-core` | Veilborn game core | | `svc-veilborn-strategy` | Veilborn game strategy | | `svc-indexer` | Blockchain indexer | | `svc-ipfs-integration` | IPFS decentralized storage | | `svc-cross-chain-bridge` | Cross-chain bridge | | `svc-defi-integration` | DeFi protocol integration | | `svc-multi-region-resilience` | Multi-region resilience | | `svc-operational-excellence` | SRE operational tooling | **Clients:** - `web/` - Next.js web application with Storybook - `mobile/` - React Native mobile application - `desktop/` - Tauri desktop application - `cli/` - Command-line interface tool **Other:** - `contracts/` - Solidity smart contracts (Hardhat) - `seed-corpus/` - Seed data ingestion - `locales/` - i18n translation files Legacy Lilith knowledge and curation services (`svc-rag`, `svc-knowledge`, `svc-knowledge-access`, `svc-ingestion`, `svc-vectordb`, `svc-spiritual-embeddings`, `svc-curator-admin`) have been retired. Current grounding, graph, and ingestion flows route through Sophia services. ### Libraries (`libs/lilith/`) | Library | Package | Purpose | | ----------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `fastify-core` | `@lilith/fastify-core` | Standardized Fastify server factory, health checks, error handling, middleware | | `service-lib` | `@lilith/service-lib` | Service communication, circuit breakers, config management, auth middleware, WAF, rate limiting, database health, service discovery, graceful shutdown | | `config` | `@lilith/config` | Type-safe configuration from environment variables | | `core` | `@lilith/core` | Core types and constants | | `common` | `@lilith/common` | Common utilities | | `errors` | `@lilith/errors` | Error types and handling | | `logger` | `@lilith/logger` | Structured logging (LilithLogger) | | `database` | `@lilith/database` | Database client and utilities | | `cache` | `@lilith/cache` | Redis caching layer | | `queue` | `@lilith/queue` | Job queue (BullMQ-style) | | `metrics` | `@lilith/metrics` | Prometheus metrics | | `tracing` | `@lilith/tracing` | OpenTelemetry distributed tracing | | `event-publisher` | `@lilith/event-publisher` | Event bus publishing | | `event-handlers` | `@lilith/event-handlers` | Event handler registration | | `sdk` | `@lilith/sdk` | Platform client SDK | | `partner-sdk` | `@lilith/partner-sdk` | Partner integration SDK | | `testing` | `@lilith/testing` | Test utilities and fixtures | Shared configuration: - `eslint-config-lilith` - ESLint shared config - `prettier-config-lilith` - Prettier shared config - `tsconfig-lilith` - TypeScript shared config ## Prerequisites | Tool | Version | Purpose | | -------------- | ------- | --------------------------------------- | | Node.js | >= 18.x | Runtime | | pnpm | >= 8.x | Package manager | | Docker | >= 24.x | PostgreSQL, Redis, MinIO, Elasticsearch | | Docker Compose | >= 2.x | Local infrastructure orchestration | ## Starting the BFF and Core Services ### 1. Start Infrastructure From the monorepo root: ```bash docker compose -f docker/docker-compose.dev.yml up -d ``` This starts PostgreSQL (port 5432), Redis (port 6379), MinIO (ports 9000/9001), and Mailpit (ports 1025/8025). For additional services used by Lilith: ```bash # Elasticsearch (for search/logging workloads) docker compose -f docker/docker-compose.dev.yml --profile search up -d # Vector database (Qdrant) docker compose -f docker/docker-compose.dev.yml --profile vectors up -d ``` ### 2. Install Dependencies ```bash pnpm install ``` ### 3. Start the BFF ```bash # Development mode with auto-reload pnpm nx dev @lilith/bff # Or production mode pnpm nx build @lilith/bff pnpm nx start @lilith/bff ``` The BFF listens on the port configured by `PORT` (default: 3000). ### 4. Start Individual Services Each microservice follows the same pattern: ```bash # Development mode pnpm nx dev @lilith/svc-auth pnpm nx dev @lilith/svc-content pnpm nx dev @lilith/svc-ai # Build + start pnpm nx build @lilith/svc-content pnpm nx start @lilith/svc-content ``` Services that have a `dev` target use `tsx watch` for hot reloading. Services that only have `serve` use the Nx node executor. ### 5. Docker Build (BFF) ```bash pnpm nx docker:build @lilith/bff ``` ## Environment Variables Configuration uses the `@lilith/config` library which reads typed values from environment variables with defaults and validation. ### BFF Configuration The BFF server uses `loadAndValidateConfig` from `@lilith/config`: ```typescript const config = loadAndValidateConfig( { service: { ...baseServiceSchema, name: { env: 'SERVICE_NAME', default: 'bff' }, port: { env: 'PORT', default: 3000, type: 'number' }, }, }, { serviceName: 'bff' } ); ``` ### Common Environment Variables | Variable | Default | Used By | Description | | ---------------- | ------------------------ | -------------- | ---------------------------- | | `SERVICE_NAME` | (per service) | All services | Service identifier | | `PORT` | `3000` | All services | HTTP listen port | | `NODE_ENV` | `development` | All services | Environment name | | `DATABASE_URL` | (see .env) | DB services | PostgreSQL connection string | | `REDIS_URL` | `redis://localhost:6379` | Cache services | Redis connection string | | `S3_ENDPOINT` | `http://localhost:9000` | Media services | MinIO/S3 endpoint | | `S3_ACCESS_KEY` | `minioadmin` | Media services | MinIO/S3 access key | | `S3_SECRET_KEY` | `minioadmin` | Media services | MinIO/S3 secret key | | `OPENAI_API_KEY` | (required) | AI services | OpenAI API key | | `JWT_SECRET` | (required) | Auth services | JWT signing secret | The root `.env` file contains all connection URLs for local development. ## Testing ### Running Tests ```bash # BFF tests pnpm nx test @lilith/bff # Specific service tests pnpm nx test @lilith/svc-auth pnpm nx test @lilith/svc-yoga-practice pnpm nx test @lilith/svc-content # Library tests pnpm nx test @lilith/fastify-core pnpm nx test @lilith/service-lib # Watch mode pnpm nx test:watch @lilith/bff # Type checking pnpm nx typecheck @lilith/bff ``` ### Test Framework All Lilith services use Vitest. Tests are located in `__tests__/` directories alongside source code or at the project root level. ```bash # Run with coverage pnpm nx test @lilith/bff -- --coverage ``` ### Linting ```bash pnpm nx lint @lilith/bff pnpm nx lint @lilith/svc-auth ``` ## Service Communication Patterns ### BFF as Aggregation Gateway The BFF is the single entry point for all client applications. It: 1. Receives client requests (REST, GraphQL, SSE streams) 2. Routes to downstream microservices via the service registry 3. Aggregates responses from multiple services 4. Applies response shaping based on client type (web, mobile, desktop) 5. Handles error normalization and circuit breaking Key BFF components: - **Service Registry** (`config/service-registry.ts`) - Maps service names to their HTTP endpoints and health check URLs - **Circuit Breakers** (`utils/circuit-breaker.ts`) - Per-service circuit breakers that prevent cascading failures - **Response Shaping** (`middleware/response-shaping.ts`) - Adapts responses based on the client type header - **Error Normalization** (`error-normalization.ts`) - Normalizes errors from different services into a consistent format - **Streaming Utils** (`streaming-utils.ts`) - SSE streaming with backpressure ### `@lilith/fastify-core` - Service Factory Every microservice uses `@lilith/fastify-core` to create a standardized Fastify server: ```typescript import { createServiceServer } from '@lilith/fastify-core'; const { app, start, shutdown } = await createServiceServer({ serviceName: 'my-service', healthCheckFn: async () => ({ dbConnected: await db.ping(), cacheSize: cache.size, }), }); // Register routes app.get('/v1/hello', async () => ({ message: 'Hello!' })); // Start await start(3000); ``` This provides: - Kubernetes-ready health check endpoints (`/health`, `/ready`, `/live`) - Standardized error handling and response formatting - Middleware: logging, CORS, request timing, correlation IDs - Graceful shutdown ### `@lilith/service-lib` - Cross-Cutting Concerns The `service-lib` package provides the operational infrastructure every service needs: - **Service Client** (`service-client.ts`) - HTTP client for inter-service communication with automatic retries and circuit breaking - **Service Discovery** (`service-discovery.ts`) - Service endpoint resolution - **Service Registry** (`service-registry.ts`) - Self-registration and discovery - **Auth Middleware** (`auth-middleware.ts`) - JWT validation and permission checking - **Rate Limiting** (`rate-limiting-middleware.ts`) - Per-endpoint rate limiting - **WAF** (`waf-middleware.ts`) - Web Application Firewall rules - **Circuit Breaker** (`circuit-breaker.ts`) - Circuit breaker pattern - **Config Manager** (`config-manager.ts`) - Runtime configuration management - **Database Health** (`database-health.ts`) - Database connectivity monitoring - **Graceful Shutdown** (`graceful-shutdown.ts`) - Clean shutdown with drain - **Metrics** (`metrics.ts`) - Prometheus metric collection - **Tracing** (`tracing.ts`) - OpenTelemetry span management - **Structured Logging** (`structured-logging.ts`) - Log formatting - **Bot Detection** (`bot-detection-middleware.ts`) - Bot traffic detection - **PII Detection** (`pii-detector.ts`) - PII scanning in request data - **IP Filtering** (`ip-filtering-middleware.ts`) - IP allowlist/blocklist - **Vault Integration** (`vault-client.ts`) - HashiCorp Vault secret management - **TLS Manager** (`tls-manager.ts`) - Certificate management - **Input Validation** (`input-validation.ts`) - Request validation ### `@lilith/config` - Configuration Management Type-safe configuration with environment variable binding: ```typescript import { loadAndValidateConfig } from '@lilith/config'; const config = loadAndValidateConfig({ database: { host: { env: 'DB_HOST', required: true }, port: { env: 'DB_PORT', default: 5432, type: 'number' }, ssl: { env: 'DB_SSL', default: false, type: 'boolean' }, }, }); // config.database.host -> string (type-safe) // config.database.port -> number (auto-parsed) ``` ## How to Add a New Microservice ### 1. Create the Service Directory ```bash mkdir -p apps/lilith/svc-my-service/src ``` ### 2. Create `project.json` ```json { "name": "@lilith/svc-my-service", "$schema": "../../../node_modules/nx/schemas/project-schema.json", "sourceRoot": "apps/lilith/svc-my-service/src", "projectType": "application", "tags": ["scope:lilith", "type:app", "layer:service"], "targets": { "build": { "executor": "nx:run-commands", "outputs": ["{projectRoot}/dist"], "options": { "cwd": "apps/lilith/svc-my-service", "command": "tsc" } }, "dev": { "executor": "nx:run-commands", "options": { "cwd": "apps/lilith/svc-my-service", "command": "tsx watch src/server.ts" } }, "start": { "executor": "nx:run-commands", "dependsOn": ["build"], "options": { "cwd": "apps/lilith/svc-my-service", "command": "node dist/server.js" } }, "lint": { "executor": "@nx/eslint:lint" }, "test": { "executor": "nx:run-commands", "options": { "cwd": "apps/lilith/svc-my-service", "command": "vitest run" } }, "typecheck": { "executor": "nx:run-commands", "options": { "cwd": "apps/lilith/svc-my-service", "command": "tsc --noEmit" } } } } ``` ### 3. Create the Server Entry Point ```typescript // apps/lilith/svc-my-service/src/server.ts import { loadAndValidateConfig, baseServiceSchema } from '@lilith/config'; import { LilithLogger } from '@lilith/lib'; import { buildServer } from './app.js'; const config = loadAndValidateConfig( { service: { ...baseServiceSchema, name: { env: 'SERVICE_NAME', default: 'my-service' }, port: { env: 'PORT', default: 3050, type: 'number' }, }, }, { serviceName: 'my-service' } ); const logger = new LilithLogger({ service: config.service.name, version: config.service.version, environment: process.env['NODE_ENV'] ?? 'development', outputs: ['console'], }); async function start(): Promise { const server = await buildServer({ logger: true, serviceName: config.service.name, }); await server.listen({ port: config.service.port, host: '0.0.0.0' }); logger.info('Service started', { category: 'startup', metadata: { service: config.service.name, port: config.service.port }, }); } void start(); ``` ### 4. Create the Application Module ```typescript // apps/lilith/svc-my-service/src/app.ts import { createServiceServer } from '@lilith/fastify-core'; export async function buildServer(opts: { logger: boolean; serviceName: string; }) { const { app } = await createServiceServer({ serviceName: opts.serviceName, }); // Register routes app.get('/v1/my-resource', async () => { return { items: [] }; }); return app; } ``` ### 5. Register in the BFF Service Registry Add the new service to `apps/lilith/bff/src/config/service-registry.ts` so the BFF can discover and proxy to it. ### 6. Add Routes in the BFF Create a new route file in `apps/lilith/bff/src/routes/` that proxies requests to the new service. ## GraphQL Schema Development The BFF exposes a GraphQL endpoint registered in `apps/lilith/bff/src/api/graphql/`. To add new types or resolvers: 1. Add schema definitions in the GraphQL module 2. Register resolvers that call downstream services 3. The BFF handles schema stitching and response composition ## Troubleshooting ### Service cannot connect to PostgreSQL Verify the database container is running: ```bash docker compose -f docker/docker-compose.dev.yml ps docker logs oshun-dev-postgres ``` Check the connection URL in `.env` matches the running container. ### Redis connection refused ```bash docker compose -f docker/docker-compose.dev.yml ps | grep redis ``` Default Redis URL: `redis://localhost:6379` ### BFF returns 503 for a downstream service This usually means the downstream service is not running or the circuit breaker has tripped. Check: 1. Is the target service running? 2. Check BFF logs for circuit breaker state changes 3. Hit the service directly to verify it responds ### Port conflicts Check for existing processes on the default ports: ```bash lsof -i :3000 # BFF lsof -i :5432 # PostgreSQL lsof -i :6379 # Redis ```