Eve library

Lilith Domain - Developer Guide

Guide

Revision 1

On this page

Lilith is the largest domain in the Oshun monorepo. It is a consciousness and spiritual wellness platform comprising a Backend-for-Frontend (BFF) aggregation layer, dozens of microservices, client applications (web, mobile, desktop), and shared libraries for configuration, logging, service communication, and testing.

Architecture Overview#

Applications (apps/lilith/)#

Core Gateway:

  • bff/ - Backend-for-Frontend (Fastify). The central aggregation gateway that proxies and composes responses from downstream services. Includes GraphQL, REST, and SSE streaming endpoints.

Microservices (selected):

Service Description
svc-auth Authentication and authorization
svc-auth-orchestrator Multi-provider auth coordination
svc-ai AI/LLM integration and reasoning
svc-content Content management
svc-catalog Content catalog and discovery
svc-conversation Conversational AI interactions
svc-curricula Learning curricula management
svc-daily-content Daily content scheduling and delivery
svc-spiritual-guidance AI spiritual guidance engine
svc-meditation-core Core meditation session management
svc-meditation-experience Meditation experience orchestration
svc-meditation-generation AI meditation content generation
svc-yoga-practice Yoga practice tracking
svc-breathwork Breathwork session management
svc-group-meditation Group meditation coordination
svc-journal Journaling service
svc-notes Notes management
svc-review Content review workflow
svc-media Media file management
svc-tts Text-to-speech synthesis
svc-stt Speech-to-text transcription
svc-voice-pipeline Voice processing pipeline
svc-notification Push/email notification delivery
svc-analytics Analytics collection and reporting
svc-moderation Content moderation
svc-sync Data synchronization
svc-offline Offline capability management
svc-webrtc WebRTC signaling and real-time communication
svc-real-time-sync Real-time state synchronization
svc-progress-sync Progress tracking synchronization
svc-user-preferences User preference management
svc-partner-api Partner API access
svc-observability Service observability endpoints
svc-error-handler Centralized error handling
svc-language-detection Language detection
svc-community-translation Community translation management
svc-anonymization Data anonymization
svc-consent-management User consent tracking
svc-data-governance Data governance enforcement
svc-safety-automation Automated safety checks
svc-content-licensing Content licensing management
svc-content-verification Content verification
svc-rights-management Digital rights management
svc-blockchain Blockchain integration
svc-native-token Native token operations
svc-token-core Token core logic
svc-token-access Token-based access control
svc-token-verification Token verification
svc-staking-mechanism Token staking
svc-dao-governance DAO governance
svc-settlement Financial settlement
svc-fiat-ramp Fiat on/off ramp
svc-payment-orchestrator Payment workflow orchestration
svc-tiered-subscription Subscription tier management
svc-micro-transaction Micro-transaction processing
svc-creator-royalty Creator royalty distribution
svc-teacher-blessing Teacher endorsement workflow
svc-metaverse Metaverse integration
svc-avatar-cosmetic Avatar cosmetic management
svc-biometric Biometric data processing
svc-veilborn-core Veilborn game core
svc-veilborn-strategy Veilborn game strategy
svc-indexer Blockchain indexer
svc-ipfs-integration IPFS decentralized storage
svc-cross-chain-bridge Cross-chain bridge
svc-defi-integration DeFi protocol integration
svc-multi-region-resilience Multi-region resilience
svc-operational-excellence SRE operational tooling

Clients:

  • web/ - Next.js web application with Storybook
  • mobile/ - React Native mobile application
  • desktop/ - Tauri desktop application
  • cli/ - Command-line interface tool

Other:

  • contracts/ - Solidity smart contracts (Hardhat)
  • seed-corpus/ - Seed data ingestion
  • locales/ - i18n translation files

Legacy Lilith knowledge and curation services (svc-rag, svc-knowledge, svc-knowledge-access, svc-ingestion, svc-vectordb, svc-spiritual-embeddings, svc-curator-admin) have been retired. Current grounding, graph, and ingestion flows route through Sophia services.

Libraries (libs/lilith/)#

Library Package Purpose
fastify-core @lilith/fastify-core Standardized Fastify server factory, health checks, error handling, middleware
service-lib @lilith/service-lib Service communication, circuit breakers, config management, auth middleware, WAF, rate limiting, database health, service discovery, graceful shutdown
config @lilith/config Type-safe configuration from environment variables
core @lilith/core Core types and constants
common @lilith/common Common utilities
errors @lilith/errors Error types and handling
logger @lilith/logger Structured logging (LilithLogger)
database @lilith/database Database client and utilities
cache @lilith/cache Redis caching layer
queue @lilith/queue Job queue (BullMQ-style)
metrics @lilith/metrics Prometheus metrics
tracing @lilith/tracing OpenTelemetry distributed tracing
event-publisher @lilith/event-publisher Event bus publishing
event-handlers @lilith/event-handlers Event handler registration
sdk @lilith/sdk Platform client SDK
partner-sdk @lilith/partner-sdk Partner integration SDK
testing @lilith/testing Test utilities and fixtures

Shared configuration:

  • eslint-config-lilith - ESLint shared config
  • prettier-config-lilith - Prettier shared config
  • tsconfig-lilith - TypeScript shared config

Prerequisites#

Tool Version Purpose
Node.js >= 18.x Runtime
pnpm >= 8.x Package manager
Docker >= 24.x PostgreSQL, Redis, MinIO, Elasticsearch
Docker Compose >= 2.x Local infrastructure orchestration

Starting the BFF and Core Services#

1. Start Infrastructure#

From the monorepo root:

bash
docker compose -f docker/docker-compose.dev.yml up -d

This starts PostgreSQL (port 5432), Redis (port 6379), MinIO (ports 9000/9001), and Mailpit (ports 1025/8025).

For additional services used by Lilith:

bash
# Elasticsearch (for search/logging workloads)
docker compose -f docker/docker-compose.dev.yml --profile search up -d

# Vector database (Qdrant)
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d

2. Install Dependencies#

bash
pnpm install

3. Start the BFF#

bash
# Development mode with auto-reload
pnpm nx dev @lilith/bff

# Or production mode
pnpm nx build @lilith/bff
pnpm nx start @lilith/bff

The BFF listens on the port configured by PORT (default: 3000).

4. Start Individual Services#

Each microservice follows the same pattern:

bash
# Development mode
pnpm nx dev @lilith/svc-auth
pnpm nx dev @lilith/svc-content
pnpm nx dev @lilith/svc-ai

# Build + start
pnpm nx build @lilith/svc-content
pnpm nx start @lilith/svc-content

Services that have a dev target use tsx watch for hot reloading. Services that only have serve use the Nx node executor.

5. Docker Build (BFF)#

bash
pnpm nx docker:build @lilith/bff

Environment Variables#

Configuration uses the @lilith/config library which reads typed values from environment variables with defaults and validation.

BFF Configuration#

The BFF server uses loadAndValidateConfig from @lilith/config:

typescript
const config = loadAndValidateConfig(
  {
    service: {
      ...baseServiceSchema,
      name: { env: 'SERVICE_NAME', default: 'bff' },
      port: { env: 'PORT', default: 3000, type: 'number' },
    },
  },
  { serviceName: 'bff' }
);

Common Environment Variables#

Variable Default Used By Description
SERVICE_NAME (per service) All services Service identifier
PORT 3000 All services HTTP listen port
NODE_ENV development All services Environment name
DATABASE_URL (see .env) DB services PostgreSQL connection string
REDIS_URL redis://localhost:6379 Cache services Redis connection string
S3_ENDPOINT http://localhost:9000 Media services MinIO/S3 endpoint
S3_ACCESS_KEY minioadmin Media services MinIO/S3 access key
S3_SECRET_KEY minioadmin Media services MinIO/S3 secret key
OPENAI_API_KEY (required) AI services OpenAI API key
JWT_SECRET (required) Auth services JWT signing secret

The root .env file contains all connection URLs for local development.

Testing#

Running Tests#

bash
# BFF tests
pnpm nx test @lilith/bff

# Specific service tests
pnpm nx test @lilith/svc-auth
pnpm nx test @lilith/svc-yoga-practice
pnpm nx test @lilith/svc-content

# Library tests
pnpm nx test @lilith/fastify-core
pnpm nx test @lilith/service-lib

# Watch mode
pnpm nx test:watch @lilith/bff

# Type checking
pnpm nx typecheck @lilith/bff

Test Framework#

All Lilith services use Vitest. Tests are located in __tests__/ directories alongside source code or at the project root level.

bash
# Run with coverage
pnpm nx test @lilith/bff -- --coverage

Linting#

bash
pnpm nx lint @lilith/bff
pnpm nx lint @lilith/svc-auth

Service Communication Patterns#

BFF as Aggregation Gateway#

The BFF is the single entry point for all client applications. It:

  1. Receives client requests (REST, GraphQL, SSE streams)
  2. Routes to downstream microservices via the service registry
  3. Aggregates responses from multiple services
  4. Applies response shaping based on client type (web, mobile, desktop)
  5. Handles error normalization and circuit breaking

Key BFF components:

  • Service Registry (config/service-registry.ts) - Maps service names to their HTTP endpoints and health check URLs
  • Circuit Breakers (utils/circuit-breaker.ts) - Per-service circuit breakers that prevent cascading failures
  • Response Shaping (middleware/response-shaping.ts) - Adapts responses based on the client type header
  • Error Normalization (error-normalization.ts) - Normalizes errors from different services into a consistent format
  • Streaming Utils (streaming-utils.ts) - SSE streaming with backpressure

@lilith/fastify-core - Service Factory#

Every microservice uses @lilith/fastify-core to create a standardized Fastify server:

typescript
import { createServiceServer } from '@lilith/fastify-core';

const { app, start, shutdown } = await createServiceServer({
  serviceName: 'my-service',
  healthCheckFn: async () => ({
    dbConnected: await db.ping(),
    cacheSize: cache.size,
  }),
});

// Register routes
app.get('/v1/hello', async () => ({ message: 'Hello!' }));

// Start
await start(3000);

This provides:

  • Kubernetes-ready health check endpoints (/health, /ready, /live)
  • Standardized error handling and response formatting
  • Middleware: logging, CORS, request timing, correlation IDs
  • Graceful shutdown

@lilith/service-lib - Cross-Cutting Concerns#

The service-lib package provides the operational infrastructure every service needs:

  • Service Client (service-client.ts) - HTTP client for inter-service communication with automatic retries and circuit breaking
  • Service Discovery (service-discovery.ts) - Service endpoint resolution
  • Service Registry (service-registry.ts) - Self-registration and discovery
  • Auth Middleware (auth-middleware.ts) - JWT validation and permission checking
  • Rate Limiting (rate-limiting-middleware.ts) - Per-endpoint rate limiting
  • WAF (waf-middleware.ts) - Web Application Firewall rules
  • Circuit Breaker (circuit-breaker.ts) - Circuit breaker pattern
  • Config Manager (config-manager.ts) - Runtime configuration management
  • Database Health (database-health.ts) - Database connectivity monitoring
  • Graceful Shutdown (graceful-shutdown.ts) - Clean shutdown with drain
  • Metrics (metrics.ts) - Prometheus metric collection
  • Tracing (tracing.ts) - OpenTelemetry span management
  • Structured Logging (structured-logging.ts) - Log formatting
  • Bot Detection (bot-detection-middleware.ts) - Bot traffic detection
  • PII Detection (pii-detector.ts) - PII scanning in request data
  • IP Filtering (ip-filtering-middleware.ts) - IP allowlist/blocklist
  • Vault Integration (vault-client.ts) - HashiCorp Vault secret management
  • TLS Manager (tls-manager.ts) - Certificate management
  • Input Validation (input-validation.ts) - Request validation

@lilith/config - Configuration Management#

Type-safe configuration with environment variable binding:

typescript
import { loadAndValidateConfig } from '@lilith/config';

const config = loadAndValidateConfig({
  database: {
    host: { env: 'DB_HOST', required: true },
    port: { env: 'DB_PORT', default: 5432, type: 'number' },
    ssl: { env: 'DB_SSL', default: false, type: 'boolean' },
  },
});

// config.database.host -> string (type-safe)
// config.database.port -> number (auto-parsed)

How to Add a New Microservice#

1. Create the Service Directory#

bash
mkdir -p apps/lilith/svc-my-service/src

2. Create project.json#

json
{
  "name": "@lilith/svc-my-service",
  "$schema": "../../../node_modules/nx/schemas/project-schema.json",
  "sourceRoot": "apps/lilith/svc-my-service/src",
  "projectType": "application",
  "tags": ["scope:lilith", "type:app", "layer:service"],
  "targets": {
    "build": {
      "executor": "nx:run-commands",
      "outputs": ["{projectRoot}/dist"],
      "options": {
        "cwd": "apps/lilith/svc-my-service",
        "command": "tsc"
      }
    },
    "dev": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "apps/lilith/svc-my-service",
        "command": "tsx watch src/server.ts"
      }
    },
    "start": {
      "executor": "nx:run-commands",
      "dependsOn": ["build"],
      "options": {
        "cwd": "apps/lilith/svc-my-service",
        "command": "node dist/server.js"
      }
    },
    "lint": {
      "executor": "@nx/eslint:lint"
    },
    "test": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "apps/lilith/svc-my-service",
        "command": "vitest run"
      }
    },
    "typecheck": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "apps/lilith/svc-my-service",
        "command": "tsc --noEmit"
      }
    }
  }
}

3. Create the Server Entry Point#

typescript
// apps/lilith/svc-my-service/src/server.ts
import { loadAndValidateConfig, baseServiceSchema } from '@lilith/config';
import { LilithLogger } from '@lilith/lib';
import { buildServer } from './app.js';

const config = loadAndValidateConfig(
  {
    service: {
      ...baseServiceSchema,
      name: { env: 'SERVICE_NAME', default: 'my-service' },
      port: { env: 'PORT', default: 3050, type: 'number' },
    },
  },
  { serviceName: 'my-service' }
);

const logger = new LilithLogger({
  service: config.service.name,
  version: config.service.version,
  environment: process.env['NODE_ENV'] ?? 'development',
  outputs: ['console'],
});

async function start(): Promise<void> {
  const server = await buildServer({
    logger: true,
    serviceName: config.service.name,
  });

  await server.listen({ port: config.service.port, host: '0.0.0.0' });
  logger.info('Service started', {
    category: 'startup',
    metadata: { service: config.service.name, port: config.service.port },
  });
}

void start();

4. Create the Application Module#

typescript
// apps/lilith/svc-my-service/src/app.ts
import { createServiceServer } from '@lilith/fastify-core';

export async function buildServer(opts: {
  logger: boolean;
  serviceName: string;
}) {
  const { app } = await createServiceServer({
    serviceName: opts.serviceName,
  });

  // Register routes
  app.get('/v1/my-resource', async () => {
    return { items: [] };
  });

  return app;
}

5. Register in the BFF Service Registry#

Add the new service to apps/lilith/bff/src/config/service-registry.ts so the BFF can discover and proxy to it.

6. Add Routes in the BFF#

Create a new route file in apps/lilith/bff/src/routes/ that proxies requests to the new service.

GraphQL Schema Development#

The BFF exposes a GraphQL endpoint registered in apps/lilith/bff/src/api/graphql/. To add new types or resolvers:

  1. Add schema definitions in the GraphQL module
  2. Register resolvers that call downstream services
  3. The BFF handles schema stitching and response composition

Troubleshooting#

Service cannot connect to PostgreSQL#

Verify the database container is running:

bash
docker compose -f docker/docker-compose.dev.yml ps
docker logs oshun-dev-postgres

Check the connection URL in .env matches the running container.

Redis connection refused#

bash
docker compose -f docker/docker-compose.dev.yml ps | grep redis

Default Redis URL: redis://localhost:6379

BFF returns 503 for a downstream service#

This usually means the downstream service is not running or the circuit breaker has tripped. Check:

  1. Is the target service running?
  2. Check BFF logs for circuit breaker state changes
  3. Hit the service directly to verify it responds

Port conflicts#

Check for existing processes on the default ports:

bash
lsof -i :3000  # BFF
lsof -i :5432  # PostgreSQL
lsof -i :6379  # Redis