Lilith is the largest domain in the Oshun monorepo. It is a consciousness and spiritual wellness platform comprising a Backend-for-Frontend (BFF) aggregation layer, dozens of microservices, client applications (web, mobile, desktop), and shared libraries for configuration, logging, service communication, and testing.
Architecture Overview#
Applications (apps/lilith/)#
Core Gateway:
bff/- Backend-for-Frontend (Fastify). The central aggregation gateway that proxies and composes responses from downstream services. Includes GraphQL, REST, and SSE streaming endpoints.
Microservices (selected):
| Service | Description |
|---|---|
svc-auth |
Authentication and authorization |
svc-auth-orchestrator |
Multi-provider auth coordination |
svc-ai |
AI/LLM integration and reasoning |
svc-content |
Content management |
svc-catalog |
Content catalog and discovery |
svc-conversation |
Conversational AI interactions |
svc-curricula |
Learning curricula management |
svc-daily-content |
Daily content scheduling and delivery |
svc-spiritual-guidance |
AI spiritual guidance engine |
svc-meditation-core |
Core meditation session management |
svc-meditation-experience |
Meditation experience orchestration |
svc-meditation-generation |
AI meditation content generation |
svc-yoga-practice |
Yoga practice tracking |
svc-breathwork |
Breathwork session management |
svc-group-meditation |
Group meditation coordination |
svc-journal |
Journaling service |
svc-notes |
Notes management |
svc-review |
Content review workflow |
svc-media |
Media file management |
svc-tts |
Text-to-speech synthesis |
svc-stt |
Speech-to-text transcription |
svc-voice-pipeline |
Voice processing pipeline |
svc-notification |
Push/email notification delivery |
svc-analytics |
Analytics collection and reporting |
svc-moderation |
Content moderation |
svc-sync |
Data synchronization |
svc-offline |
Offline capability management |
svc-webrtc |
WebRTC signaling and real-time communication |
svc-real-time-sync |
Real-time state synchronization |
svc-progress-sync |
Progress tracking synchronization |
svc-user-preferences |
User preference management |
svc-partner-api |
Partner API access |
svc-observability |
Service observability endpoints |
svc-error-handler |
Centralized error handling |
svc-language-detection |
Language detection |
svc-community-translation |
Community translation management |
svc-anonymization |
Data anonymization |
svc-consent-management |
User consent tracking |
svc-data-governance |
Data governance enforcement |
svc-safety-automation |
Automated safety checks |
svc-content-licensing |
Content licensing management |
svc-content-verification |
Content verification |
svc-rights-management |
Digital rights management |
svc-blockchain |
Blockchain integration |
svc-native-token |
Native token operations |
svc-token-core |
Token core logic |
svc-token-access |
Token-based access control |
svc-token-verification |
Token verification |
svc-staking-mechanism |
Token staking |
svc-dao-governance |
DAO governance |
svc-settlement |
Financial settlement |
svc-fiat-ramp |
Fiat on/off ramp |
svc-payment-orchestrator |
Payment workflow orchestration |
svc-tiered-subscription |
Subscription tier management |
svc-micro-transaction |
Micro-transaction processing |
svc-creator-royalty |
Creator royalty distribution |
svc-teacher-blessing |
Teacher endorsement workflow |
svc-metaverse |
Metaverse integration |
svc-avatar-cosmetic |
Avatar cosmetic management |
svc-biometric |
Biometric data processing |
svc-veilborn-core |
Veilborn game core |
svc-veilborn-strategy |
Veilborn game strategy |
svc-indexer |
Blockchain indexer |
svc-ipfs-integration |
IPFS decentralized storage |
svc-cross-chain-bridge |
Cross-chain bridge |
svc-defi-integration |
DeFi protocol integration |
svc-multi-region-resilience |
Multi-region resilience |
svc-operational-excellence |
SRE operational tooling |
Clients:
web/- Next.js web application with Storybookmobile/- React Native mobile applicationdesktop/- Tauri desktop applicationcli/- Command-line interface tool
Other:
contracts/- Solidity smart contracts (Hardhat)seed-corpus/- Seed data ingestionlocales/- i18n translation files
Legacy Lilith knowledge and curation services (svc-rag, svc-knowledge,
svc-knowledge-access, svc-ingestion, svc-vectordb,
svc-spiritual-embeddings, svc-curator-admin) have been retired. Current
grounding, graph, and ingestion flows route through Sophia services.
Libraries (libs/lilith/)#
| Library | Package | Purpose |
|---|---|---|
fastify-core |
@lilith/fastify-core |
Standardized Fastify server factory, health checks, error handling, middleware |
service-lib |
@lilith/service-lib |
Service communication, circuit breakers, config management, auth middleware, WAF, rate limiting, database health, service discovery, graceful shutdown |
config |
@lilith/config |
Type-safe configuration from environment variables |
core |
@lilith/core |
Core types and constants |
common |
@lilith/common |
Common utilities |
errors |
@lilith/errors |
Error types and handling |
logger |
@lilith/logger |
Structured logging (LilithLogger) |
database |
@lilith/database |
Database client and utilities |
cache |
@lilith/cache |
Redis caching layer |
queue |
@lilith/queue |
Job queue (BullMQ-style) |
metrics |
@lilith/metrics |
Prometheus metrics |
tracing |
@lilith/tracing |
OpenTelemetry distributed tracing |
event-publisher |
@lilith/event-publisher |
Event bus publishing |
event-handlers |
@lilith/event-handlers |
Event handler registration |
sdk |
@lilith/sdk |
Platform client SDK |
partner-sdk |
@lilith/partner-sdk |
Partner integration SDK |
testing |
@lilith/testing |
Test utilities and fixtures |
Shared configuration:
eslint-config-lilith- ESLint shared configprettier-config-lilith- Prettier shared configtsconfig-lilith- TypeScript shared config
Prerequisites#
| Tool | Version | Purpose |
|---|---|---|
| Node.js | >= 18.x | Runtime |
| pnpm | >= 8.x | Package manager |
| Docker | >= 24.x | PostgreSQL, Redis, MinIO, Elasticsearch |
| Docker Compose | >= 2.x | Local infrastructure orchestration |
Starting the BFF and Core Services#
1. Start Infrastructure#
From the monorepo root:
docker compose -f docker/docker-compose.dev.yml up -d
This starts PostgreSQL (port 5432), Redis (port 6379), MinIO (ports 9000/9001), and Mailpit (ports 1025/8025).
For additional services used by Lilith:
# Elasticsearch (for search/logging workloads)
docker compose -f docker/docker-compose.dev.yml --profile search up -d
# Vector database (Qdrant)
docker compose -f docker/docker-compose.dev.yml --profile vectors up -d
2. Install Dependencies#
pnpm install
3. Start the BFF#
# Development mode with auto-reload
pnpm nx dev @lilith/bff
# Or production mode
pnpm nx build @lilith/bff
pnpm nx start @lilith/bff
The BFF listens on the port configured by PORT (default: 3000).
4. Start Individual Services#
Each microservice follows the same pattern:
# Development mode
pnpm nx dev @lilith/svc-auth
pnpm nx dev @lilith/svc-content
pnpm nx dev @lilith/svc-ai
# Build + start
pnpm nx build @lilith/svc-content
pnpm nx start @lilith/svc-content
Services that have a dev target use tsx watch for hot reloading. Services
that only have serve use the Nx node executor.
5. Docker Build (BFF)#
pnpm nx docker:build @lilith/bff
Environment Variables#
Configuration uses the @lilith/config library which reads typed values from
environment variables with defaults and validation.
BFF Configuration#
The BFF server uses loadAndValidateConfig from @lilith/config:
const config = loadAndValidateConfig(
{
service: {
...baseServiceSchema,
name: { env: 'SERVICE_NAME', default: 'bff' },
port: { env: 'PORT', default: 3000, type: 'number' },
},
},
{ serviceName: 'bff' }
);
Common Environment Variables#
| Variable | Default | Used By | Description |
|---|---|---|---|
SERVICE_NAME |
(per service) | All services | Service identifier |
PORT |
3000 |
All services | HTTP listen port |
NODE_ENV |
development |
All services | Environment name |
DATABASE_URL |
(see .env) | DB services | PostgreSQL connection string |
REDIS_URL |
redis://localhost:6379 |
Cache services | Redis connection string |
S3_ENDPOINT |
http://localhost:9000 |
Media services | MinIO/S3 endpoint |
S3_ACCESS_KEY |
minioadmin |
Media services | MinIO/S3 access key |
S3_SECRET_KEY |
minioadmin |
Media services | MinIO/S3 secret key |
OPENAI_API_KEY |
(required) | AI services | OpenAI API key |
JWT_SECRET |
(required) | Auth services | JWT signing secret |
The root .env file contains all connection URLs for local development.
Testing#
Running Tests#
# BFF tests
pnpm nx test @lilith/bff
# Specific service tests
pnpm nx test @lilith/svc-auth
pnpm nx test @lilith/svc-yoga-practice
pnpm nx test @lilith/svc-content
# Library tests
pnpm nx test @lilith/fastify-core
pnpm nx test @lilith/service-lib
# Watch mode
pnpm nx test:watch @lilith/bff
# Type checking
pnpm nx typecheck @lilith/bff
Test Framework#
All Lilith services use Vitest. Tests are located in __tests__/ directories
alongside source code or at the project root level.
# Run with coverage
pnpm nx test @lilith/bff -- --coverage
Linting#
pnpm nx lint @lilith/bff
pnpm nx lint @lilith/svc-auth
Service Communication Patterns#
BFF as Aggregation Gateway#
The BFF is the single entry point for all client applications. It:
- Receives client requests (REST, GraphQL, SSE streams)
- Routes to downstream microservices via the service registry
- Aggregates responses from multiple services
- Applies response shaping based on client type (web, mobile, desktop)
- Handles error normalization and circuit breaking
Key BFF components:
- Service Registry (
config/service-registry.ts) - Maps service names to their HTTP endpoints and health check URLs - Circuit Breakers (
utils/circuit-breaker.ts) - Per-service circuit breakers that prevent cascading failures - Response Shaping (
middleware/response-shaping.ts) - Adapts responses based on the client type header - Error Normalization (
error-normalization.ts) - Normalizes errors from different services into a consistent format - Streaming Utils (
streaming-utils.ts) - SSE streaming with backpressure
@lilith/fastify-core - Service Factory#
Every microservice uses @lilith/fastify-core to create a standardized Fastify
server:
import { createServiceServer } from '@lilith/fastify-core';
const { app, start, shutdown } = await createServiceServer({
serviceName: 'my-service',
healthCheckFn: async () => ({
dbConnected: await db.ping(),
cacheSize: cache.size,
}),
});
// Register routes
app.get('/v1/hello', async () => ({ message: 'Hello!' }));
// Start
await start(3000);
This provides:
- Kubernetes-ready health check endpoints (
/health,/ready,/live) - Standardized error handling and response formatting
- Middleware: logging, CORS, request timing, correlation IDs
- Graceful shutdown
@lilith/service-lib - Cross-Cutting Concerns#
The service-lib package provides the operational infrastructure every service
needs:
- Service Client (
service-client.ts) - HTTP client for inter-service communication with automatic retries and circuit breaking - Service Discovery (
service-discovery.ts) - Service endpoint resolution - Service Registry (
service-registry.ts) - Self-registration and discovery - Auth Middleware (
auth-middleware.ts) - JWT validation and permission checking - Rate Limiting (
rate-limiting-middleware.ts) - Per-endpoint rate limiting - WAF (
waf-middleware.ts) - Web Application Firewall rules - Circuit Breaker (
circuit-breaker.ts) - Circuit breaker pattern - Config Manager (
config-manager.ts) - Runtime configuration management - Database Health (
database-health.ts) - Database connectivity monitoring - Graceful Shutdown (
graceful-shutdown.ts) - Clean shutdown with drain - Metrics (
metrics.ts) - Prometheus metric collection - Tracing (
tracing.ts) - OpenTelemetry span management - Structured Logging (
structured-logging.ts) - Log formatting - Bot Detection (
bot-detection-middleware.ts) - Bot traffic detection - PII Detection (
pii-detector.ts) - PII scanning in request data - IP Filtering (
ip-filtering-middleware.ts) - IP allowlist/blocklist - Vault Integration (
vault-client.ts) - HashiCorp Vault secret management - TLS Manager (
tls-manager.ts) - Certificate management - Input Validation (
input-validation.ts) - Request validation
@lilith/config - Configuration Management#
Type-safe configuration with environment variable binding:
import { loadAndValidateConfig } from '@lilith/config';
const config = loadAndValidateConfig({
database: {
host: { env: 'DB_HOST', required: true },
port: { env: 'DB_PORT', default: 5432, type: 'number' },
ssl: { env: 'DB_SSL', default: false, type: 'boolean' },
},
});
// config.database.host -> string (type-safe)
// config.database.port -> number (auto-parsed)
How to Add a New Microservice#
1. Create the Service Directory#
mkdir -p apps/lilith/svc-my-service/src
2. Create project.json#
{
"name": "@lilith/svc-my-service",
"$schema": "../../../node_modules/nx/schemas/project-schema.json",
"sourceRoot": "apps/lilith/svc-my-service/src",
"projectType": "application",
"tags": ["scope:lilith", "type:app", "layer:service"],
"targets": {
"build": {
"executor": "nx:run-commands",
"outputs": ["{projectRoot}/dist"],
"options": {
"cwd": "apps/lilith/svc-my-service",
"command": "tsc"
}
},
"dev": {
"executor": "nx:run-commands",
"options": {
"cwd": "apps/lilith/svc-my-service",
"command": "tsx watch src/server.ts"
}
},
"start": {
"executor": "nx:run-commands",
"dependsOn": ["build"],
"options": {
"cwd": "apps/lilith/svc-my-service",
"command": "node dist/server.js"
}
},
"lint": {
"executor": "@nx/eslint:lint"
},
"test": {
"executor": "nx:run-commands",
"options": {
"cwd": "apps/lilith/svc-my-service",
"command": "vitest run"
}
},
"typecheck": {
"executor": "nx:run-commands",
"options": {
"cwd": "apps/lilith/svc-my-service",
"command": "tsc --noEmit"
}
}
}
}
3. Create the Server Entry Point#
// apps/lilith/svc-my-service/src/server.ts
import { loadAndValidateConfig, baseServiceSchema } from '@lilith/config';
import { LilithLogger } from '@lilith/lib';
import { buildServer } from './app.js';
const config = loadAndValidateConfig(
{
service: {
...baseServiceSchema,
name: { env: 'SERVICE_NAME', default: 'my-service' },
port: { env: 'PORT', default: 3050, type: 'number' },
},
},
{ serviceName: 'my-service' }
);
const logger = new LilithLogger({
service: config.service.name,
version: config.service.version,
environment: process.env['NODE_ENV'] ?? 'development',
outputs: ['console'],
});
async function start(): Promise<void> {
const server = await buildServer({
logger: true,
serviceName: config.service.name,
});
await server.listen({ port: config.service.port, host: '0.0.0.0' });
logger.info('Service started', {
category: 'startup',
metadata: { service: config.service.name, port: config.service.port },
});
}
void start();
4. Create the Application Module#
// apps/lilith/svc-my-service/src/app.ts
import { createServiceServer } from '@lilith/fastify-core';
export async function buildServer(opts: {
logger: boolean;
serviceName: string;
}) {
const { app } = await createServiceServer({
serviceName: opts.serviceName,
});
// Register routes
app.get('/v1/my-resource', async () => {
return { items: [] };
});
return app;
}
5. Register in the BFF Service Registry#
Add the new service to apps/lilith/bff/src/config/service-registry.ts so the
BFF can discover and proxy to it.
6. Add Routes in the BFF#
Create a new route file in apps/lilith/bff/src/routes/ that proxies requests
to the new service.
GraphQL Schema Development#
The BFF exposes a GraphQL endpoint registered in
apps/lilith/bff/src/api/graphql/. To add new types or resolvers:
- Add schema definitions in the GraphQL module
- Register resolvers that call downstream services
- The BFF handles schema stitching and response composition
Troubleshooting#
Service cannot connect to PostgreSQL#
Verify the database container is running:
docker compose -f docker/docker-compose.dev.yml ps
docker logs oshun-dev-postgres
Check the connection URL in .env matches the running container.
Redis connection refused#
docker compose -f docker/docker-compose.dev.yml ps | grep redis
Default Redis URL: redis://localhost:6379
BFF returns 503 for a downstream service#
This usually means the downstream service is not running or the circuit breaker has tripped. Check:
- Is the target service running?
- Check BFF logs for circuit breaker state changes
- Hit the service directly to verify it responds
Port conflicts#
Check for existing processes on the default ports:
lsof -i :3000 # BFF
lsof -i :5432 # PostgreSQL
lsof -i :6379 # Redis