Overview#
This document describes the HashiCorp Vault implementation for centralized secret management in the Lilith platform. Vault provides secure storage, dynamic secrets, encryption as a service, and fine-grained access control.
Architecture#
Components#
- Vault Server: Core secret management service
- Secrets Engines: KV v2, Transit, PKI
- Policies: Role-based access control
- Authentication: Token-based auth for services
- Audit Logging: Comprehensive access logging
Directory Structure#
text
vault/
├── config.hcl # Vault configuration
├── policies/
│ └── policies.hcl # Access control policies
├── scripts/
│ └── setup.sh # Container initialization
├── docker-entrypoint.sh # Custom entrypoint
├── tokens/ # Service tokens
├── backups/ # Backup storage
└── logs/ # Audit logs
Quick Start#
1. Start Vault#
bash
# Start with development environment
docker compose -f docker-compose.dev.yml up -d vault
# Check Vault status
curl http://localhost:8200/v1/sys/health
2. Initialize Vault#
bash
# Run complete setup
./scripts/vault-setup.sh full
# Or step by step
./scripts/vault-setup.sh init
./scripts/vault-setup.sh secrets
./scripts/vault-setup.sh policies
./scripts/vault-setup.sh accounts
3. Access Vault UI#
- URL: http://localhost:8200
- Token: dev-token
- Method: Token
Configuration#
Vault Configuration (config.hcl)#
Key configuration options:
hcl
# UI and API
ui = true
api_addr = "http://0.0.0.0:8200"
# Backend
backend "file" {
path = "/vault/data"
}
# Listener
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = true
}
# Audit logging
audit "file" {
path = "/vault/logs/audit.log"
format = "json"
}
# Lease TTLs
default_lease_ttl = "168h"
max_lease_ttl = "720h"
Environment Variables#
| Variable | Default | Description |
|---|---|---|
VAULT_ADDR |
http://localhost:8200 |
Vault server URL |
VAULT_TOKEN |
dev-token |
Root token for development |
VAULT_MODE |
dev |
Server mode (dev/server) |
VAULT_DEV_ROOT_TOKEN_ID |
dev-token |
Development root token |
Secrets Management#
Secrets Engines#
1. KV v2 (Key-Value)#
General purpose secret storage:
bash
# Store a secret
vault kv put secret/lilith/ai/config \
openai_api_key="sk-..." \
anthropic_api_key="sk-ant-..."
# Read a secret
vault kv get secret/lilith/ai/config
# List secrets
vault kv list secret/lilith/
2. Transit#
Encryption as a service:
bash
# Create encryption key
vault write -f transit/keys/data
# Encrypt data
vault write transit/encrypt/data \
plaintext=$(base64 <<< "sensitive data")
# Decrypt data
vault write transit/decrypt/data \
ciphertext="vault:v1:..."
3. PKI#
Certificate management:
bash
# Generate certificate
vault write pki/issue/lilith-server \
common_name="bff.lilith.local" \
ttl="24h"
Secret Structure#
text
secret/lilith/
├── shared/
│ ├── database # Database credentials
│ ├── jwt # JWT secrets
│ ├── storage # S3/MinIO keys
│ ├── redis # Redis configuration
│ ├── vector-db # Milvus configuration
│ └── external-apis # Third-party API keys
├── bff/ # BFF-specific secrets
├── ai/ # AI service secrets
├── auth/ # Auth service secrets
├── content/ # Content service secrets
├── media/ # Media service secrets
├── notification/ # Notification service secrets
├── moderation/ # Moderation service secrets
├── rag/ # RAG service secrets
├── tts/ # TTS service secrets
└── stt/ # STT service secrets
Access Control#
Policies#
Vault policies define what each service can access:
Example Service Policy (BFF)#
hcl
# Read service-specific secrets
path "secret/lilith/bff/*" {
capabilities = ["read"]
}
# Read shared secrets
path "secret/lilith/shared/jwt" {
capabilities = ["read"]
}
path "secret/lilith/shared/database" {
capabilities = ["read"]
}
Service Accounts#
Each service gets a dedicated token with appropriate permissions:
bash
# Create service token
vault token create -policy=bff -ttl=720h
# Service uses token to access secrets
export VAULT_TOKEN="service-token"
vault kv get secret/lilith/bff/config
Policy Hierarchy#
- admin: Full access to everything
- operator: System management, limited secret access
- service policies: Service-specific secret access
- readonly: Read-only access to shared secrets
Integration with Services#
Service Configuration#
Services are configured to use Vault through environment variables and configuration files:
bash
# .env.vault for each service
VAULT_ADDR=http://vault:8200
VAULT_TOKEN_FILE=/vault/tokens/bff.token
VAULT_SECRET_PATH=secret/lilith/bff
VAULT_SHARED_PATH=secret/lilith/shared
Node.js Integration Example#
javascript
// vault-client.js
const axios = require('axios');
class VaultClient {
constructor(opts = {}) {
this.addr = opts.addr || process.env.VAULT_ADDR;
this.token = opts.token || process.env.VAULT_TOKEN;
this.secretPath = opts.secretPath || process.env.VAULT_SECRET_PATH;
}
async getSecret(path) {
const response = await axios.get(
`${this.addr}/v1/${this.secretPath}/${path}`,
{
headers: { 'X-Vault-Token': this.token },
}
);
return response.data.data.data;
}
async getSharedSecret(path) {
const response = await axios.get(
`${this.addr}/v1/secret/lilith/shared/${path}`,
{
headers: { 'X-Vault-Token': this.token },
}
);
return response.data.data.data;
}
}
module.exports = VaultClient;
Service Usage#
javascript
// In service startup
const VaultClient = require('./vault-client');
const vault = new VaultClient();
async function loadConfig() {
try {
const dbConfig = await vault.getSharedSecret('database');
const jwtConfig = await vault.getSharedSecret('jwt');
return {
database: dbConfig,
jwt: jwtConfig,
};
} catch (error) {
console.error('Failed to load config from Vault:', error);
throw error;
}
}
Security Features#
Encryption#
- Data at Rest: Encrypted storage backend
- Data in Transit: TLS encryption (in production)
- Transit Engine: Encryption as a service
- Key Management: Automatic key rotation
Audit Logging#
All Vault operations are logged:
json
{
"time": "2025-01-06T12:00:00Z",
"type": "request",
"operation": "read",
"path": "secret/data/lilith/shared/database",
"auth": {
"client_token": "hvs.CAES...",
"policies": ["bff"]
}
}
Token Management#
- Service Tokens: Long-lived tokens for services
- Time-to-Live (TTL): Automatic token expiration
- Renewal: Token renewal capabilities
- Revocation: Immediate token invalidation
Operations#
Monitoring#
bash
# Check Vault status
./scripts/vault-setup.sh status
# Monitor health
curl -s http://localhost:8200/v1/sys/health | jq
# Check token usage
vault token lookup -accessor=your-token-accessor
Backup and Recovery#
bash
# Create backup
./scripts/vault-backup.sh
# Manual backup
vault operator raft snapshot save backup-$(date +%Y%m%d).snap
# Restore (raft mode)
vault operator raft snapshot restore backup-20250106.snap
Maintenance#
bash
# Renew token
vault token renew -increment=24h
# Rotate encryption key
vault write -f transit/keys/data/rotate
# Update policies
vault policy write bff vault/policies/bff.hcl
Environment-Specific Configuration#
Development#
yaml
# docker-compose.dev.yml
vault:
environment:
- VAULT_MODE=dev
- VAULT_DEV_ROOT_TOKEN_ID=dev-token
volumes:
- ./vault:/vault
Production#
yaml
# docker-compose.prod.yml
vault:
environment:
- VAULT_MODE=server
- VAULT_ADDR=https://vault.example.com
volumes:
- ./vault/config:/vault/config
- vault-data:/vault/data
Best Practices#
1. Security#
- Use least-privilege policies
- Regular token rotation
- Enable audit logging
- Use TLS in production
- Monitor access patterns
2. Operations#
- Regular backups
- Monitor health and performance
- Test disaster recovery
- Document procedures
- Automate where possible
3. Development#
- Use development mode for local development
- Never commit real secrets to git
- Use environment-specific configurations
- Test secret rotation procedures
- Implement proper error handling
Troubleshooting#
Common Issues#
-
Vault not accessible
bash# Check if running docker ps | grep vault # Check logs docker logs lilith-vault-1 -
Permission denied errors
bash# Check token policies vault token lookup $VAULT_TOKEN # Verify policy syntax vault policy read policy-name -
Secret not found
bash# List available secrets vault kv list secret/ # Check path structure vault kv get secret/path/to/secret
Health Check#
bash
# Comprehensive health check
./scripts/health-monitor.sh utilities
# Vault-specific health
curl -s http://localhost:8200/v1/sys/health | jq '.'
Debug Mode#
bash
# Enable debug logging
export VAULT_LOG_LEVEL=debug
# Check configuration
vault operator init -status
Migration#
From Environment Variables#
-
Create secrets in Vault
bashvault kv put secret/lilith/ai/config \ openai_api_key="$OPENAI_API_KEY" \ anthropic_api_key="$ANTHROPIC_API_KEY" -
Update service configuration
bash# Replace environment variables with Vault client # Remove secret values from docker-compose.yml -
Deploy updated services
bashdocker compose up -d --force-recreate ai-service
Migration Checklist#
- Identify all secret environment variables
- Create corresponding Vault secrets
- Update service configurations
- Implement Vault client in services
- Test secret retrieval
- Remove secrets from environment files
- Update documentation
- Train development team