# Deployment Procedures

**Purpose**: Safe deployment practices for Lilith microservices using GitOps and
rolling updates

## Deployment Architecture Overview

### GitOps Workflow

```
Developer → Git Push → CI Pipeline → Image Build → GitOps Repo Update → ArgoCD → Kubernetes Deployment
    ↓           ↓           ↓             ↓              ↓              ↓              ↓
  Feature    Trigger     Tests        Container       Manifest      Sync         Rolling
   Branch    Build      Pass         Registry        Update       Deploy        Update
```

### Deployment Environments

```
Development → Staging → Production
     ↓           ↓           ↓
  Feature    Integration  Production
  Testing    Testing      Workloads
     ↓           ↓           ↓
  Immediate  Automated    Manual
  Deploy     Deploy       Approval
```

## Pre-Deployment Checklist

### Code Review Requirements

- [ ] Code review approved by at least 2 engineers
- [ ] All tests passing (unit, integration, e2e)
- [ ] Security scan completed without high/critical issues
- [ ] Documentation updated for API changes
- [ ] Database migrations tested in staging

### Infrastructure Readiness

- [ ] Target environment healthy (all services green)
- [ ] Resource capacity sufficient for new deployment
- [ ] Database migrations completed if required
- [ ] External dependencies verified
- [ ] Monitoring and alerting configured

### Deployment Planning

- [ ] Deployment window scheduled (avoid peak hours)
- [ ] Rollback plan documented and tested
- [ ] Feature flags configured for gradual rollout
- [ ] Communication plan for user-facing changes
- [ ] On-call engineer available during deployment

## Deployment Types and Procedures

### Standard Rolling Deployment

#### For Stateless Services (BFF, Auth, Conversation)

```bash
# 1. Verify current deployment health
kubectl get deployments -n lilith-bff
kubectl rollout status deployment bff-deployment -n lilith-bff

# 2. Update image tag in ArgoCD application
argocd app set lilith-bff --parameter image.tag=v2.1.3

# 3. Sync deployment
argocd app sync lilith-bff

# 4. Monitor rollout progress
kubectl rollout status deployment bff-deployment -n lilith-bff --timeout=600s

# 5. Verify health after deployment
curl -f https://bff.lilith.com/v1/health
./scripts/post-deploy-validation.sh bff
```

#### Rolling Update Configuration

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bff-deployment
  namespace: lilith-bff
spec:
  replicas: 6
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1 # Only 1 pod down at a time
      maxSurge: 2 # Up to 2 extra pods during update
  template:
    spec:
      containers:
        - name: bff
          image: lilith/bff:v2.1.3
          readinessProbe:
            httpGet:
              path: /v1/health
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /v1/health
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 30
```

### Blue-Green Deployment

#### For Critical Services Requiring Zero Downtime

```bash
# 1. Deploy to green environment (inactive)
kubectl apply -f manifests/bff-green-deployment.yaml

# 2. Wait for green deployment to be ready
kubectl wait --for=condition=Available deployment bff-green-deployment -n lilith-bff --timeout=600s

# 3. Run validation tests against green environment
./scripts/validate-green-deployment.sh bff

# 4. Switch traffic to green (update service selector)
kubectl patch service bff-service -n lilith-bff \
  --patch='{"spec":{"selector":{"version":"green"}}}'

# 5. Monitor for any issues
sleep 300
./scripts/monitor-deployment-health.sh bff 300

# 6. If successful, scale down blue deployment
kubectl scale deployment bff-blue-deployment --replicas=0 -n lilith-bff

# 7. If issues detected, rollback immediately
# kubectl patch service bff-service -n lilith-bff \
#   --patch='{"spec":{"selector":{"version":"blue"}}}'
```

### Canary Deployment

#### For Gradual Feature Rollouts

```bash
# 1. Deploy canary version (5% traffic)
kubectl apply -f - <<EOF
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: bff-rollout
  namespace: lilith-bff
spec:
  replicas: 10
  strategy:
    canary:
      steps:
      - setWeight: 5
      - pause: {duration: 10m}
      - setWeight: 20
      - pause: {duration: 10m}
      - setWeight: 50
      - pause: {duration: 10m}
      - setWeight: 100
      canaryService: bff-canary-service
      stableService: bff-stable-service
      trafficRouting:
        istio:
          virtualService:
            name: bff-vs
            routes:
            - primary
  selector:
    matchLabels:
      app: bff
  template:
    spec:
      containers:
      - name: bff
        image: lilith/bff:v2.1.3-canary
EOF

# 2. Monitor canary metrics
kubectl argo rollouts get rollout bff-rollout -n lilith-bff --watch

# 3. Promote if metrics look good, or abort if issues
# kubectl argo rollouts promote bff-rollout -n lilith-bff
# kubectl argo rollouts abort bff-rollout -n lilith-bff
```

## Database Migrations

### Safe Migration Procedures

#### Pre-Migration Validation

```bash
# 1. Backup database before migration
kubectl exec -it postgres-primary-0 -n database -- \
  pg_dump lilith > /backups/pre-migration-$(date +%Y%m%d-%H%M).sql

# 2. Test migration on staging database
kubectl exec -it postgres-staging-0 -n staging -- \
  psql lilith < migrations/v2.1.3-migration.sql

# 3. Validate migration results
kubectl exec -it postgres-staging-0 -n staging -- \
  psql lilith -c "SELECT count(*) FROM new_table;"

# 4. Test application compatibility with new schema
./scripts/test-app-with-new-schema.sh staging
```

#### Production Migration Execution

```bash
# 1. Put application in maintenance mode (if required)
kubectl scale deployment --replicas=0 -n lilith-bff
kubectl apply -f manifests/maintenance-page.yaml

# 2. Run migration with monitoring
kubectl exec -it postgres-primary-0 -n database -- \
  psql lilith < migrations/v2.1.3-migration.sql | tee migration.log

# 3. Validate migration success
kubectl exec -it postgres-primary-0 -n database -- \
  psql lilith -c "\d+ new_table;"

# 4. Deploy application with new schema support
kubectl scale deployment bff-deployment --replicas=6 -n lilith-bff
kubectl wait --for=condition=Available deployment bff-deployment -n lilith-bff --timeout=600s

# 5. Remove maintenance mode
kubectl delete -f manifests/maintenance-page.yaml
```

### Backward-Compatible Migration Pattern

```sql
-- Phase 1: Add new column (nullable)
ALTER TABLE users ADD COLUMN new_feature_flag BOOLEAN DEFAULT NULL;

-- Deploy application version that can handle both old and new schema

-- Phase 2: Backfill data (in next deployment)
UPDATE users SET new_feature_flag = false WHERE new_feature_flag IS NULL;

-- Phase 3: Make column NOT NULL (in subsequent deployment)
ALTER TABLE users ALTER COLUMN new_feature_flag SET NOT NULL;
ALTER TABLE users ALTER COLUMN new_feature_flag SET DEFAULT false;
```

## Rollback Procedures

### Automated Rollback Triggers

```bash
#!/bin/bash
# Automated rollback script based on health metrics

SERVICE=$1
NAMESPACE="lilith-${SERVICE}"
ROLLBACK_THRESHOLD_ERROR_RATE=5.0
ROLLBACK_THRESHOLD_RESPONSE_TIME=5000

# Get current error rate and response time
error_rate=$(kubectl exec -n monitoring prometheus-0 -- \
  promtool query instant "rate(http_requests_total{job=\"$SERVICE\",status=~\"5..\"}[5m]) * 100")

response_time=$(kubectl exec -n monitoring prometheus-0 -- \
  promtool query instant "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"$SERVICE\"}[5m])) * 1000")

# Check if rollback is needed
if (( $(echo "$error_rate > $ROLLBACK_THRESHOLD_ERROR_RATE" | bc -l) )) || \
   (( $(echo "$response_time > $ROLLBACK_THRESHOLD_RESPONSE_TIME" | bc -l) )); then

  echo "Rollback triggered for $SERVICE: error_rate=$error_rate%, response_time=${response_time}ms"

  # Execute rollback
  kubectl rollout undo deployment ${SERVICE}-deployment -n $NAMESPACE

  # Wait for rollback to complete
  kubectl rollout status deployment ${SERVICE}-deployment -n $NAMESPACE --timeout=300s

  # Notify team
  ./scripts/send-alert.sh "🔄 Automatic rollback executed for $SERVICE due to performance degradation"

  # Create incident for investigation
  ./scripts/create-incident.sh "Automatic rollback: $SERVICE performance degradation"
else
  echo "No rollback needed for $SERVICE: error_rate=$error_rate%, response_time=${response_time}ms"
fi
```

### Manual Rollback Process

```bash
# 1. Identify current and previous versions
kubectl rollout history deployment bff-deployment -n lilith-bff

# 2. Rollback to previous version
kubectl rollout undo deployment bff-deployment -n lilith-bff

# 3. Or rollback to specific revision
kubectl rollout undo deployment bff-deployment --to-revision=3 -n lilith-bff

# 4. Monitor rollback progress
kubectl rollout status deployment bff-deployment -n lilith-bff

# 5. Verify service health after rollback
curl -f https://bff.lilith.com/v1/health
./scripts/post-rollback-validation.sh bff

# 6. Update ArgoCD to match rolled-back state
argocd app set lilith-bff --parameter image.tag=v2.1.2
```

## Feature Flag Management

### Feature Flag Deployment Pattern

```bash
# 1. Deploy new feature with flag disabled
kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: feature-flags
  namespace: lilith-bff
data:
  NEW_CHAT_INTERFACE: "false"
  ADVANCED_SEARCH: "false"
  VOICE_CHAT_V2: "false"
EOF

# 2. Deploy application code that checks feature flags
kubectl set image deployment/bff-deployment bff=lilith/bff:v2.1.3-with-flags -n lilith-bff

# 3. Gradually enable feature for user segments
kubectl patch configmap feature-flags -n lilith-bff \
  --patch='{"data":{"NEW_CHAT_INTERFACE":"true:beta-users"}}'

# 4. Monitor feature performance
./scripts/monitor-feature-flags.sh NEW_CHAT_INTERFACE

# 5. Full rollout if metrics are good
kubectl patch configmap feature-flags -n lilith-bff \
  --patch='{"data":{"NEW_CHAT_INTERFACE":"true"}}'

# 6. Emergency disable if issues
kubectl patch configmap feature-flags -n lilith-bff \
  --patch='{"data":{"NEW_CHAT_INTERFACE":"false"}}'
```

## Deployment Monitoring and Validation

### Post-Deployment Validation Script

```bash
#!/bin/bash
# Comprehensive post-deployment validation

SERVICE=$1
NAMESPACE="lilith-${SERVICE}"

echo "Starting post-deployment validation for $SERVICE..."

# 1. Check pod health
echo "Checking pod health..."
kubectl get pods -n $NAMESPACE -l app=$SERVICE
if [ $(kubectl get pods -n $NAMESPACE -l app=$SERVICE --field-selector=status.phase=Running --no-headers | wc -l) -eq 0 ]; then
  echo "❌ No running pods found for $SERVICE"
  exit 1
fi

# 2. Check service endpoints
echo "Checking service endpoints..."
kubectl get endpoints -n $NAMESPACE
if [ $(kubectl get endpoints $SERVICE-service -n $NAMESPACE -o jsonpath='{.subsets[0].addresses}' | jq length) -eq 0 ]; then
  echo "❌ No service endpoints available for $SERVICE"
  exit 1
fi

# 3. Health check endpoint
echo "Testing health endpoint..."
if ! curl -f https://$SERVICE.lilith.com/v1/health > /dev/null 2>&1; then
  echo "❌ Health endpoint check failed for $SERVICE"
  exit 1
fi

# 4. Functional tests
echo "Running functional tests..."
if ! ./tests/functional-tests-$SERVICE.sh; then
  echo "❌ Functional tests failed for $SERVICE"
  exit 1
fi

# 5. Performance baseline check
echo "Checking performance metrics..."
response_time=$(curl -w "%{time_total}" -s -o /dev/null https://$SERVICE.lilith.com/v1/health)
if (( $(echo "$response_time > 2.0" | bc -l) )); then
  echo "⚠️ Response time higher than expected: ${response_time}s"
fi

# 6. Check for new errors
echo "Checking for deployment-related errors..."
error_count=$(kubectl logs -n $NAMESPACE -l app=$SERVICE --since=10m | grep -i error | wc -l)
if [ $error_count -gt 10 ]; then
  echo "⚠️ High error count detected: $error_count errors in last 10 minutes"
fi

echo "✅ Post-deployment validation completed successfully for $SERVICE"
```

### Deployment Metrics Dashboard

```bash
#!/bin/bash
# Create deployment metrics dashboard

create_deployment_dashboard() {
  local service=$1

  curl -X POST http://grafana.lilith.com/api/dashboards/db \
    -H "Content-Type: application/json" \
    -d '{
      "dashboard": {
        "title": "'$service' Deployment Metrics",
        "panels": [
          {
            "title": "Response Time",
            "type": "graph",
            "targets": [
              {
                "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\"'$service'\"}[5m]))"
              }
            ]
          },
          {
            "title": "Error Rate",
            "type": "graph",
            "targets": [
              {
                "expr": "rate(http_requests_total{job=\"'$service'\",status=~\"5..\"}[5m]) * 100"
              }
            ]
          },
          {
            "title": "Pod Restarts",
            "type": "graph",
            "targets": [
              {
                "expr": "rate(kube_pod_container_status_restarts_total{pod=~\"'$service'-.*\"}[5m])"
              }
            ]
          }
        ]
      }
    }'
}

# Create dashboards for all services
for service in bff auth conversation content media ai; do
  create_deployment_dashboard $service
done
```

## Deployment Automation

### CI/CD Pipeline Integration

```yaml
# .github/workflows/deploy.yml
name: Deploy to Production

on:
  push:
    tags:
      - 'v*'

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

      - name: Run security scan
        run: |
          trivy fs --severity HIGH,CRITICAL .

      - name: Build and push image
        run: |
          docker build -t lilith/${{ github.event.repository.name }}:${{ github.ref_name }} .
          docker push lilith/${{ github.event.repository.name }}:${{ github.ref_name }}

      - name: Update GitOps repo
        run: |
          git clone https://github.com/lilith/gitops-manifests.git
          cd gitops-manifests
          yq eval '.spec.template.spec.containers[0].image = "lilith/${{ github.event.repository.name }}:${{ github.ref_name }}"' \
            -i ${{ github.event.repository.name }}/deployment.yaml
          git commit -am "Deploy ${{ github.event.repository.name }}:${{ github.ref_name }}"
          git push

      - name: Wait for ArgoCD sync
        run: |
          argocd app wait lilith-${{ github.event.repository.name }} --timeout 600

      - name: Run post-deployment tests
        run: |
          ./scripts/post-deploy-validation.sh ${{ github.event.repository.name }}

      - name: Notify team
        if: failure()
        run: |
          ./scripts/send-alert.sh "❌ Deployment failed for ${{ github.event.repository.name }}:${{ github.ref_name }}"
```

---

## Deployment Troubleshooting Guide

### Common Issues and Solutions

#### 1. Container Fails to Start

**Symptoms:**

- Pod in `CrashLoopBackOff` or `Error` state
- Service health check fails immediately

**Diagnosis:**

```bash
# Check pod events
kubectl describe pod <pod-name> -n <namespace>

# View container logs
kubectl logs <pod-name> -n <namespace> --previous

# Check for resource constraints
kubectl top pods -n <namespace>
```

**Common Causes and Fixes:** | Cause | Solution | |-------|----------| | Missing
environment variables | Check ConfigMaps and Secrets are mounted | | Port
already in use | Verify port conflicts in deployment spec | | Insufficient
memory/CPU | Increase resource limits in deployment | | Failed health probe |
Check probe paths and initial delays |

#### 2. Database Connection Failures

**Symptoms:**

- Service starts but returns 500 errors
- Logs show connection refused or timeout

**Diagnosis:**

```bash
# Check database service is running
kubectl get svc -n lilith-databases

# Test connectivity from pod
kubectl exec -it <pod> -- nc -z postgres 5432

# Check connection pool status
kubectl exec -it <pod> -- curl localhost:8080/metrics | grep db_pool
```

**Fixes:**

```bash
# Restart database if needed (CAUTION: may cause brief downtime)
kubectl rollout restart statefulset postgres -n lilith-databases

# Check database credentials
kubectl get secret db-credentials -n <namespace> -o yaml

# Verify network policies allow traffic
kubectl get networkpolicy -n <namespace>
```

#### 3. Service-to-Service Communication Failures

**Symptoms:**

- Intermittent 503 errors
- Timeouts on internal service calls

**Diagnosis:**

```bash
# Check service discovery
kubectl get endpoints <service-name> -n <namespace>

# Test internal DNS resolution
kubectl exec -it <pod> -- nslookup <service>.<namespace>.svc.cluster.local

# Check network policies
kubectl describe networkpolicy -n <namespace>
```

**Fixes:**

- Verify service selectors match pod labels
- Check if circuit breaker is open (see logs for circuit breaker state)
- Increase timeout values if services are under load

#### 4. Memory/CPU Issues

**Symptoms:**

- Pod OOMKilled
- Slow response times
- Throttling warnings in metrics

**Diagnosis:**

```bash
# Check resource usage
kubectl top pods -n <namespace>

# Check limits vs requests
kubectl describe pod <pod-name> -n <namespace> | grep -A 5 "Limits\|Requests"

# Check for memory leaks
kubectl exec -it <pod> -- curl localhost:8080/metrics | grep nodejs_heap
```

**Fixes:**

```yaml
# Increase resources in deployment
resources:
  requests:
    memory: '512Mi'
    cpu: '250m'
  limits:
    memory: '1Gi'
    cpu: '500m'
```

#### 5. Docker Compose (Local Development) Issues

**Symptoms:**

- Services fail to start locally
- Port conflicts
- Volume mounting issues

**Common Fixes:**

```bash
# Clean up and restart
docker compose -f docker-compose.dev.yml down -v
docker system prune -f
docker compose -f docker-compose.dev.yml up -d --build

# Port conflict - find what's using the port
lsof -i :3000

# Volume permissions
sudo chown -R $(id -u):$(id -g) ./volumes

# Clear npm cache if node_modules issues
rm -rf node_modules
npm ci
```

#### 6. Database Migrations Fail

**Symptoms:**

- Service starts but throws migration errors
- Schema mismatch errors

**Diagnosis:**

```bash
# Check migration status
docker exec -it lilith-postgres-1 psql -U lilith -d lilith_dev -c "SELECT * FROM schema_migrations;"

# View migration logs
kubectl logs <pod-name> -n <namespace> | grep migration
```

**Fixes:**

```bash
# Rollback last migration (if safe)
npm run migrate:rollback

# Reset migrations (CAUTION: data loss in dev only)
npm run migrate:reset

# Manual fix in database
docker exec -it lilith-postgres-1 psql -U lilith -d lilith_dev
```

### Emergency Contacts

| Role             | Contact          | When to Use            |
| ---------------- | ---------------- | ---------------------- |
| On-Call Engineer | PagerDuty        | Production issues      |
| Database Admin   | #database-team   | Data integrity issues  |
| Security Team    | #security-oncall | Security incidents     |
| Platform Lead    | @platform-lead   | Architecture decisions |

### Quick Reference Commands

```bash
# Full stack restart (local)
npm run deploy:restart

# View all service logs
npm run deploy:logs

# Health check all services
npm run health:check

# Force rebuild single service
npm run build:service -- <service-name>
```

---

**Remember**: Deployments should be boring and predictable. Always have a
rollback plan, monitor actively during deployments, and validate thoroughly
before considering a deployment complete.
