Eve library

CI/CD Pipeline

Guide

Revision 1

On this page

Comprehensive guide for the Continuous Integration and Continuous Deployment pipeline for the Lilith platform.

Table of Contents#

Overview#

Pipeline Flow#

text
┌─────────────────────────────────────────────────────────────────────────┐
│                           CI/CD Pipeline                                 │
│                                                                          │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐   │
│  │  Push   │──▶│  Lint   │──▶│  Test   │──▶│  Build  │──▶│  Scan   │   │
│  │         │   │  Check  │   │  Suite  │   │  Image  │   │  Image  │   │
│  └─────────┘   └─────────┘   └─────────┘   └─────────┘   └─────────┘   │
│                                                               │          │
│  ┌─────────────────────────────────────────────────────────────┘         │
│  │                                                                       │
│  ▼                                                                       │
│  ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐                  │
│  │  Push   │──▶│ ArgoCD  │──▶│ Staging │──▶│  Prod   │                  │
│  │  ECR    │   │  Sync   │   │ Deploy  │   │ Deploy  │                  │
│  └─────────┘   └─────────┘   └─────────┘   └─────────┘                  │
│                                     │             │                      │
│                              ┌──────▼─────┐ ┌─────▼──────┐              │
│                              │  Smoke     │ │  Approval  │              │
│                              │  Tests     │ │  Required  │              │
│                              └────────────┘ └────────────┘              │
└─────────────────────────────────────────────────────────────────────────┘

Key Principles#

Principle Implementation
Fast Feedback Parallel jobs, early failure, caching
Consistency Docker builds, immutable artifacts
Security Image scanning, secret management, OIDC
Observability Detailed logs, metrics, notifications

Pipeline Architecture#

GitHub Actions Workflows#

Workflow Trigger Purpose
ci.yml Push, PR Lint, test, quality checks
deploy.yml Push to main, tags Build, push, deploy
deploy-promote.yaml Manual Promote between environments
maintenance.yml Schedule Dependency updates, cleanup

Job Dependencies#

yaml
# Typical job flow
prelim-checks ──▶ quality-checks ──▶ unit-tests ──▶ build-images
│                                 │ ▼                                 ▼
security-scan              integration-tests │                                 │
└────────────┬────────────────────┘ ▼ deploy-staging │ ▼ smoke-tests │ ▼
deploy-production

Continuous Integration#

CI Workflow Configuration#

yaml
# .github/workflows/ci.yml
name: Continuous Integration

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]
  schedule:
    - cron: '0 2 * * *' # Nightly builds

env:
  NODE_VERSION: '18'
  CACHE_VERSION: 'v1'

jobs:
  prelim-checks:
    name: Preliminary Checks
    runs-on: ubuntu-latest
    outputs:
      affected-services: ${{ steps.detect-changes.outputs.services }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}
          cache: 'npm'

      - name: Cache dependencies
        uses: actions/cache@v4
        with:
          path: |
            node_modules
            services/*/node_modules
          key:
            ${{ env.CACHE_VERSION }}-deps-${{ hashFiles('**/package-lock.json')
            }}

      - name: Install dependencies
        run: npm ci

      - name: Detect changed services
        id: detect-changes
        run: |
          CHANGED=$(git diff --name-only ${{ github.event.pull_request.base.sha }}..${{ github.sha }} | \
            grep -E '^services/[^/]+/' | cut -d'/' -f2 | sort -u | tr '\n' ',')
          echo "services=${CHANGED%,}" >> $GITHUB_OUTPUT

Quality Checks#

yaml
quality-checks:
  name: Quality & Security
  needs: prelim-checks
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Run ESLint
      run: npm run lint

    - name: Check TypeScript
      run: npm run typecheck

    - name: Check formatting
      run: npm run format:check

    - name: Run security audit
      run: npm audit --audit-level=high

    - name: Check for secrets
      uses: trufflesecurity/trufflehog@main
      with:
        path: ./
        base: ${{ github.event.repository.default_branch }}

Unit Tests#

yaml
unit-tests:
  name: Unit Tests
  needs: prelim-checks
  runs-on: ubuntu-latest
  strategy:
    matrix:
      service: [auth, bff, content, conversation, media, ai]
  steps:
    - uses: actions/checkout@v4

    - name: Setup Node.js
      uses: actions/setup-node@v4
      with:
        node-version: ${{ env.NODE_VERSION }}

    - name: Run tests
      run: |
        cd services/${{ matrix.service }}
        npm ci
        npm test -- --coverage

    - name: Upload coverage
      uses: codecov/codecov-action@v3
      with:
        files: services/${{ matrix.service }}/coverage/lcov.info
        flags: ${{ matrix.service }}

Integration Tests#

yaml
integration-tests:
  name: Integration Tests
  needs: [unit-tests, build-images]
  runs-on: ubuntu-latest
  services:
    postgres:
      image: postgres:16
      env:
        POSTGRES_PASSWORD: test
      options: >-
        --health-cmd pg_isready --health-interval 10s
    redis:
      image: redis:7
      options: >-
        --health-cmd "redis-cli ping"
  steps:
    - uses: actions/checkout@v4

    - name: Run integration tests
      run: npm run test:integration
      env:
        DATABASE_URL: postgresql://postgres:test@localhost/test
        REDIS_URL: redis://localhost:6379

Continuous Deployment#

Deployment Workflow#

yaml
# .github/workflows/deploy.yml
name: Continuous Deployment

on:
  push:
    branches: [main]
    tags: ['v*.*.*']
  workflow_dispatch:
    inputs:
      environment:
        description: 'Target environment'
        required: true
        type: choice
        options: [staging, production]
      service:
        description: 'Service to deploy (empty for all)'
        required: false

env:
  AWS_REGION: us-east-1
  ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com

jobs:
  build-and-push:
    name: Build & Push
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service:
          [
            auth,
            bff,
            content,
            conversation,
            media,
            ai,
            notification,
            moderation,
          ]
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume:
            arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: services/${{ matrix.service }}/Dockerfile
          push: true
          tags: |
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Scan image
        run: |
          aws ecr wait image-scan-complete \
            --repository-name lilith/${{ matrix.service }} \
            --image-id imageTag=${{ github.sha }}

Deployment Strategies#

Strategy Use Case Rollback Time
Rolling Standard deployments 2-5 minutes
Blue-Green Zero-downtime critical Instant
Canary High-risk changes Gradual

Rolling Deployment#

yaml
deploy-staging:
  name: Deploy to Staging
  needs: build-and-push
  runs-on: ubuntu-latest
  environment:
    name: staging
    url: https://api.staging.lilith.app
  steps:
    - name: Update ArgoCD application
      run: |
        argocd app set lilith-staging \
          --helm-set image.tag=${{ github.sha }}
        argocd app sync lilith-staging --prune
        argocd app wait lilith-staging --health

Production Deployment#

yaml
deploy-production:
  name: Deploy to Production
  needs: [deploy-staging, smoke-tests]
  runs-on: ubuntu-latest
  environment:
    name: production
    url: https://api.lilith.app
  steps:
    - name: Deploy to production
      run: |
        argocd app set lilith-production \
          --helm-set image.tag=${{ github.sha }}
        argocd app sync lilith-production --prune
        argocd app wait lilith-production --health

    - name: Notify on success
      uses: slackapi/slack-github-action@v1
      with:
        channel-id: ${{ secrets.SLACK_DEPLOY_CHANNEL }}
        payload: |
          {
            "text": "✅ Production deployment successful",
            "blocks": [
              {
                "type": "section",
                "text": {
                  "type": "mrkdwn",
                  "text": "*Deployment Complete*\nVersion: `${{ github.sha }}`\nEnvironment: Production"
                }
              }
            ]
          }

Environment Strategy#

Environment Configuration#

Environment Branch Auto-Deploy Approval
Development develop Yes No
Staging main Yes No
Production tags (v*) No Required

Environment Protection#

yaml
# GitHub Environment settings
environments:
  staging:
    deployment_branch_policy:
      protected_branches: true
    reviewers: []

  production:
    deployment_branch_policy:
      protected_branches: true
    reviewers:
      - platform-team
    wait_timer: 5 # 5 minutes

Environment Variables#

yaml
# Per-environment configuration
env:
  staging:
    REPLICAS: 2
    LOG_LEVEL: debug
    FEATURE_FLAGS: experimental

  production:
    REPLICAS: 5
    LOG_LEVEL: info
    FEATURE_FLAGS: stable

GitOps with ArgoCD#

Application Definition#

yaml
# deploy/argocd/apps/lilith-staging.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: lilith-staging
  namespace: argocd
spec:
  project: lilith
  source:
    repoURL: https://github.com/org/lilith
    targetRevision: HEAD
    path: deploy/helm/lilith-service
    helm:
      valueFiles:
        - values.yaml
        - values-staging.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: lilith-staging
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

ApplicationSet for Multiple Services#

yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: lilith-services
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - service: auth
          - service: bff
          - service: content
          - service: ai
  template:
    metadata:
      name: 'lilith-{{service}}-staging'
    spec:
      project: lilith
      source:
        repoURL: https://github.com/org/lilith
        path: 'deploy/helm/lilith-service'
        helm:
          values: |
            service:
              name: {{service}}
            image:
              repository: ECR_REGISTRY/lilith/{{service}}
      destination:
        server: https://kubernetes.default.svc
        namespace: lilith-staging

Sync Waves#

yaml
# Control deployment order
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"  # Deploy first

# Dependencies
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"  # Deploy after wave 1

Secrets Management#

GitHub Secrets#

Secret Purpose Scope
AWS_ACCOUNT_ID ECR registry Organization
SLACK_WEBHOOK Notifications Repository
ARGOCD_TOKEN ArgoCD API Environment

OIDC Authentication#

yaml
# No long-lived credentials needed
- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume:
      arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
    aws-region: us-east-1

IAM Role for GitHub Actions#

hcl
resource "aws_iam_role" "github_actions" {
  name = "github-actions-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:oidc-provider/token.actions.githubusercontent.com"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringEquals = {
            "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com"
          }
          StringLike = {
            "token.actions.githubusercontent.com:sub" = "repo:your-org/lilith:*"
          }
        }
      }
    ]
  })
}

Rollback Procedures#

Quick Rollback#

bash
# Rollback to previous version
argocd app history lilith-production
argocd app rollback lilith-production <REVISION>

# Or via Helm
helm rollback lilith-auth -n lilith

Automated Rollback#

yaml
# ArgoCD sync with rollback on failure
syncPolicy:
  automated:
    prune: true
    selfHeal: true
  retry:
    limit: 3
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 3m

Emergency Rollback Script#

bash
#!/bin/bash
# scripts/emergency-rollback.sh

SERVICE=$1
ENVIRONMENT=${2:-production}
PREVIOUS_TAG=$(aws ecr describe-images \
  --repository-name lilith/${SERVICE} \
  --query 'sort_by(imageDetails,& imagePushedAt)[-2].imageTags[0]' \
  --output text)

echo "Rolling back ${SERVICE} to ${PREVIOUS_TAG}"

argocd app set lilith-${SERVICE}-${ENVIRONMENT} \
  --helm-set image.tag=${PREVIOUS_TAG}
argocd app sync lilith-${SERVICE}-${ENVIRONMENT}

Monitoring & Alerting#

Deployment Metrics#

yaml
# Prometheus metrics for deployments
- name: deployment_status
  help: 'Deployment status (1=success, 0=failure)'
  type: gauge
  labels:
    - service
    - environment
    - version

- name: deployment_duration_seconds
  help: 'Deployment duration in seconds'
  type: histogram
  labels:
    - service
    - environment

Slack Notifications#

yaml
- name: Notify deployment
  uses: slackapi/slack-github-action@v1
  with:
    channel-id: deployments
    payload: |
      {
        "blocks": [
          {
            "type": "section",
            "text": {
              "type": "mrkdwn",
              "text": "${{ job.status == 'success' && '✅' || '❌' }} *${{ github.repository }}*"
            }
          },
          {
            "type": "context",
            "elements": [
              {
                "type": "mrkdwn",
                "text": "Environment: `${{ env.ENVIRONMENT }}`"
              },
              {
                "type": "mrkdwn",
                "text": "Version: `${{ github.sha }}`"
              }
            ]
          }
        ]
      }

Failure Alerts#

yaml
- name: Alert on failure
  if: failure()
  uses: slackapi/slack-github-action@v1
  with:
    channel-id: platform-alerts
    payload: |
      {
        "text": "🚨 Deployment failed!",
        "attachments": [
          {
            "color": "danger",
            "fields": [
              {"title": "Service", "value": "${{ matrix.service }}", "short": true},
              {"title": "Environment", "value": "${{ env.ENVIRONMENT }}", "short": true},
              {"title": "Commit", "value": "${{ github.sha }}", "short": true},
              {"title": "Author", "value": "${{ github.actor }}", "short": true}
            ]
          }
        ]
      }

Troubleshooting#

Common Issues#

Issue Cause Solution
Build timeout Large images Enable caching, use multi-stage
Test failures Flaky tests Add retries, isolate tests
Deploy stuck ArgoCD sync issue Check ArgoCD logs, manual sync
Permission denied Missing IAM Update OIDC trust policy

Debug Commands#

bash
# Check workflow runs
gh run list --workflow=ci.yml

# View workflow logs
gh run view <RUN_ID> --log

# Re-run failed jobs
gh run rerun <RUN_ID> --failed

# Check ArgoCD sync status
argocd app get lilith-staging
argocd app sync lilith-staging --dry-run