# Container Strategy

Comprehensive guide for containerization and container registry management in
the Lilith platform.

## Table of Contents

- [Overview](#overview)
- [Base Images](#base-images)
- [Dockerfile Standards](#dockerfile-standards)
- [Multi-Stage Builds](#multi-stage-builds)
- [Image Registry (ECR)](#image-registry-ecr)
- [Image Scanning](#image-scanning)
- [Tagging Strategy](#tagging-strategy)
- [Build Pipeline](#build-pipeline)
- [Best Practices](#best-practices)

## Overview

The Lilith platform uses containerized microservices deployed to Amazon EKS. All
services follow consistent containerization patterns for security, efficiency,
and maintainability.

### Container Architecture

```
┌─────────────────────────────────────────────────────────────────────┐
│                        Container Registry (ECR)                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐                 │
│  │ lilith/auth │  │ lilith/bff  │  │ lilith/ai   │  ...            │
│  │  :v1.2.3    │  │  :v1.2.3    │  │  :v1.2.3    │                 │
│  └─────────────┘  └─────────────┘  └─────────────┘                 │
└─────────────────────────────────────────────────────────────────────┘
                              │
                    ┌─────────▼─────────┐
                    │   EKS Cluster     │
                    │  ┌─────┐ ┌─────┐  │
                    │  │ Pod │ │ Pod │  │
                    │  └─────┘ └─────┘  │
                    └───────────────────┘
```

### Key Principles

| Principle       | Implementation                                            |
| --------------- | --------------------------------------------------------- |
| **Security**    | Non-root user, minimal base image, vulnerability scanning |
| **Efficiency**  | Multi-stage builds, layer caching, small image size       |
| **Reliability** | Health checks, proper signal handling, graceful shutdown  |
| **Consistency** | Standardized Dockerfile template across all services      |

## Base Images

### Standard Base Image

All Node.js services use `node:20-alpine` as the base image:

```dockerfile
FROM node:20-alpine
```

### Why Alpine?

| Feature         | Alpine  | Debian-based |
| --------------- | ------- | ------------ |
| Base size       | ~5MB    | ~120MB       |
| Final image     | ~150MB  | ~400MB+      |
| Attack surface  | Minimal | Larger       |
| Package manager | apk     | apt          |

### Required System Packages

```dockerfile
# Runtime dependencies
RUN apk add --no-cache dumb-init

# Build dependencies (builder stage only)
RUN apk add --no-cache python3 make g++
```

## Dockerfile Standards

### Standard Template

All services use this standardized Dockerfile structure:

```dockerfile
# Multi-stage build for production-ready service
FROM node:20-alpine AS builder

# Install build dependencies
RUN apk add --no-cache python3 make g++

# Set working directory
WORKDIR /app

# Copy package files for layer caching
COPY services/SERVICE_NAME/package*.json ./services/SERVICE_NAME/
COPY services/lib/package*.json ./services/lib/
COPY services/common/package*.json ./services/common/

# Install dependencies
WORKDIR /app/services/SERVICE_NAME
RUN npm ci --only=production || npm install --only=production

# Production stage
FROM node:20-alpine

# Install dumb-init for signal handling
RUN apk add --no-cache dumb-init

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

# Copy from builder
COPY --from=builder --chown=nodejs:nodejs /app/services/SERVICE_NAME/node_modules ./services/SERVICE_NAME/node_modules

# Copy source
COPY --chown=nodejs:nodejs services/SERVICE_NAME ./services/SERVICE_NAME
COPY --chown=nodejs:nodejs services/lib ./services/lib
COPY --chown=nodejs:nodejs services/common ./services/common

# Switch to non-root
USER nodejs

# Environment
ENV NODE_ENV=production
ENV SERVICE_NAME=SERVICE_NAME
ENV PORT=8080

EXPOSE 8080

# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
    CMD node -e "require('http').get('http://localhost:8080/health', (r) => process.exit(r.statusCode === 200 ? 0 : 1))"

WORKDIR /app/services/SERVICE_NAME

# Use dumb-init for proper signal handling
ENTRYPOINT ["dumb-init", "--"]

CMD ["node", "src/server.js"]
```

### Required Elements

| Element           | Purpose           | Required |
| ----------------- | ----------------- | -------- |
| Multi-stage build | Reduce image size | Yes      |
| Non-root user     | Security          | Yes      |
| dumb-init         | Signal handling   | Yes      |
| HEALTHCHECK       | Container health  | Yes      |
| Layer caching     | Build efficiency  | Yes      |

## Multi-Stage Builds

### Build Stage

The builder stage includes all build tools and dependencies:

```dockerfile
FROM node:20-alpine AS builder

# Build dependencies needed for native modules
RUN apk add --no-cache python3 make g++

# Install all dependencies including devDependencies if needed
RUN npm ci
```

### Production Stage

The production stage contains only runtime requirements:

```dockerfile
FROM node:20-alpine

# Only runtime dependencies
RUN apk add --no-cache dumb-init

# Copy only production node_modules
COPY --from=builder /app/node_modules ./node_modules

# Copy application code
COPY --chown=nodejs:nodejs . .
```

### Image Size Comparison

| Build Type              | Image Size             |
| ----------------------- | ---------------------- |
| Single-stage (dev deps) | ~800MB                 |
| Multi-stage (prod only) | ~150MB                 |
| With build caching      | ~150MB (faster builds) |

## Image Registry (ECR)

### Repository Structure

```
AWS ECR
├── lilith/auth
├── lilith/bff
├── lilith/content
├── lilith/conversation
├── lilith/media
├── lilith/ai
├── lilith/notification
├── lilith/moderation
└── lilith/base (shared base images)
```

### Terraform Configuration

```hcl
resource "aws_ecr_repository" "services" {
  for_each = toset([
    "auth", "bff", "content", "conversation",
    "media", "ai", "notification", "moderation"
  ])

  name                 = "lilith/${each.key}"
  image_tag_mutability = "IMMUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }

  encryption_configuration {
    encryption_type = "KMS"
    kms_key         = aws_kms_key.ecr.arn
  }

  tags = {
    Service     = each.key
    Environment = var.environment
    Project     = "lilith"
  }
}

# Lifecycle policy to manage image retention
resource "aws_ecr_lifecycle_policy" "services" {
  for_each   = aws_ecr_repository.services
  repository = each.value.name

  policy = jsonencode({
    rules = [
      {
        rulePriority = 1
        description  = "Keep last 30 production images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["v"]
          countType     = "imageCountMoreThan"
          countNumber   = 30
        }
        action = {
          type = "expire"
        }
      },
      {
        rulePriority = 2
        description  = "Keep last 10 dev images"
        selection = {
          tagStatus     = "tagged"
          tagPrefixList = ["dev-", "pr-"]
          countType     = "imageCountMoreThan"
          countNumber   = 10
        }
        action = {
          type = "expire"
        }
      },
      {
        rulePriority = 3
        description  = "Delete untagged images after 7 days"
        selection = {
          tagStatus   = "untagged"
          countType   = "sinceImagePushed"
          countUnit   = "days"
          countNumber = 7
        }
        action = {
          type = "expire"
        }
      }
    ]
  })
}
```

### Cross-Account Access

For multi-account setups:

```hcl
resource "aws_ecr_repository_policy" "cross_account" {
  repository = aws_ecr_repository.services["auth"].name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowPull"
        Effect = "Allow"
        Principal = {
          AWS = [
            "arn:aws:iam::${var.staging_account_id}:root",
            "arn:aws:iam::${var.prod_account_id}:root"
          ]
        }
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
      }
    ]
  })
}
```

## Image Scanning

### Automated Scanning

ECR automatically scans images on push:

```hcl
image_scanning_configuration {
  scan_on_push = true
}
```

### Scanning Results

Check scan results via CLI:

```bash
# Get scan findings
aws ecr describe-image-scan-findings \
  --repository-name lilith/auth \
  --image-id imageTag=v1.2.3

# List images with vulnerabilities
aws ecr describe-images \
  --repository-name lilith/auth \
  --query 'imageDetails[?imageScanStatus.status==`COMPLETE`].{Tag:imageTags[0],Critical:imageScanFindingsSummary.findingSeverityCounts.CRITICAL,High:imageScanFindingsSummary.findingSeverityCounts.HIGH}'
```

### CI/CD Integration

```yaml
# .github/workflows/build.yml
- name: Scan image
  run: |
    # Wait for scan to complete
    aws ecr wait image-scan-complete \
      --repository-name lilith/${{ matrix.service }} \
      --image-id imageTag=${{ github.sha }}

    # Get findings
    CRITICAL=$(aws ecr describe-image-scan-findings \
      --repository-name lilith/${{ matrix.service }} \
      --image-id imageTag=${{ github.sha }} \
      --query 'imageScanFindings.findingSeverityCounts.CRITICAL' \
      --output text)

    # Fail if critical vulnerabilities
    if [ "$CRITICAL" != "None" ] && [ "$CRITICAL" -gt 0 ]; then
      echo "Critical vulnerabilities found!"
      exit 1
    fi
```

### Trivy Integration

For additional scanning:

```yaml
- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref:
      '${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}'
    format: 'sarif'
    output: 'trivy-results.sarif'
    severity: 'CRITICAL,HIGH'
    exit-code: '1'
```

## Tagging Strategy

### Tag Format

| Environment | Tag Format                 | Example           |
| ----------- | -------------------------- | ----------------- |
| Production  | `v{major}.{minor}.{patch}` | `v1.2.3`          |
| Staging     | `staging-{sha}`            | `staging-abc1234` |
| Development | `dev-{sha}`                | `dev-abc1234`     |
| PR          | `pr-{number}`              | `pr-123`          |
| Latest      | `latest`                   | `latest`          |

### Semantic Versioning

```bash
# Tag format: v{MAJOR}.{MINOR}.{PATCH}
# MAJOR: Breaking changes
# MINOR: New features (backward compatible)
# PATCH: Bug fixes

# Examples
v1.0.0  # Initial release
v1.1.0  # New feature added
v1.1.1  # Bug fix
v2.0.0  # Breaking change
```

### Git-Based Tags

```yaml
# Automatic tagging based on git
- name: Generate tags
  id: meta
  uses: docker/metadata-action@v5
  with:
    images: ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}
    tags: |
      type=semver,pattern=v{{version}}
      type=semver,pattern=v{{major}}.{{minor}}
      type=sha,prefix=
      type=ref,event=branch
      type=ref,event=pr
```

## Build Pipeline

### GitHub Actions Workflow

```yaml
name: Build and Push

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main]

env:
  AWS_REGION: us-east-1
  ECR_REGISTRY: ${{ secrets.AWS_ACCOUNT_ID }}.dkr.ecr.us-east-1.amazonaws.com

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service:
          [
            auth,
            bff,
            content,
            conversation,
            media,
            ai,
            notification,
            moderation,
          ]

    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume:
            arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/github-actions
          aws-region: ${{ env.AWS_REGION }}

      - name: Login to ECR
        uses: aws-actions/amazon-ecr-login@v2

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: services/${{ matrix.service }}/Dockerfile
          push: true
          tags: |
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:${{ github.sha }}
            ${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:latest
          cache-from: type=gha
          cache-to: type=gha,mode=max
          platforms: linux/amd64
```

### Build Caching

```yaml
# GitHub Actions cache
cache-from: type=gha
cache-to: type=gha,mode=max

# ECR cache (for self-hosted runners)
cache-from: type=registry,ref=${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:buildcache
cache-to: type=registry,ref=${{ env.ECR_REGISTRY }}/lilith/${{ matrix.service }}:buildcache,mode=max
```

### Local Build

```bash
# Build single service
docker build -t lilith/auth:local -f services/auth/Dockerfile .

# Build with BuildKit (faster)
DOCKER_BUILDKIT=1 docker build -t lilith/auth:local -f services/auth/Dockerfile .

# Build all services
for service in auth bff content conversation media ai notification moderation; do
  docker build -t lilith/${service}:local -f services/${service}/Dockerfile .
done
```

## Best Practices

### Security

1. **Non-root user**: Always run as non-root

   ```dockerfile
   RUN adduser -S nodejs -u 1001
   USER nodejs
   ```

2. **Read-only filesystem**: Use read-only root filesystem in Kubernetes

   ```yaml
   securityContext:
     readOnlyRootFilesystem: true
   ```

3. **No secrets in images**: Never bake secrets into images

   ```dockerfile
   # BAD - Don't do this
   ENV API_KEY=secret123

   # GOOD - Use runtime environment
   ENV API_KEY=""
   ```

4. **Pin versions**: Pin base image versions
   ```dockerfile
   FROM node:20.10.0-alpine3.18
   ```

### Efficiency

1. **Layer ordering**: Put frequently changing layers last

   ```dockerfile
   COPY package*.json ./    # Changes rarely
   RUN npm ci               # Cached when package.json unchanged
   COPY . .                 # Changes frequently
   ```

2. **Minimize layers**: Combine RUN commands

   ```dockerfile
   RUN apk add --no-cache dumb-init && \
       addgroup -g 1001 -S nodejs && \
       adduser -S nodejs -u 1001
   ```

3. **Use .dockerignore**: Exclude unnecessary files
   ```
   # .dockerignore
   node_modules
   .git
   *.md
   tests
   coverage
   .env*
   ```

### Reliability

1. **Health checks**: Always include health checks

   ```dockerfile
   HEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost:8080/health || exit 1
   ```

2. **Signal handling**: Use dumb-init or tini

   ```dockerfile
   ENTRYPOINT ["dumb-init", "--"]
   CMD ["node", "src/server.js"]
   ```

3. **Graceful shutdown**: Handle SIGTERM in application
   ```javascript
   process.on('SIGTERM', async () => {
     await server.close();
     process.exit(0);
   });
   ```

## Troubleshooting

### Common Issues

| Issue             | Cause                     | Solution                         |
| ----------------- | ------------------------- | -------------------------------- |
| Large image size  | Dev dependencies included | Use multi-stage build            |
| Build failures    | Missing native deps       | Add build tools in builder stage |
| Container crashes | Signal handling           | Use dumb-init                    |
| Permission denied | Running as root           | Add non-root user                |

### Debug Commands

```bash
# Inspect image layers
docker history lilith/auth:latest

# Check image size
docker images lilith/auth --format "{{.Size}}"

# Run container interactively
docker run -it --rm lilith/auth:latest sh

# View container logs
docker logs -f <container-id>

# Inspect running container
docker exec -it <container-id> sh
```

## Related Documentation

- [AWS Setup](./aws-setup.md) - ECR configuration
- [EKS Setup](./eks.md) - Kubernetes deployment
- [CI/CD Pipeline](../ci-cd.md) - Build automation
- [Deployment Process](../runbooks/procedures/deployment.md) - Deployment
  procedures
