Eve library

Iris AI Platform - Production Infrastructure

Other

Revision 1

On this page

Terraform configuration for the Iris AI Platform production environment on AWS EKS.

Overview#

This configuration provisions production-grade infrastructure:

  • VPC: Multi-AZ network with public, private, and database subnets (3 NAT gateways)
  • EKS: Kubernetes cluster with ON_DEMAND managed node groups
  • RDS: PostgreSQL 16 with pgvector, Multi-AZ, read replica
  • ElastiCache: Redis 7.1 replication group (3 clusters)
  • S3: Encrypted buckets with cross-region replication
  • ECR: Container registries with immutable tags
  • IAM: IRSA roles for Kubernetes workloads
  • Blue-Green: CodeDeploy for zero-downtime deployments
  • Auto-Scaling: Cluster Autoscaler and HPA
  • Services: API, conversation, memory, agent, voice, and worker runtimes

Prerequisites#

  1. AWS CLI configured with production credentials
  2. Terraform >= 1.10.0
  3. S3 bucket for Terraform state (oshun-terraform-state)
  4. DynamoDB table for state locking (oshun-terraform-locks)
  5. Production approval permissions

Production Configuration#

Component Production Configuration
EKS Nodes ON_DEMAND instances
NAT Gateways 3 (one per AZ for HA)
RDS db.r6g.large, Multi-AZ + Replica
ElastiCache cache.r6g.large, 3 clusters
Node Scaling 3-15 nodes
HPA 3-20 replicas
Deletion Protection Enabled
Image Mutability Immutable

Deployment Process#

1. Infrastructure Changes (Requires Approval)#

bash
# Plan changes
terraform plan -out=tfplan

# Apply with approval
terraform apply tfplan

2. Application Deployment#

Production deployments use the GitHub Actions workflow with:

  1. Manual Trigger: Requires version input
  2. Approval Gate: Requires production environment approval
  3. Image Validation: Confirms the requested tag exists in all six ECR repositories
  4. Canary Analysis: Starts a shadow API canary and evaluates health and metrics
  5. Blue-Green Rollout: Applies the production Kubernetes overlay with immutable images
  6. Smoke Tests: Automated health verification
  7. Automatic Rollback: Reverts every Iris deployment on smoke-test failure
bash
# Trigger via GitHub CLI
gh workflow run iris-production-deploy.yml \
  -f version=v1.2.3 \
  -f deploy_infra=false

3. Emergency Rollback#

bash
# Rollback to previous version
gh workflow run iris-production-deploy.yml \
  -f rollback_version=v1.2.2 \
  -f skip_canary=true

Blue-Green Deployment#

The production environment uses CodeDeploy for blue-green deployments:

  1. Blue Environment: Current production
  2. Green Environment: New version
  3. Traffic Shift: Linear 10% per minute
  4. Validation: Lambda hook validates new environment
  5. Termination: Old environment terminated after 60 minutes
text
[Blue (v1.2.2)] ──┐
                  ├──> [ALB] ──> Users
[Green (v1.2.3)]──┘

Traffic: 100% Blue → 90/10 → 80/20 → ... → 100% Green

Auto-Scaling#

Cluster Autoscaler#

  • Scale nodes based on pending pods
  • Scale-down delay: 10 minutes
  • Expander: Least waste

Horizontal Pod Autoscaler#

  • Services: iris-api, iris-conversation, iris-memory, iris-agent, iris-voice, iris-worker
  • CPU threshold: 70%
  • Memory threshold: 80%
  • Min replicas: 3
  • Max replicas: 20
  • Scale-down stabilization: 5 minutes

Disaster Recovery#

Metric Target
RPO 1 hour
RTO 4 hours

DR Components#

  • RDS automated backups (30 days)
  • RDS read replica for fast failover
  • S3 cross-region replication to us-west-2
  • ElastiCache snapshots (7 days)
  • EKS cluster configuration in Git

Failover Procedure#

  1. Promote RDS read replica in DR region
  2. Update Route53 to DR region ALB
  3. Deploy from ECR replicated images
  4. Verify smoke tests pass

Security#

  • All data encrypted at rest (KMS)
  • All data encrypted in transit (TLS 1.2+)
  • VPC endpoints for AWS services
  • Network policies for pod isolation
  • IRSA for fine-grained IAM
  • GuardDuty for threat detection
  • AWS Config for compliance
  • Immutable container images
  • Secrets in AWS Secrets Manager

Monitoring#

CloudWatch Dashboards#

  • EKS cluster metrics
  • RDS Performance Insights
  • ElastiCache metrics
  • Application metrics

Alarms#

Alarm Threshold Action
RDS CPU > 80% SNS + Scale
RDS Connections > 400 SNS
ElastiCache Memory > 80% SNS
Smoke Test Failure > 0 SNS + Page
5XX Error Rate > 1% Rollback
Deployment Failure Any Rollback

Cost Optimization#

While production prioritizes reliability over cost:

  • Reserved instances for predictable workloads
  • Savings Plans for compute
  • S3 Intelligent Tiering for long-term storage
  • Right-sizing recommendations from Cost Explorer

Runbooks#

Scale Cluster Manually#

bash
# Update desired capacity
aws eks update-nodegroup-config \
  --cluster-name iris-prod \
  --nodegroup-name iris-prod-general \
  --scaling-config minSize=5,maxSize=20,desiredSize=10

Force Rollback#

bash
# Get previous revision
kubectl rollout history deployment/iris-api -n iris

# Rollback to specific revision
kubectl rollout undo deployment/iris-api -n iris --to-revision=2

# Rollback all Iris service deployments to their previous revisions
for service in api conversation memory agent voice worker; do
  kubectl rollout undo deployment/iris-${service} -n iris
done

Database Failover#

bash
# Promote read replica (creates new instance)
aws rds promote-read-replica \
  --db-instance-identifier iris-prod-postgres-replica

# Update application configuration
kubectl set env deployment/iris-api \
  DATABASE_HOST=new-primary-endpoint \
  -n iris

Compliance#

  • SOC 2 Type II logging enabled
  • HIPAA-eligible services configured
  • PCI DSS network isolation
  • GDPR data residency controls

Support#

  • On-Call: PagerDuty rotation
  • Escalation: #iris-production Slack channel
  • Runbooks: Confluence/iris-runbooks

Architecture#

text
                    ┌─────────────────────────────────────────────────────────┐
                    │                     Route53                              │
                    └─────────────────────┬───────────────────────────────────┘
                                          │
                    ┌─────────────────────▼───────────────────────────────────┐
                    │                   CloudFront                             │
                    └─────────────────────┬───────────────────────────────────┘
                                          │
┌─────────────────────────────────────────┼─────────────────────────────────────┐
│                         VPC (10.120.0.0/16)                                   │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                    Public Subnets (3 AZs)                              │  │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌───────────────────────────┐  │  │
│  │  │ NAT GW  │  │ NAT GW  │  │ NAT GW  │  │   Application Load        │  │  │
│  │  │  (AZ-A) │  │  (AZ-B) │  │  (AZ-C) │  │   Balancer                │  │  │
│  │  └─────────┘  └─────────┘  └─────────┘  └───────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                    Private Subnets (3 AZs)                             │  │
│  │  ┌──────────────────────────────────────────────────────────────────┐  │  │
│  │  │                      EKS Node Group                              │  │  │
│  │  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐            │  │  │
│  │  │  │ iris-api │ │convers.  │ │ memory   │ │ agent    │            │  │  │
│  │  │  │ (3-20)   │ │ (3-20)   │ │ (3-20)   │ │ (3-20)   │            │  │  │
│  │  │  └──────────┘ └──────────┘ └──────────┘ └──────────┘            │  │  │
│  │  │  ┌──────────┐ ┌──────────┐                                      │  │  │
│  │  │  │ voice    │ │ worker   │                                      │  │  │
│  │  │  │ (3-20)   │ │ (3-20)   │                                      │  │  │
│  │  │  └──────────┘ └──────────┘                                      │  │  │
│  │  └──────────────────────────────────────────────────────────────────┘  │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
│  ┌────────────────────────────────────────────────────────────────────────┐  │
│  │                   Database Subnets (Isolated)                          │  │
│  │  ┌────────────────────────────┐  ┌────────────────────────────────┐   │  │
│  │  │    RDS PostgreSQL          │  │   ElastiCache Redis            │   │  │
│  │  │    Primary (Multi-AZ)      │  │   Primary + 2 Replicas         │   │  │
│  │  │         +                  │  │                                │   │  │
│  │  │    Read Replica            │  │                                │   │  │
│  │  └────────────────────────────┘  └────────────────────────────────┘   │  │
│  └────────────────────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────────────────────┘

                    ┌─────────────────────────────────────────────────────────┐
                    │                DR Region (us-west-2)                     │
                    │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐      │
                    │  │ S3 Replica  │  │ ECR Replica │  │ RDS Snapshot│      │
                    │  └─────────────┘  └─────────────┘  └─────────────┘      │
                    └─────────────────────────────────────────────────────────┘