# Cost Management Runbook

This runbook provides guidance for monitoring, optimizing, and managing cloud
costs for the Oshun infrastructure.

## Cost Overview

### Monthly Cost Breakdown (Estimated)

| Service          | Staging       | Production     | % of Total |
| ---------------- | ------------- | -------------- | ---------- |
| ECS (Fargate)    | $200-400      | $800-1500      | 25-30%     |
| RDS (PostgreSQL) | $100-200      | $400-800       | 15-20%     |
| RunPod (GPU)     | $100-300      | $500-2000      | 20-40%     |
| S3 Storage       | $20-50        | $50-150        | 2-5%       |
| CloudWatch       | $50-100       | $100-300       | 5-8%       |
| ALB              | $20-40        | $40-100        | 2-4%       |
| Data Transfer    | $50-100       | $200-500       | 5-10%      |
| Other            | $50-100       | $100-300       | 5-10%      |
| **Total**        | **$600-1300** | **$2200-5600** |            |

## Monitoring Costs

### AWS Cost Explorer

**Monthly cost by service:**

```bash
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of this month' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.BlendedCost.Amount}' \
  --output table
```

**Daily cost trend:**

```bash
aws ce get-cost-and-usage \
  --time-period Start=$(date -d '7 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity DAILY \
  --metrics BlendedCost \
  --query 'ResultsByTime[*].{Date:TimePeriod.Start,Cost:Total.BlendedCost.Amount}' \
  --output table
```

**Cost by tag (environment):**

```bash
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of this month' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=TAG,Key=Environment \
  --query 'ResultsByTime[0].Groups[*].{Environment:Keys[0],Cost:Metrics.BlendedCost.Amount}'
```

### AWS Budgets

**Create monthly budget alert:**

```bash
aws budgets create-budget \
  --account-id $(aws sts get-caller-identity --query Account --output text) \
  --budget '{
    "BudgetName": "oshun-monthly",
    "BudgetLimit": {"Amount": "5000", "Unit": "USD"},
    "BudgetType": "COST",
    "TimeUnit": "MONTHLY"
  }' \
  --notifications-with-subscribers '[
    {
      "Notification": {
        "NotificationType": "ACTUAL",
        "ComparisonOperator": "GREATER_THAN",
        "Threshold": 80,
        "ThresholdType": "PERCENTAGE"
      },
      "Subscribers": [
        {"SubscriptionType": "EMAIL", "Address": "ops@oshun.ai"}
      ]
    }
  ]'
```

### RunPod Costs

**Check RunPod spending:**

```bash
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "{ myself { spending { totalSpend currentMonthSpend } } }"}'
```

## Cost Optimization Strategies

### ECS Optimization

#### Use Fargate Spot (Staging)

Fargate Spot provides up to 70% discount for interruptible workloads.

```hcl
# In Terraform
capacity_provider_strategy {
  capacity_provider = "FARGATE_SPOT"
  weight            = 4
  base              = 0
}

capacity_provider_strategy {
  capacity_provider = "FARGATE"
  weight            = 1
  base              = 1  # At least 1 on-demand task
}
```

#### Right-Size Tasks

**Analyze CPU/memory utilization:**

```bash
# Average CPU utilization over 7 days
aws cloudwatch get-metric-statistics \
  --namespace AWS/ECS \
  --metric-name CPUUtilization \
  --dimensions Name=ClusterName,Value=oshun-production Name=ServiceName,Value=oshun-api \
  --start-time $(date -d '7 days ago' -u +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 86400 \
  --statistics Average Maximum p99

# If average CPU < 30%, consider reducing task size
# If max CPU > 80%, consider increasing
```

**Task sizing recommendations:**

| Avg CPU | Avg Memory | Recommendation        |
| ------- | ---------- | --------------------- |
| < 20%   | < 40%      | Reduce by 50%         |
| 20-40%  | 40-60%     | May reduce by 25%     |
| 40-60%  | 60-80%     | Well-sized            |
| > 70%   | > 80%      | Increase or scale out |

#### Scheduled Scaling

Reduce costs during off-peak hours:

```bash
# Scale down at night (10 PM UTC)
aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --scheduled-action-name night-scale-down \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --schedule "cron(0 22 ? * * *)" \
  --scalable-target-action MinCapacity=1,MaxCapacity=4

# Scale up in morning (8 AM UTC)
aws application-autoscaling put-scheduled-action \
  --service-namespace ecs \
  --scheduled-action-name morning-scale-up \
  --resource-id service/oshun-production/oshun-api \
  --scalable-dimension ecs:service:DesiredCount \
  --schedule "cron(0 8 ? * MON-FRI *)" \
  --scalable-target-action MinCapacity=2,MaxCapacity=10
```

### RDS Optimization

#### Reserved Instances

For predictable database workloads, RIs save 30-60%:

```bash
# Check current on-demand usage
aws ce get-reservation-coverage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --filter '{"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Relational Database Service"]}}'

# Purchase RI via console or CLI
aws rds purchase-reserved-db-instances-offering \
  --reserved-db-instances-offering-id <offering-id> \
  --reserved-db-instance-id oshun-production-ri
```

#### Storage Optimization

**Enable storage autoscaling to avoid over-provisioning:**

```bash
aws rds modify-db-instance \
  --db-instance-identifier oshun-production \
  --max-allocated-storage 500  # GB
```

**Delete old snapshots:**

```bash
# List snapshots older than 30 days
aws rds describe-db-snapshots \
  --db-instance-identifier oshun-production \
  --query "DBSnapshots[?SnapshotCreateTime<='$(date -d '30 days ago' +%Y-%m-%d)'].DBSnapshotIdentifier"

# Delete old manual snapshots (automated retained per policy)
aws rds delete-db-snapshot --db-snapshot-identifier <snapshot-id>
```

### S3 Optimization

#### Lifecycle Policies

Move infrequently accessed data to cheaper storage:

```bash
aws s3api put-bucket-lifecycle-configuration \
  --bucket oshun-production-assets \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "MoveToIA",
        "Status": "Enabled",
        "Filter": {"Prefix": "exports/"},
        "Transitions": [
          {"Days": 30, "StorageClass": "STANDARD_IA"},
          {"Days": 90, "StorageClass": "GLACIER"}
        ]
      },
      {
        "ID": "DeleteTemp",
        "Status": "Enabled",
        "Filter": {"Prefix": "temp/"},
        "Expiration": {"Days": 7}
      }
    ]
  }'
```

#### Intelligent-Tiering

For unpredictable access patterns:

```bash
# Enable Intelligent-Tiering by default
aws s3api put-bucket-intelligent-tiering-configuration \
  --bucket oshun-production-assets \
  --id entire-bucket \
  --intelligent-tiering-configuration '{
    "Id": "entire-bucket",
    "Status": "Enabled",
    "Tierings": [
      {"Days": 90, "AccessTier": "ARCHIVE_ACCESS"},
      {"Days": 180, "AccessTier": "DEEP_ARCHIVE_ACCESS"}
    ]
  }'
```

### RunPod Optimization

#### Idle Timeout Tuning

| Traffic Pattern | Recommended Timeout | Impact          |
| --------------- | ------------------- | --------------- |
| Continuous      | 120-300s            | Low cold starts |
| Bursty          | 60-120s             | Balance         |
| Infrequent      | 30-60s              | Lower cost      |
| Very Infrequent | 0-30s               | Lowest cost     |

```bash
# Reduce idle timeout for low-traffic endpoint
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{
    "query": "mutation { updateEndpoint(input: { id: \"ENDPOINT_ID\", idleTimeout: 30 }) { id idleTimeout } }"
  }'
```

#### GPU Selection

| Use Case           | Recommended GPU | Price/Hour | Notes             |
| ------------------ | --------------- | ---------- | ----------------- |
| SD1.5              | RTX 4090        | $0.50      | Fast, cheap       |
| SDXL               | A40             | $0.86      | Good balance      |
| Flux               | A100 40GB       | $1.58      | Required for Flux |
| Training/Fine-tune | A100 80GB       | $2.00      | Max VRAM          |

### CloudWatch Optimization

#### Reduce Log Retention

```bash
# Set 14-day retention for staging
aws logs put-retention-policy \
  --log-group-name /ecs/oshun-staging/api \
  --retention-in-days 14

# Set 30-day retention for less critical production logs
aws logs put-retention-policy \
  --log-group-name /ecs/oshun-production/api \
  --retention-in-days 30
```

#### Optimize Metrics Collection

Disable high-cardinality metrics if not needed:

```bash
# Review custom metrics
aws cloudwatch list-metrics --namespace Oshun/Custom

# Delete unused custom metrics (stop publishing)
```

### Data Transfer Optimization

**Use VPC endpoints to avoid NAT costs:**

```bash
# S3 Gateway Endpoint (free)
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-xxx

# ECR, CloudWatch Interface Endpoints
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxx \
  --service-name com.amazonaws.us-east-1.ecr.api \
  --vpc-endpoint-type Interface \
  --subnet-ids subnet-xxx
```

## Cost Alerts

### Setting Up Alerts

**80% budget threshold:**

```bash
aws cloudwatch put-metric-alarm \
  --alarm-name oshun-cost-warning \
  --alarm-description "AWS costs exceeding 80% of budget" \
  --metric-name EstimatedCharges \
  --namespace AWS/Billing \
  --statistic Maximum \
  --period 21600 \
  --threshold 4000 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=Currency,Value=USD \
  --evaluation-periods 1 \
  --alarm-actions arn:aws:sns:us-east-1:123456789:billing-alerts
```

### Anomaly Detection

```bash
aws cloudwatch put-anomaly-detector \
  --namespace AWS/Billing \
  --metric-name EstimatedCharges \
  --stat Maximum \
  --dimensions Name=Currency,Value=USD
```

## Monthly Cost Review

### Review Checklist

- [ ] Review AWS Cost Explorer for unexpected spikes
- [ ] Check RunPod spending dashboard
- [ ] Review underutilized resources
- [ ] Check for orphaned resources (unused EBS, old snapshots)
- [ ] Verify Reserved Instance coverage
- [ ] Review data transfer costs
- [ ] Update budget forecasts if needed

### Cost Reporting

**Generate monthly report:**

```bash
#!/bin/bash
# Monthly cost report script

echo "=== Oshun Monthly Cost Report ==="
echo "Period: $(date -d 'first day of last month' +%B\ %Y)"
echo ""

echo "=== AWS Costs by Service ==="
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[*].{Service:Keys[0],Cost:Metrics.BlendedCost.Amount}' \
  --output table

echo ""
echo "=== Costs by Environment ==="
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --group-by Type=TAG,Key=Environment \
  --query 'ResultsByTime[0].Groups[*].{Environment:Keys[0],Cost:Metrics.BlendedCost.Amount}' \
  --output table

echo ""
echo "=== Total ==="
aws ce get-cost-and-usage \
  --time-period Start=$(date -d 'first day of last month' +%Y-%m-%d),End=$(date -d 'first day of this month' +%Y-%m-%d) \
  --granularity MONTHLY \
  --metrics BlendedCost \
  --query 'ResultsByTime[0].Total.BlendedCost.Amount'
```

## Emergency Cost Reduction

If costs are spiking unexpectedly:

### 1. Immediate Actions

```bash
# Scale down non-critical services
aws ecs update-service --cluster oshun-staging --service oshun-api --desired-count 0

# Reduce RunPod max workers
curl -X POST https://api.runpod.io/graphql \
  -H "Authorization: Bearer $RUNPOD_API_KEY" \
  -d '{"query": "mutation { updateEndpoint(input: { id: \"ENDPOINT_ID\", workersMax: 1 }) { id } }"}'
```

### 2. Identify Cost Source

```bash
# Check for cost spikes by service
aws ce get-cost-and-usage \
  --time-period Start=$(date -d '3 days ago' +%Y-%m-%d),End=$(date +%Y-%m-%d) \
  --granularity DAILY \
  --metrics BlendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --output table
```

### 3. Common Causes

| Symptom             | Likely Cause                 | Fix                      |
| ------------------- | ---------------------------- | ------------------------ |
| ECS costs doubled   | Auto-scaling spike           | Check scaling policies   |
| Data transfer spike | Large file uploads/downloads | Review CloudFront/CDN    |
| RunPod costs high   | Workers staying alive        | Reduce idle timeout      |
| S3 costs increased  | Lifecycle not applied        | Check lifecycle policies |
| RDS costs increased | Storage autoscale            | Review storage usage     |

## Related Documentation

- [ECS Architecture](../../infrastructure/ecs-architecture.md)
- [RunPod Integration](../../infrastructure/runpod-integration.md)
- [Scaling Runbook](./scaling.md)