Eve library

Aphrodite Chaos Engineering Test Suite

Other

Revision 1

On this page

Comprehensive chaos engineering tests for the Aphrodite adult live streaming platform. These tests validate system resilience, failure recovery, and graceful degradation under adverse conditions.

Overview#

This test suite implements chaos engineering principles to verify that the Aphrodite platform can:

  • Maintain service availability during partial failures
  • Recover gracefully from service outages
  • Prevent cascade failures across dependent services
  • Maintain data consistency during network partitions
  • Handle latency injection without data loss

Test Categories#

Service Failure Simulation (service-failure.test.ts)#

Tests system behavior when individual services become unavailable:

Test Description
Streaming Service Failure Validates viewer handling during stream outage
Payment Service Failure Tests tip processing during payment downtime
Chat Service Failure Verifies stream continuation without chat
Device Service Failure Tests tip-triggered device queue handling
Broadcaster Service Failure Validates viewer notification and cleanup

Key validations:

  • Graceful degradation behavior
  • Dependent service isolation
  • Circuit breaker activation
  • Queue preservation during outages
  • Error message quality

Network Partition Tests (network-partition.test.ts)#

Tests system behavior during network isolation scenarios:

Test Description
Payment Isolation Isolates payment service from other services
Chat-Streaming Split Brain Tests message synchronization during partition
Full Partition Simulates complete service isolation
Intermittent Connectivity Tests retry logic and reconnection
Latency Injection Validates timeout handling

Key validations:

  • Split-brain prevention
  • Data reconciliation after partition heals
  • Timeout handling
  • Retry behavior
  • Message ordering preservation

Recovery Validation Tests (recovery-validation.test.ts)#

Tests system recovery after failures:

Test Description
Time-to-Recovery (TTR) Measures recovery time for each service
Data Consistency Validates transaction logs and event ordering
Queue Drainage Verifies pending messages are processed
Metrics Normalization Confirms return to baseline performance
Dependency Recovery Tests dependent service restoration

Key validations:

  • Recovery within SLA timeouts
  • No data loss during recovery
  • Queue depth returns to zero
  • Response times normalize
  • All dependencies healthy post-recovery

Service Configuration#

Services are configured in config.ts with the following properties:

typescript
{
  name: 'Streaming',
  url: 'http://localhost:3101',
  healthEndpoint: '/health',
  criticalityLevel: 'critical',  // critical | high | medium | low
  dependencies: ['payment', 'chat', 'devices'],
  recoveryTimeoutMs: 30000,
  gracefulDegradation: true,
}

Criticality Levels#

Level Services Recovery SLA Approval Required
Critical streaming, broadcaster, payment 30-60s Yes (payment)
High viewer, chat 20-30s No
Medium devices 15s No

Pre-defined Experiments#

The suite includes pre-configured chaos experiments:

Service Failure Experiments#

  • streaming_service_failure - Complete streaming outage
  • payment_service_failure - Payment service unavailable
  • chat_service_failure - Chat service down
  • device_service_failure - Device service unavailable

Network Partition Experiments#

  • network_partition_payment - Isolate payment service
  • network_partition_chat_streaming - Split chat and streaming

Latency Injection Experiments#

  • latency_payment - 2000ms latency with 500ms jitter
  • latency_devices - 1000ms latency with 300ms jitter

Advanced Experiments#

  • cascade_streaming_failure - Gradual degradation cascade
  • database_payment_failure - Payment database unavailability
  • cache_failure - Redis cache failure

Running Tests#

Prerequisites#

  1. Ensure all Aphrodite services are running:

    bash
    # Start services
    docker compose -f docker/docker-compose.dev.yml up -d
    
  2. Install dependencies:

    bash
    pnpm install
    

Run All Chaos Tests#

bash
# Run all chaos engineering tests
pnpm nx test testing-chaos-aphrodite

# Or using vitest directly
pnpm vitest run testing/chaos/aphrodite/

Run Specific Test Suites#

bash
# Service failure tests only
pnpm vitest run testing/chaos/aphrodite/service-failure.test.ts

# Network partition tests only
pnpm vitest run testing/chaos/aphrodite/network-partition.test.ts

# Recovery validation tests only
pnpm vitest run testing/chaos/aphrodite/recovery-validation.test.ts

Run with Verbose Output#

bash
pnpm vitest run testing/chaos/aphrodite/ --reporter=verbose

Environment Configuration#

Configure service URLs via environment variables:

bash
# Core services
export APHRODITE_STREAMING_URL=http://localhost:3101
export APHRODITE_BROADCASTER_URL=http://localhost:3102
export APHRODITE_VIEWER_URL=http://localhost:3103
export APHRODITE_CHAT_URL=http://localhost:3104
export APHRODITE_PAYMENT_URL=http://localhost:3105
export APHRODITE_DEVICES_URL=http://localhost:3106

# Chaos API (if using external chaos orchestrator)
export CHAOS_API_URL=http://localhost:3500/v1/chaos
export OBSERVABILITY_URL=http://localhost:3500

Safety Controls#

All experiments include safety controls:

typescript
safetyControls: {
  automaticRollback: true,      // Auto-rollback on critical errors
  monitoringRequired: true,      // Require observability
  approvalRequired: true,        // Manual approval (critical services)
  rollbackTriggers: [            // Automatic rollback conditions
    'error_rate > 50%',
    'cascade_detected',
    'data_inconsistency',
  ],
}

Rollback Triggers#

Trigger Description
error_rate > N% Error rate exceeds threshold
cascade_detected Cascade failure detected
data_inconsistency Data integrity violation
timeout > Ns Operation timeout exceeded
queue_depth > N Message queue overflow
revenue_loss Payment processing affected

Success Criteria#

Each experiment defines success criteria:

typescript
successCriteria: {
  maxErrorRate: 0.1,           // Max 10% error rate during recovery
  maxResponseTimeMs: 2000,     // Max response time
  recoveryTimeMs: 30000,       // Time to full recovery
  dataConsistencyCheck: true,  // Require data consistency validation
}

Integration with CI/CD#

GitHub Actions Example#

yaml
chaos-tests:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - name: Start services
      run: docker compose up -d

    - name: Run chaos tests
      run: pnpm vitest run testing/chaos/aphrodite/

    - name: Upload results
      uses: actions/upload-artifact@v4
      with:
        name: chaos-test-results
        path: coverage/chaos/

Scheduled Chaos Testing#

Run chaos tests on a schedule to validate ongoing resilience:

yaml
on:
  schedule:
    - cron: '0 2 * * *' # Daily at 2 AM

Metrics and Observability#

Tests emit metrics for monitoring:

  • chaos.experiment.started - Experiment start event
  • chaos.experiment.completed - Experiment completion
  • chaos.recovery.time_ms - Time to recovery
  • chaos.error.rate - Error rate during experiment
  • chaos.cascade.detected - Cascade failure events

Best Practices#

  1. Start Small: Begin with single-service failures before testing cascades

  2. Monitor During Tests: Always have observability active during chaos tests

  3. Document Findings: Record unexpected behaviors and system limits

  4. Iterate: Use findings to improve system resilience

  5. Production Chaos: Graduate to production chaos testing only after extensive staging validation

Troubleshooting#

Tests Failing to Connect#

Verify services are running:

bash
curl http://localhost:3101/health
curl http://localhost:3105/health

Timeout Errors#

Increase test timeout in vitest config or individual tests:

typescript
it('should recover', async () => { ... }, 120000);

Cascade Detection False Positives#

Adjust cascade detection thresholds in config.ts:

typescript
rollbackTriggers: ['error_rate > 60%'],  // Increase threshold