# Integration Testing Guide

This guide covers integration testing for the Lilith platform, including
database testing with test containers, fixtures, and cleanup utilities.

## Overview

The `@lilith/testing` package provides comprehensive utilities for integration
testing:

- **Test Containers**: Real database instances using testcontainers
- **Database Fixtures**: Seeding utilities for test data
- **Cleanup Scripts**: Utilities for test isolation
- **Helpers**: Assertion and utility functions

## Installation

The testing package is available as a workspace package:

```typescript
import {
  PostgresContainer,
  RedisContainer,
  MilvusContainer,
  Neo4jContainer,
} from '@lilith/testing/containers';

import {
  PostgresFixtures,
  RedisFixtures,
  Neo4jFixtures,
} from '@lilith/testing/fixtures';

import {
  cleanupPostgres,
  cleanupRedis,
  cleanupNeo4j,
  CleanupManager,
} from '@lilith/testing/helpers';
```

## Test Containers

### PostgreSQL Container

```typescript
import { PostgresContainer } from '@lilith/testing/containers';
import { describe, beforeAll, afterAll, it } from 'vitest';

describe('User Repository', () => {
  let container: PostgresContainer;

  beforeAll(async () => {
    container = new PostgresContainer({
      database: 'users_test',
      enablePgVector: true, // Enable pgvector extension
    });
    await container.start();
  }, 60000); // 60 second timeout for container startup

  afterAll(async () => {
    await container.stop();
  });

  it('should connect to database', async () => {
    const connectionString = container.getConnectionString();
    // Use with pg, drizzle, prisma, etc.
    expect(connectionString).toContain('postgresql://');
  });

  it('should execute SQL', async () => {
    const result = await container.exec('SELECT 1 as test');
    expect(result.exitCode).toBe(0);
  });
});
```

#### PostgreSQL Options

```typescript
interface PostgresContainerOptions {
  imageTag?: string; // Default: '16-alpine'
  database?: string; // Default: 'test_db'
  username?: string; // Default: 'test_user'
  password?: string; // Default: 'test_password'
  enablePgVector?: boolean; // Default: true
  startupTimeout?: number; // Default: 60000
}
```

### Redis Container

```typescript
import { RedisContainer } from '@lilith/testing/containers';

describe('Cache Service', () => {
  let container: RedisContainer;

  beforeAll(async () => {
    container = new RedisContainer({
      password: 'test_password', // Optional
    });
    await container.start();
  });

  afterAll(async () => {
    await container.stop();
  });

  it('should set and get values', async () => {
    await container.set('key', 'value', 60);
    const result = await container.get('key');
    expect(result).toBe('value');
  });

  it('should flush all data', async () => {
    await container.flushAll();
  });
});
```

### Neo4j Container

```typescript
import { Neo4jContainer } from '@lilith/testing/containers';

describe('Graph Queries', () => {
  let container: Neo4jContainer;

  beforeAll(async () => {
    container = new Neo4jContainer({
      imageTag: '5-community',
      password: 'test_password',
    });
    await container.start();
    await container.waitForReady();
  }, 90000);

  afterAll(async () => {
    await container.stop();
  });

  it('should execute Cypher queries', async () => {
    await container.execCypher(`
      CREATE (u:User {name: 'Alice'})
      RETURN u
    `);

    const result = await container.execCypher(`
      MATCH (u:User) RETURN count(u) as count
    `);
    expect(result).toBeDefined();
  });

  it('should clear database', async () => {
    await container.clearDatabase();
  });
});
```

### Milvus Container

```typescript
import { MilvusContainer } from '@lilith/testing/containers';

describe('Vector Search', () => {
  let container: MilvusContainer;

  beforeAll(async () => {
    container = new MilvusContainer({
      imageTag: 'v2.3-latest',
    });
    await container.start();
    await container.waitForReady();
  }, 120000); // Milvus needs more time

  afterAll(async () => {
    await container.stop();
  });

  it('should provide connection info', () => {
    const address = container.getAddress();
    expect(address).toMatch(/localhost:\d+/);
  });
});
```

## Database Fixtures

### PostgreSQL Fixtures

```typescript
import { PostgresContainer } from '@lilith/testing/containers';
import { PostgresFixtures } from '@lilith/testing/fixtures';

describe('Content API', () => {
  let container: PostgresContainer;
  let fixtures: PostgresFixtures;

  beforeAll(async () => {
    container = new PostgresContainer();
    await container.start();

    fixtures = new PostgresFixtures(container);
    await fixtures.createAllTables();
  });

  beforeEach(async () => {
    // Seed test data
    const userIds = await fixtures.seedUsers(5);
    await fixtures.seedContent(10, userIds[0]);
  });

  afterEach(async () => {
    await fixtures.clearAll();
  });

  it('should query seeded data', async () => {
    const result = await container.exec('SELECT COUNT(*) FROM users');
    expect(result.stdout).toContain('5');
  });
});
```

#### Available PostgreSQL Fixture Methods

- `createUsersTable()` - Create users schema
- `createContentTable()` - Create content schema
- `createConversationsTable()` - Create conversations and messages schema
- `createEmbeddingsTable(dimensions)` - Create vector embeddings table
- `createAllTables()` - Create all base tables
- `seedUsers(count)` - Insert test users
- `seedContent(count, authorId?)` - Insert test content
- `seedConversation(userId, messageCount)` - Insert conversation with messages
- `seedEmbeddings(count, dimensions)` - Insert vector embeddings
- `clearAll()` - Truncate all tables

### Redis Fixtures

```typescript
import { RedisContainer } from '@lilith/testing/containers';
import { RedisFixtures } from '@lilith/testing/fixtures';

describe('Session Service', () => {
  let container: RedisContainer;
  let fixtures: RedisFixtures;

  beforeAll(async () => {
    container = new RedisContainer();
    await container.start();
    fixtures = new RedisFixtures(container);
  });

  it('should seed sessions', async () => {
    const sessionIds = await fixtures.seedSessions(10, 3600);
    expect(sessionIds).toHaveLength(10);
  });

  it('should seed feature flags', async () => {
    await fixtures.seedFeatureFlags({
      'dark-mode': true,
      'beta-features': false,
    });
  });
});
```

### Neo4j Fixtures

```typescript
import { Neo4jContainer } from '@lilith/testing/containers';
import { Neo4jFixtures } from '@lilith/testing/fixtures';

describe('Recommendation Engine', () => {
  let container: Neo4jContainer;
  let fixtures: Neo4jFixtures;

  beforeAll(async () => {
    container = new Neo4jContainer();
    await container.start();
    await container.waitForReady();

    fixtures = new Neo4jFixtures(container);
  });

  it('should seed knowledge graph', async () => {
    const graph = await fixtures.seedKnowledgeGraph({
      userCount: 10,
      contentCount: 20,
      topics: ['meditation', 'yoga', 'mindfulness'],
    });

    expect(graph.userIds).toHaveLength(10);
    expect(graph.contentIds).toHaveLength(20);
  });
});
```

## Cleanup Utilities

### Basic Cleanup

```typescript
import {
  cleanupPostgres,
  cleanupRedis,
  cleanupNeo4j,
} from '@lilith/testing/helpers';

afterEach(async () => {
  await cleanupPostgres(postgresContainer);
  await cleanupRedis(redisContainer);
  await cleanupNeo4j(neo4jContainer);
});
```

### Cleanup Manager

For managing multiple databases:

```typescript
import { CleanupManager } from '@lilith/testing/helpers';

describe('Multi-Database Test', () => {
  let cleanupManager: CleanupManager;
  let postgres: PostgresContainer;
  let redis: RedisContainer;

  beforeAll(async () => {
    cleanupManager = new CleanupManager();

    postgres = new PostgresContainer();
    redis = new RedisContainer();

    await postgres.start();
    await redis.start();

    cleanupManager.registerPostgres(postgres);
    cleanupManager.registerRedis(redis);
  });

  afterEach(async () => {
    const result = await cleanupManager.cleanupAll();
    if (!result.success) {
      console.warn('Cleanup errors:', result.errors);
    }
  });

  afterAll(async () => {
    await postgres.stop();
    await redis.stop();
  });
});
```

### Cleanup Options

```typescript
// PostgreSQL cleanup options
await cleanupPostgres(container, {
  excludeTables: ['migrations'], // Don't truncate these tables
  includeTables: ['users', 'posts'], // Only truncate these tables
  resetSequences: true, // Reset auto-increment sequences
  vacuum: true, // Run VACUUM ANALYZE
});

// Redis cleanup options
await cleanupRedis(container, {
  patterns: ['session:*', 'cache:*'], // Only delete matching keys
  flushAll: false, // Don't flush entire database
});

// Neo4j cleanup options
await cleanupNeo4j(container, {
  excludeLabels: ['Config'], // Don't delete these node labels
  includeLabels: ['User', 'Content'], // Only delete these labels
  dropConstraints: false, // Keep constraints
  dropIndexes: false, // Keep indexes
});
```

## Best Practices

### 1. Container Timeouts

Always set appropriate timeouts for container startup:

```typescript
beforeAll(async () => {
  container = new PostgresContainer();
  await container.start();
}, 60000); // 60 seconds for Postgres

beforeAll(async () => {
  container = new MilvusContainer();
  await container.start();
}, 120000); // 120 seconds for Milvus
```

### 2. Parallel Test Isolation

Use unique databases for parallel test files:

```typescript
const testId = process.env.VITEST_POOL_ID || '0';

container = new PostgresContainer({
  database: `test_db_${testId}`,
});
```

### 3. Reuse Containers in Test Suites

Don't restart containers between tests:

```typescript
// Good - reuse container
beforeAll(async () => {
  container = new PostgresContainer();
  await container.start();
});

afterEach(async () => {
  await cleanupPostgres(container);
});

afterAll(async () => {
  await container.stop();
});
```

### 4. Use Fixtures for Complex Data

```typescript
// Good - use fixtures for complex seeding
const { userIds, contentIds, topics } = await fixtures.seedKnowledgeGraph({
  userCount: 10,
  contentCount: 50,
});

// Avoid - manual seeding for each test
for (let i = 0; i < 10; i++) {
  await container.exec(`INSERT INTO users...`);
}
```

### 5. Health Checks Before Tests

```typescript
beforeAll(async () => {
  container = new PostgresContainer();
  await container.start();

  // Wait for container to be fully ready
  const isHealthy = await container.isHealthy();
  expect(isHealthy).toBe(true);
});
```

## CI/CD Configuration

### GitHub Actions

```yaml
name: Integration Tests

on: [push, pull_request]

jobs:
  integration-tests:
    runs-on: ubuntu-latest

    services:
      # Note: testcontainers manages its own containers,
      # but you may need Docker available

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run integration tests
        run: npm run test:integration
        env:
          TESTCONTAINERS_RYUK_DISABLED: 'true' # If using Ryuk causes issues
```

### Environment Variables

```bash
# Container configuration
TESTCONTAINERS_RYUK_DISABLED=true  # Disable resource reaper if needed
DOCKER_HOST=unix:///var/run/docker.sock

# Test configuration
TEST_POSTGRES_IMAGE=postgres:16-alpine
TEST_REDIS_IMAGE=redis:7-alpine
TEST_MILVUS_IMAGE=milvusdb/milvus:v2.3-latest
TEST_NEO4J_IMAGE=neo4j:5-community
```

## Troubleshooting

### Container Startup Failures

1. **Check Docker is running**: `docker info`
2. **Check available memory**: Containers need sufficient RAM
3. **Check port conflicts**: Other services might use required ports
4. **Increase timeouts**: Some containers take longer on first pull

### Test Isolation Issues

1. **Always use cleanup**: Call cleanup after each test
2. **Check for shared state**: Ensure tests don't depend on each other
3. **Use unique identifiers**: Generate unique IDs for test data

### Slow Tests

1. **Reuse containers**: Don't restart between tests
2. **Use lightweight images**: Alpine variants are faster
3. **Batch operations**: Seed data in batches
4. **Parallelize carefully**: Balance between speed and resource usage

## See Also

- [External Integration Tests](./external-integration-tests.md) - Testing
  external APIs
- [Unit Testing Guide](./unit-testing.md) - Mocking and unit tests
- [E2E Testing Guide](./e2e-testing.md) - End-to-end tests with Playwright
