# ElevenLabs Integration Guide

This guide covers integration with the ElevenLabs API for text-to-speech synthesis, voice cloning, and real-time streaming.

## Overview

The `ElevenLabsProvider` provides a comprehensive interface to ElevenLabs capabilities:

- **Text-to-Speech**: Multiple models from ultra-low latency to highly expressive
- **Voice Library**: 1000+ premade voices with filtering and search
- **Voice Cloning**: Instant and professional voice cloning
- **Voice Design**: Generate new voices from text descriptions
- **WebSocket Streaming**: Real-time audio generation
- **Audio Tags**: Control pauses, emphasis, and delivery (Eleven v3)

## Installation & Setup

### Environment Variables

```bash
export ELEVENLABS_API_KEY="your-api-key"
```

### Basic Initialization

```typescript
import { ElevenLabsProvider } from '../providers/elevenlabs-provider.js';

const provider = new ElevenLabsProvider({
  apiKey: process.env.ELEVENLABS_API_KEY,
  defaultVoiceId: 'EXAVITQu4vr4xnSDxMaL',  // Sarah
  defaultModel: 'eleven_v3',
  timeout: 30000
});
```

### Configuration Options

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `apiKey` | string | `ELEVENLABS_API_KEY` env | Your ElevenLabs API key |
| `baseUrl` | string | `https://api.elevenlabs.io/v1` | REST API base URL |
| `websocketUrl` | string | `wss://api.elevenlabs.io/v1/text-to-speech` | WebSocket URL |
| `defaultVoiceId` | string | - | Default voice ID |
| `defaultModel` | ElevenLabsModelId | `eleven_v3` | Default TTS model |
| `timeout` | number | `30000` | Request timeout (ms) |
| `maxRetries` | number | `3` | Max retry attempts |
| `voiceCacheTTLSeconds` | number | `300` | Voice list cache TTL |
| `websocketTimeoutSeconds` | number | `20` | WebSocket inactivity timeout |
| `logRequests` | boolean | `false` | Enable request logging |

## Text-to-Speech

### Basic Synthesis

```typescript
const response = await provider.synthesize({
  text: 'Welcome to your meditation practice.',
  voice_id: 'EXAVITQu4vr4xnSDxMaL'  // Sarah
});

// response.audio is a Buffer containing MP3 data
fs.writeFileSync('output.mp3', response.audio);
```

### With Voice Settings

```typescript
const response = await provider.synthesize({
  text: 'Take a deep breath and relax.',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  model_id: 'eleven_v3',
  voice_settings: {
    stability: 0.7,        // 0-1: Higher = more consistent
    similarity_boost: 0.8, // 0-1: Higher = more similar to original
    style: 0.3,            // 0-1: Style exaggeration (v3 only)
    use_speaker_boost: true
  }
});
```

### Audio Output Formats

```typescript
// High quality MP3
const mp3 = await provider.synthesize({
  text: 'Hello',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  output_format: 'mp3_44100_192'  // 192kbps
});

// Raw PCM for processing
const pcm = await provider.synthesize({
  text: 'Hello',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  output_format: 'pcm_44100'
});
```

### Available Formats

| Format | Description |
|--------|-------------|
| `mp3_22050_32` | MP3 22.05kHz 32kbps |
| `mp3_44100_32` | MP3 44.1kHz 32kbps |
| `mp3_44100_64` | MP3 44.1kHz 64kbps |
| `mp3_44100_96` | MP3 44.1kHz 96kbps |
| `mp3_44100_128` | MP3 44.1kHz 128kbps |
| `mp3_44100_192` | MP3 44.1kHz 192kbps (highest) |
| `pcm_16000` | PCM 16kHz |
| `pcm_22050` | PCM 22.05kHz |
| `pcm_24000` | PCM 24kHz |
| `pcm_44100` | PCM 44.1kHz |
| `ulaw_8000` | μ-law 8kHz (telephony) |

### With Word Timestamps

```typescript
const response = await provider.synthesize({
  text: 'Welcome to your guided meditation.',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  with_timestamps: true
});

// Word-level timing
response.alignment?.words?.forEach(word => {
  console.log(`${word.word}: ${word.start_time}s - ${word.end_time}s`);
});
```

## Models

### Available Models

| Model | Latency | Languages | Best For |
|-------|---------|-----------|----------|
| `eleven_v3` | Standard | 32 | Highest quality, audio tags |
| `eleven_flash_v2_5` | Ultra-low (~75ms) | 32 | Real-time applications |
| `eleven_turbo_v2_5` | Low | 32 | Fast with good quality |
| `eleven_multilingual_v2` | Standard | 29 | Multi-language content |
| `eleven_english_v1` | Low | English | English-only content |

### Model Selection

```typescript
// Real-time conversational AI
const flash = await provider.synthesize({
  text: 'Response text',
  voice_id: 'voice_id',
  model_id: 'eleven_flash_v2_5'
});

// High-quality meditation content
const v3 = await provider.synthesize({
  text: 'Meditation script',
  voice_id: 'voice_id',
  model_id: 'eleven_v3'
});
```

### Get Available Models

```typescript
const models = await provider.getModels();
models.forEach(model => {
  console.log(`${model.model_id}: ${model.name}`);
  console.log(`  Languages: ${model.languages?.length || 0}`);
  console.log(`  Can use style: ${model.can_use_style}`);
});
```

## Audio Tags (Eleven v3)

Audio tags provide fine-grained control over speech delivery. Only supported by `eleven_v3`.

### Pauses

```typescript
// Short pause (200ms)
const text = 'Take a breath. <break time="0.2s" /> Now relax.';

// Longer pause (2 seconds)
const text = 'Close your eyes. <break time="2s" /> Begin your journey.';
```

### Pronunciation

```typescript
// Phonetic spelling
const text = 'The word <phoneme alphabet="ipa" ph="naːmɑːsteɪ">namaste</phoneme> means...';

// Alternative pronunciation
const text = 'Say <say-as interpret-as="characters">ABC</say-as>';
```

### Emphasis

```typescript
// Emphasize words
const text = 'This is <emphasis level="strong">very</emphasis> important.';
```

### Full Example

```typescript
const script = `
Welcome to your meditation. <break time="1s" />
Find a comfortable position. <break time="2s" />
Take a <emphasis level="moderate">deep</emphasis> breath in...
<break time="3s" />
And slowly exhale. <break time="3s" />
<phoneme alphabet="ipa" ph="naːmɑːsteɪ">Namaste</phoneme>.
`;

const response = await provider.synthesize({
  text: script,
  voice_id: 'meditation_voice_id',
  model_id: 'eleven_v3'
});
```

## Voice Management

### List Available Voices

```typescript
const voices = await provider.listVoices();
console.log(`Found ${voices.length} voices`);

voices.forEach(voice => {
  console.log(`${voice.name} (${voice.voice_id})`);
  console.log(`  Category: ${voice.category}`);
  console.log(`  Labels: ${JSON.stringify(voice.labels)}`);
});
```

### Filter Voices

```typescript
// Get voices with specific characteristics
const voices = await provider.listVoices();
const calmVoices = voices.filter(v =>
  v.labels?.['accent'] === 'american' &&
  v.labels?.['description']?.includes('calm')
);
```

### Get Voice Details

```typescript
const voice = await provider.getVoice('EXAVITQu4vr4xnSDxMaL');
console.log(`Name: ${voice.name}`);
console.log(`Preview: ${voice.preview_url}`);
console.log(`Settings: ${JSON.stringify(voice.settings)}`);
```

### Get Voice Settings

```typescript
const settings = await provider.getVoiceSettings('voice_id');
console.log(`Stability: ${settings.stability}`);
console.log(`Similarity: ${settings.similarity_boost}`);
```

### Update Voice Settings

```typescript
await provider.updateVoiceSettings('voice_id', {
  stability: 0.8,
  similarity_boost: 0.75,
  style: 0.2,
  use_speaker_boost: true
});
```

## Voice Cloning

### Instant Voice Clone

```typescript
const clonedVoice = await provider.cloneVoiceInstant({
  name: 'My Custom Voice',
  files: [
    fs.readFileSync('sample1.mp3'),
    fs.readFileSync('sample2.mp3')
  ],
  description: 'A calm meditation guide voice',
  labels: {
    accent: 'american',
    age: 'middle-aged',
    gender: 'male'
  },
  remove_background_noise: true
});

// Use the cloned voice
const response = await provider.synthesize({
  text: 'Hello from my cloned voice.',
  voice_id: clonedVoice.voice_id
});
```

### Professional Voice Clone

For higher quality cloning with more samples:

```typescript
const proVoice = await provider.cloneVoiceProfessional({
  name: 'Professional Voice',
  files: [/* 30+ minutes of audio */],
  description: 'Professional meditation instructor',
  labels: { ... }
});
```

## Voice Design

Generate new voices from text descriptions:

```typescript
// Preview a generated voice
const preview = await provider.designVoice({
  gender: 'female',
  age: 'middle_aged',
  accent: 'british',
  accent_strength: 1.2,
  text: 'Sample text for preview'
});

// Listen to preview audio
fs.writeFileSync('preview.mp3', preview.audio);

// Save the voice permanently
if (satisfied) {
  const savedVoice = await provider.saveVoiceDesign({
    voice_description: 'A calm British meditation instructor',
    voice_name: 'Meditation Guide',
    generated_voice_id: preview.generated_voice_id,
    labels: {
      accent: 'british',
      use_case: 'meditation'
    }
  });
}
```

## WebSocket Streaming

### Real-time Streaming

```typescript
const session = await provider.startStreaming({
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  model_id: 'eleven_flash_v2_5',  // Low latency for streaming
  output_format: 'pcm_24000',
  inactivity_timeout: 30
});

// Send text chunks
await provider.sendStreamingText(session.sessionId, 'Hello ');
await provider.sendStreamingText(session.sessionId, 'world!');

// Listen for audio chunks
provider.on('streaming-audio', (event) => {
  if (event.sessionId === session.sessionId) {
    // Process audio chunk
    playAudio(event.audio);
  }
});

// Close when done
await provider.closeStreaming(session.sessionId);
```

### Streaming Events

```typescript
provider.on('streaming-audio', (event) => {
  // Audio chunk received
  console.log(`Audio chunk: ${event.audio.length} bytes`);
});

provider.on('streaming-error', (event) => {
  console.error(`Stream error: ${event.error}`);
});

provider.on('streaming-closed', (event) => {
  console.log(`Stream closed: ${event.sessionId}`);
});
```

## Usage & Quota Tracking

### Get Usage Info

```typescript
const usage = await provider.getUsage();
console.log(`Characters used: ${usage.character_count}`);
console.log(`Character limit: ${usage.character_limit}`);
console.log(`Can extend: ${usage.can_extend_character_limit}`);
```

### Get Subscription Info

```typescript
const subscription = await provider.getSubscription();
console.log(`Tier: ${subscription.tier}`);
console.log(`Next reset: ${subscription.next_invoice?.date}`);
```

### Track Statistics

```typescript
const stats = provider.getStats();
console.log(`Total requests: ${stats.totalRequests}`);
console.log(`Success rate: ${stats.successfulRequests / stats.totalRequests * 100}%`);
console.log(`Characters synthesized: ${stats.charactersSynthesized}`);
console.log(`Average latency: ${stats.averageLatencyMs}ms`);

// Per-model breakdown
console.log('Requests by model:', stats.requestsByModel);

// Per-voice breakdown
console.log('Requests by voice:', stats.requestsByVoice);
```

## Event Handling

```typescript
// Synthesis events
provider.on('synthesis-start', (event) => {
  console.log(`Starting synthesis: ${event.voiceId}`);
});

provider.on('synthesis-complete', (event) => {
  console.log(`Completed: ${event.characterCount} chars in ${event.latencyMs}ms`);
});

provider.on('synthesis-error', (event) => {
  console.error(`Error: ${event.error}`);
});

// Streaming events
provider.on('streaming-started', (event) => {
  console.log(`Stream started: ${event.sessionId}`);
});

provider.on('streaming-audio', (event) => {
  // Handle audio chunk
});

// Voice events
provider.on('voice-cloned', (event) => {
  console.log(`Voice cloned: ${event.voiceId}`);
});
```

## Error Handling

```typescript
import { ElevenLabsProviderError } from '../providers/elevenlabs-provider.js';

try {
  await provider.synthesize({ text: 'Hello', voice_id: 'invalid' });
} catch (error) {
  if (error instanceof ElevenLabsProviderError) {
    switch (error.code) {
      case 'INVALID_API_KEY':
        console.log('Check your API key');
        break;
      case 'VOICE_NOT_FOUND':
        console.log('Voice ID is invalid');
        break;
      case 'QUOTA_EXCEEDED':
        console.log('Character limit reached');
        break;
      case 'RATE_LIMITED':
        console.log('Too many requests, wait and retry');
        break;
      case 'MODEL_NOT_AVAILABLE':
        console.log('Model temporarily unavailable');
        break;
      default:
        console.log(`API error: ${error.message}`);
    }
  }
}
```

## Best Practices

### 1. Choose the Right Model

```typescript
// Real-time conversation: Flash for lowest latency
model_id: 'eleven_flash_v2_5'

// Pre-generated content: v3 for highest quality
model_id: 'eleven_v3'

// Multi-language: Multilingual v2
model_id: 'eleven_multilingual_v2'
```

### 2. Optimize Voice Settings

```typescript
// For meditation (calm, consistent)
voice_settings: {
  stability: 0.8,        // High stability for calm delivery
  similarity_boost: 0.7,
  style: 0.2,            // Low style for neutral tone
  use_speaker_boost: true
}

// For storytelling (expressive)
voice_settings: {
  stability: 0.5,        // Lower for variation
  similarity_boost: 0.8,
  style: 0.6,            // Higher for expression
  use_speaker_boost: true
}
```

### 3. Use Audio Tags Wisely

```typescript
// Add natural pauses
const script = 'Take a deep breath. <break time="2s" /> Now exhale slowly.';

// Don't overuse - it can sound unnatural
// Bad: 'Take <break time="0.1s" /> a <break time="0.1s" /> breath'
```

### 4. Handle Long Text

```typescript
// Split long text into chunks
const chunks = splitTextIntoChunks(longText, 5000);
const audioBuffers = await Promise.all(
  chunks.map(chunk => provider.synthesize({ text: chunk, voice_id }))
);
const combined = Buffer.concat(audioBuffers.map(r => r.audio));
```

### 5. Cache Voices

```typescript
// Voice list is cached automatically (voiceCacheTTLSeconds)
// Force refresh when needed
await provider.listVoices({ forceRefresh: true });
```

## Recommended Voices for Meditation

| Voice | ID | Style | Best For |
|-------|-----|-------|----------|
| Sarah | `EXAVITQu4vr4xnSDxMaL` | Calm, warm | Guided meditation |
| Daniel | `onwK4e9ZLuTAKqWW03F9` | Soothing male | Sleep stories |
| Charlotte | `XB0fDUnXU5powFXDhCwa` | Gentle British | Mindfulness |
| Adam | `pNInz6obpgDQGcFmaJgB` | Deep, calm | Body scan |

## Related Resources

- [OpenRouter Integration Guide](./openrouter-integration.md)
- [Content Generation API Docs](./content-generation-api.md)
- [ElevenLabs API Docs](https://docs.elevenlabs.io/)