Eve library

ElevenLabs Integration Guide

Guide

Revision 1

On this page

This guide covers integration with the ElevenLabs API for text-to-speech synthesis, voice cloning, and real-time streaming.

Overview#

The ElevenLabsProvider provides a comprehensive interface to ElevenLabs capabilities:

  • Text-to-Speech: Multiple models from ultra-low latency to highly expressive
  • Voice Library: 1000+ premade voices with filtering and search
  • Voice Cloning: Instant and professional voice cloning
  • Voice Design: Generate new voices from text descriptions
  • WebSocket Streaming: Real-time audio generation
  • Audio Tags: Control pauses, emphasis, and delivery (Eleven v3)

Installation & Setup#

Environment Variables#

bash
export ELEVENLABS_API_KEY="your-api-key"

Basic Initialization#

typescript
import { ElevenLabsProvider } from '../providers/elevenlabs-provider.js';

const provider = new ElevenLabsProvider({
  apiKey: process.env.ELEVENLABS_API_KEY,
  defaultVoiceId: 'EXAVITQu4vr4xnSDxMaL',  // Sarah
  defaultModel: 'eleven_v3',
  timeout: 30000
});

Configuration Options#

Option Type Default Description
apiKey string ELEVENLABS_API_KEY env Your ElevenLabs API key
baseUrl string https://api.elevenlabs.io/v1 REST API base URL
websocketUrl string wss://api.elevenlabs.io/v1/text-to-speech WebSocket URL
defaultVoiceId string - Default voice ID
defaultModel ElevenLabsModelId eleven_v3 Default TTS model
timeout number 30000 Request timeout (ms)
maxRetries number 3 Max retry attempts
voiceCacheTTLSeconds number 300 Voice list cache TTL
websocketTimeoutSeconds number 20 WebSocket inactivity timeout
logRequests boolean false Enable request logging

Text-to-Speech#

Basic Synthesis#

typescript
const response = await provider.synthesize({
  text: 'Welcome to your meditation practice.',
  voice_id: 'EXAVITQu4vr4xnSDxMaL'  // Sarah
});

// response.audio is a Buffer containing MP3 data
fs.writeFileSync('output.mp3', response.audio);

With Voice Settings#

typescript
const response = await provider.synthesize({
  text: 'Take a deep breath and relax.',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  model_id: 'eleven_v3',
  voice_settings: {
    stability: 0.7,        // 0-1: Higher = more consistent
    similarity_boost: 0.8, // 0-1: Higher = more similar to original
    style: 0.3,            // 0-1: Style exaggeration (v3 only)
    use_speaker_boost: true
  }
});

Audio Output Formats#

typescript
// High quality MP3
const mp3 = await provider.synthesize({
  text: 'Hello',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  output_format: 'mp3_44100_192'  // 192kbps
});

// Raw PCM for processing
const pcm = await provider.synthesize({
  text: 'Hello',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  output_format: 'pcm_44100'
});

Available Formats#

Format Description
mp3_22050_32 MP3 22.05kHz 32kbps
mp3_44100_32 MP3 44.1kHz 32kbps
mp3_44100_64 MP3 44.1kHz 64kbps
mp3_44100_96 MP3 44.1kHz 96kbps
mp3_44100_128 MP3 44.1kHz 128kbps
mp3_44100_192 MP3 44.1kHz 192kbps (highest)
pcm_16000 PCM 16kHz
pcm_22050 PCM 22.05kHz
pcm_24000 PCM 24kHz
pcm_44100 PCM 44.1kHz
ulaw_8000 μ-law 8kHz (telephony)

With Word Timestamps#

typescript
const response = await provider.synthesize({
  text: 'Welcome to your guided meditation.',
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  with_timestamps: true
});

// Word-level timing
response.alignment?.words?.forEach(word => {
  console.log(`${word.word}: ${word.start_time}s - ${word.end_time}s`);
});

Models#

Available Models#

Model Latency Languages Best For
eleven_v3 Standard 32 Highest quality, audio tags
eleven_flash_v2_5 Ultra-low (~75ms) 32 Real-time applications
eleven_turbo_v2_5 Low 32 Fast with good quality
eleven_multilingual_v2 Standard 29 Multi-language content
eleven_english_v1 Low English English-only content

Model Selection#

typescript
// Real-time conversational AI
const flash = await provider.synthesize({
  text: 'Response text',
  voice_id: 'voice_id',
  model_id: 'eleven_flash_v2_5'
});

// High-quality meditation content
const v3 = await provider.synthesize({
  text: 'Meditation script',
  voice_id: 'voice_id',
  model_id: 'eleven_v3'
});

Get Available Models#

typescript
const models = await provider.getModels();
models.forEach(model => {
  console.log(`${model.model_id}: ${model.name}`);
  console.log(`  Languages: ${model.languages?.length || 0}`);
  console.log(`  Can use style: ${model.can_use_style}`);
});

Audio Tags (Eleven v3)#

Audio tags provide fine-grained control over speech delivery. Only supported by eleven_v3.

Pauses#

typescript
// Short pause (200ms)
const text = 'Take a breath. <break time="0.2s" /> Now relax.';

// Longer pause (2 seconds)
const text = 'Close your eyes. <break time="2s" /> Begin your journey.';

Pronunciation#

typescript
// Phonetic spelling
const text = 'The word <phoneme alphabet="ipa" ph="naːmɑːsteɪ">namaste</phoneme> means...';

// Alternative pronunciation
const text = 'Say <say-as interpret-as="characters">ABC</say-as>';

Emphasis#

typescript
// Emphasize words
const text = 'This is <emphasis level="strong">very</emphasis> important.';

Full Example#

typescript
const script = `
Welcome to your meditation. <break time="1s" />
Find a comfortable position. <break time="2s" />
Take a <emphasis level="moderate">deep</emphasis> breath in...
<break time="3s" />
And slowly exhale. <break time="3s" />
<phoneme alphabet="ipa" ph="naːmɑːsteɪ">Namaste</phoneme>.
`;

const response = await provider.synthesize({
  text: script,
  voice_id: 'meditation_voice_id',
  model_id: 'eleven_v3'
});

Voice Management#

List Available Voices#

typescript
const voices = await provider.listVoices();
console.log(`Found ${voices.length} voices`);

voices.forEach(voice => {
  console.log(`${voice.name} (${voice.voice_id})`);
  console.log(`  Category: ${voice.category}`);
  console.log(`  Labels: ${JSON.stringify(voice.labels)}`);
});

Filter Voices#

typescript
// Get voices with specific characteristics
const voices = await provider.listVoices();
const calmVoices = voices.filter(v =>
  v.labels?.['accent'] === 'american' &&
  v.labels?.['description']?.includes('calm')
);

Get Voice Details#

typescript
const voice = await provider.getVoice('EXAVITQu4vr4xnSDxMaL');
console.log(`Name: ${voice.name}`);
console.log(`Preview: ${voice.preview_url}`);
console.log(`Settings: ${JSON.stringify(voice.settings)}`);

Get Voice Settings#

typescript
const settings = await provider.getVoiceSettings('voice_id');
console.log(`Stability: ${settings.stability}`);
console.log(`Similarity: ${settings.similarity_boost}`);

Update Voice Settings#

typescript
await provider.updateVoiceSettings('voice_id', {
  stability: 0.8,
  similarity_boost: 0.75,
  style: 0.2,
  use_speaker_boost: true
});

Voice Cloning#

Instant Voice Clone#

typescript
const clonedVoice = await provider.cloneVoiceInstant({
  name: 'My Custom Voice',
  files: [
    fs.readFileSync('sample1.mp3'),
    fs.readFileSync('sample2.mp3')
  ],
  description: 'A calm meditation guide voice',
  labels: {
    accent: 'american',
    age: 'middle-aged',
    gender: 'male'
  },
  remove_background_noise: true
});

// Use the cloned voice
const response = await provider.synthesize({
  text: 'Hello from my cloned voice.',
  voice_id: clonedVoice.voice_id
});

Professional Voice Clone#

For higher quality cloning with more samples:

typescript
const proVoice = await provider.cloneVoiceProfessional({
  name: 'Professional Voice',
  files: [/* 30+ minutes of audio */],
  description: 'Professional meditation instructor',
  labels: { ... }
});

Voice Design#

Generate new voices from text descriptions:

typescript
// Preview a generated voice
const preview = await provider.designVoice({
  gender: 'female',
  age: 'middle_aged',
  accent: 'british',
  accent_strength: 1.2,
  text: 'Sample text for preview'
});

// Listen to preview audio
fs.writeFileSync('preview.mp3', preview.audio);

// Save the voice permanently
if (satisfied) {
  const savedVoice = await provider.saveVoiceDesign({
    voice_description: 'A calm British meditation instructor',
    voice_name: 'Meditation Guide',
    generated_voice_id: preview.generated_voice_id,
    labels: {
      accent: 'british',
      use_case: 'meditation'
    }
  });
}

WebSocket Streaming#

Real-time Streaming#

typescript
const session = await provider.startStreaming({
  voice_id: 'EXAVITQu4vr4xnSDxMaL',
  model_id: 'eleven_flash_v2_5',  // Low latency for streaming
  output_format: 'pcm_24000',
  inactivity_timeout: 30
});

// Send text chunks
await provider.sendStreamingText(session.sessionId, 'Hello ');
await provider.sendStreamingText(session.sessionId, 'world!');

// Listen for audio chunks
provider.on('streaming-audio', (event) => {
  if (event.sessionId === session.sessionId) {
    // Process audio chunk
    playAudio(event.audio);
  }
});

// Close when done
await provider.closeStreaming(session.sessionId);

Streaming Events#

typescript
provider.on('streaming-audio', (event) => {
  // Audio chunk received
  console.log(`Audio chunk: ${event.audio.length} bytes`);
});

provider.on('streaming-error', (event) => {
  console.error(`Stream error: ${event.error}`);
});

provider.on('streaming-closed', (event) => {
  console.log(`Stream closed: ${event.sessionId}`);
});

Usage & Quota Tracking#

Get Usage Info#

typescript
const usage = await provider.getUsage();
console.log(`Characters used: ${usage.character_count}`);
console.log(`Character limit: ${usage.character_limit}`);
console.log(`Can extend: ${usage.can_extend_character_limit}`);

Get Subscription Info#

typescript
const subscription = await provider.getSubscription();
console.log(`Tier: ${subscription.tier}`);
console.log(`Next reset: ${subscription.next_invoice?.date}`);

Track Statistics#

typescript
const stats = provider.getStats();
console.log(`Total requests: ${stats.totalRequests}`);
console.log(`Success rate: ${stats.successfulRequests / stats.totalRequests * 100}%`);
console.log(`Characters synthesized: ${stats.charactersSynthesized}`);
console.log(`Average latency: ${stats.averageLatencyMs}ms`);

// Per-model breakdown
console.log('Requests by model:', stats.requestsByModel);

// Per-voice breakdown
console.log('Requests by voice:', stats.requestsByVoice);

Event Handling#

typescript
// Synthesis events
provider.on('synthesis-start', (event) => {
  console.log(`Starting synthesis: ${event.voiceId}`);
});

provider.on('synthesis-complete', (event) => {
  console.log(`Completed: ${event.characterCount} chars in ${event.latencyMs}ms`);
});

provider.on('synthesis-error', (event) => {
  console.error(`Error: ${event.error}`);
});

// Streaming events
provider.on('streaming-started', (event) => {
  console.log(`Stream started: ${event.sessionId}`);
});

provider.on('streaming-audio', (event) => {
  // Handle audio chunk
});

// Voice events
provider.on('voice-cloned', (event) => {
  console.log(`Voice cloned: ${event.voiceId}`);
});

Error Handling#

typescript
import { ElevenLabsProviderError } from '../providers/elevenlabs-provider.js';

try {
  await provider.synthesize({ text: 'Hello', voice_id: 'invalid' });
} catch (error) {
  if (error instanceof ElevenLabsProviderError) {
    switch (error.code) {
      case 'INVALID_API_KEY':
        console.log('Check your API key');
        break;
      case 'VOICE_NOT_FOUND':
        console.log('Voice ID is invalid');
        break;
      case 'QUOTA_EXCEEDED':
        console.log('Character limit reached');
        break;
      case 'RATE_LIMITED':
        console.log('Too many requests, wait and retry');
        break;
      case 'MODEL_NOT_AVAILABLE':
        console.log('Model temporarily unavailable');
        break;
      default:
        console.log(`API error: ${error.message}`);
    }
  }
}

Best Practices#

1. Choose the Right Model#

typescript
// Real-time conversation: Flash for lowest latency
model_id: 'eleven_flash_v2_5'

// Pre-generated content: v3 for highest quality
model_id: 'eleven_v3'

// Multi-language: Multilingual v2
model_id: 'eleven_multilingual_v2'

2. Optimize Voice Settings#

typescript
// For meditation (calm, consistent)
voice_settings: {
  stability: 0.8,        // High stability for calm delivery
  similarity_boost: 0.7,
  style: 0.2,            // Low style for neutral tone
  use_speaker_boost: true
}

// For storytelling (expressive)
voice_settings: {
  stability: 0.5,        // Lower for variation
  similarity_boost: 0.8,
  style: 0.6,            // Higher for expression
  use_speaker_boost: true
}

3. Use Audio Tags Wisely#

typescript
// Add natural pauses
const script = 'Take a deep breath. <break time="2s" /> Now exhale slowly.';

// Don't overuse - it can sound unnatural
// Bad: 'Take <break time="0.1s" /> a <break time="0.1s" /> breath'

4. Handle Long Text#

typescript
// Split long text into chunks
const chunks = splitTextIntoChunks(longText, 5000);
const audioBuffers = await Promise.all(
  chunks.map(chunk => provider.synthesize({ text: chunk, voice_id }))
);
const combined = Buffer.concat(audioBuffers.map(r => r.audio));

5. Cache Voices#

typescript
// Voice list is cached automatically (voiceCacheTTLSeconds)
// Force refresh when needed
await provider.listVoices({ forceRefresh: true });
Voice ID Style Best For
Sarah EXAVITQu4vr4xnSDxMaL Calm, warm Guided meditation
Daniel onwK4e9ZLuTAKqWW03F9 Soothing male Sleep stories
Charlotte XB0fDUnXU5powFXDhCwa Gentle British Mindfulness
Adam pNInz6obpgDQGcFmaJgB Deep, calm Body scan