This guide covers integration with the ElevenLabs API for text-to-speech synthesis, voice cloning, and real-time streaming.
Overview#
The ElevenLabsProvider provides a comprehensive interface to ElevenLabs capabilities:
- Text-to-Speech: Multiple models from ultra-low latency to highly expressive
- Voice Library: 1000+ premade voices with filtering and search
- Voice Cloning: Instant and professional voice cloning
- Voice Design: Generate new voices from text descriptions
- WebSocket Streaming: Real-time audio generation
- Audio Tags: Control pauses, emphasis, and delivery (Eleven v3)
Installation & Setup#
Environment Variables#
bash
export ELEVENLABS_API_KEY="your-api-key"
Basic Initialization#
typescript
import { ElevenLabsProvider } from '../providers/elevenlabs-provider.js';
const provider = new ElevenLabsProvider({
apiKey: process.env.ELEVENLABS_API_KEY,
defaultVoiceId: 'EXAVITQu4vr4xnSDxMaL', // Sarah
defaultModel: 'eleven_v3',
timeout: 30000
});
Configuration Options#
| Option | Type | Default | Description |
|---|---|---|---|
apiKey |
string | ELEVENLABS_API_KEY env |
Your ElevenLabs API key |
baseUrl |
string | https://api.elevenlabs.io/v1 |
REST API base URL |
websocketUrl |
string | wss://api.elevenlabs.io/v1/text-to-speech |
WebSocket URL |
defaultVoiceId |
string | - | Default voice ID |
defaultModel |
ElevenLabsModelId | eleven_v3 |
Default TTS model |
timeout |
number | 30000 |
Request timeout (ms) |
maxRetries |
number | 3 |
Max retry attempts |
voiceCacheTTLSeconds |
number | 300 |
Voice list cache TTL |
websocketTimeoutSeconds |
number | 20 |
WebSocket inactivity timeout |
logRequests |
boolean | false |
Enable request logging |
Text-to-Speech#
Basic Synthesis#
typescript
const response = await provider.synthesize({
text: 'Welcome to your meditation practice.',
voice_id: 'EXAVITQu4vr4xnSDxMaL' // Sarah
});
// response.audio is a Buffer containing MP3 data
fs.writeFileSync('output.mp3', response.audio);
With Voice Settings#
typescript
const response = await provider.synthesize({
text: 'Take a deep breath and relax.',
voice_id: 'EXAVITQu4vr4xnSDxMaL',
model_id: 'eleven_v3',
voice_settings: {
stability: 0.7, // 0-1: Higher = more consistent
similarity_boost: 0.8, // 0-1: Higher = more similar to original
style: 0.3, // 0-1: Style exaggeration (v3 only)
use_speaker_boost: true
}
});
Audio Output Formats#
typescript
// High quality MP3
const mp3 = await provider.synthesize({
text: 'Hello',
voice_id: 'EXAVITQu4vr4xnSDxMaL',
output_format: 'mp3_44100_192' // 192kbps
});
// Raw PCM for processing
const pcm = await provider.synthesize({
text: 'Hello',
voice_id: 'EXAVITQu4vr4xnSDxMaL',
output_format: 'pcm_44100'
});
Available Formats#
| Format | Description |
|---|---|
mp3_22050_32 |
MP3 22.05kHz 32kbps |
mp3_44100_32 |
MP3 44.1kHz 32kbps |
mp3_44100_64 |
MP3 44.1kHz 64kbps |
mp3_44100_96 |
MP3 44.1kHz 96kbps |
mp3_44100_128 |
MP3 44.1kHz 128kbps |
mp3_44100_192 |
MP3 44.1kHz 192kbps (highest) |
pcm_16000 |
PCM 16kHz |
pcm_22050 |
PCM 22.05kHz |
pcm_24000 |
PCM 24kHz |
pcm_44100 |
PCM 44.1kHz |
ulaw_8000 |
μ-law 8kHz (telephony) |
With Word Timestamps#
typescript
const response = await provider.synthesize({
text: 'Welcome to your guided meditation.',
voice_id: 'EXAVITQu4vr4xnSDxMaL',
with_timestamps: true
});
// Word-level timing
response.alignment?.words?.forEach(word => {
console.log(`${word.word}: ${word.start_time}s - ${word.end_time}s`);
});
Models#
Available Models#
| Model | Latency | Languages | Best For |
|---|---|---|---|
eleven_v3 |
Standard | 32 | Highest quality, audio tags |
eleven_flash_v2_5 |
Ultra-low (~75ms) | 32 | Real-time applications |
eleven_turbo_v2_5 |
Low | 32 | Fast with good quality |
eleven_multilingual_v2 |
Standard | 29 | Multi-language content |
eleven_english_v1 |
Low | English | English-only content |
Model Selection#
typescript
// Real-time conversational AI
const flash = await provider.synthesize({
text: 'Response text',
voice_id: 'voice_id',
model_id: 'eleven_flash_v2_5'
});
// High-quality meditation content
const v3 = await provider.synthesize({
text: 'Meditation script',
voice_id: 'voice_id',
model_id: 'eleven_v3'
});
Get Available Models#
typescript
const models = await provider.getModels();
models.forEach(model => {
console.log(`${model.model_id}: ${model.name}`);
console.log(` Languages: ${model.languages?.length || 0}`);
console.log(` Can use style: ${model.can_use_style}`);
});
Audio Tags (Eleven v3)#
Audio tags provide fine-grained control over speech delivery. Only supported by eleven_v3.
Pauses#
typescript
// Short pause (200ms)
const text = 'Take a breath. <break time="0.2s" /> Now relax.';
// Longer pause (2 seconds)
const text = 'Close your eyes. <break time="2s" /> Begin your journey.';
Pronunciation#
typescript
// Phonetic spelling
const text = 'The word <phoneme alphabet="ipa" ph="naːmɑːsteɪ">namaste</phoneme> means...';
// Alternative pronunciation
const text = 'Say <say-as interpret-as="characters">ABC</say-as>';
Emphasis#
typescript
// Emphasize words
const text = 'This is <emphasis level="strong">very</emphasis> important.';
Full Example#
typescript
const script = `
Welcome to your meditation. <break time="1s" />
Find a comfortable position. <break time="2s" />
Take a <emphasis level="moderate">deep</emphasis> breath in...
<break time="3s" />
And slowly exhale. <break time="3s" />
<phoneme alphabet="ipa" ph="naːmɑːsteɪ">Namaste</phoneme>.
`;
const response = await provider.synthesize({
text: script,
voice_id: 'meditation_voice_id',
model_id: 'eleven_v3'
});
Voice Management#
List Available Voices#
typescript
const voices = await provider.listVoices();
console.log(`Found ${voices.length} voices`);
voices.forEach(voice => {
console.log(`${voice.name} (${voice.voice_id})`);
console.log(` Category: ${voice.category}`);
console.log(` Labels: ${JSON.stringify(voice.labels)}`);
});
Filter Voices#
typescript
// Get voices with specific characteristics
const voices = await provider.listVoices();
const calmVoices = voices.filter(v =>
v.labels?.['accent'] === 'american' &&
v.labels?.['description']?.includes('calm')
);
Get Voice Details#
typescript
const voice = await provider.getVoice('EXAVITQu4vr4xnSDxMaL');
console.log(`Name: ${voice.name}`);
console.log(`Preview: ${voice.preview_url}`);
console.log(`Settings: ${JSON.stringify(voice.settings)}`);
Get Voice Settings#
typescript
const settings = await provider.getVoiceSettings('voice_id');
console.log(`Stability: ${settings.stability}`);
console.log(`Similarity: ${settings.similarity_boost}`);
Update Voice Settings#
typescript
await provider.updateVoiceSettings('voice_id', {
stability: 0.8,
similarity_boost: 0.75,
style: 0.2,
use_speaker_boost: true
});
Voice Cloning#
Instant Voice Clone#
typescript
const clonedVoice = await provider.cloneVoiceInstant({
name: 'My Custom Voice',
files: [
fs.readFileSync('sample1.mp3'),
fs.readFileSync('sample2.mp3')
],
description: 'A calm meditation guide voice',
labels: {
accent: 'american',
age: 'middle-aged',
gender: 'male'
},
remove_background_noise: true
});
// Use the cloned voice
const response = await provider.synthesize({
text: 'Hello from my cloned voice.',
voice_id: clonedVoice.voice_id
});
Professional Voice Clone#
For higher quality cloning with more samples:
typescript
const proVoice = await provider.cloneVoiceProfessional({
name: 'Professional Voice',
files: [/* 30+ minutes of audio */],
description: 'Professional meditation instructor',
labels: { ... }
});
Voice Design#
Generate new voices from text descriptions:
typescript
// Preview a generated voice
const preview = await provider.designVoice({
gender: 'female',
age: 'middle_aged',
accent: 'british',
accent_strength: 1.2,
text: 'Sample text for preview'
});
// Listen to preview audio
fs.writeFileSync('preview.mp3', preview.audio);
// Save the voice permanently
if (satisfied) {
const savedVoice = await provider.saveVoiceDesign({
voice_description: 'A calm British meditation instructor',
voice_name: 'Meditation Guide',
generated_voice_id: preview.generated_voice_id,
labels: {
accent: 'british',
use_case: 'meditation'
}
});
}
WebSocket Streaming#
Real-time Streaming#
typescript
const session = await provider.startStreaming({
voice_id: 'EXAVITQu4vr4xnSDxMaL',
model_id: 'eleven_flash_v2_5', // Low latency for streaming
output_format: 'pcm_24000',
inactivity_timeout: 30
});
// Send text chunks
await provider.sendStreamingText(session.sessionId, 'Hello ');
await provider.sendStreamingText(session.sessionId, 'world!');
// Listen for audio chunks
provider.on('streaming-audio', (event) => {
if (event.sessionId === session.sessionId) {
// Process audio chunk
playAudio(event.audio);
}
});
// Close when done
await provider.closeStreaming(session.sessionId);
Streaming Events#
typescript
provider.on('streaming-audio', (event) => {
// Audio chunk received
console.log(`Audio chunk: ${event.audio.length} bytes`);
});
provider.on('streaming-error', (event) => {
console.error(`Stream error: ${event.error}`);
});
provider.on('streaming-closed', (event) => {
console.log(`Stream closed: ${event.sessionId}`);
});
Usage & Quota Tracking#
Get Usage Info#
typescript
const usage = await provider.getUsage();
console.log(`Characters used: ${usage.character_count}`);
console.log(`Character limit: ${usage.character_limit}`);
console.log(`Can extend: ${usage.can_extend_character_limit}`);
Get Subscription Info#
typescript
const subscription = await provider.getSubscription();
console.log(`Tier: ${subscription.tier}`);
console.log(`Next reset: ${subscription.next_invoice?.date}`);
Track Statistics#
typescript
const stats = provider.getStats();
console.log(`Total requests: ${stats.totalRequests}`);
console.log(`Success rate: ${stats.successfulRequests / stats.totalRequests * 100}%`);
console.log(`Characters synthesized: ${stats.charactersSynthesized}`);
console.log(`Average latency: ${stats.averageLatencyMs}ms`);
// Per-model breakdown
console.log('Requests by model:', stats.requestsByModel);
// Per-voice breakdown
console.log('Requests by voice:', stats.requestsByVoice);
Event Handling#
typescript
// Synthesis events
provider.on('synthesis-start', (event) => {
console.log(`Starting synthesis: ${event.voiceId}`);
});
provider.on('synthesis-complete', (event) => {
console.log(`Completed: ${event.characterCount} chars in ${event.latencyMs}ms`);
});
provider.on('synthesis-error', (event) => {
console.error(`Error: ${event.error}`);
});
// Streaming events
provider.on('streaming-started', (event) => {
console.log(`Stream started: ${event.sessionId}`);
});
provider.on('streaming-audio', (event) => {
// Handle audio chunk
});
// Voice events
provider.on('voice-cloned', (event) => {
console.log(`Voice cloned: ${event.voiceId}`);
});
Error Handling#
typescript
import { ElevenLabsProviderError } from '../providers/elevenlabs-provider.js';
try {
await provider.synthesize({ text: 'Hello', voice_id: 'invalid' });
} catch (error) {
if (error instanceof ElevenLabsProviderError) {
switch (error.code) {
case 'INVALID_API_KEY':
console.log('Check your API key');
break;
case 'VOICE_NOT_FOUND':
console.log('Voice ID is invalid');
break;
case 'QUOTA_EXCEEDED':
console.log('Character limit reached');
break;
case 'RATE_LIMITED':
console.log('Too many requests, wait and retry');
break;
case 'MODEL_NOT_AVAILABLE':
console.log('Model temporarily unavailable');
break;
default:
console.log(`API error: ${error.message}`);
}
}
}
Best Practices#
1. Choose the Right Model#
typescript
// Real-time conversation: Flash for lowest latency
model_id: 'eleven_flash_v2_5'
// Pre-generated content: v3 for highest quality
model_id: 'eleven_v3'
// Multi-language: Multilingual v2
model_id: 'eleven_multilingual_v2'
2. Optimize Voice Settings#
typescript
// For meditation (calm, consistent)
voice_settings: {
stability: 0.8, // High stability for calm delivery
similarity_boost: 0.7,
style: 0.2, // Low style for neutral tone
use_speaker_boost: true
}
// For storytelling (expressive)
voice_settings: {
stability: 0.5, // Lower for variation
similarity_boost: 0.8,
style: 0.6, // Higher for expression
use_speaker_boost: true
}
3. Use Audio Tags Wisely#
typescript
// Add natural pauses
const script = 'Take a deep breath. <break time="2s" /> Now exhale slowly.';
// Don't overuse - it can sound unnatural
// Bad: 'Take <break time="0.1s" /> a <break time="0.1s" /> breath'
4. Handle Long Text#
typescript
// Split long text into chunks
const chunks = splitTextIntoChunks(longText, 5000);
const audioBuffers = await Promise.all(
chunks.map(chunk => provider.synthesize({ text: chunk, voice_id }))
);
const combined = Buffer.concat(audioBuffers.map(r => r.audio));
5. Cache Voices#
typescript
// Voice list is cached automatically (voiceCacheTTLSeconds)
// Force refresh when needed
await provider.listVoices({ forceRefresh: true });
Recommended Voices for Meditation#
| Voice | ID | Style | Best For |
|---|---|---|---|
| Sarah | EXAVITQu4vr4xnSDxMaL |
Calm, warm | Guided meditation |
| Daniel | onwK4e9ZLuTAKqWW03F9 |
Soothing male | Sleep stories |
| Charlotte | XB0fDUnXU5powFXDhCwa |
Gentle British | Mindfulness |
| Adam | pNInz6obpgDQGcFmaJgB |
Deep, calm | Body scan |