Reference documentation for the ElevenLabs AI voice provider supporting text-to-speech, streaming, voice cloning, and audio tags.
Overview#
The ElevenLabs provider offers:
- Text-to-Speech: Multiple models (Flash, Turbo, Multilingual, v3)
- Streaming Audio: Real-time audio generation
- Voice Management: Listing, preview, cloning
- Audio Tags (v3): Performance cues for emotional delivery
- Model Listing: Available models and capabilities
- Retry Logic: Exponential backoff with rate limit handling
Configuration#
Environment Variables#
| Variable | Description | Required |
|---|---|---|
ELEVENLABS_API_KEY |
ElevenLabs API key | Yes |
Provider Initialization#
import { ElevenLabsProvider } from './providers/elevenlabs-provider';
const provider = new ElevenLabsProvider({
apiKey: process.env.ELEVENLABS_API_KEY,
baseURL: 'https://api.elevenlabs.io/v1',
maxRetries: 3,
retryDelayMs: 1000,
timeout: 30000,
defaultModel: 'eleven_multilingual_v2',
defaultVoiceId: '21m00Tcm4TlvDq8ikWAM',
defaultOutputFormat: 'mp3_44100_128',
});
Configuration Options#
| Option | Type | Default | Description |
|---|---|---|---|
apiKey |
string | ELEVENLABS_API_KEY |
API key |
baseURL |
string | https://api.elevenlabs.io/v1 |
API base URL |
maxRetries |
number | 3 | Max retry attempts |
retryDelayMs |
number | 1000 | Base retry delay (ms) |
timeout |
number | 30000 | Request timeout (ms) |
defaultModel |
string | eleven_multilingual_v2 |
Default TTS model |
defaultVoiceId |
string | - | Default voice ID |
defaultOutputFormat |
OutputFormat | mp3_44100_128 |
Default audio format |
Text-to-Speech#
Basic TTS#
const response = await provider.textToSpeech({
text: 'Welcome to your guided meditation.',
voiceId: 'pNInz6obpgDQGcFmaJgB', // Adam
modelId: 'eleven_multilingual_v2',
});
// response.audio is a Buffer containing the audio
fs.writeFileSync('output.mp3', response.audio);
TTS Request Parameters#
| Parameter | Type | Description |
|---|---|---|
text |
string | Text to convert to speech |
voiceId |
string | Voice ID to use |
modelId |
string | Model ID (optional) |
voiceSettings |
VoiceSettings | Voice customization |
outputFormat |
OutputFormat | Audio format |
languageCode |
string | Language override |
seed |
number | For reproducible output |
previousText |
string | Context from previous segment |
nextText |
string | Context for next segment |
optimizeStreamingLatency |
0-4 | Latency optimization level |
Voice Settings#
const response = await provider.textToSpeech({
text: 'This is a calm, slow meditation.',
voiceId: 'voice_id',
voiceSettings: {
stability: 0.8, // 0-1, higher = more consistent
similarity_boost: 0.75, // 0-1, higher = closer to original
style: 0.3, // 0-1, v2+ models only
use_speaker_boost: true, // Enhance voice clarity
speed: 0.8, // 0.25-4.0, speech rate
},
});
Output Formats#
| Format | Description |
|---|---|
mp3_22050_32 |
MP3, 22.05kHz, 32kbps |
mp3_44100_32 |
MP3, 44.1kHz, 32kbps |
mp3_44100_64 |
MP3, 44.1kHz, 64kbps |
mp3_44100_96 |
MP3, 44.1kHz, 96kbps |
mp3_44100_128 |
MP3, 44.1kHz, 128kbps (default) |
mp3_44100_192 |
MP3, 44.1kHz, 192kbps |
pcm_16000 |
PCM, 16kHz |
pcm_22050 |
PCM, 22.05kHz |
pcm_24000 |
PCM, 24kHz |
pcm_44100 |
PCM, 44.1kHz |
ulaw_8000 |
uLaw, 8kHz (telephony) |
alaw_8000 |
aLaw, 8kHz (telephony) |
opus_48000_64 |
Opus, 48kHz, 64kbps |
opus_48000_128 |
Opus, 48kHz, 128kbps |
TTS Response#
interface TTSResponse {
audio: Buffer;
contentType: string;
requestId?: string;
characterCount?: number;
historyItemId?: string;
}
Streaming Audio#
Stream TTS#
const stream = await provider.textToSpeechStream({
text: 'This is a long meditation script...',
voiceId: 'voice_id',
});
for await (const chunk of stream) {
// chunk.audio is a Buffer
// chunk.isFinal indicates last chunk
// chunk.alignment contains timing info
socket.write(chunk.audio);
if (chunk.alignment) {
console.log('Characters:', chunk.alignment.chars);
console.log('Timings:', chunk.alignment.charStartTimesMs);
}
}
Stream Chunk#
interface TTSStreamChunk {
audio: Buffer;
isFinal: boolean;
alignment?: {
chars: string[];
charStartTimesMs: number[];
charDurationsMs: number[];
};
}
Latency Optimization#
// Level 0: No optimization (highest quality)
// Level 1: Slight latency reduction
// Level 2: Balanced
// Level 3: Optimized for real-time
// Level 4: Maximum speed (lowest quality)
const response = await provider.textToSpeech({
text: 'Hello!',
voiceId: 'voice_id',
optimizeStreamingLatency: 3,
});
Voice Management#
List Voices#
const response = await provider.listVoices({
page_size: 20,
search: 'calm',
sort: 'name',
sort_direction: 'asc',
voice_type: 'default',
category: 'premade',
});
for (const voice of response.voices) {
console.log(`${voice.name} (${voice.voice_id})`);
console.log(` Category: ${voice.category}`);
console.log(` Labels: ${JSON.stringify(voice.labels)}`);
console.log(` Preview: ${voice.preview_url}`);
}
Voice Properties#
interface Voice {
voice_id: string;
name: string;
category: 'premade' | 'cloned' | 'generated' | 'professional';
labels?: Record<string, string>;
description?: string;
preview_url?: string;
settings?: VoiceSettings;
samples?: VoiceSample[];
high_quality_base_model_ids?: string[];
}
Voice Search Parameters#
| Parameter | Type | Description |
|---|---|---|
page_size |
number | Results per page |
search |
string | Search query |
sort |
string | Sort field (created_at, name) |
sort_direction |
string | asc or desc |
voice_type |
string | personal, community, default, workspace |
category |
string | premade, cloned, generated, professional |
Get Voice Sample#
const sampleUrl = await provider.getVoiceSample('voice_id');
// Returns URL to audio sample
Voice Cloning#
Comprehensive voice cloning support with Instant Voice Cloning (IVC), Professional Voice Cloning (PVC), consent management, and quality analysis.
Supported Audio Formats#
| Format | MIME Type | Extension |
|---|---|---|
| MP3 | audio/mpeg |
.mp3 |
| WAV | audio/wav, audio/x-wav |
.wav |
| M4A | audio/mp4, audio/x-m4a |
.m4a |
| OGG | audio/ogg |
.ogg |
| FLAC | audio/flac |
.flac |
| WebM | audio/webm |
.webm |
Instant Voice Cloning (IVC)#
const response = await provider.cloneVoice({
name: 'My Custom Voice',
files: [
{ data: audioBuffer1, filename: 'sample1.mp3' },
{ data: audioBuffer2, filename: 'sample2.mp3' },
],
description: 'A calm, meditative voice',
labels: { use_case: 'meditation', accent: 'american' },
removeBackgroundNoise: true,
});
console.log('Created voice:', response.voice_id);
Voice Cloning Request#
interface VoiceCloningRequest {
name: string; // Voice name (required)
files: VoiceCloningSample[]; // Audio samples (required)
description?: string; // Voice description
labels?: Record<string, string>; // Categorization labels
removeBackgroundNoise?: boolean; // Remove noise from samples
}
interface VoiceCloningSample {
data: Buffer; // Audio data
filename: string; // Used for MIME type detection
mimeType?: string; // Override MIME type
}
Voice Cloning Response#
interface VoiceCloningResponse {
voice_id: string; // Unique voice identifier
}
Edit Voice Metadata#
await provider.editVoice('voice_id', {
name: 'Updated Name',
description: 'New description',
labels: { style: 'calming' },
});
Update Voice Settings#
await provider.updateVoiceSettings('voice_id', {
stability: 0.8,
similarity_boost: 0.7,
style: 0.3,
use_speaker_boost: true,
});
Add Samples to Voice#
await provider.addVoiceSamples('voice_id', {
files: [{ data: newAudioBuffer, filename: 'new_sample.mp3' }],
removeBackgroundNoise: true,
});
Delete Voice Sample#
await provider.deleteVoiceSample('voice_id', 'sample_id');
Delete Voice#
await provider.deleteVoice('voice_id');
Professional Voice Cloning (PVC)#
Higher quality cloning with manual verification (requires 30+ minutes of audio):
const response = await provider.professionalVoiceClone({
name: 'Professional Voice',
files: [{ data: highQualityAudio, filename: 'studio_recording.wav' }],
description: 'Professional narrator voice',
consentVerificationId: 'consent_id_from_flow',
});
// Check status
if (response.status === 'processing') {
console.log('Estimated completion:', response.estimated_completion);
}
PVC Status Values#
| Status | Description |
|---|---|
pending_verification |
Awaiting consent verification |
processing |
Voice is being trained |
completed |
Voice ready for use |
failed |
Training failed |
rejected |
Consent rejected |
Check Cloning Capabilities#
// Check if IVC is available
const canUseIVC = await provider.canUseInstantVoiceCloning();
// Check if PVC is available
const canUsePVC = await provider.canUseProfessionalVoiceCloning();
// Get detailed limits
const limits = await provider.getVoiceCloningLimits();
// {
// voiceLimit: 10,
// currentVoiceCount: 3,
// maxVoiceAddEdits: 100,
// voiceAddEditCounter: 15,
// canUseIVC: true,
// canUsePVC: false
// }
Consent Management#
Consent flow for ethical voice cloning with verification and tamper detection.
Generate Consent Statement#
const consent = provider.generateConsentStatement({
voiceOwnerName: 'John Smith',
clonedVoiceName: 'Meditation Guide',
intendedUse: 'Meditation and wellness applications',
expirationDate: new Date('2026-01-01'),
});
console.log(consent.consentText);
console.log(consent.checksum); // For verification
Consent Statement Structure#
interface ConsentStatement {
version: string; // Statement format version
timestamp: string; // When consent was given
voiceOwnerName: string; // Voice owner's name
clonedVoiceName: string; // Name for cloned voice
intendedUse: string; // Description of use
expirationDate?: string; // Optional expiration
consentText: string; // Full consent agreement
checksum?: string; // Tamper verification
}
Verify Consent Statement#
const result = provider.verifyConsentStatement(consent);
if (result.valid) {
console.log('Consent is valid');
} else {
console.log('Errors:', result.errors);
if (result.isExpired) {
console.log('Consent has expired');
}
}
Consent Verification Result#
interface ConsentVerificationResult {
valid: boolean; // Overall validity
errors: string[]; // Validation errors
isExpired: boolean; // Expiration status
}
Sample Validation#
Utility functions for validating audio samples before cloning.
Validate Single Sample#
import { validateVoiceCloningSample } from './providers/elevenlabs-provider';
const sample = { data: audioBuffer, filename: 'sample.mp3' };
const result = validateVoiceCloningSample(sample);
if (!result.valid) {
console.log('Validation errors:', result.errors);
}
Validation Rules#
| Check | Requirement |
|---|---|
| Data | Non-empty buffer |
| Filename | Required, non-empty |
| Format | Supported MIME type |
| Min Size | 10KB minimum |
| Max Size | 50MB maximum |
Analyze Sample Quality#
import { analyzeVoiceCloningSamples } from './providers/elevenlabs-provider';
const samples = [
{ data: buffer1, filename: 'sample1.wav' },
{ data: buffer2, filename: 'sample2.wav' },
];
const report = analyzeVoiceCloningSamples(samples);
console.log('Quality Score:', report.qualityScore);
console.log('Recommendations:', report.recommendations);
console.log('Meets Requirements:', report.meetsMinimumRequirements);
console.log('Details:', report.details);
Quality Report Structure#
interface VoiceCloningQualityReport {
qualityScore: number; // 0-100 score
recommendations: string[]; // Improvement suggestions
meetsMinimumRequirements: boolean; // Minimum met
details: {
totalDuration: number; // Estimated seconds
sampleCount: number; // Number of samples
averageSampleSize: number; // Bytes
formatConsistency: boolean; // Same format
};
}
Best Practices#
Recommended Settings#
import { VOICE_CLONING_BEST_PRACTICES } from './providers/elevenlabs-provider';
// {
// minSampleDuration: 60, // 60 seconds minimum
// optimalSampleDuration: 300, // 5 minutes optimal
// maxSampleDuration: 1800, // 30 minutes max
// recommendedSampleCount: 3, // 3 samples
// recommendedSampleRate: 44100,
// recommendedBitDepth: 16,
// tips: [...]
// }
Recording Tips#
| Tip | Description |
|---|---|
| Environment | Quiet room, minimal background noise |
| Microphone | Good quality, consistent distance |
| Speaking | Natural pace and tone |
| Content | Diverse text, varied expressions |
| Format | WAV or FLAC for best quality |
| Audio | Mono, not stereo |
| Volume | Consistent levels across samples |
MIME Type Detection#
import { getMimeTypeFromFilename } from './providers/elevenlabs-provider';
const mimeType = getMimeTypeFromFilename('sample.wav');
// 'audio/wav'
const mimeType2 = getMimeTypeFromFilename('audio.m4a');
// 'audio/mp4'
Complete Voice Cloning Workflow#
import {
validateVoiceCloningSample,
analyzeVoiceCloningSamples,
VOICE_CLONING_BEST_PRACTICES,
} from './providers/elevenlabs-provider';
// 1. Validate individual samples
const samples = [
{ data: buffer1, filename: 'sample1.wav' },
{ data: buffer2, filename: 'sample2.wav' },
];
for (const sample of samples) {
const validation = validateVoiceCloningSample(sample);
if (!validation.valid) {
throw new Error(`Invalid sample: ${validation.errors.join(', ')}`);
}
}
// 2. Analyze overall quality
const report = analyzeVoiceCloningSamples(samples);
if (!report.meetsMinimumRequirements) {
console.warn('Recommendations:', report.recommendations);
}
// 3. Generate consent
const consent = provider.generateConsentStatement({
voiceOwnerName: 'Jane Doe',
clonedVoiceName: 'Wellness Guide',
intendedUse: 'Meditation app narration',
});
// 4. Verify consent
const verification = provider.verifyConsentStatement(consent);
if (!verification.valid) {
throw new Error(`Invalid consent: ${verification.errors.join(', ')}`);
}
// 5. Check account capabilities
const canClone = await provider.canUseInstantVoiceCloning();
if (!canClone) {
throw new Error('Voice cloning not available on this account');
}
// 6. Clone the voice
const response = await provider.cloneVoice({
name: consent.clonedVoiceName,
files: samples,
description: 'Created with proper consent',
labels: { consent_verified: 'true' },
removeBackgroundNoise: true,
});
console.log('Voice created:', response.voice_id);
Voice Design#
Create unique AI voices from text descriptions without audio samples.
Design a Voice#
const response = await provider.designVoice({
voiceDescription:
'A calm, soothing voice ideal for meditation. Warm tone with natural pauses.',
text: 'Welcome to your guided meditation session. Take a deep breath in...',
guidanceScale: 6,
autoGenerateText: false,
});
// Returns 3 previews to choose from
for (const preview of response.previews) {
console.log('Voice ID:', preview.generatedVoiceId);
console.log('Duration:', preview.durationSecs, 'seconds');
// preview.audioBase64 contains the audio data
}
Voice Design Request#
interface VoiceDesignRequest {
voiceDescription: string; // 20-1000 characters (required)
text?: string; // 100-1000 character preview text
modelId?: VoiceDesignModelId; // Model to use
autoGenerateText?: boolean; // Auto-generate preview text
guidanceScale?: number; // 0-10, how closely AI follows prompt
loudness?: number; // -1 to 1, volume level
seed?: number; // For reproducibility
shouldEnhance?: boolean; // AI enhancement of description
quality?: number; // Output quality vs. variety
referenceAudioBase64?: string; // Reference audio (v3 only)
promptStrength?: number; // 0-1, prompt vs. reference balance
outputFormat?: OutputFormat; // Audio format
}
Voice Design Model IDs#
| Model | Description |
|---|---|
eleven_multilingual_ttv_v2 |
Multilingual voice design (v2) |
eleven_ttv_v3 |
Latest voice design model (v3) |
Voice Design Response#
interface VoiceDesignResponse {
previews: VoiceDesignPreview[]; // Typically 3 previews
text: string; // Text used for previews
}
interface VoiceDesignPreview {
audioBase64: string; // Base64-encoded audio
generatedVoiceId: string; // ID for saving
mediaType: string; // Audio MIME type
durationSecs: number; // Audio duration
language?: string; // Detected language
}
Save Designed Voice#
const savedVoice = await provider.saveDesignedVoice({
voiceName: 'Meditation Guide',
voiceDescription: 'A calm, soothing voice for meditation',
generatedVoiceId: response.previews[0].generatedVoiceId,
labels: { category: 'meditation', mood: 'calm' },
playedNotSelectedVoiceIds: [
// Feedback for ElevenLabs
response.previews[1].generatedVoiceId,
response.previews[2].generatedVoiceId,
],
});
console.log('Saved voice:', savedVoice.voice_id);
Voice Save Request#
interface VoiceSaveRequest {
voiceName: string; // Name (required)
voiceDescription: string; // Description
generatedVoiceId: string; // From preview (required)
labels?: Record<string, string>; // Metadata
playedNotSelectedVoiceIds?: string[]; // Feedback
}
Voice Save Response#
interface VoiceSaveResponse {
voice_id: string;
name: string;
category: 'generated' | 'cloned' | 'premade' | 'professional';
description?: string;
settings?: VoiceSettings;
labels?: Record<string, string>;
created_at_unix?: number;
}
Convenience Methods#
Create Voice from Description#
Design and save in one call:
const voice = await provider.createVoiceFromDescription({
voiceDescription: 'A wise, grounded voice for spiritual teachings',
voiceName: 'Spiritual Teacher',
previewText: 'The journey inward is the most important journey...',
guidanceScale: 7,
labels: { category: 'spiritual', style: 'wise' },
});
Create Voice from Preset#
Use built-in presets:
const voice = await provider.createVoiceFromPreset(
'Meditation Guide',
'My Meditation Voice',
{
labels: { app: 'meditation-app' },
previewText: 'Custom preview text...',
}
);
Voice Design Presets#
Pre-configured presets for common use cases.
Available Presets#
| Preset | Category | Description |
|---|---|---|
| Meditation Guide | meditation | Calm, soothing, nurturing |
| Spiritual Teacher | spiritual | Wise, resonant, grounded |
| Yoga Instructor | meditation | Balanced, encouraging, peaceful |
| Sleep Narrator | meditation | Soft, gentle, dreamlike |
| Philosophical Narrator | educational | Thoughtful, articulate |
| Mystical Storyteller | storytelling | Enchanting, otherworldly |
Design from Preset#
const response = await provider.designVoiceFromPreset('Meditation Guide', {
text: 'Custom preview text instead of preset sample...',
guidanceScale: 7, // Override preset guidance
});
Preset Structure#
interface VoiceDesignPreset {
name: string;
description: string;
category:
| 'meditation'
| 'spiritual'
| 'educational'
| 'storytelling'
| 'character'
| 'custom';
voiceDescription: string;
guidanceScale: number;
recommendedSettings: Partial<VoiceSettings>;
sampleTexts: string[];
}
Get Preset by Name#
import { getVoiceDesignPreset } from './providers/elevenlabs-provider';
const preset = getVoiceDesignPreset('Meditation Guide');
console.log(preset?.voiceDescription);
console.log(preset?.sampleTexts[0]);
Get Presets by Category#
import { getVoiceDesignPresetsByCategory } from './providers/elevenlabs-provider';
const meditationPresets = getVoiceDesignPresetsByCategory('meditation');
// ['Meditation Guide', 'Yoga Instructor', 'Sleep Narrator']
Create Custom Preset#
import { createVoiceDesignPreset } from './providers/elevenlabs-provider';
const customPreset = createVoiceDesignPreset(
'Zen Master',
'A profound, deeply peaceful voice that speaks with the wisdom of ages...',
{
category: 'spiritual',
description: 'Ancient wisdom voice',
guidanceScale: 8,
recommendedSettings: {
stability: 0.75,
similarity_boost: 0.65,
style: 0.4,
speed: 0.85,
},
sampleTexts: [
'In the silence between thoughts, truth reveals itself.',
'The path to enlightenment begins with a single breath.',
],
}
);
Built-in Preset Details#
Meditation Guide:
{
voiceDescription: 'A calm, serene voice with a gentle, soothing quality...',
guidanceScale: 6,
recommendedSettings: {
stability: 0.7,
similarity_boost: 0.6,
style: 0.3,
speed: 0.85
}
}
Spiritual Teacher:
{
voiceDescription: 'A wise, deeply resonant voice that conveys ancient wisdom...',
guidanceScale: 7,
recommendedSettings: {
stability: 0.65,
similarity_boost: 0.7,
style: 0.4,
speed: 0.9
}
}
Sleep Narrator:
{
voiceDescription: 'An extremely soft, gentle voice with a dreamlike quality...',
guidanceScale: 8,
recommendedSettings: {
stability: 0.8,
similarity_boost: 0.5,
style: 0.2,
speed: 0.75
}
}
Complete Voice Design Workflow#
import {
VOICE_DESIGN_PRESETS,
getVoiceDesignPreset,
getVoiceDesignPresetsByCategory,
createVoiceDesignPreset,
} from './providers/elevenlabs-provider';
// 1. List available presets
console.log(
'Available presets:',
VOICE_DESIGN_PRESETS.map((p) => p.name)
);
// 2. Filter by category
const spiritualPresets = getVoiceDesignPresetsByCategory('spiritual');
// 3. Get a specific preset
const preset = getVoiceDesignPreset('Meditation Guide')!;
// 4. Generate voice previews
const designResponse = await provider.designVoiceFromPreset(
'Meditation Guide',
{
text: preset.sampleTexts[0],
}
);
// 5. Review previews (play audio to user)
for (const preview of designResponse.previews) {
const audioBuffer = Buffer.from(preview.audioBase64, 'base64');
// Play or analyze audio...
}
// 6. Save the best preview
const selectedPreview = designResponse.previews[0];
const voice = await provider.saveDesignedVoice({
voiceName: 'My Meditation Guide',
voiceDescription: preset.voiceDescription,
generatedVoiceId: selectedPreview.generatedVoiceId,
labels: {
category: preset.category,
source: 'voice_design',
},
playedNotSelectedVoiceIds: designResponse.previews
.filter((p) => p.generatedVoiceId !== selectedPreview.generatedVoiceId)
.map((p) => p.generatedVoiceId),
});
// 7. Apply recommended settings
await provider.updateVoiceSettings(voice.voice_id, preset.recommendedSettings);
// 8. Use the voice for TTS
const audio = await provider.textToSpeech({
text: 'Welcome to your meditation...',
voiceId: voice.voice_id,
});
Model Management#
List Models#
const models = await provider.listModels();
for (const model of models) {
console.log(`${model.name} (${model.model_id})`);
console.log(` TTS: ${model.can_do_text_to_speech}`);
console.log(` Voice Conversion: ${model.can_do_voice_conversion}`);
console.log(` Style: ${model.can_use_style}`);
console.log(` Languages: ${model.languages?.map((l) => l.name).join(', ')}`);
}
Available Models#
| Model ID | Description |
|---|---|
eleven_monolingual_v1 |
English only, original |
eleven_multilingual_v1 |
Multi-language support |
eleven_multilingual_v2 |
Improved multi-language |
eleven_turbo_v2 |
Fast generation |
eleven_turbo_v2_5 |
Latest turbo model |
eleven_flash_v2 |
Ultra-fast generation |
eleven_flash_v2_5 |
Latest flash model |
Usage Stats#
const stats = await provider.getUsageStats();
console.log(
`Characters used: ${stats.character_count}/${stats.character_limit}`
);
console.log(`Voices: ${stats.voice_add_edit_counter}/${stats.voice_limit}`);
console.log(`Reset: ${new Date(stats.next_character_count_reset_unix * 1000)}`);
Audio Tags (v3)#
Audio tags control emotional delivery in the v3 model.
Tag Categories#
| Category | Description | Examples |
|---|---|---|
emotion |
Emotional states | [sad], [happy], [calm] |
delivery |
Speech delivery | [whispers], [shouts] |
reaction |
Vocal reactions | [sighs], [laughs], [gasps] |
cognitive |
Thinking patterns | [pauses], [hesitates] |
sound_effect |
Non-speech sounds | [applause], [thunder] |
accent |
Speaking styles | [French accent], [pirate voice] |
pacing |
Speed control | [slowly], [quickly] |
intensity |
Strength modifiers | [softly], [loudly] |
character |
Character voices | [old man voice], [narrator] |
Emotion Tags#
const emotionTags = [
'[sad]',
'[happy]',
'[angry]',
'[excited]',
'[nervous]',
'[calm]',
'[sorrowful]',
'[hopeful]',
'[fearful]',
'[surprised]',
'[tender]',
'[loving]',
'[contemplative]',
'[serene]',
'[determined]',
];
Delivery Tags#
const deliveryTags = [
'[whispers]',
'[shouts]',
'[murmurs]',
'[speaks softly]',
'[speaks firmly]',
'[monotone]',
'[dramatic]',
'[matter of fact]',
];
Reaction Tags#
const reactionTags = [
'[sighs]',
'[laughs]',
'[nervous laugh]',
'[gasps]',
'[gulps]',
'[clears throat]',
'[coughs]',
'[sniffles]',
'[yawns]',
'[groans]',
'[exhales]',
'[inhales]',
];
Using Audio Tags#
const text =
'[calm] Welcome to this meditation. [pauses] ' +
'Take a deep breath. [exhales] ' +
'Let your body relax. [slowly] ' +
'Feel the peace within. [serene]';
const response = await provider.textToSpeech({
text,
voiceId: 'voice_id',
modelId: 'eleven_multilingual_v2', // v3 model supports tags
});
Parse Audio Tags#
import { parseAudioTags } from './providers/elevenlabs-audio-tags';
const result = parseAudioTags('[calm] Hello [pauses] there.');
console.log(result.cleanText); // "Hello there."
console.log(result.tags);
// [
// { tag: '[calm]', innerText: 'calm', category: 'emotion', isValid: true },
// { tag: '[pauses]', innerText: 'pauses', category: 'cognitive', isValid: true }
// ]
Validate Tags#
import {
validateTag,
validateTaggedText,
} from './providers/elevenlabs-audio-tags';
const validation = validateTag('whispers');
// { tag: 'whispers', isValid: true, category: 'delivery' }
const result = validateTaggedText('[calm] Hello [unknown tag] world');
// { isValid: true, warnings: ['Tag "[unknown tag]": Custom tag (not in known list)'] }
Audio Tag Templates#
Pre-built templates for different content types:
import {
MEDITATION_TEMPLATE,
SLEEP_TEMPLATE,
ENERGIZING_TEMPLATE,
STORYTELLING_TEMPLATE,
EDUCATIONAL_TEMPLATE,
} from './providers/elevenlabs-audio-tags';
Meditation Template:
{
openingTags: ['[calm]', '[serene]', '[gently]'],
transitionTags: ['[pauses]', '[softly]', '[deep breath]'],
closingTags: ['[tenderly]', '[peacefully]', '[slowly]'],
paragraphTags: ['[calmly]', '[serenely]'],
emphasisTags: ['[gently]', '[lovingly]'],
pauseTags: ['[pauses]', '[long pause]', '[exhales]']
}
Sleep Template:
{
openingTags: ['[slowly]', '[softly]', '[whispers]'],
transitionTags: ['[long pause]', '[exhales]', '[yawns]'],
closingTags: ['[whispers]', '[trails off]', '[slowly]'],
paragraphTags: ['[softly]', '[murmurs]'],
emphasisTags: ['[tenderly]', '[whispers]'],
pauseTags: ['[long pause]', '[pauses]', '[exhales deeply]']
}
Inject Tags Automatically#
import {
injectMeditationTags,
injectSleepTags,
injectEnergizingTags,
} from './providers/elevenlabs-audio-tags';
const meditation = injectMeditationTags(
'Welcome to this practice. Close your eyes.',
{ pauseFrequency: 2 }
);
// "[calm] Welcome to this practice. [pauses] Close your eyes. [tenderly]"
const sleep = injectSleepTags('Let your body relax. Feel yourself sinking.');
// "[slowly] Let your body relax. [long pause] Feel yourself sinking. [whispers]"
Custom Tag Injection#
import { injectAudioTags } from './providers/elevenlabs-audio-tags';
const result = injectAudioTags(text, {
template: MEDITATION_TEMPLATE,
addOpeningTag: true,
addClosingTag: true,
addTransitions: true,
pauseAfterSentences: true,
pauseFrequency: 3,
addEmphasis: true,
emphasisKeywords: ['peace', 'calm', 'relax'],
customReplacements: new Map([['breath', '[inhales] breath']]),
});
Utility Functions#
import {
extractTags,
removeAudioTags,
countAudioTags,
hasAudioTags,
formatTag,
combineTags,
getTagsByCategory,
getSuggestedTags,
} from './providers/elevenlabs-audio-tags';
// Extract tags
const tags = extractTags('[calm] Hello [pauses]');
// ['[calm]', '[pauses]']
// Remove tags
const clean = removeAudioTags('[calm] Hello [pauses]');
// 'Hello'
// Count tags
const count = countAudioTags('[calm] Hello [pauses]');
// 2
// Check for tags
const hasTags = hasAudioTags('[calm] Hello');
// true
// Format tag
const formatted = formatTag('whispers');
// '[whispers]'
// Combine tags
const combined = combineTags('calm', 'slowly');
// '[calm][slowly]'
// Get tags by category
const emotions = getTagsByCategory('emotion');
// Get suggested tags for content type
const suggestions = getSuggestedTags('meditation');
Conversational AI (WebRTC/WebSocket)#
Real-time conversational AI with bidirectional audio for interactive voice agents.
Connection Types#
| Type | Description | Use Case |
|---|---|---|
websocket |
WebSocket connection | Server-side, full control |
webrtc |
WebRTC peer connection | Client-side, browser |
Get Conversation Token (WebRTC)#
const tokenResponse = await provider.getConversationToken(
'agent_abc123',
'User Name' // Optional participant name
);
console.log('Token:', tokenResponse.token);
// Use token to establish WebRTC connection
Get Signed WebSocket URL#
const urlResponse = await provider.getConversationSignedUrl('agent_abc123');
const ws = new WebSocket(urlResponse.signedUrl);
Get Public Agent WebSocket URL#
For public agents that don't require authentication:
const wsUrl = provider.getPublicAgentWebSocketUrl('agent_abc123');
// 'wss://api.elevenlabs.io/v1/convai/conversation?agent_id=agent_abc123'
Create Conversation Session#
const session = provider.createConversationSession({
agentId: 'agent_abc123',
connectionType: 'websocket',
participantName: 'User',
initialContext: 'The user is interested in meditation.',
audioFormat: 'pcm',
sampleRate: 16000,
});
console.log('Session ID:', session.conversationId);
console.log('Status:', session.status); // 'connecting'
console.log('Mode:', session.mode); // 'listening'
Conversation Configuration#
interface ConversationConfig {
agentId: string; // Agent ID (required)
connectionType?: 'websocket' | 'webrtc'; // Connection type
participantName?: string; // User's name
customMetadata?: Record<string, unknown>; // Custom data
initialContext?: string; // Initial context
audioFormat?: 'pcm' | 'mp3' | 'opus'; // Audio format
sampleRate?: number; // Sample rate (Hz)
}
Conversation Session#
interface ConversationSession {
conversationId: string;
agentId: string;
connectionType: 'websocket' | 'webrtc';
status: ConversationStatus;
mode: ConversationMode;
startedAt: Date;
participantName?: string;
}
type ConversationStatus = 'connecting' | 'connected' | 'disconnected' | 'error';
type ConversationMode = 'speaking' | 'listening';
Server-to-Client Events#
| Event Type | Description |
|---|---|
user_transcript |
User speech transcription |
agent_response |
Agent's text response |
agent_response_correction |
Corrected agent response |
audio |
Agent audio chunk |
interruption |
User interrupted agent |
ping |
Keep-alive ping |
conversation_initiation_metadata |
Session metadata |
Parse Server Events#
const event = provider.parseServerEvent(rawMessage);
if (event) {
switch (event.type) {
case 'user_transcript':
console.log('User said:', event.userTranscript);
console.log('Final:', event.isFinal);
break;
case 'agent_response':
console.log('Agent:', event.agentResponse);
break;
case 'audio':
const audioBuffer = Buffer.from(event.audioBase64, 'base64');
playAudio(audioBuffer);
break;
case 'ping':
ws.send(JSON.stringify(provider.createPongEvent(event.eventId)));
break;
case 'interruption':
console.log('Interrupted:', event.reason);
break;
case 'conversation_initiation_metadata':
console.log('Conversation ID:', event.conversationId);
break;
}
}
Server Event Types#
interface UserTranscriptEvent {
type: 'user_transcript';
userTranscript: string;
isFinal: boolean;
}
interface AgentResponseEvent {
type: 'agent_response';
agentResponse: string;
}
interface ConversationAudioEvent {
type: 'audio';
audioBase64: string;
eventId: number;
}
interface InterruptionEvent {
type: 'interruption';
reason: string;
}
interface PingEvent {
type: 'ping';
eventId: number;
pingMs?: number;
}
interface ConversationInitiationMetadata {
type: 'conversation_initiation_metadata';
conversationId: string;
agentConfig?: Record<string, unknown>;
}
Client-to-Server Events#
Send Audio Chunk#
// Audio: PCM 16-bit mono at 16kHz, base64 encoded
const audioEvent = provider.createUserAudioChunkEvent(audioBase64);
ws.send(JSON.stringify(audioEvent));
Send Text Message#
const textEvent = provider.createUserTextMessageEvent('Hello, how are you?');
ws.send(JSON.stringify(textEvent));
Send Contextual Update#
Non-interrupting information for the agent:
const contextEvent = provider.createContextualUpdateEvent(
'User just started a breathing exercise'
);
ws.send(JSON.stringify(contextEvent));
Send User Activity#
Prevent agent interruption during user actions:
const activityEvent = provider.createUserActivityEvent();
ws.send(JSON.stringify(activityEvent));
Send Pong Response#
Reply to ping events:
const pongEvent = provider.createPongEvent(pingEvent.eventId);
ws.send(JSON.stringify(pongEvent));
Client Event Types#
interface ContextualUpdateEvent {
type: 'contextual_update';
text: string;
}
interface UserAudioChunkEvent {
userAudioChunk: string; // base64 PCM
}
interface UserTextMessageEvent {
type: 'user_message';
text: string;
}
interface UserActivityEvent {
type: 'user_activity';
}
interface PongEvent {
type: 'pong';
eventId: number;
}
Conversation Messages#
Create and track conversation messages:
const message = provider.createConversationMessage(
'user', // 'user' or 'agent'
'Hello there!', // text
true, // isFinal
audioBase64 // optional audio
);
// {
// id: 'msg_1704067200_abc123',
// role: 'user',
// text: 'Hello there!',
// timestamp: Date,
// isFinal: true,
// audioBase64: '...'
// }
Conversation History#
const history = provider.createConversationHistory(
'conv_123',
messages,
session.startedAt,
new Date() // endedAt
);
// {
// conversationId: 'conv_123',
// messages: [...],
// durationSec: 120,
// startedAt: Date,
// endedAt: Date
// }
Complete WebSocket Example#
async function startConversation(agentId: string) {
// 1. Get signed URL
const { signedUrl } = await provider.getConversationSignedUrl(agentId);
// 2. Create session
const session = provider.createConversationSession({ agentId });
const messages: ConversationMessage[] = [];
// 3. Connect WebSocket
const ws = new WebSocket(signedUrl);
ws.onopen = () => {
console.log('Connected');
session.status = 'connected';
};
ws.onmessage = (event) => {
const parsed = provider.parseServerEvent(event.data);
if (!parsed) return;
switch (parsed.type) {
case 'user_transcript':
if (parsed.isFinal) {
messages.push(
provider.createConversationMessage(
'user',
parsed.userTranscript,
true
)
);
}
break;
case 'agent_response':
messages.push(
provider.createConversationMessage(
'agent',
parsed.agentResponse,
true
)
);
break;
case 'audio':
// Play audio through speakers
playAudio(Buffer.from(parsed.audioBase64, 'base64'));
break;
case 'ping':
ws.send(JSON.stringify(provider.createPongEvent(parsed.eventId)));
break;
}
};
ws.onclose = () => {
session.status = 'disconnected';
const history = provider.createConversationHistory(
session.conversationId,
messages,
session.startedAt,
new Date()
);
console.log('Conversation duration:', history.durationSec, 'seconds');
};
// 4. Send user audio (from microphone)
startMicrophoneCapture((audioChunk) => {
const base64 = audioChunk.toString('base64');
ws.send(JSON.stringify(provider.createUserAudioChunkEvent(base64)));
});
// 5. Send context updates as needed
ws.send(
JSON.stringify(
provider.createContextualUpdateEvent(
'User is in a quiet room ready for meditation guidance'
)
)
);
}
Sound Effects#
Generate AI sound effects from text descriptions for meditation and ambient content.
Generate Sound Effect#
const response = await provider.generateSoundEffect({
text: 'A Tibetan singing bowl being gently struck',
durationSeconds: 8,
promptInfluence: 0.5,
loop: false,
outputFormat: 'mp3_44100_128',
});
fs.writeFileSync('singing-bowl.mp3', response.audio);
Sound Effects Request#
interface SoundEffectsRequest {
text: string; // Description (required)
modelId?: SoundEffectsModelId; // Model to use
durationSeconds?: number; // 0.5-30 seconds
promptInfluence?: number; // 0-1 (default 0.3)
loop?: boolean; // Seamless loop (v2 only)
outputFormat?: OutputFormat; // Audio format
}
type SoundEffectsModelId =
| 'eleven_text_to_sound_v1'
| 'eleven_text_to_sound_v2';
Sound Effects Response#
interface SoundEffectsResponse {
audio: Buffer; // Generated audio
contentType: string; // MIME type
requestId?: string; // For tracking
durationSeconds?: number;
isLooped?: boolean;
}
Generate from Preset#
const response = await provider.generateSoundEffectFromPreset(
'Tibetan Singing Bowl',
{
durationSeconds: 10, // Override preset duration
loop: true, // Override preset loop setting
}
);
Batch Generation#
// Multiple custom requests
const responses = await provider.generateSoundEffectBatch([
{ text: 'Gentle rain', durationSeconds: 20, loop: true },
{ text: 'Temple bell', durationSeconds: 6 },
{ text: 'Wind chimes', durationSeconds: 10 },
]);
// Multiple presets
const presetResults = await provider.generateSoundEffectBatchFromPresets(
['Tibetan Singing Bowl', 'Ocean Waves', 'Forest Birds'],
{ outputFormat: 'mp3_44100_128' }
);
for (const result of presetResults) {
if (result.response) {
fs.writeFileSync(`${result.preset}.mp3`, result.response.audio);
} else {
console.log(`Error generating ${result.preset}:`, result.error);
}
}
Sound Effects Presets#
17 built-in presets for meditation and ambient content.
Preset Categories#
| Category | Presets |
|---|---|
meditation |
Tibetan Singing Bowl, Temple Bell, Meditation Chime, Om Drone |
nature |
Gentle Rain, Ocean Waves, Forest Birds, Flowing Stream, Wind in Trees, Campfire Crackling |
ambient |
Ethereal Pad, Crystal Resonance, Deep Space |
transition |
Breath Transition, Shimmer, Gong Wash |
notification |
Mindful Bell, Soft Chime |
Available Presets#
| Preset | Duration | Loop | Category |
|---|---|---|---|
| Tibetan Singing Bowl | 8s | No | meditation |
| Temple Bell | 6s | No | meditation |
| Meditation Chime | 4s | No | meditation |
| Om Drone | 20s | Yes | meditation |
| Gentle Rain | 20s | Yes | nature |
| Ocean Waves | 20s | Yes | nature |
| Forest Birds | 20s | Yes | nature |
| Flowing Stream | 20s | Yes | nature |
| Wind in Trees | 20s | Yes | nature |
| Campfire Crackling | 20s | Yes | nature |
| Ethereal Pad | 20s | Yes | ambient |
| Crystal Resonance | 15s | Yes | ambient |
| Deep Space | 20s | Yes | ambient |
| Breath Transition | 2s | No | transition |
| Shimmer | 2s | No | transition |
| Gong Wash | 8s | No | transition |
| Mindful Bell | 3s | No | notification |
| Soft Chime | 2s | No | notification |
Preset Structure#
interface SoundEffectsPreset {
name: string;
description: string;
category:
| 'meditation'
| 'nature'
| 'ambient'
| 'transition'
| 'notification'
| 'musical'
| 'custom';
prompt: string;
recommendedDuration: number;
shouldLoop: boolean;
promptInfluence: number;
}
Preset Utilities#
import {
SOUND_EFFECTS_PRESETS,
getSoundEffectsPreset,
getSoundEffectsPresetsByCategory,
createSoundEffectsPreset,
} from './providers/elevenlabs-provider';
// Get preset by name
const preset = getSoundEffectsPreset('Tibetan Singing Bowl');
// Get presets by category
const naturePresets = getSoundEffectsPresetsByCategory('nature');
// Create custom preset
const customPreset = createSoundEffectsPreset(
'Zen Garden',
'Water feature trickling in a peaceful zen garden with soft wind chimes',
{
category: 'ambient',
description: 'Peaceful zen garden ambiance',
recommendedDuration: 20,
shouldLoop: true,
promptInfluence: 0.4,
}
);
Music Generation#
Generate AI music from text descriptions for meditation and wellness content.
Generate Music#
const response = await provider.generateMusic({
prompt: 'Create slow, ambient meditation music with soft synthesizer pads',
durationMs: 180000, // 3 minutes
includeVocals: false,
});
fs.writeFileSync('meditation-music.mp3', response.audio);
Music Generation Request#
interface MusicGenerationRequest {
prompt: string; // Description (required)
durationMs?: number; // 10000-300000ms (10s-5min)
compositionPlan?: MusicCompositionPlan; // Structured plan
includeVocals?: boolean; // Include vocals
vocalsLanguage?: 'en' | 'es' | 'de' | 'ja' | 'fr' | 'pt' | 'it' | 'zh';
}
Music Generation Response#
interface MusicGenerationResponse {
audio: Buffer; // MP3 audio
contentType: string; // 'audio/mpeg'
requestId?: string; // For tracking
durationMs?: number;
filename?: string;
compositionPlan?: MusicCompositionPlan; // If detailed
}
Generate with Composition Plan#
// First generate a composition plan
const plan = await provider.generateCompositionPlan({
prompt: 'Peaceful meditation music with a gentle intro and calming outro',
durationMs: 120000,
});
// Then generate music with the plan
const response = await provider.generateMusic({
prompt: 'Meditation music',
durationMs: 120000,
compositionPlan: plan,
});
Composition Plan Structure#
interface MusicCompositionPlan {
positiveGlobalStyles: string[]; // Global positive attributes
negativeGlobalStyles: string[]; // Styles to avoid
sections: MusicSection[]; // Track structure
}
interface MusicSection {
sectionName: string; // 'intro', 'verse', 'outro', etc.
positiveLocalStyles: string[]; // Section-specific attributes
negativeLocalStyles: string[]; // Section-specific avoids
durationMs: number; // Section duration
lines?: string[]; // Lyrics (if vocals)
}
Detailed Generation#
Get both audio and composition plan:
const response = await provider.generateMusicDetailed({
prompt: 'Ambient yoga music with soft sitar and tabla',
durationMs: 180000,
});
console.log('Plan:', response.compositionPlan);
console.log('Filename:', response.filename);
Generate from Preset#
const response = await provider.generateMusicFromPreset('Deep Meditation', {
durationMs: 300000, // Override to 5 minutes
});
Batch Generation#
// Multiple custom tracks
const tracks = await provider.generateMusicBatch([
{ prompt: 'Morning meditation', durationMs: 180000 },
{ prompt: 'Evening relaxation', durationMs: 240000 },
]);
// Multiple presets
const presetTracks = await provider.generateMusicBatchFromPresets(
['Deep Meditation', 'Yoga Flow', 'Deep Sleep'],
{ durationMs: 180000 }
);
for (const result of presetTracks) {
if (result.response) {
fs.writeFileSync(`${result.preset}.mp3`, result.response.audio);
}
}
Music Generation Presets#
14 built-in presets for meditation, yoga, sleep, and spiritual content.
Preset Categories#
| Category | Presets |
|---|---|
meditation |
Deep Meditation, Guided Meditation Background, Zen Garden |
yoga |
Yoga Flow, Restorative Yoga |
sleep |
Deep Sleep, Lullaby Dreams |
focus |
Focus Flow |
spiritual |
Sacred Om, Temple Atmosphere |
nature |
Forest Meditation, Ocean Waves Music |
relaxation |
Spa Relaxation, Evening Unwind |
Available Presets#
| Preset | Duration | Vocals | BPM | Category |
|---|---|---|---|---|
| Deep Meditation | 5min | No | 50-70 | meditation |
| Guided Meditation Background | 5min | No | 50-60 | meditation |
| Zen Garden | 4min | No | 50-65 | meditation |
| Yoga Flow | 5min | No | 75-95 | yoga |
| Restorative Yoga | 5min | No | 45-60 | yoga |
| Deep Sleep | 5min | No | 40-55 | sleep |
| Lullaby Dreams | 4min | No | 50-65 | sleep |
| Focus Flow | 5min | No | 65-85 | focus |
| Sacred Om | 5min | Yes | 55-70 | spiritual |
| Temple Atmosphere | 4min | No | 50-65 | spiritual |
| Forest Meditation | 4min | No | 50-65 | nature |
| Ocean Waves Music | 4min | No | 50-60 | nature |
| Spa Relaxation | 4min | No | 55-70 | relaxation |
| Evening Unwind | 4min | No | 50-65 | relaxation |
Preset Structure#
interface MusicGenerationPreset {
name: string;
description: string;
category:
| 'meditation'
| 'yoga'
| 'sleep'
| 'focus'
| 'relaxation'
| 'spiritual'
| 'nature'
| 'custom';
prompt: string;
recommendedDurationMs: number;
includeVocals: boolean;
bpmRange?: { min: number; max: number };
musicalElements?: string[];
}
Preset Utilities#
import {
MUSIC_GENERATION_PRESETS,
getMusicGenerationPreset,
getMusicGenerationPresetsByCategory,
createMusicGenerationPreset,
} from './providers/elevenlabs-provider';
// Get preset by name
const preset = getMusicGenerationPreset('Deep Meditation');
// Get presets by category
const sleepPresets = getMusicGenerationPresetsByCategory('sleep');
// Create custom preset
const customPreset = createMusicGenerationPreset(
'Morning Awakening',
'Gentle, uplifting music to start the day with soft strings and light percussion',
{
category: 'meditation',
description: 'Energizing morning music',
recommendedDurationMs: 180000,
includeVocals: false,
bpmRange: { min: 65, max: 80 },
musicalElements: ['strings', 'light percussion', 'soft piano'],
}
);
Dubbing#
Translate and dub audio/video content into multiple languages.
Create Dubbing Job#
// From file
const response = await provider.createDubbing({
file: audioBuffer,
filename: 'meditation.mp3',
targetLang: 'es',
sourceLang: 'en',
name: 'Meditation Guide - Spanish',
});
// From URL
const response = await provider.createDubbing({
sourceUrl: 'https://example.com/video.mp4',
targetLang: 'fr',
name: 'Tutorial - French',
});
console.log('Dubbing ID:', response.dubbingId);
console.log('Expected duration:', response.expectedDurationSec, 'seconds');
Dubbing Request#
interface DubbingRequest {
targetLang: DubbingLanguage | string; // Target language (required)
sourceLang?: DubbingLanguage | string; // Source (default: auto-detect)
file?: Buffer; // Audio/video file
filename?: string; // Filename for file
sourceUrl?: string; // Alternative to file
name?: string; // Project name
targetAccent?: string; // Accent preference
numSpeakers?: number; // 0 = auto-detect (max 9)
mode?: DubbingMode; // 'automatic' | 'manual'
watermark?: boolean; // Apply watermark
highestResolution?: boolean; // Use highest resolution
dropBackgroundAudio?: boolean; // Remove background
useProfanityFilter?: boolean; // Censor profanities
disableVoiceCloning?: boolean; // Use voice library
dubbingStudio?: boolean; // Enable studio edits
startTime?: number; // Clip start (ms)
endTime?: number; // Clip end (ms)
}
Dubbing Create Response#
interface DubbingCreateResponse {
dubbingId: string;
expectedDurationSec: number;
}
Get Dubbing Status#
const metadata = await provider.getDubbingMetadata(dubbingId);
console.log('Status:', metadata.status); // 'dubbing' | 'dubbed' | 'failed'
console.log('Target Languages:', metadata.targetLanguages);
console.log('Editable:', metadata.editable);
Dubbing Metadata#
interface DubbingMetadata {
dubbingId: string;
name: string;
status: DubbingStatus;
targetLanguages: string[];
editable: boolean;
createdAt: Date;
mediaMetadata?: {
contentType: string;
duration: number;
};
error?: string;
}
type DubbingStatus = 'dubbing' | 'dubbed' | 'failed' | 'cloning';
Wait for Completion#
const metadata = await provider.waitForDubbing(dubbingId, {
pollIntervalMs: 5000, // Check every 5 seconds
timeoutMs: 600000, // 10 minute timeout
});
if (metadata.status === 'dubbed') {
console.log('Dubbing complete!');
} else if (metadata.status === 'failed') {
console.log('Error:', metadata.error);
}
Get Dubbed Audio#
const audio = await provider.getDubbedAudio(dubbingId, 'es');
fs.writeFileSync('meditation-spanish.mp3', audio.audio);
console.log('Content type:', audio.contentType);
console.log('Language:', audio.languageCode);
Dubbed Audio Response#
interface DubbedAudioResponse {
audio: Buffer;
contentType: string;
languageCode: string;
}
Get Transcript#
const transcript = await provider.getDubbingTranscript(dubbingId, 'es');
console.log('Full text:', transcript.text);
for (const segment of transcript.segments) {
console.log(`[${segment.startMs}-${segment.endMs}ms] ${segment.text}`);
if (segment.words) {
for (const word of segment.words) {
console.log(` ${word.word} (${word.confidence})`);
}
}
}
Transcript Response#
interface DubbingTranscriptResponse {
languageCode: string;
text: string;
segments: TranscriptSegment[];
}
interface TranscriptSegment {
text: string;
speakerId?: string;
startMs: number;
endMs: number;
words?: TranscriptWord[];
}
interface TranscriptWord {
word: string;
startMs: number;
endMs: number;
confidence?: number;
}
Delete Dubbing Project#
await provider.deleteDubbing(dubbingId);
Supported Dubbing Languages#
32 languages supported for dubbing.
| Code | Language | Code | Language |
|---|---|---|---|
en |
English | ko |
Korean |
es |
Spanish | hu |
Hungarian |
fr |
French | hi |
Hindi |
de |
German | sv |
Swedish |
it |
Italian | da |
Danish |
pt |
Portuguese | fi |
Finnish |
pl |
Polish | no |
Norwegian |
tr |
Turkish | el |
Greek |
ru |
Russian | he |
Hebrew |
nl |
Dutch | ro |
Romanian |
cs |
Czech | uk |
Ukrainian |
ar |
Arabic | id |
Indonesian |
zh |
Chinese | ms |
Malay |
ja |
Japanese | vi |
Vietnamese |
th |
Thai | bg |
Bulgarian |
sk |
Slovak | hr |
Croatian |
Language Utilities#
import {
DUBBING_LANGUAGES,
getDubbingLanguageName,
isDubbingLanguageSupported,
} from './providers/elevenlabs-provider';
// Get language name
const name = getDubbingLanguageName('ja'); // 'Japanese'
// Check if supported
if (isDubbingLanguageSupported('es')) {
console.log('Spanish is supported');
}
// List all languages
for (const [code, name] of Object.entries(DUBBING_LANGUAGES)) {
console.log(`${code}: ${name}`);
}
Error Handling#
ElevenLabsError#
interface ElevenLabsError extends Error {
code: string;
status: number;
detail?: string;
}
Error Handling Example#
try {
const response = await provider.textToSpeech({
text: 'Hello',
voiceId: 'invalid_id',
});
} catch (error) {
if (error.status === 401) {
console.log('Invalid API key');
} else if (error.status === 429) {
console.log('Rate limited, retry after cooldown');
} else if (error.status === 400) {
console.log('Bad request:', error.detail);
}
}
Related Documentation#
- OpenRouter Integration - LLM provider
- AI Service - Main AI orchestration
- Prompt Management - Template system