This document provides comprehensive documentation for the Lilith Media Service Audio Processing Pipeline, including transcoding, normalization, silence detection/removal, waveform generation, and audio fingerprinting.
Overview#
The Audio Processing Pipeline provides:
- Audio Transcoding: Convert between formats (MP3, AAC, Opus, WAV, FLAC, OGG)
- Audio Normalization: LUFS-based loudness normalization with platform presets
- Silence Processing: Detection, removal, trimming, and splitting at silence
- Waveform Generation: Visual waveform images and JSON peak data
- Audio Fingerprinting: Chromaprint-based fingerprinting for duplicate detection
Table of Contents#
Audio Transcoding#
Supported Formats#
| Format | Codec | Extension | Default Bitrate | Use Case |
|---|---|---|---|---|
mp3 |
libmp3lame | .mp3 | 192 kbps | General audio, broad compatibility |
aac |
aac (LC-AAC) | .m4a | 128 kbps | Mobile streaming, Apple ecosystem |
opus |
libopus | .opus | 96 kbps | Voice, VoIP, web streaming |
wav |
pcm_s16le | .wav | Lossless | Editing, archival |
flac |
flac | .flac | Lossless | Lossless archival |
ogg |
libvorbis | .ogg | 128 kbps | Open source, web audio |
Transcode Options#
interface AudioTranscodeOptions {
format: 'mp3' | 'aac' | 'opus' | 'wav' | 'flac' | 'ogg';
bitrate?: number; // kbps (e.g., 128, 192, 256, 320)
sampleRate?: number; // Hz (e.g., 44100, 48000)
channels?: number; // 1 = mono, 2 = stereo
codec?: string; // Override default codec
onProgress?: (progress: FFmpegProgress) => void;
timeout?: number; // ms (default: 30 minutes)
}
Transcoding Presets#
| Preset | Format | Bitrate | Sample Rate | Channels | Use Case |
|---|---|---|---|---|---|
mp3_high |
mp3 | 320 kbps | 44.1 kHz | stereo | Music archival |
mp3_standard |
mp3 | 192 kbps | 44.1 kHz | stereo | General use |
mp3_voice |
mp3 | 96 kbps | 44.1 kHz | mono | Podcasts, voice |
aac_streaming |
aac | 128 kbps | 44.1 kHz | stereo | Mobile streaming |
aac_high |
aac | 256 kbps | 48 kHz | stereo | High-quality AAC |
opus_voice |
opus | 32 kbps | 48 kHz | mono | Voice chat, VoIP |
opus_music |
opus | 128 kbps | 48 kHz | stereo | Music streaming |
wav_lossless |
wav | Lossless | 44.1 kHz | stereo | Editing source |
flac_lossless |
flac | Lossless | 44.1 kHz | stereo | Lossless archival |
Usage Example#
import {
transcodeAudio,
transcodeAudioBuffer,
batchTranscode,
TRANSCODE_PRESETS,
} from './processing/audio-transcoder.js';
// Single file transcoding
const result = await transcodeAudio(
'/path/to/input.wav',
'/path/to/output.mp3',
{
format: 'mp3',
bitrate: 320,
sampleRate: 44100,
onProgress: (progress) => {
console.log(`Progress: ${progress.percent}%`);
},
}
);
console.log(
`Transcoded: ${result.outputFormat} at ${result.outputBitrate}kbps`
);
// Using presets
const voiceResult = await transcodeAudio(
'/path/to/voice.wav',
'/path/to/voice.mp3',
TRANSCODE_PRESETS.mp3_voice
);
// Buffer transcoding
const { outputBuffer, result } = await transcodeAudioBuffer(
inputBuffer,
'.wav',
{ format: 'aac', bitrate: 128 }
);
// Batch transcoding
const files = [
{ inputPath: 'track1.wav', outputPath: 'track1.mp3' },
{ inputPath: 'track2.wav', outputPath: 'track2.mp3' },
];
const results = await batchTranscode(files, { format: 'mp3', bitrate: 192 });
Audio Normalization#
Overview#
Audio normalization adjusts loudness to target levels using the EBU R128 standard (LUFS measurement). This ensures consistent playback volume across different audio files.
Normalization Options#
interface NormalizationOptions {
targetLoudness?: number; // LUFS (default: -16)
truePeak?: number; // dBTP (default: -1.5)
loudnessRange?: number; // LU (default: 11)
dualPass?: boolean; // Two-pass for accuracy (default: true)
linear?: boolean; // No compression (default: false)
onProgress?: (progress: FFmpegProgress) => void;
timeout?: number;
}
Platform Presets#
| Preset | Target LUFS | True Peak | LRA | Description |
|---|---|---|---|---|
spotify |
-14 LUFS | -1 dBTP | - | Spotify recommendation |
youtube |
-14 LUFS | -1 dBTP | - | YouTube recommendation |
apple |
-16 LUFS | -1 dBTP | - | Apple Music / iTunes |
podcast |
-16 LUFS | -1.5 dBTP | 8 LU | Podcast standard |
voice |
-16 LUFS | -3 dBTP | 5 LU | Voice/speech (tight DR) |
broadcast |
-23 LUFS | -1 dBTP | 15 LU | EBU R128 broadcast |
peak_only |
-16 LUFS | -1 dBTP | - | Simple peak limiting |
Loudness Analysis#
interface LoudnessInfo {
integratedLoudness: number; // LUFS
truePeak: number; // dBTP
loudnessRange: number; // LU
threshold: number;
targetOffset?: number;
}
Usage Example#
import {
normalizeAudio,
analyzeLoudness,
needsNormalization,
applyLimiter,
applyCompressor,
NORMALIZATION_PRESETS,
} from './processing/audio-normalizer.js';
// Analyze loudness
const loudness = await analyzeLoudness('/path/to/audio.mp3');
console.log(`Current loudness: ${loudness.integratedLoudness} LUFS`);
console.log(`True peak: ${loudness.truePeak} dBTP`);
// Check if normalization is needed
const check = await needsNormalization('/path/to/audio.mp3', -16);
if (check.needsNormalization) {
console.log(`Needs adjustment: ${check.delta} dB`);
}
// Normalize audio
const result = await normalizeAudio(
'/path/to/input.mp3',
'/path/to/output.wav',
{
targetLoudness: -14,
truePeak: -1,
dualPass: true,
}
);
console.log(`Gain applied: ${result.gainAdjustment} dB`);
console.log(
`Output loudness: ${result.outputLoudness?.integratedLoudness} LUFS`
);
// Using platform presets
const spotifyResult = await normalizeAudio(
'/path/to/input.mp3',
'/path/to/spotify-ready.wav',
NORMALIZATION_PRESETS.spotify
);
// Apply limiter
await applyLimiter('/path/to/input.wav', '/path/to/limited.wav', {
threshold: -1, // dB
attack: 5, // ms
release: 50, // ms
makeup: 0, // dB
});
// Apply compressor
await applyCompressor('/path/to/input.wav', '/path/to/compressed.wav', {
threshold: -20, // dB
ratio: 4, // 4:1 compression
attack: 20, // ms
release: 250, // ms
knee: 2.82843, // dB
});
Silence Processing#
Silence Detection#
Detect silence segments in audio files based on noise threshold and minimum duration.
interface SilenceDetectionResult {
segments: SilenceSegment[];
totalSilenceDuration: number;
audioDuration: number;
silencePercentage: number;
}
interface SilenceSegment {
start: number; // seconds
end: number; // seconds
duration: number; // seconds
}
Silence Removal Options#
interface SilenceRemovalOptions {
noiseThreshold?: number; // dB (default: -50)
minDuration?: number; // seconds (default: 0.5)
padding?: number; // Keep around speech (default: 0.1s)
maxSilence?: number; // Max silence between speech (default: 0.5s)
trimStart?: boolean; // Remove from start only
trimEnd?: boolean; // Remove from end only
}
Content Presets#
| Preset | Threshold | Min Duration | Padding | Max Silence | Use Case |
|---|---|---|---|---|---|
podcast |
-40 dB | 0.3s | 0.05s | 0.3s | Aggressive removal |
music |
-60 dB | 2.0s | 0.2s | 1.0s | Preserve dynamics |
voice |
-45 dB | 0.5s | 0.1s | 0.5s | Moderate removal |
audiobook |
-45 dB | 1.0s | 0.15s | 0.8s | Keep natural pauses |
transcription |
-35 dB | 0.2s | 0.02s | 0.2s | Prep for STT |
Usage Example#
import {
detectSilence,
removeSilence,
trimSilence,
compressSilence,
splitAtSilence,
getSilenceStats,
SILENCE_PRESETS,
} from './processing/silence-processor.js';
// Detect silence segments
const silence = await detectSilence('/path/to/audio.mp3', {
noiseThreshold: -50,
minDuration: 0.5,
});
console.log(`Found ${silence.segments.length} silence segments`);
console.log(`${silence.silencePercentage.toFixed(1)}% of audio is silence`);
// Get detailed statistics
const stats = getSilenceStats(silence);
console.log(`Average silence: ${stats.averageDuration.toFixed(2)}s`);
console.log(`Longest silence: ${stats.maxDuration.toFixed(2)}s`);
// Remove all silence
const result = await removeSilence(
'/path/to/input.mp3',
'/path/to/no-silence.wav',
{ noiseThreshold: -45, minDuration: 0.3 }
);
console.log(`Removed ${result.durationRemoved.toFixed(2)}s of silence`);
// Trim silence from start/end only
await trimSilence('/path/to/input.mp3', '/path/to/trimmed.wav', {
padding: 0.1,
});
// Compress pauses (keep shorter silence)
await compressSilence('/path/to/input.mp3', '/path/to/compressed.wav', {
maxSilence: 0.5,
});
// Split at silence points
const { segments, silenceInfo } = await splitAtSilence(
'/path/to/audio.mp3',
'/path/to/output/',
{
...SILENCE_PRESETS.podcast,
outputFormat: 'wav',
minSegmentDuration: 2.0,
}
);
console.log(`Created ${segments.length} segments`);
segments.forEach((seg, i) => {
console.log(`Segment ${i + 1}: ${seg.duration.toFixed(2)}s`);
});
Waveform Generation#
Overview#
Generate visual waveform representations as images (PNG) or data arrays for frontend rendering.
Waveform Options#
interface WaveformOptions {
samples?: number; // Number of bars (default: 800)
width?: number; // Image width px (default: 1800)
height?: number; // Image height px (default: 140)
color?: string; // Hex color (default: '#5865F2')
backgroundColor?: string; // Hex or 'transparent'
mirror?: boolean; // Mirrored waveform (default: true)
scale?: 'lin' | 'log' | 'sqrt'; // Scale type (default: 'sqrt')
}
Waveform Presets#
| Preset | Samples | Dimensions | Scale | Use Case |
|---|---|---|---|---|
detailed |
2000 | 3600x280 | linear | Full detail view |
player |
800 | 1800x140 | sqrt | Audio player UI |
thumbnail |
200 | 400x60 | sqrt | List thumbnails |
mobile |
100 | 300x40 | sqrt | Mobile devices |
Waveform Data Format#
interface WaveformDataResult {
peaks: number[]; // Amplitude values (0.0 to 1.0)
duration: number; // Audio duration in seconds
sampleRate: number; // Samples per second
channels: number; // Channel count
bitsPerSample: number;
}
Usage Example#
import {
generateWaveformImage,
generateWaveformData,
generateWaveform,
peaksToSvgPath,
downsamplePeaks,
WAVEFORM_PRESETS,
} from './processing/waveform-generator.js';
// Generate waveform image
const imageResult = await generateWaveformImage(
'/path/to/audio.mp3',
'/path/to/waveform.png',
{
width: 1800,
height: 140,
color: '#5865F2',
backgroundColor: 'transparent',
mirror: true,
scale: 'sqrt',
}
);
console.log(`Generated waveform: ${imageResult.width}x${imageResult.height}`);
// Generate waveform data (for frontend)
const data = await generateWaveformData('/path/to/audio.mp3', {
samples: 800,
scale: 'sqrt',
});
console.log(`${data.peaks.length} samples, ${data.duration}s duration`);
// Generate both image and data
const { image, data: waveformData } = await generateWaveform(
'/path/to/audio.mp3',
'/path/to/waveform.png',
WAVEFORM_PRESETS.player
);
// Convert peaks to SVG path
const svgPath = peaksToSvgPath(data.peaks, 800, 100, {
mirror: true,
smooth: true,
});
// Use in SVG:
// <path d="${svgPath}" fill="#5865F2" />
// Downsample for different views
const mobilePeaks = downsamplePeaks(data.peaks, 100);
// Generate from buffer
const { imageBuffer, result } = await generateWaveformImageFromBuffer(
audioBuffer,
'.mp3',
WAVEFORM_PRESETS.thumbnail
);
Audio Fingerprinting#
Overview#
Generate audio fingerprints using Chromaprint (fpcalc) for audio identification, duplicate detection, and integration with music recognition services like AcoustID.
Prerequisites#
Requires Chromaprint's fpcalc utility:
- macOS:
brew install chromaprint - Ubuntu:
apt-get install libchromaprint-tools - Windows: Download from https://acoustid.org/chromaprint
Fingerprint Result#
interface FingerprintResult {
fingerprint: string; // Raw fingerprint
duration: number; // Duration analyzed
fingerprintBase64: string; // Base64 encoded
fingerprintHash: string; // SHA-256 hash
algorithm: number; // Algorithm version
}
Fingerprint Options#
interface FingerprintOptions {
maxDuration?: number; // Max seconds to analyze (default: 120)
startOffset?: number; // Start offset in seconds (default: 0)
algorithm?: number; // Algorithm version (default: 2)
overlap?: number; // Overlap factor (default: 0)
}
Usage Example#
import {
generateFingerprint,
compareFingerprints,
compareRawFingerprints,
areDuplicates,
findDuplicates,
batchFingerprint,
prepareForAcoustId,
isFpcalcAvailable,
} from './processing/audio-fingerprint.js';
// Check if fpcalc is available
if (await isFpcalcAvailable()) {
console.log('Chromaprint is installed');
}
// Generate fingerprint
const fingerprint = await generateFingerprint('/path/to/audio.mp3', {
maxDuration: 120,
algorithm: 2,
});
console.log(`Fingerprint hash: ${fingerprint.fingerprintHash}`);
console.log(`Duration analyzed: ${fingerprint.duration}s`);
// Compare two fingerprints
const fp1 = await generateFingerprint('/path/to/song1.mp3');
const fp2 = await generateFingerprint('/path/to/song2.mp3');
const comparison = compareFingerprints(fp1, fp2, {
threshold: 0.9,
method: 'hash',
});
console.log(`Similarity: ${(comparison.similarity * 100).toFixed(1)}%`);
console.log(`Match: ${comparison.match}`);
// Check if files are duplicates
const duplicateCheck = await areDuplicates(
'/path/to/file1.mp3',
'/path/to/file2.mp3',
{ threshold: 0.85 }
);
if (duplicateCheck.isDuplicate) {
console.log(
`Duplicates! Similarity: ${(duplicateCheck.similarity * 100).toFixed(1)}%`
);
}
// Batch fingerprint multiple files
const fingerprints = await batchFingerprint(
['song1.mp3', 'song2.mp3', 'song3.mp3'],
{
maxDuration: 120,
concurrency: 4,
onProgress: (i, total, file) => {
console.log(`Processing ${i}/${total}: ${file}`);
},
}
);
// Find duplicates in collection
const duplicates = await findDuplicates(
['song1.mp3', 'song2.mp3', 'song3.mp3', 'song4.mp3'],
{
threshold: 0.85,
onProgress: (current, total) => {
console.log(`Comparing: ${current}/${total}`);
},
}
);
duplicates.forEach((dup) => {
console.log(`Duplicate pair: ${dup.files[0]} <-> ${dup.files[1]}`);
console.log(`Similarity: ${(dup.similarity * 100).toFixed(1)}%`);
});
// Prepare for AcoustID lookup
const acoustId = prepareForAcoustId(fingerprint);
console.log(
`AcoustID fingerprint: ${acoustId.acoustIdFingerprint.substring(0, 50)}...`
);
console.log(`Duration: ${acoustId.acoustIdDuration}s`);
Processing Pipeline Integration#
Requesting Processing Jobs#
Use the Asset API to request audio processing:
curl -X POST "https://api.lilith.app/v1/assets/asset-abc123/process" \
-H "Authorization: Bearer <access_token>" \
-H "Content-Type: application/json" \
-d '{
"type": "transcode",
"config": {
"format": "mp3",
"bitrate": 192,
"normalize": true,
"targetLoudness": -14
}
}'
Processing Job Types#
| Type | Description |
|---|---|
transcode |
Convert audio format |
normalization |
Apply loudness normalization |
silence_removal |
Remove or compress silence |
waveform |
Generate waveform visualization |
fingerprint |
Generate audio fingerprint |
metadata_extraction |
Extract detailed audio metadata |
SDK Integration#
TypeScript/JavaScript#
import { LilithClient } from '@lilith/sdk';
const client = new LilithClient({
apiKey: process.env.LILITH_API_KEY,
});
// Request transcoding
const job = await client.media.requestProcessing('asset-abc123', {
type: 'transcode',
config: {
format: 'mp3',
bitrate: 320,
normalize: true,
targetLoudness: -14,
},
});
// Monitor job progress
const status = await client.media.getProcessingJob(job.jobId);
console.log(`Status: ${status.status}, Progress: ${status.progress}%`);
// Get waveform data
const waveform = await client.media.getWaveform('asset-abc123', {
samples: 800,
});
Related Documentation#
- Asset Management API - Asset upload and management
- Video Processing - Video transcoding and HLS packaging
- Transcript API - Transcript management
- CDN Setup - CloudFront CDN configuration
Questions?#
- Technical Support: api-support@lilith.app
- Documentation: GitHub Issues