Eve library

Audio Processing Pipeline

Guide

Revision 1

On this page

This document provides comprehensive documentation for the Lilith Media Service Audio Processing Pipeline, including transcoding, normalization, silence detection/removal, waveform generation, and audio fingerprinting.

Overview#

The Audio Processing Pipeline provides:

  • Audio Transcoding: Convert between formats (MP3, AAC, Opus, WAV, FLAC, OGG)
  • Audio Normalization: LUFS-based loudness normalization with platform presets
  • Silence Processing: Detection, removal, trimming, and splitting at silence
  • Waveform Generation: Visual waveform images and JSON peak data
  • Audio Fingerprinting: Chromaprint-based fingerprinting for duplicate detection

Table of Contents#

  1. Audio Transcoding
  2. Audio Normalization
  3. Silence Processing
  4. Waveform Generation
  5. Audio Fingerprinting

Audio Transcoding#

Supported Formats#

Format Codec Extension Default Bitrate Use Case
mp3 libmp3lame .mp3 192 kbps General audio, broad compatibility
aac aac (LC-AAC) .m4a 128 kbps Mobile streaming, Apple ecosystem
opus libopus .opus 96 kbps Voice, VoIP, web streaming
wav pcm_s16le .wav Lossless Editing, archival
flac flac .flac Lossless Lossless archival
ogg libvorbis .ogg 128 kbps Open source, web audio

Transcode Options#

typescript
interface AudioTranscodeOptions {
  format: 'mp3' | 'aac' | 'opus' | 'wav' | 'flac' | 'ogg';
  bitrate?: number; // kbps (e.g., 128, 192, 256, 320)
  sampleRate?: number; // Hz (e.g., 44100, 48000)
  channels?: number; // 1 = mono, 2 = stereo
  codec?: string; // Override default codec
  onProgress?: (progress: FFmpegProgress) => void;
  timeout?: number; // ms (default: 30 minutes)
}

Transcoding Presets#

Preset Format Bitrate Sample Rate Channels Use Case
mp3_high mp3 320 kbps 44.1 kHz stereo Music archival
mp3_standard mp3 192 kbps 44.1 kHz stereo General use
mp3_voice mp3 96 kbps 44.1 kHz mono Podcasts, voice
aac_streaming aac 128 kbps 44.1 kHz stereo Mobile streaming
aac_high aac 256 kbps 48 kHz stereo High-quality AAC
opus_voice opus 32 kbps 48 kHz mono Voice chat, VoIP
opus_music opus 128 kbps 48 kHz stereo Music streaming
wav_lossless wav Lossless 44.1 kHz stereo Editing source
flac_lossless flac Lossless 44.1 kHz stereo Lossless archival

Usage Example#

typescript
import {
  transcodeAudio,
  transcodeAudioBuffer,
  batchTranscode,
  TRANSCODE_PRESETS,
} from './processing/audio-transcoder.js';

// Single file transcoding
const result = await transcodeAudio(
  '/path/to/input.wav',
  '/path/to/output.mp3',
  {
    format: 'mp3',
    bitrate: 320,
    sampleRate: 44100,
    onProgress: (progress) => {
      console.log(`Progress: ${progress.percent}%`);
    },
  }
);

console.log(
  `Transcoded: ${result.outputFormat} at ${result.outputBitrate}kbps`
);

// Using presets
const voiceResult = await transcodeAudio(
  '/path/to/voice.wav',
  '/path/to/voice.mp3',
  TRANSCODE_PRESETS.mp3_voice
);

// Buffer transcoding
const { outputBuffer, result } = await transcodeAudioBuffer(
  inputBuffer,
  '.wav',
  { format: 'aac', bitrate: 128 }
);

// Batch transcoding
const files = [
  { inputPath: 'track1.wav', outputPath: 'track1.mp3' },
  { inputPath: 'track2.wav', outputPath: 'track2.mp3' },
];

const results = await batchTranscode(files, { format: 'mp3', bitrate: 192 });

Audio Normalization#

Overview#

Audio normalization adjusts loudness to target levels using the EBU R128 standard (LUFS measurement). This ensures consistent playback volume across different audio files.

Normalization Options#

typescript
interface NormalizationOptions {
  targetLoudness?: number; // LUFS (default: -16)
  truePeak?: number; // dBTP (default: -1.5)
  loudnessRange?: number; // LU (default: 11)
  dualPass?: boolean; // Two-pass for accuracy (default: true)
  linear?: boolean; // No compression (default: false)
  onProgress?: (progress: FFmpegProgress) => void;
  timeout?: number;
}

Platform Presets#

Preset Target LUFS True Peak LRA Description
spotify -14 LUFS -1 dBTP - Spotify recommendation
youtube -14 LUFS -1 dBTP - YouTube recommendation
apple -16 LUFS -1 dBTP - Apple Music / iTunes
podcast -16 LUFS -1.5 dBTP 8 LU Podcast standard
voice -16 LUFS -3 dBTP 5 LU Voice/speech (tight DR)
broadcast -23 LUFS -1 dBTP 15 LU EBU R128 broadcast
peak_only -16 LUFS -1 dBTP - Simple peak limiting

Loudness Analysis#

typescript
interface LoudnessInfo {
  integratedLoudness: number; // LUFS
  truePeak: number; // dBTP
  loudnessRange: number; // LU
  threshold: number;
  targetOffset?: number;
}

Usage Example#

typescript
import {
  normalizeAudio,
  analyzeLoudness,
  needsNormalization,
  applyLimiter,
  applyCompressor,
  NORMALIZATION_PRESETS,
} from './processing/audio-normalizer.js';

// Analyze loudness
const loudness = await analyzeLoudness('/path/to/audio.mp3');
console.log(`Current loudness: ${loudness.integratedLoudness} LUFS`);
console.log(`True peak: ${loudness.truePeak} dBTP`);

// Check if normalization is needed
const check = await needsNormalization('/path/to/audio.mp3', -16);
if (check.needsNormalization) {
  console.log(`Needs adjustment: ${check.delta} dB`);
}

// Normalize audio
const result = await normalizeAudio(
  '/path/to/input.mp3',
  '/path/to/output.wav',
  {
    targetLoudness: -14,
    truePeak: -1,
    dualPass: true,
  }
);

console.log(`Gain applied: ${result.gainAdjustment} dB`);
console.log(
  `Output loudness: ${result.outputLoudness?.integratedLoudness} LUFS`
);

// Using platform presets
const spotifyResult = await normalizeAudio(
  '/path/to/input.mp3',
  '/path/to/spotify-ready.wav',
  NORMALIZATION_PRESETS.spotify
);

// Apply limiter
await applyLimiter('/path/to/input.wav', '/path/to/limited.wav', {
  threshold: -1, // dB
  attack: 5, // ms
  release: 50, // ms
  makeup: 0, // dB
});

// Apply compressor
await applyCompressor('/path/to/input.wav', '/path/to/compressed.wav', {
  threshold: -20, // dB
  ratio: 4, // 4:1 compression
  attack: 20, // ms
  release: 250, // ms
  knee: 2.82843, // dB
});

Silence Processing#

Silence Detection#

Detect silence segments in audio files based on noise threshold and minimum duration.

typescript
interface SilenceDetectionResult {
  segments: SilenceSegment[];
  totalSilenceDuration: number;
  audioDuration: number;
  silencePercentage: number;
}

interface SilenceSegment {
  start: number; // seconds
  end: number; // seconds
  duration: number; // seconds
}

Silence Removal Options#

typescript
interface SilenceRemovalOptions {
  noiseThreshold?: number; // dB (default: -50)
  minDuration?: number; // seconds (default: 0.5)
  padding?: number; // Keep around speech (default: 0.1s)
  maxSilence?: number; // Max silence between speech (default: 0.5s)
  trimStart?: boolean; // Remove from start only
  trimEnd?: boolean; // Remove from end only
}

Content Presets#

Preset Threshold Min Duration Padding Max Silence Use Case
podcast -40 dB 0.3s 0.05s 0.3s Aggressive removal
music -60 dB 2.0s 0.2s 1.0s Preserve dynamics
voice -45 dB 0.5s 0.1s 0.5s Moderate removal
audiobook -45 dB 1.0s 0.15s 0.8s Keep natural pauses
transcription -35 dB 0.2s 0.02s 0.2s Prep for STT

Usage Example#

typescript
import {
  detectSilence,
  removeSilence,
  trimSilence,
  compressSilence,
  splitAtSilence,
  getSilenceStats,
  SILENCE_PRESETS,
} from './processing/silence-processor.js';

// Detect silence segments
const silence = await detectSilence('/path/to/audio.mp3', {
  noiseThreshold: -50,
  minDuration: 0.5,
});

console.log(`Found ${silence.segments.length} silence segments`);
console.log(`${silence.silencePercentage.toFixed(1)}% of audio is silence`);

// Get detailed statistics
const stats = getSilenceStats(silence);
console.log(`Average silence: ${stats.averageDuration.toFixed(2)}s`);
console.log(`Longest silence: ${stats.maxDuration.toFixed(2)}s`);

// Remove all silence
const result = await removeSilence(
  '/path/to/input.mp3',
  '/path/to/no-silence.wav',
  { noiseThreshold: -45, minDuration: 0.3 }
);

console.log(`Removed ${result.durationRemoved.toFixed(2)}s of silence`);

// Trim silence from start/end only
await trimSilence('/path/to/input.mp3', '/path/to/trimmed.wav', {
  padding: 0.1,
});

// Compress pauses (keep shorter silence)
await compressSilence('/path/to/input.mp3', '/path/to/compressed.wav', {
  maxSilence: 0.5,
});

// Split at silence points
const { segments, silenceInfo } = await splitAtSilence(
  '/path/to/audio.mp3',
  '/path/to/output/',
  {
    ...SILENCE_PRESETS.podcast,
    outputFormat: 'wav',
    minSegmentDuration: 2.0,
  }
);

console.log(`Created ${segments.length} segments`);
segments.forEach((seg, i) => {
  console.log(`Segment ${i + 1}: ${seg.duration.toFixed(2)}s`);
});

Waveform Generation#

Overview#

Generate visual waveform representations as images (PNG) or data arrays for frontend rendering.

Waveform Options#

typescript
interface WaveformOptions {
  samples?: number; // Number of bars (default: 800)
  width?: number; // Image width px (default: 1800)
  height?: number; // Image height px (default: 140)
  color?: string; // Hex color (default: '#5865F2')
  backgroundColor?: string; // Hex or 'transparent'
  mirror?: boolean; // Mirrored waveform (default: true)
  scale?: 'lin' | 'log' | 'sqrt'; // Scale type (default: 'sqrt')
}

Waveform Presets#

Preset Samples Dimensions Scale Use Case
detailed 2000 3600x280 linear Full detail view
player 800 1800x140 sqrt Audio player UI
thumbnail 200 400x60 sqrt List thumbnails
mobile 100 300x40 sqrt Mobile devices

Waveform Data Format#

typescript
interface WaveformDataResult {
  peaks: number[]; // Amplitude values (0.0 to 1.0)
  duration: number; // Audio duration in seconds
  sampleRate: number; // Samples per second
  channels: number; // Channel count
  bitsPerSample: number;
}

Usage Example#

typescript
import {
  generateWaveformImage,
  generateWaveformData,
  generateWaveform,
  peaksToSvgPath,
  downsamplePeaks,
  WAVEFORM_PRESETS,
} from './processing/waveform-generator.js';

// Generate waveform image
const imageResult = await generateWaveformImage(
  '/path/to/audio.mp3',
  '/path/to/waveform.png',
  {
    width: 1800,
    height: 140,
    color: '#5865F2',
    backgroundColor: 'transparent',
    mirror: true,
    scale: 'sqrt',
  }
);

console.log(`Generated waveform: ${imageResult.width}x${imageResult.height}`);

// Generate waveform data (for frontend)
const data = await generateWaveformData('/path/to/audio.mp3', {
  samples: 800,
  scale: 'sqrt',
});

console.log(`${data.peaks.length} samples, ${data.duration}s duration`);

// Generate both image and data
const { image, data: waveformData } = await generateWaveform(
  '/path/to/audio.mp3',
  '/path/to/waveform.png',
  WAVEFORM_PRESETS.player
);

// Convert peaks to SVG path
const svgPath = peaksToSvgPath(data.peaks, 800, 100, {
  mirror: true,
  smooth: true,
});

// Use in SVG:
// <path d="${svgPath}" fill="#5865F2" />

// Downsample for different views
const mobilePeaks = downsamplePeaks(data.peaks, 100);

// Generate from buffer
const { imageBuffer, result } = await generateWaveformImageFromBuffer(
  audioBuffer,
  '.mp3',
  WAVEFORM_PRESETS.thumbnail
);

Audio Fingerprinting#

Overview#

Generate audio fingerprints using Chromaprint (fpcalc) for audio identification, duplicate detection, and integration with music recognition services like AcoustID.

Prerequisites#

Requires Chromaprint's fpcalc utility:

  • macOS: brew install chromaprint
  • Ubuntu: apt-get install libchromaprint-tools
  • Windows: Download from https://acoustid.org/chromaprint

Fingerprint Result#

typescript
interface FingerprintResult {
  fingerprint: string; // Raw fingerprint
  duration: number; // Duration analyzed
  fingerprintBase64: string; // Base64 encoded
  fingerprintHash: string; // SHA-256 hash
  algorithm: number; // Algorithm version
}

Fingerprint Options#

typescript
interface FingerprintOptions {
  maxDuration?: number; // Max seconds to analyze (default: 120)
  startOffset?: number; // Start offset in seconds (default: 0)
  algorithm?: number; // Algorithm version (default: 2)
  overlap?: number; // Overlap factor (default: 0)
}

Usage Example#

typescript
import {
  generateFingerprint,
  compareFingerprints,
  compareRawFingerprints,
  areDuplicates,
  findDuplicates,
  batchFingerprint,
  prepareForAcoustId,
  isFpcalcAvailable,
} from './processing/audio-fingerprint.js';

// Check if fpcalc is available
if (await isFpcalcAvailable()) {
  console.log('Chromaprint is installed');
}

// Generate fingerprint
const fingerprint = await generateFingerprint('/path/to/audio.mp3', {
  maxDuration: 120,
  algorithm: 2,
});

console.log(`Fingerprint hash: ${fingerprint.fingerprintHash}`);
console.log(`Duration analyzed: ${fingerprint.duration}s`);

// Compare two fingerprints
const fp1 = await generateFingerprint('/path/to/song1.mp3');
const fp2 = await generateFingerprint('/path/to/song2.mp3');

const comparison = compareFingerprints(fp1, fp2, {
  threshold: 0.9,
  method: 'hash',
});

console.log(`Similarity: ${(comparison.similarity * 100).toFixed(1)}%`);
console.log(`Match: ${comparison.match}`);

// Check if files are duplicates
const duplicateCheck = await areDuplicates(
  '/path/to/file1.mp3',
  '/path/to/file2.mp3',
  { threshold: 0.85 }
);

if (duplicateCheck.isDuplicate) {
  console.log(
    `Duplicates! Similarity: ${(duplicateCheck.similarity * 100).toFixed(1)}%`
  );
}

// Batch fingerprint multiple files
const fingerprints = await batchFingerprint(
  ['song1.mp3', 'song2.mp3', 'song3.mp3'],
  {
    maxDuration: 120,
    concurrency: 4,
    onProgress: (i, total, file) => {
      console.log(`Processing ${i}/${total}: ${file}`);
    },
  }
);

// Find duplicates in collection
const duplicates = await findDuplicates(
  ['song1.mp3', 'song2.mp3', 'song3.mp3', 'song4.mp3'],
  {
    threshold: 0.85,
    onProgress: (current, total) => {
      console.log(`Comparing: ${current}/${total}`);
    },
  }
);

duplicates.forEach((dup) => {
  console.log(`Duplicate pair: ${dup.files[0]} <-> ${dup.files[1]}`);
  console.log(`Similarity: ${(dup.similarity * 100).toFixed(1)}%`);
});

// Prepare for AcoustID lookup
const acoustId = prepareForAcoustId(fingerprint);
console.log(
  `AcoustID fingerprint: ${acoustId.acoustIdFingerprint.substring(0, 50)}...`
);
console.log(`Duration: ${acoustId.acoustIdDuration}s`);

Processing Pipeline Integration#

Requesting Processing Jobs#

Use the Asset API to request audio processing:

bash
curl -X POST "https://api.lilith.app/v1/assets/asset-abc123/process" \
  -H "Authorization: Bearer <access_token>" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "transcode",
    "config": {
      "format": "mp3",
      "bitrate": 192,
      "normalize": true,
      "targetLoudness": -14
    }
  }'

Processing Job Types#

Type Description
transcode Convert audio format
normalization Apply loudness normalization
silence_removal Remove or compress silence
waveform Generate waveform visualization
fingerprint Generate audio fingerprint
metadata_extraction Extract detailed audio metadata

SDK Integration#

TypeScript/JavaScript#

typescript
import { LilithClient } from '@lilith/sdk';

const client = new LilithClient({
  apiKey: process.env.LILITH_API_KEY,
});

// Request transcoding
const job = await client.media.requestProcessing('asset-abc123', {
  type: 'transcode',
  config: {
    format: 'mp3',
    bitrate: 320,
    normalize: true,
    targetLoudness: -14,
  },
});

// Monitor job progress
const status = await client.media.getProcessingJob(job.jobId);
console.log(`Status: ${status.status}, Progress: ${status.progress}%`);

// Get waveform data
const waveform = await client.media.getWaveform('asset-abc123', {
  samples: 800,
});


Questions?#

  • Technical Support: api-support@lilith.app
  • Documentation: GitHub Issues