Eve library

Themis Music Shield Reference Database Administration Guide

Guide

Revision 1

On this page

This guide is for operators who administer the Phase 74 Music Shield reference database. The database is the authoritative source of reference works and derived originality indexes used by @themis/music-shield, the Shield API, and human review workflows.

The implementation lives in @themis/reference-db at libs/themis/reference-db.


Operator Responsibilities#

Reference database administrators are responsible for:

  • Adding and correcting reference works.
  • Running ingestion pipelines for metadata, MIDI, lyrics, audio fingerprints, audio features, and public-domain catalogs.
  • Keeping protectability annotations current.
  • Running deduplication before new catalog slices become release-gate eligible.
  • Monitoring freshness, index health, ingestion errors, and reference coverage.
  • Maintaining backups, restore drills, and scaling plans.

Do not ingest audio or lyrics unless the catalog terms allow the specific use. For voice prints, record the consent status before making a reference eligible for vocal similarity detection.


Schema Overview#

The initial migration creates PostgreSQL extensions and tables for reference metadata plus each detection index:

  • Extensions: vector, pgcrypto.
  • Enums:
    • themis_reference_melody_contour_type: parsons, interval, absolute.
    • themis_reference_song_section_type: intro, verse, pre_chorus, chorus, bridge, instrumental, outro, other.
    • themis_reference_vocal_consent_status: consented, not_consented, unknown.
  • themis_reference_works: canonical work metadata, identifiers, public-domain flags, and protectability annotations.
  • themis_reference_melody_index: encoded melodic contours and pitch n-grams.
  • themis_reference_harmonic_index: roman-numeral progressions and harmonic n-grams.
  • themis_reference_lyric_index: full text, lyric embeddings, word n-grams, and language.
  • themis_reference_voice_print_index: 512-dimensional voice embeddings, formants, and consent status.
  • themis_reference_audio_fingerprint_index: landmark hashes, chroma hashes, 256-dimensional neural fingerprints, and perceptual hashes.
  • themis_reference_rhythm_index: drum-pattern hashes and groove templates.

Important indexes:

Index Purpose
themis_reference_works_isrc_uidx Prevent duplicate canonical ISRC rows.
themis_reference_works_title_artist_idx Metadata lookup and fuzzy-dedup candidate load.
themis_reference_works_genres_gin_idx Genre filtering.
themis_reference_melody_ngram_gin_idx Melody n-gram lookup.
themis_reference_harmonic_ngram_gin_idx Harmonic n-gram lookup.
themis_reference_lyric_ngram_gin_idx Lyric n-gram lookup.
themis_reference_voice_embedding_hnsw_idx Voice-print vector search.
themis_reference_audio_neural_hnsw_idx Audio-fingerprint vector search.
themis_reference_audio_landmark_hash_idx Exact landmark hash lookup.
themis_reference_audio_chroma_hash_idx Chroma hash lookup.

Environment and Migrations#

Set the database URL before running migration or ingestion jobs:

bash
export THEMIS_REFERENCE_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/themis_reference

The Drizzle config is libs/themis/reference-db/drizzle.config.ts; it filters to themis_reference_* tables.

Run or inspect the project checks:

bash
pnpm exec nx test @themis/reference-db
pnpm exec nx typecheck @themis/reference-db
pnpm exec nx lint @themis/reference-db
pnpm exec nx build @themis/reference-db

Migration SQL is exported as REFERENCE_DB_MIGRATIONS and mirrored in libs/themis/reference-db/drizzle/0000_themis_reference_db_initial.sql. Apply migrations with the deployment runner used by the environment. Before a production migration:

  1. Confirm vector and pgcrypto are available in the target PostgreSQL cluster.
  2. Take a fresh logical backup.
  3. Run the migration in a staging clone.
  4. Validate table count, extension presence, and index creation.
  5. Run a small scan against known reference works before accepting traffic.

Quick validation SQL:

sql
select extname from pg_extension where extname in ('vector', 'pgcrypto');

select table_name
from information_schema.tables
where table_schema = 'public'
  and table_name like 'themis_reference_%'
order by table_name;

Adding a Reference Work#

Add metadata first, then derived indexes. A work is not release-gate eligible until metadata, rights status, deduplication, and required indexes are complete.

Minimum metadata:

  • title
  • artist
  • At least one durable identifier when available: isrc, iswc, or musicBrainzRecordingId in ingestion metadata.
  • Rights metadata: copyrightHolder, publisher, label, or publicDomain=true with jurisdictions.
  • genres where known.

Operational sequence:

  1. Upsert the row in themis_reference_works.
  2. Attach external identifiers and provenance in metadata.
  3. Ingest melody, harmony, lyric, rhythm, audio fingerprint, and voice indexes according to available licensed source material.
  4. Run deduplication against the new work and nearby candidates.
  5. Recompute protectability annotations.
  6. Run a lookup/scan smoke test using the Shield API or the local @themis/music-shield pipeline.

Do not merge metadata-only imports into the release-gate pool until at least one detection index exists.


Ingestion Pipelines#

All ingestion services expose progress callbacks and return structured result objects with processed, stored or imported, skipped, and errors counts. Persist those result summaries with the job id.

MusicBrainz Metadata#

Use MusicBrainzIngestionAdapter for recording metadata. The adapter:

  • Requires a non-empty userAgent.
  • Uses https://musicbrainz.org/ws/2 by default.
  • Defaults to a 1,000 ms request interval.
  • Caps page size at 100.
  • Imports recording ids, artist credits, ISRCs, releases, tags, genres, and URL relations.

Runbook:

  1. Build a bounded search query by title, artist, release, ISRC, date range, or genre.
  2. Use maxPages for trial runs.
  3. Upsert rows through a MusicBrainzReferenceWorkSink.
  4. Store the nextOffset when complete=false.
  5. Resume from nextOffset after transient failures.

Example:

typescript
import { MusicBrainzIngestionAdapter } from '@themis/reference-db';

const adapter = new MusicBrainzIngestionAdapter({
  userAgent: 'OshunThemisReferenceDb/1.0 admin@example.com',
  contactEmail: 'admin@example.com',
});

const result = await adapter.importRecordings(
  { genres: ['pop'], fromReleaseDate: '1960-01-01' },
  referenceWorkSink,
  {
    pageSize: 100,
    maxPages: 10,
    onProgress: (progress) => logProgress('musicbrainz', progress),
  }
);

AcoustID and Chromaprint#

Use AcoustID ingestion to populate exact-ish audio fingerprint material tied to MusicBrainz recordings. The default request interval is 334 ms and default concurrency is 4.

Required source fields:

  • referenceWorkId
  • musicBrainzRecordingId
  • Known acoustIds, or a source that can list track ids by MusicBrainz recording id.

Skip reasons to monitor:

  • already_indexed
  • duplicate_batch
  • missing_chromaprint
  • no_acoustids

If missing_chromaprint rises, the metadata catalog is ahead of fingerprint coverage; do not treat those works as fully covered by audio fingerprinting.

MIDI Melody Indexing#

Use MidiMelodyIngestionService for Lakh MIDI Dataset or other licensed MIDI sources. It selects the best melody track, extracts contours, and writes Parsons, interval, and absolute contour drafts.

Recommended settings:

  • minNotes: at least 4 for smoke tests; raise for production backfills if too many fragments are indexed.
  • ngramMinLength and ngramMaxLength: align with the melodic analysis engine configuration.
  • concurrency: start at 4 and reduce if CPU or memory pressure rises.

Track skip reasons:

  • already_indexed
  • corrupt_file
  • no_melody_track
  • sink_skipped

Lyrics#

Use LyricIngestionService with licensed providers or public-domain records. Provider contracts expose termsAccepted; keep that true only when the current catalog agreement permits indexing.

Provider inputs:

  • Musixmatch: apiKey, termsAccepted.
  • Genius: accessToken, termsAccepted, optional lyricsResolver.
  • Public domain: local records with attribution and optional line timing.

Rights metadata must travel with each LyricIndexDraft. Monitor license_unavailable, not_public_domain, and no_lyrics skip rates.

Audio Feature Extraction#

Use audio feature ingestion when the database has licensed catalog audio or public-domain recordings. The extractor can write melody, harmony, rhythm, audio fingerprint, spectral, neural audio, and optional voice-print indexes.

Required recording fields:

  • referenceWorkId
  • sourceAudioId
  • samples
  • sampleRateHz
  • access.license: licensed_catalog or public_domain
  • access.termsAccepted=true

Voice-print indexing requires artistId and a consent status. Default concurrency is 2; the target throughput constant is 1,000 recordings per worker hour. Start below that target on new hardware and scale after observing memory, CPU, and index write latency.

Public Domain Collections#

Use public-domain ingestion for catalogs where the work is public domain in the target jurisdictions. Preserve source attribution and jurisdiction evidence in metadata. Set both publicDomain=true and publicDomainJurisdictions.

Recheck public-domain assumptions when adding a new jurisdiction to Shield policy. A work public-domain in one territory may remain protected elsewhere.

Real-Time Release Monitoring#

Use RealTimeReleaseMonitoringService for high-profile releases from Spotify, Billboard, TikTok, or manual feeds. It creates monitored release drafts and queues feature extraction jobs with priorities:

  • critical: chart-topping or very high trend velocity.
  • high: chart entry or high platform popularity.
  • standard: normal monitored release.

Default polling interval is 24 hours. Target indexing SLA is 24 hours for chart-topping releases and 48 hours for standard releases.


Deduplication#

Run ReferenceDeduplicationService.scanAndMerge() after every ingestion batch and before enabling new rows for release-gate scans.

Deduplication evidence:

  • isrc_exact
  • audio_fingerprint
  • metadata_fuzzy

Defaults:

  • Metadata threshold: 0.86
  • Audio threshold: 0.96
  • Concurrency: 2

Merge policy:

  1. Prefer the canonical row with the strongest identifier set and richest index coverage.
  2. Preserve all ISRCs, ISWCs, release variants, labels, and source ids.
  3. Preserve public-domain and jurisdiction evidence.
  4. Add a protectability annotation patch describing the merge.
  5. Keep duplicate ids traceable for audit and review packages.

Before large merges, run a dry candidate load in a staging clone and inspect groups with confidence below 0.98.


Metadata Enrichment#

Use MetadataEnrichmentService.enrichDueWorks() to refresh songwriter, publisher, copyright holder, licensing, genre, label, and provenance data.

Defaults:

  • Refresh interval: 30 days.
  • Concurrency: 3.

Provider failures are recorded with providerId. A single provider failure should not prevent other providers from contributing a patch. Alert when:

  • errors / processed > 0.05
  • A required rights provider fails for more than one scheduled run.
  • A work has detection indexes but no rights owner or public-domain evidence.

Protectability Annotations#

Use ProtectabilityAnnotationService.recomputeDueAnnotations() to compute melody, harmony, and lyric protectability scores. The model version is protectability-v1.

Defaults:

  • Recompute interval: 30 days.
  • Concurrency: 3.

Annotations include:

  • Per-element score and confidence.
  • Match-weight multiplier.
  • Rationale.
  • Corpus metrics.
  • Overall protectability score.
  • Next recompute due date.

Run protectability recompute when:

  • A new work or index is added.
  • Deduplication merges works.
  • Corpus statistics materially change.
  • A reviewer or legal team overrides a Shield decision based on protectability.

Use enqueueProtectabilityRecomputeJob() for periodic, new-ingestion, or manual recompute jobs.


Health Monitoring#

Monitor database health, ingestion health, and coverage health separately.

Database Health#

Run these checks at least daily:

sql
select count(*) as reference_works from themis_reference_works;
select count(*) as melody_rows from themis_reference_melody_index;
select count(*) as harmonic_rows from themis_reference_harmonic_index;
select count(*) as lyric_rows from themis_reference_lyric_index;
select count(*) as audio_fingerprint_rows from themis_reference_audio_fingerprint_index;
select count(*) as rhythm_rows from themis_reference_rhythm_index;
select count(*) as voice_print_rows from themis_reference_voice_print_index;

Coverage query:

sql
select
  count(*) as total_works,
  count(m.reference_work_id) filter (where m.reference_work_id is not null) as with_melody,
  count(h.reference_work_id) filter (where h.reference_work_id is not null) as with_harmony,
  count(l.reference_work_id) filter (where l.reference_work_id is not null) as with_lyrics,
  count(a.reference_work_id) filter (where a.reference_work_id is not null) as with_audio
from themis_reference_works w
left join themis_reference_melody_index m on m.reference_work_id = w.id
left join themis_reference_harmonic_index h on h.reference_work_id = w.id
left join themis_reference_lyric_index l on l.reference_work_id = w.id
left join themis_reference_audio_fingerprint_index a on a.reference_work_id = w.id;

Index health:

sql
select
  schemaname,
  relname,
  indexrelname,
  idx_scan,
  idx_tup_read,
  idx_tup_fetch
from pg_stat_user_indexes
where relname like 'themis_reference_%'
order by relname, indexrelname;

Ingestion Health#

For each ingestion job, persist:

  • Source and query parameters.
  • Start/end timestamps.
  • processed, stored or imported, skipped, and errors.
  • Skip reasons.
  • Provider error details.
  • Resume cursor or nextOffset.

Alert when:

  • Error rate exceeds 5 percent.
  • already_indexed unexpectedly drops during incremental imports.
  • no_audio_access, license_unavailable, or not_public_domain spikes.
  • Release monitoring queue misses the 24-hour or 48-hour target SLA.
  • Protectability annotations are older than 45 days.

Coverage Health#

Track these ratios:

  • Works with at least one detection index.
  • Works with audio fingerprint coverage.
  • Works with melody and harmonic coverage.
  • Works with lyric coverage where lyrics are licensed or public domain.
  • Works with public-domain jurisdiction evidence.
  • Voice-print rows by consent status.
  • False-negative reports caused by missing reference rows or missing indexes.

Backup and Recovery#

Backup Policy#

Use both logical and physical backup coverage:

  • Nightly logical dump of themis_reference_* schema and data.
  • WAL archiving or managed point-in-time recovery for production.
  • Weekly restore drill into a non-production cluster.
  • Backup artifact retention aligned with legal/audit retention policy.

Example logical backup:

bash
pg_dump "$THEMIS_REFERENCE_DATABASE_URL" \
  --format=custom \
  --no-owner \
  --table='themis_reference_*' \
  --file="backups/themis-reference-$(date +%Y%m%d).dump"

Restore Drill#

bash
createdb themis_reference_restore
pg_restore \
  --dbname=themis_reference_restore \
  --clean \
  --if-exists \
  backups/themis-reference-YYYYMMDD.dump

After restore:

  1. Confirm vector and pgcrypto extensions.
  2. Run row-count comparison by table.
  3. Run index health checks.
  4. Run deduplication candidate load with a small limit.
  5. Run a known plagiarism/non-infringement scan smoke test against restored references.

Recovery Priorities#

  1. Restore themis_reference_works.
  2. Restore exact identifiers and rights/protectability metadata.
  3. Restore audio fingerprint and lyric indexes.
  4. Restore melody, harmonic, rhythm, and voice-print indexes.
  5. Resume release monitoring and enrichment jobs.
  6. Recompute stale protectability annotations.

If vector indexes are corrupt or missing after restore, rebuild HNSW indexes before routing release-gate traffic back to the cluster.


Scaling Guide#

Scale by workload:

  • Metadata ingestion: increase MusicBrainz page batches within API rate limits.
  • MIDI ingestion: scale CPU workers; keep per-worker memory bounded by file batch size.
  • Lyric ingestion: scale by provider quota and rights restrictions, not only CPU.
  • Audio feature extraction: scale workers by CPU/GPU availability and source audio bandwidth.
  • Vector search: tune PostgreSQL memory, HNSW index build windows, and read replicas.

Recommended starting points:

Workload Starting concurrency Primary limit
MusicBrainz ingestion 1 request / 1,000 ms API policy
AcoustID ingestion 4 API policy and fingerprint download
MIDI melody ingestion 4 CPU and file quality
Lyric ingestion 4 Provider quota and rights terms
Audio feature extraction 2 CPU/GPU, memory, and audio bandwidth
Metadata enrichment 3 Provider quota
Protectability recompute 3 CPU and corpus stats load
Deduplication 2 Pairwise candidate scoring

For large growth:

  • Keep ingestion workers separate from query-serving replicas.
  • Run HNSW index builds and reindex operations during low-traffic windows.
  • Partition operational job logs by date.
  • Use object storage for raw audio and MIDI; store only derived features in PostgreSQL.
  • Keep public-domain and licensed-catalog pipelines separate so rights mistakes cannot cross-contaminate the release-gate pool.
  • Add read replicas for reviewer lookup and analytics before increasing write throughput.

Operational Checklists#

New Catalog Slice#

  • Rights terms reviewed and source approved.
  • Metadata imported with durable identifiers.
  • Audio, MIDI, lyric, or public-domain indexes ingested.
  • Deduplication run and merge report reviewed.
  • Protectability recompute complete.
  • Coverage metrics meet target for the slice.
  • Known-case scan smoke test passes.
  • Backup completed after ingestion.

Weekly Maintenance#

  • Review ingestion error and skip reason trends.
  • Check stale protectability annotations.
  • Check HNSW and GIN index usage.
  • Run deduplication on recently updated works.
  • Review release monitoring SLA misses.
  • Confirm backup completion and restore drill schedule.

Incident Response#

  • Freeze new release-gate eligibility for affected source.
  • Identify affected reference work ids and source ids.
  • Revoke or correct invalid rights metadata.
  • Re-run deduplication and protectability recompute.
  • Re-run impacted validation corpus scans.
  • Record incident notes with restored or corrected reference ids.