This guide is for operators who administer the Phase 74 Music Shield reference
database. The database is the authoritative source of reference works and
derived originality indexes used by @themis/music-shield, the Shield API, and
human review workflows.
The implementation lives in @themis/reference-db at
libs/themis/reference-db.
Operator Responsibilities#
Reference database administrators are responsible for:
- Adding and correcting reference works.
- Running ingestion pipelines for metadata, MIDI, lyrics, audio fingerprints, audio features, and public-domain catalogs.
- Keeping protectability annotations current.
- Running deduplication before new catalog slices become release-gate eligible.
- Monitoring freshness, index health, ingestion errors, and reference coverage.
- Maintaining backups, restore drills, and scaling plans.
Do not ingest audio or lyrics unless the catalog terms allow the specific use. For voice prints, record the consent status before making a reference eligible for vocal similarity detection.
Schema Overview#
The initial migration creates PostgreSQL extensions and tables for reference metadata plus each detection index:
- Extensions:
vector,pgcrypto. - Enums:
themis_reference_melody_contour_type:parsons,interval,absolute.themis_reference_song_section_type:intro,verse,pre_chorus,chorus,bridge,instrumental,outro,other.themis_reference_vocal_consent_status:consented,not_consented,unknown.
themis_reference_works: canonical work metadata, identifiers, public-domain flags, and protectability annotations.themis_reference_melody_index: encoded melodic contours and pitch n-grams.themis_reference_harmonic_index: roman-numeral progressions and harmonic n-grams.themis_reference_lyric_index: full text, lyric embeddings, word n-grams, and language.themis_reference_voice_print_index: 512-dimensional voice embeddings, formants, and consent status.themis_reference_audio_fingerprint_index: landmark hashes, chroma hashes, 256-dimensional neural fingerprints, and perceptual hashes.themis_reference_rhythm_index: drum-pattern hashes and groove templates.
Important indexes:
| Index | Purpose |
|---|---|
themis_reference_works_isrc_uidx |
Prevent duplicate canonical ISRC rows. |
themis_reference_works_title_artist_idx |
Metadata lookup and fuzzy-dedup candidate load. |
themis_reference_works_genres_gin_idx |
Genre filtering. |
themis_reference_melody_ngram_gin_idx |
Melody n-gram lookup. |
themis_reference_harmonic_ngram_gin_idx |
Harmonic n-gram lookup. |
themis_reference_lyric_ngram_gin_idx |
Lyric n-gram lookup. |
themis_reference_voice_embedding_hnsw_idx |
Voice-print vector search. |
themis_reference_audio_neural_hnsw_idx |
Audio-fingerprint vector search. |
themis_reference_audio_landmark_hash_idx |
Exact landmark hash lookup. |
themis_reference_audio_chroma_hash_idx |
Chroma hash lookup. |
Environment and Migrations#
Set the database URL before running migration or ingestion jobs:
export THEMIS_REFERENCE_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/themis_reference
The Drizzle config is libs/themis/reference-db/drizzle.config.ts; it filters
to themis_reference_* tables.
Run or inspect the project checks:
pnpm exec nx test @themis/reference-db
pnpm exec nx typecheck @themis/reference-db
pnpm exec nx lint @themis/reference-db
pnpm exec nx build @themis/reference-db
Migration SQL is exported as REFERENCE_DB_MIGRATIONS and mirrored in
libs/themis/reference-db/drizzle/0000_themis_reference_db_initial.sql. Apply
migrations with the deployment runner used by the environment. Before a
production migration:
- Confirm
vectorandpgcryptoare available in the target PostgreSQL cluster. - Take a fresh logical backup.
- Run the migration in a staging clone.
- Validate table count, extension presence, and index creation.
- Run a small scan against known reference works before accepting traffic.
Quick validation SQL:
select extname from pg_extension where extname in ('vector', 'pgcrypto');
select table_name
from information_schema.tables
where table_schema = 'public'
and table_name like 'themis_reference_%'
order by table_name;
Adding a Reference Work#
Add metadata first, then derived indexes. A work is not release-gate eligible until metadata, rights status, deduplication, and required indexes are complete.
Minimum metadata:
titleartist- At least one durable identifier when available:
isrc,iswc, ormusicBrainzRecordingIdin ingestion metadata. - Rights metadata:
copyrightHolder,publisher,label, orpublicDomain=truewith jurisdictions. genreswhere known.
Operational sequence:
- Upsert the row in
themis_reference_works. - Attach external identifiers and provenance in metadata.
- Ingest melody, harmony, lyric, rhythm, audio fingerprint, and voice indexes according to available licensed source material.
- Run deduplication against the new work and nearby candidates.
- Recompute protectability annotations.
- Run a lookup/scan smoke test using the Shield API or the local
@themis/music-shieldpipeline.
Do not merge metadata-only imports into the release-gate pool until at least one detection index exists.
Ingestion Pipelines#
All ingestion services expose progress callbacks and return structured result
objects with processed, stored or imported, skipped, and errors
counts. Persist those result summaries with the job id.
MusicBrainz Metadata#
Use MusicBrainzIngestionAdapter for recording metadata. The adapter:
- Requires a non-empty
userAgent. - Uses
https://musicbrainz.org/ws/2by default. - Defaults to a 1,000 ms request interval.
- Caps page size at 100.
- Imports recording ids, artist credits, ISRCs, releases, tags, genres, and URL relations.
Runbook:
- Build a bounded search query by title, artist, release, ISRC, date range, or genre.
- Use
maxPagesfor trial runs. - Upsert rows through a
MusicBrainzReferenceWorkSink. - Store the
nextOffsetwhencomplete=false. - Resume from
nextOffsetafter transient failures.
Example:
import { MusicBrainzIngestionAdapter } from '@themis/reference-db';
const adapter = new MusicBrainzIngestionAdapter({
userAgent: 'OshunThemisReferenceDb/1.0 admin@example.com',
contactEmail: 'admin@example.com',
});
const result = await adapter.importRecordings(
{ genres: ['pop'], fromReleaseDate: '1960-01-01' },
referenceWorkSink,
{
pageSize: 100,
maxPages: 10,
onProgress: (progress) => logProgress('musicbrainz', progress),
}
);
AcoustID and Chromaprint#
Use AcoustID ingestion to populate exact-ish audio fingerprint material tied to MusicBrainz recordings. The default request interval is 334 ms and default concurrency is 4.
Required source fields:
referenceWorkIdmusicBrainzRecordingId- Known
acoustIds, or a source that can list track ids by MusicBrainz recording id.
Skip reasons to monitor:
already_indexedduplicate_batchmissing_chromaprintno_acoustids
If missing_chromaprint rises, the metadata catalog is ahead of fingerprint
coverage; do not treat those works as fully covered by audio fingerprinting.
MIDI Melody Indexing#
Use MidiMelodyIngestionService for Lakh MIDI Dataset or other licensed MIDI
sources. It selects the best melody track, extracts contours, and writes
Parsons, interval, and absolute contour drafts.
Recommended settings:
minNotes: at least 4 for smoke tests; raise for production backfills if too many fragments are indexed.ngramMinLengthandngramMaxLength: align with the melodic analysis engine configuration.concurrency: start at 4 and reduce if CPU or memory pressure rises.
Track skip reasons:
already_indexedcorrupt_fileno_melody_tracksink_skipped
Lyrics#
Use LyricIngestionService with licensed providers or public-domain records.
Provider contracts expose termsAccepted; keep that true only when the current
catalog agreement permits indexing.
Provider inputs:
- Musixmatch:
apiKey,termsAccepted. - Genius:
accessToken,termsAccepted, optionallyricsResolver. - Public domain: local records with attribution and optional line timing.
Rights metadata must travel with each LyricIndexDraft. Monitor
license_unavailable, not_public_domain, and no_lyrics skip rates.
Audio Feature Extraction#
Use audio feature ingestion when the database has licensed catalog audio or public-domain recordings. The extractor can write melody, harmony, rhythm, audio fingerprint, spectral, neural audio, and optional voice-print indexes.
Required recording fields:
referenceWorkIdsourceAudioIdsamplessampleRateHzaccess.license:licensed_catalogorpublic_domainaccess.termsAccepted=true
Voice-print indexing requires artistId and a consent status. Default
concurrency is 2; the target throughput constant is 1,000 recordings per worker
hour. Start below that target on new hardware and scale after observing memory,
CPU, and index write latency.
Public Domain Collections#
Use public-domain ingestion for catalogs where the work is public domain in the
target jurisdictions. Preserve source attribution and jurisdiction evidence in
metadata. Set both publicDomain=true and publicDomainJurisdictions.
Recheck public-domain assumptions when adding a new jurisdiction to Shield policy. A work public-domain in one territory may remain protected elsewhere.
Real-Time Release Monitoring#
Use RealTimeReleaseMonitoringService for high-profile releases from Spotify,
Billboard, TikTok, or manual feeds. It creates monitored release drafts and
queues feature extraction jobs with priorities:
critical: chart-topping or very high trend velocity.high: chart entry or high platform popularity.standard: normal monitored release.
Default polling interval is 24 hours. Target indexing SLA is 24 hours for chart-topping releases and 48 hours for standard releases.
Deduplication#
Run ReferenceDeduplicationService.scanAndMerge() after every ingestion batch
and before enabling new rows for release-gate scans.
Deduplication evidence:
isrc_exactaudio_fingerprintmetadata_fuzzy
Defaults:
- Metadata threshold:
0.86 - Audio threshold:
0.96 - Concurrency:
2
Merge policy:
- Prefer the canonical row with the strongest identifier set and richest index coverage.
- Preserve all ISRCs, ISWCs, release variants, labels, and source ids.
- Preserve public-domain and jurisdiction evidence.
- Add a protectability annotation patch describing the merge.
- Keep duplicate ids traceable for audit and review packages.
Before large merges, run a dry candidate load in a staging clone and inspect
groups with confidence below 0.98.
Metadata Enrichment#
Use MetadataEnrichmentService.enrichDueWorks() to refresh songwriter,
publisher, copyright holder, licensing, genre, label, and provenance data.
Defaults:
- Refresh interval: 30 days.
- Concurrency: 3.
Provider failures are recorded with providerId. A single provider failure
should not prevent other providers from contributing a patch. Alert when:
errors / processed > 0.05- A required rights provider fails for more than one scheduled run.
- A work has detection indexes but no rights owner or public-domain evidence.
Protectability Annotations#
Use ProtectabilityAnnotationService.recomputeDueAnnotations() to compute
melody, harmony, and lyric protectability scores. The model version is
protectability-v1.
Defaults:
- Recompute interval: 30 days.
- Concurrency: 3.
Annotations include:
- Per-element score and confidence.
- Match-weight multiplier.
- Rationale.
- Corpus metrics.
- Overall protectability score.
- Next recompute due date.
Run protectability recompute when:
- A new work or index is added.
- Deduplication merges works.
- Corpus statistics materially change.
- A reviewer or legal team overrides a Shield decision based on protectability.
Use enqueueProtectabilityRecomputeJob() for periodic, new-ingestion, or manual
recompute jobs.
Health Monitoring#
Monitor database health, ingestion health, and coverage health separately.
Database Health#
Run these checks at least daily:
select count(*) as reference_works from themis_reference_works;
select count(*) as melody_rows from themis_reference_melody_index;
select count(*) as harmonic_rows from themis_reference_harmonic_index;
select count(*) as lyric_rows from themis_reference_lyric_index;
select count(*) as audio_fingerprint_rows from themis_reference_audio_fingerprint_index;
select count(*) as rhythm_rows from themis_reference_rhythm_index;
select count(*) as voice_print_rows from themis_reference_voice_print_index;
Coverage query:
select
count(*) as total_works,
count(m.reference_work_id) filter (where m.reference_work_id is not null) as with_melody,
count(h.reference_work_id) filter (where h.reference_work_id is not null) as with_harmony,
count(l.reference_work_id) filter (where l.reference_work_id is not null) as with_lyrics,
count(a.reference_work_id) filter (where a.reference_work_id is not null) as with_audio
from themis_reference_works w
left join themis_reference_melody_index m on m.reference_work_id = w.id
left join themis_reference_harmonic_index h on h.reference_work_id = w.id
left join themis_reference_lyric_index l on l.reference_work_id = w.id
left join themis_reference_audio_fingerprint_index a on a.reference_work_id = w.id;
Index health:
select
schemaname,
relname,
indexrelname,
idx_scan,
idx_tup_read,
idx_tup_fetch
from pg_stat_user_indexes
where relname like 'themis_reference_%'
order by relname, indexrelname;
Ingestion Health#
For each ingestion job, persist:
- Source and query parameters.
- Start/end timestamps.
processed,storedorimported,skipped, anderrors.- Skip reasons.
- Provider error details.
- Resume cursor or
nextOffset.
Alert when:
- Error rate exceeds 5 percent.
already_indexedunexpectedly drops during incremental imports.no_audio_access,license_unavailable, ornot_public_domainspikes.- Release monitoring queue misses the 24-hour or 48-hour target SLA.
- Protectability annotations are older than 45 days.
Coverage Health#
Track these ratios:
- Works with at least one detection index.
- Works with audio fingerprint coverage.
- Works with melody and harmonic coverage.
- Works with lyric coverage where lyrics are licensed or public domain.
- Works with public-domain jurisdiction evidence.
- Voice-print rows by consent status.
- False-negative reports caused by missing reference rows or missing indexes.
Backup and Recovery#
Backup Policy#
Use both logical and physical backup coverage:
- Nightly logical dump of
themis_reference_*schema and data. - WAL archiving or managed point-in-time recovery for production.
- Weekly restore drill into a non-production cluster.
- Backup artifact retention aligned with legal/audit retention policy.
Example logical backup:
pg_dump "$THEMIS_REFERENCE_DATABASE_URL" \
--format=custom \
--no-owner \
--table='themis_reference_*' \
--file="backups/themis-reference-$(date +%Y%m%d).dump"
Restore Drill#
createdb themis_reference_restore
pg_restore \
--dbname=themis_reference_restore \
--clean \
--if-exists \
backups/themis-reference-YYYYMMDD.dump
After restore:
- Confirm
vectorandpgcryptoextensions. - Run row-count comparison by table.
- Run index health checks.
- Run deduplication candidate load with a small limit.
- Run a known plagiarism/non-infringement scan smoke test against restored references.
Recovery Priorities#
- Restore
themis_reference_works. - Restore exact identifiers and rights/protectability metadata.
- Restore audio fingerprint and lyric indexes.
- Restore melody, harmonic, rhythm, and voice-print indexes.
- Resume release monitoring and enrichment jobs.
- Recompute stale protectability annotations.
If vector indexes are corrupt or missing after restore, rebuild HNSW indexes before routing release-gate traffic back to the cluster.
Scaling Guide#
Scale by workload:
- Metadata ingestion: increase MusicBrainz page batches within API rate limits.
- MIDI ingestion: scale CPU workers; keep per-worker memory bounded by file batch size.
- Lyric ingestion: scale by provider quota and rights restrictions, not only CPU.
- Audio feature extraction: scale workers by CPU/GPU availability and source audio bandwidth.
- Vector search: tune PostgreSQL memory, HNSW index build windows, and read replicas.
Recommended starting points:
| Workload | Starting concurrency | Primary limit |
|---|---|---|
| MusicBrainz ingestion | 1 request / 1,000 ms | API policy |
| AcoustID ingestion | 4 | API policy and fingerprint download |
| MIDI melody ingestion | 4 | CPU and file quality |
| Lyric ingestion | 4 | Provider quota and rights terms |
| Audio feature extraction | 2 | CPU/GPU, memory, and audio bandwidth |
| Metadata enrichment | 3 | Provider quota |
| Protectability recompute | 3 | CPU and corpus stats load |
| Deduplication | 2 | Pairwise candidate scoring |
For large growth:
- Keep ingestion workers separate from query-serving replicas.
- Run HNSW index builds and reindex operations during low-traffic windows.
- Partition operational job logs by date.
- Use object storage for raw audio and MIDI; store only derived features in PostgreSQL.
- Keep public-domain and licensed-catalog pipelines separate so rights mistakes cannot cross-contaminate the release-gate pool.
- Add read replicas for reviewer lookup and analytics before increasing write throughput.
Operational Checklists#
New Catalog Slice#
- Rights terms reviewed and source approved.
- Metadata imported with durable identifiers.
- Audio, MIDI, lyric, or public-domain indexes ingested.
- Deduplication run and merge report reviewed.
- Protectability recompute complete.
- Coverage metrics meet target for the slice.
- Known-case scan smoke test passes.
- Backup completed after ingestion.
Weekly Maintenance#
- Review ingestion error and skip reason trends.
- Check stale protectability annotations.
- Check HNSW and GIN index usage.
- Run deduplication on recently updated works.
- Review release monitoring SLA misses.
- Confirm backup completion and restore drill schedule.
Incident Response#
- Freeze new release-gate eligibility for affected source.
- Identify affected reference work ids and source ids.
- Revoke or correct invalid rights metadata.
- Re-run deduplication and protectability recompute.
- Re-run impacted validation corpus scans.
- Record incident notes with restored or corrected reference ids.