# Themis Music Shield Reference Database Administration Guide

This guide is for operators who administer the Phase 74 Music Shield reference
database. The database is the authoritative source of reference works and
derived originality indexes used by `@themis/music-shield`, the Shield API, and
human review workflows.

The implementation lives in `@themis/reference-db` at
`libs/themis/reference-db`.

---

## Operator Responsibilities

Reference database administrators are responsible for:

- Adding and correcting reference works.
- Running ingestion pipelines for metadata, MIDI, lyrics, audio fingerprints,
  audio features, and public-domain catalogs.
- Keeping protectability annotations current.
- Running deduplication before new catalog slices become release-gate eligible.
- Monitoring freshness, index health, ingestion errors, and reference coverage.
- Maintaining backups, restore drills, and scaling plans.

Do not ingest audio or lyrics unless the catalog terms allow the specific use.
For voice prints, record the consent status before making a reference eligible
for vocal similarity detection.

---

## Schema Overview

The initial migration creates PostgreSQL extensions and tables for reference
metadata plus each detection index:

- Extensions: `vector`, `pgcrypto`.
- Enums:
  - `themis_reference_melody_contour_type`: `parsons`, `interval`, `absolute`.
  - `themis_reference_song_section_type`: `intro`, `verse`, `pre_chorus`,
    `chorus`, `bridge`, `instrumental`, `outro`, `other`.
  - `themis_reference_vocal_consent_status`: `consented`, `not_consented`,
    `unknown`.
- `themis_reference_works`: canonical work metadata, identifiers, public-domain
  flags, and protectability annotations.
- `themis_reference_melody_index`: encoded melodic contours and pitch n-grams.
- `themis_reference_harmonic_index`: roman-numeral progressions and harmonic
  n-grams.
- `themis_reference_lyric_index`: full text, lyric embeddings, word n-grams, and
  language.
- `themis_reference_voice_print_index`: 512-dimensional voice embeddings,
  formants, and consent status.
- `themis_reference_audio_fingerprint_index`: landmark hashes, chroma hashes,
  256-dimensional neural fingerprints, and perceptual hashes.
- `themis_reference_rhythm_index`: drum-pattern hashes and groove templates.

Important indexes:

| Index                                       | Purpose                                         |
| ------------------------------------------- | ----------------------------------------------- |
| `themis_reference_works_isrc_uidx`          | Prevent duplicate canonical ISRC rows.          |
| `themis_reference_works_title_artist_idx`   | Metadata lookup and fuzzy-dedup candidate load. |
| `themis_reference_works_genres_gin_idx`     | Genre filtering.                                |
| `themis_reference_melody_ngram_gin_idx`     | Melody n-gram lookup.                           |
| `themis_reference_harmonic_ngram_gin_idx`   | Harmonic n-gram lookup.                         |
| `themis_reference_lyric_ngram_gin_idx`      | Lyric n-gram lookup.                            |
| `themis_reference_voice_embedding_hnsw_idx` | Voice-print vector search.                      |
| `themis_reference_audio_neural_hnsw_idx`    | Audio-fingerprint vector search.                |
| `themis_reference_audio_landmark_hash_idx`  | Exact landmark hash lookup.                     |
| `themis_reference_audio_chroma_hash_idx`    | Chroma hash lookup.                             |

---

## Environment and Migrations

Set the database URL before running migration or ingestion jobs:

```bash
export THEMIS_REFERENCE_DATABASE_URL=postgresql://oshun:oshun_dev@localhost:5432/themis_reference
```

The Drizzle config is `libs/themis/reference-db/drizzle.config.ts`; it filters
to `themis_reference_*` tables.

Run or inspect the project checks:

```bash
pnpm exec nx test @themis/reference-db
pnpm exec nx typecheck @themis/reference-db
pnpm exec nx lint @themis/reference-db
pnpm exec nx build @themis/reference-db
```

Migration SQL is exported as `REFERENCE_DB_MIGRATIONS` and mirrored in
`libs/themis/reference-db/drizzle/0000_themis_reference_db_initial.sql`. Apply
migrations with the deployment runner used by the environment. Before a
production migration:

1. Confirm `vector` and `pgcrypto` are available in the target PostgreSQL
   cluster.
2. Take a fresh logical backup.
3. Run the migration in a staging clone.
4. Validate table count, extension presence, and index creation.
5. Run a small scan against known reference works before accepting traffic.

Quick validation SQL:

```sql
select extname from pg_extension where extname in ('vector', 'pgcrypto');

select table_name
from information_schema.tables
where table_schema = 'public'
  and table_name like 'themis_reference_%'
order by table_name;
```

---

## Adding a Reference Work

Add metadata first, then derived indexes. A work is not release-gate eligible
until metadata, rights status, deduplication, and required indexes are complete.

Minimum metadata:

- `title`
- `artist`
- At least one durable identifier when available: `isrc`, `iswc`, or
  `musicBrainzRecordingId` in ingestion metadata.
- Rights metadata: `copyrightHolder`, `publisher`, `label`, or
  `publicDomain=true` with jurisdictions.
- `genres` where known.

Operational sequence:

1. Upsert the row in `themis_reference_works`.
2. Attach external identifiers and provenance in metadata.
3. Ingest melody, harmony, lyric, rhythm, audio fingerprint, and voice indexes
   according to available licensed source material.
4. Run deduplication against the new work and nearby candidates.
5. Recompute protectability annotations.
6. Run a lookup/scan smoke test using the Shield API or the local
   `@themis/music-shield` pipeline.

Do not merge metadata-only imports into the release-gate pool until at least one
detection index exists.

---

## Ingestion Pipelines

All ingestion services expose progress callbacks and return structured result
objects with `processed`, `stored` or `imported`, `skipped`, and `errors`
counts. Persist those result summaries with the job id.

### MusicBrainz Metadata

Use `MusicBrainzIngestionAdapter` for recording metadata. The adapter:

- Requires a non-empty `userAgent`.
- Uses `https://musicbrainz.org/ws/2` by default.
- Defaults to a 1,000 ms request interval.
- Caps page size at 100.
- Imports recording ids, artist credits, ISRCs, releases, tags, genres, and URL
  relations.

Runbook:

1. Build a bounded search query by title, artist, release, ISRC, date range, or
   genre.
2. Use `maxPages` for trial runs.
3. Upsert rows through a `MusicBrainzReferenceWorkSink`.
4. Store the `nextOffset` when `complete=false`.
5. Resume from `nextOffset` after transient failures.

Example:

```typescript
import { MusicBrainzIngestionAdapter } from '@themis/reference-db';

const adapter = new MusicBrainzIngestionAdapter({
  userAgent: 'OshunThemisReferenceDb/1.0 admin@example.com',
  contactEmail: 'admin@example.com',
});

const result = await adapter.importRecordings(
  { genres: ['pop'], fromReleaseDate: '1960-01-01' },
  referenceWorkSink,
  {
    pageSize: 100,
    maxPages: 10,
    onProgress: (progress) => logProgress('musicbrainz', progress),
  }
);
```

### AcoustID and Chromaprint

Use AcoustID ingestion to populate exact-ish audio fingerprint material tied to
MusicBrainz recordings. The default request interval is 334 ms and default
concurrency is 4.

Required source fields:

- `referenceWorkId`
- `musicBrainzRecordingId`
- Known `acoustIds`, or a source that can list track ids by MusicBrainz
  recording id.

Skip reasons to monitor:

- `already_indexed`
- `duplicate_batch`
- `missing_chromaprint`
- `no_acoustids`

If `missing_chromaprint` rises, the metadata catalog is ahead of fingerprint
coverage; do not treat those works as fully covered by audio fingerprinting.

### MIDI Melody Indexing

Use `MidiMelodyIngestionService` for Lakh MIDI Dataset or other licensed MIDI
sources. It selects the best melody track, extracts contours, and writes
Parsons, interval, and absolute contour drafts.

Recommended settings:

- `minNotes`: at least 4 for smoke tests; raise for production backfills if too
  many fragments are indexed.
- `ngramMinLength` and `ngramMaxLength`: align with the melodic analysis engine
  configuration.
- `concurrency`: start at 4 and reduce if CPU or memory pressure rises.

Track skip reasons:

- `already_indexed`
- `corrupt_file`
- `no_melody_track`
- `sink_skipped`

### Lyrics

Use `LyricIngestionService` with licensed providers or public-domain records.
Provider contracts expose `termsAccepted`; keep that true only when the current
catalog agreement permits indexing.

Provider inputs:

- Musixmatch: `apiKey`, `termsAccepted`.
- Genius: `accessToken`, `termsAccepted`, optional `lyricsResolver`.
- Public domain: local records with attribution and optional line timing.

Rights metadata must travel with each `LyricIndexDraft`. Monitor
`license_unavailable`, `not_public_domain`, and `no_lyrics` skip rates.

### Audio Feature Extraction

Use audio feature ingestion when the database has licensed catalog audio or
public-domain recordings. The extractor can write melody, harmony, rhythm, audio
fingerprint, spectral, neural audio, and optional voice-print indexes.

Required recording fields:

- `referenceWorkId`
- `sourceAudioId`
- `samples`
- `sampleRateHz`
- `access.license`: `licensed_catalog` or `public_domain`
- `access.termsAccepted=true`

Voice-print indexing requires `artistId` and a consent status. Default
concurrency is 2; the target throughput constant is 1,000 recordings per worker
hour. Start below that target on new hardware and scale after observing memory,
CPU, and index write latency.

### Public Domain Collections

Use public-domain ingestion for catalogs where the work is public domain in the
target jurisdictions. Preserve source attribution and jurisdiction evidence in
metadata. Set both `publicDomain=true` and `publicDomainJurisdictions`.

Recheck public-domain assumptions when adding a new jurisdiction to Shield
policy. A work public-domain in one territory may remain protected elsewhere.

### Real-Time Release Monitoring

Use `RealTimeReleaseMonitoringService` for high-profile releases from Spotify,
Billboard, TikTok, or manual feeds. It creates monitored release drafts and
queues feature extraction jobs with priorities:

- `critical`: chart-topping or very high trend velocity.
- `high`: chart entry or high platform popularity.
- `standard`: normal monitored release.

Default polling interval is 24 hours. Target indexing SLA is 24 hours for
chart-topping releases and 48 hours for standard releases.

---

## Deduplication

Run `ReferenceDeduplicationService.scanAndMerge()` after every ingestion batch
and before enabling new rows for release-gate scans.

Deduplication evidence:

- `isrc_exact`
- `audio_fingerprint`
- `metadata_fuzzy`

Defaults:

- Metadata threshold: `0.86`
- Audio threshold: `0.96`
- Concurrency: `2`

Merge policy:

1. Prefer the canonical row with the strongest identifier set and richest index
   coverage.
2. Preserve all ISRCs, ISWCs, release variants, labels, and source ids.
3. Preserve public-domain and jurisdiction evidence.
4. Add a protectability annotation patch describing the merge.
5. Keep duplicate ids traceable for audit and review packages.

Before large merges, run a dry candidate load in a staging clone and inspect
groups with confidence below `0.98`.

---

## Metadata Enrichment

Use `MetadataEnrichmentService.enrichDueWorks()` to refresh songwriter,
publisher, copyright holder, licensing, genre, label, and provenance data.

Defaults:

- Refresh interval: 30 days.
- Concurrency: 3.

Provider failures are recorded with `providerId`. A single provider failure
should not prevent other providers from contributing a patch. Alert when:

- `errors / processed > 0.05`
- A required rights provider fails for more than one scheduled run.
- A work has detection indexes but no rights owner or public-domain evidence.

---

## Protectability Annotations

Use `ProtectabilityAnnotationService.recomputeDueAnnotations()` to compute
melody, harmony, and lyric protectability scores. The model version is
`protectability-v1`.

Defaults:

- Recompute interval: 30 days.
- Concurrency: 3.

Annotations include:

- Per-element score and confidence.
- Match-weight multiplier.
- Rationale.
- Corpus metrics.
- Overall protectability score.
- Next recompute due date.

Run protectability recompute when:

- A new work or index is added.
- Deduplication merges works.
- Corpus statistics materially change.
- A reviewer or legal team overrides a Shield decision based on protectability.

Use `enqueueProtectabilityRecomputeJob()` for periodic, new-ingestion, or manual
recompute jobs.

---

## Health Monitoring

Monitor database health, ingestion health, and coverage health separately.

### Database Health

Run these checks at least daily:

```sql
select count(*) as reference_works from themis_reference_works;
select count(*) as melody_rows from themis_reference_melody_index;
select count(*) as harmonic_rows from themis_reference_harmonic_index;
select count(*) as lyric_rows from themis_reference_lyric_index;
select count(*) as audio_fingerprint_rows from themis_reference_audio_fingerprint_index;
select count(*) as rhythm_rows from themis_reference_rhythm_index;
select count(*) as voice_print_rows from themis_reference_voice_print_index;
```

Coverage query:

```sql
select
  count(*) as total_works,
  count(m.reference_work_id) filter (where m.reference_work_id is not null) as with_melody,
  count(h.reference_work_id) filter (where h.reference_work_id is not null) as with_harmony,
  count(l.reference_work_id) filter (where l.reference_work_id is not null) as with_lyrics,
  count(a.reference_work_id) filter (where a.reference_work_id is not null) as with_audio
from themis_reference_works w
left join themis_reference_melody_index m on m.reference_work_id = w.id
left join themis_reference_harmonic_index h on h.reference_work_id = w.id
left join themis_reference_lyric_index l on l.reference_work_id = w.id
left join themis_reference_audio_fingerprint_index a on a.reference_work_id = w.id;
```

Index health:

```sql
select
  schemaname,
  relname,
  indexrelname,
  idx_scan,
  idx_tup_read,
  idx_tup_fetch
from pg_stat_user_indexes
where relname like 'themis_reference_%'
order by relname, indexrelname;
```

### Ingestion Health

For each ingestion job, persist:

- Source and query parameters.
- Start/end timestamps.
- `processed`, `stored` or `imported`, `skipped`, and `errors`.
- Skip reasons.
- Provider error details.
- Resume cursor or `nextOffset`.

Alert when:

- Error rate exceeds 5 percent.
- `already_indexed` unexpectedly drops during incremental imports.
- `no_audio_access`, `license_unavailable`, or `not_public_domain` spikes.
- Release monitoring queue misses the 24-hour or 48-hour target SLA.
- Protectability annotations are older than 45 days.

### Coverage Health

Track these ratios:

- Works with at least one detection index.
- Works with audio fingerprint coverage.
- Works with melody and harmonic coverage.
- Works with lyric coverage where lyrics are licensed or public domain.
- Works with public-domain jurisdiction evidence.
- Voice-print rows by consent status.
- False-negative reports caused by missing reference rows or missing indexes.

---

## Backup and Recovery

### Backup Policy

Use both logical and physical backup coverage:

- Nightly logical dump of `themis_reference_*` schema and data.
- WAL archiving or managed point-in-time recovery for production.
- Weekly restore drill into a non-production cluster.
- Backup artifact retention aligned with legal/audit retention policy.

Example logical backup:

```bash
pg_dump "$THEMIS_REFERENCE_DATABASE_URL" \
  --format=custom \
  --no-owner \
  --table='themis_reference_*' \
  --file="backups/themis-reference-$(date +%Y%m%d).dump"
```

### Restore Drill

```bash
createdb themis_reference_restore
pg_restore \
  --dbname=themis_reference_restore \
  --clean \
  --if-exists \
  backups/themis-reference-YYYYMMDD.dump
```

After restore:

1. Confirm `vector` and `pgcrypto` extensions.
2. Run row-count comparison by table.
3. Run index health checks.
4. Run deduplication candidate load with a small limit.
5. Run a known plagiarism/non-infringement scan smoke test against restored
   references.

### Recovery Priorities

1. Restore `themis_reference_works`.
2. Restore exact identifiers and rights/protectability metadata.
3. Restore audio fingerprint and lyric indexes.
4. Restore melody, harmonic, rhythm, and voice-print indexes.
5. Resume release monitoring and enrichment jobs.
6. Recompute stale protectability annotations.

If vector indexes are corrupt or missing after restore, rebuild HNSW indexes
before routing release-gate traffic back to the cluster.

---

## Scaling Guide

Scale by workload:

- Metadata ingestion: increase MusicBrainz page batches within API rate limits.
- MIDI ingestion: scale CPU workers; keep per-worker memory bounded by file
  batch size.
- Lyric ingestion: scale by provider quota and rights restrictions, not only
  CPU.
- Audio feature extraction: scale workers by CPU/GPU availability and source
  audio bandwidth.
- Vector search: tune PostgreSQL memory, HNSW index build windows, and read
  replicas.

Recommended starting points:

| Workload                 | Starting concurrency | Primary limit                        |
| ------------------------ | -------------------- | ------------------------------------ |
| MusicBrainz ingestion    | 1 request / 1,000 ms | API policy                           |
| AcoustID ingestion       | 4                    | API policy and fingerprint download  |
| MIDI melody ingestion    | 4                    | CPU and file quality                 |
| Lyric ingestion          | 4                    | Provider quota and rights terms      |
| Audio feature extraction | 2                    | CPU/GPU, memory, and audio bandwidth |
| Metadata enrichment      | 3                    | Provider quota                       |
| Protectability recompute | 3                    | CPU and corpus stats load            |
| Deduplication            | 2                    | Pairwise candidate scoring           |

For large growth:

- Keep ingestion workers separate from query-serving replicas.
- Run HNSW index builds and reindex operations during low-traffic windows.
- Partition operational job logs by date.
- Use object storage for raw audio and MIDI; store only derived features in
  PostgreSQL.
- Keep public-domain and licensed-catalog pipelines separate so rights mistakes
  cannot cross-contaminate the release-gate pool.
- Add read replicas for reviewer lookup and analytics before increasing write
  throughput.

---

## Operational Checklists

### New Catalog Slice

- [ ] Rights terms reviewed and source approved.
- [ ] Metadata imported with durable identifiers.
- [ ] Audio, MIDI, lyric, or public-domain indexes ingested.
- [ ] Deduplication run and merge report reviewed.
- [ ] Protectability recompute complete.
- [ ] Coverage metrics meet target for the slice.
- [ ] Known-case scan smoke test passes.
- [ ] Backup completed after ingestion.

### Weekly Maintenance

- [ ] Review ingestion error and skip reason trends.
- [ ] Check stale protectability annotations.
- [ ] Check HNSW and GIN index usage.
- [ ] Run deduplication on recently updated works.
- [ ] Review release monitoring SLA misses.
- [ ] Confirm backup completion and restore drill schedule.

### Incident Response

- [ ] Freeze new release-gate eligibility for affected source.
- [ ] Identify affected reference work ids and source ids.
- [ ] Revoke or correct invalid rights metadata.
- [ ] Re-run deduplication and protectability recompute.
- [ ] Re-run impacted validation corpus scans.
- [ ] Record incident notes with restored or corrected reference ids.
