# Themis Universal Shield Reference Database Administration Guide

This guide is for operators who administer the all-media reference database used
by the Phase 75 Universal Originality Shield. The database is the authoritative
source of reference works, rightsholder metadata, protectability annotations,
consent evidence, provenance records, and media-specific detection indexes.

The Music Shield reference database guide remains the detailed runbook for
music-specific ingestion. This guide defines the universal operating model for
all media types.

---

## Operator Responsibilities

Reference database administrators are responsible for:

- Accepting or rejecting reference work ingestion requests.
- Verifying rights, license, public-domain, consent, and source terms before a
  record becomes release-gate eligible.
- Running media-specific ingestion pipelines and index builders.
- Preserving provenance and source catalog evidence for every reference work.
- Running deduplication, merge review, and stale-record cleanup.
- Keeping protectability and review annotations current.
- Monitoring index coverage, ingestion failures, freshness, latency, and storage
  growth.
- Supporting takedown, erasure, legal hold, and discovery exports.
- Planning scale-out for vector search, object storage, and high-volume
  backfills.

Do not make a reference work eligible for production scans until metadata,
rights status, deduplication status, and at least one relevant detection index
are complete.

---

## Canonical Reference Record

Every reference work should have one canonical record keyed by
`referenceWorkId`. The record must be media-neutral so compound works can link
multiple media components to the same rights object.

Required fields:

| Field                 | Purpose                                             |
| --------------------- | --------------------------------------------------- |
| `referenceWorkId`     | Stable internal id used by scans and reviews        |
| `mediaTypes`          | One or more Universal Shield media types            |
| `title`               | Human-readable title or catalog label               |
| `creatorNames`        | Creator, artist, author, studio, designer           |
| `rightsholderNames`   | Parties with clearance or notice interest           |
| `sourceCatalog`       | Provider, internal catalog, claim, or manual load   |
| `sourceRecordIds`     | External ids, URLs, ISRC/ISWC, ISBN, SKU, etc.      |
| `rightsStatus`        | licensed, public_domain, owned, claim, unknown      |
| `licenseScope`        | permitted use, territory, dates, restrictions       |
| `protectabilityNotes` | expression, functional limits, public-domain caveat |
| `provenance`          | source ingest time, operator, checksums, lineage    |
| `eligibilityStatus`   | quarantine, indexed, eligible, disabled, held       |
| `retentionClass`      | ordinary, consent-bound, legal-hold, delete-by      |

Optional fields should include consent ids, public-domain jurisdictions, content
credential manifests, version links, duplicate ids, takedown history, and
reviewer annotations.

---

## Eligibility State Machine

Reference records move through a strict lifecycle:

1. `quarantine`: metadata was received but cannot be used for release gates.
2. `metadata_verified`: identity, title, source, and creator fields are valid.
3. `rights_verified`: license, ownership, public-domain, or claim status is
   recorded.
4. `indexed`: one or more media-specific detection indexes are complete.
5. `deduped`: duplicate and merge candidates have been reviewed.
6. `eligible`: adapters may use the reference for production release gates.
7. `disabled`: record is retained but not used in scans.
8. `legal_hold`: deletion and mutation are suspended by counsel.
9. `erasure_pending`: deletion or anonymization is queued where legally
   available.

Only `eligible` records may influence automated release decisions. `quarantine`,
`disabled`, `legal_hold`, and `erasure_pending` records may be visible to
operators and counsel but must not silently affect publication gates.

---

## Ingestion Pipelines By Media Type

Each ingestion job must persist a job id, source, operator or service principal,
input checksum, processed count, stored count, skipped count, errors, and
eligibility transition.

| Media type    | Ingestion sources                               | Indexes to build                                      |
| ------------- | ----------------------------------------------- | ----------------------------------------------------- |
| `audio`       | licensed audio, public-domain recordings        | fingerprints, chroma, spectral, embeddings            |
| `music`       | MusicBrainz, ISRC/ISWC, MIDI, lyrics, stems     | melody, harmony, rhythm, lyric, audio, structure      |
| `voice`       | consented voice references, claims, removals    | voice embeddings, speaker metadata, consent scope     |
| `image`       | licensed images, claim evidence, owned catalogs | perceptual hash, visual embeddings, segmentation      |
| `text`        | books, scripts, lyrics, articles, claim packets | n-grams, embeddings, paraphrase maps, language        |
| `video`       | clips, film references, storyboards, claims     | keyframes, shot graph, audio/text sub-index links     |
| `model_3d`    | GLB, glTF, OBJ, FBX, marketplace claims         | topology, silhouette, material, UV, embedding indexes |
| `motion`      | mocap clips, animation packs, choreography      | skeleton, key pose, timing curve, loop signatures     |
| `character`   | character bibles, turnarounds, franchise claims | trait matrix, silhouette, costume, lore embeddings    |
| `fashion`     | patterns, tech packs, lookbooks, claims         | pattern diff, silhouette, textile print embeddings    |
| `environment` | level layouts, scene graphs, venue claims       | map topology, landmarks, path graph, scene embedding  |

### Audio, Music, And Voice

- Follow the Music Shield reference database guide for MusicBrainz, AcoustID,
  MIDI, lyric, audio feature, public-domain, and release-monitoring pipelines.
- Store recording rights separately from composition rights.
- Require explicit consent metadata before voice references become eligible for
  voice-likeness detection.
- Disable voice references immediately when consent is revoked unless legal hold
  requires preservation outside scan eligibility.

### Image

- Preserve original dimensions, color profile, crop history, source URL, and
  content credential manifest when available.
- Build perceptual hash and embedding indexes from normalized images.
- Store segmentation, subject placement, and composition summaries separately
  from raw media.
- Mark public-domain images by jurisdiction and distinguish work rights from
  photograph or scan rights.

### Text

- Preserve source edition, translation, language, chapter/scene/page offsets,
  and license scope.
- Build n-gram, phrase, embedding, and paraphrase indexes.
- Keep public-domain editions separate from modern translations and annotated
  editions.
- For scripts and lyrics, preserve line ordering because sequence similarity is
  part of the detection signal.

### Video

- Extract keyframes, shot boundaries, audio tracks, transcripts, captions, and
  scene metadata.
- Link video references to audio, music, text, character, and image sub-indexes
  when applicable.
- Store frame hashes and shot graph signatures with timecode ranges.
- Preserve rights for source clip, soundtrack, performers, and embedded works
  separately.

### 3D Models

- Store source format, unit scale, vertex count, mesh names, material names,
  texture hashes, rig presence, and export tool.
- Build topology, silhouette, material, UV, and normalized embedding indexes.
- Keep marketplace license, purchased asset id, and derivative-work status in
  metadata.
- Review functional or mechanical parts for protectability before enabling
  release-gate use.

### Motion

- Normalize skeletons to a canonical rig before indexing.
- Store frame rate, duration, joint hierarchy, key poses, timing curves, and
  loop signatures.
- Preserve performer, mocap pack, choreography, and consent/license metadata.
- Reindex after retargeting pipeline changes because timing and pose features
  can drift.

### Character

- Ingest turnarounds, model sheets, costume sheets, character bibles, name
  variants, catchphrases, props, and lore references.
- Build silhouette, trait, costume, color-blocking, prop, and lore indexes.
- Keep parody, license, internal IP, and franchise claim status separate.
- Link character references to image, 3D, fashion, motion, and text components.

### Fashion

- Store garment category, season, collection, pattern files, textile art,
  hardware, ornament, and lookbook context.
- Build garment silhouette, pattern, textile print, and tech-pack indexes.
- Distinguish functional garment shapes from protectable pattern art or trade
  dress annotations.
- Track brand, designer, collection, and license metadata.

### Environment

- Ingest map layouts, scene graphs, prop placement, landmark lists, navigation
  paths, lighting plans, and venue references.
- Build topology, path graph, landmark, scene embedding, and prop-cluster
  indexes.
- Store source game, venue, architecture, asset pack, and license metadata.
- Review real-world venue or branded environment rights before eligibility.

---

## Deduplication And Merge Policy

Run deduplication after every ingestion batch and before an index becomes
eligible.

Deduplication evidence by media:

- exact identifiers: ISRC, ISWC, ISBN, catalog id, marketplace id, claim id;
- perceptual hashes: audio, image, video keyframes, texture images;
- vector similarity: audio, visual, text, 3D, motion, character, scene;
- metadata fuzzy match: title, creator, rightsholder, release date, collection;
- component overlap: same source work appearing in several media indexes.

Merge policy:

1. Prefer the record with verified rights, richer source identifiers, and
   complete provenance.
2. Preserve every external id and duplicate id as aliases.
3. Preserve stricter rights, consent, and retention limits.
4. Do not merge public-domain and licensed edition records unless counsel has
   confirmed the rights object is the same.
5. Record merge rationale, operator, timestamp, and confidence.
6. Recompute indexes and protectability annotations after merge.

Low-confidence merge candidates should remain separate but linked through a
duplicate-candidate relationship for reviewer visibility.

---

## Maintenance Procedures

### Daily

- Review failed ingestion jobs and high-volume skip reasons.
- Check queue age for reference indexing and release-monitoring feeds.
- Verify scan adapters can read current eligible index versions.
- Confirm no `quarantine` records are accidentally marked `eligible`.

### Weekly

- Run deduplication on new and changed records.
- Review missing rights, public-domain, and consent metadata.
- Recompute stale protectability annotations for high-traffic catalogs.
- Sample review false positives and false negatives for database gaps.
- Check object storage integrity and media hash drift.

### Monthly

- Refresh external catalog metadata.
- Rotate stale source credentials.
- Run restore drills in a staging clone.
- Review vector index bloat, slow queries, and partition growth.
- Audit disabled, legal-hold, and erasure-pending records.
- Update coverage dashboards by media type and domain.

### Before Large Backfills

1. Confirm source terms allow the intended indexing use.
2. Create a staging ingest and validate sample scan behavior.
3. Set batch size, concurrency, and retry limits.
4. Reserve object storage and vector index capacity.
5. Take a database backup.
6. Run ingestion into `quarantine`.
7. Validate metadata, indexes, deduplication, and rights.
8. Promote to `eligible` only after smoke scans pass.

---

## Monitoring

Monitor database health, ingestion health, coverage, quality, and compliance
separately.

| Metric                                | Alert threshold                       |
| ------------------------------------- | ------------------------------------- |
| ingestion job failure rate            | over 5 percent in a rolling hour      |
| skipped for missing rights            | over 10 percent of a production batch |
| eligible records without index        | any record                            |
| indexed records without rights status | any record                            |
| p95 reference lookup latency          | above release-gate budget             |
| vector index recall smoke test        | below baseline                        |
| false negative database gap rate      | above policy threshold                |
| consent-revoked eligible references   | any voice/likeness record             |
| stale metadata age                    | above media-specific refresh interval |
| legal-hold mutation attempts          | any attempt                           |
| storage growth                        | above forecast by 20 percent          |

Dashboards should break down metrics by media type, source catalog, ingestion
worker, index version, and eligibility status.

---

## Scaling

Use these defaults before optimizing:

- Partition large tables by media type and index version.
- Keep ingestion workers separate from query-serving replicas.
- Use object storage for raw media and store only hashes, pointers, and derived
  features in the database.
- Use read replicas for reviewer lookup and analytics dashboards.
- Build new vector indexes side by side, then atomically switch adapter read
  versions.
- Run backfills at lower priority than publication-gate scans.
- Cap worker concurrency per media type instead of using one global queue.

Scaling guidance by workload:

| Workload           | Primary bottleneck                        | Scaling action                                      |
| ------------------ | ----------------------------------------- | --------------------------------------------------- |
| audio/music ingest | provider rate limits, CPU feature extract | provider-aware queues, CPU workers, batch writes    |
| image ingest       | embedding GPU, object storage bandwidth   | GPU worker pool, thumbnail cache, staged writes     |
| text ingest        | tokenization and embedding throughput     | shard by language and corpus, cache normalized text |
| video ingest       | decode/keyframe extraction                | transcode queue, frame sampling, derived artifacts  |
| 3D ingest          | mesh normalization and topology analysis  | CPU workers, file size caps, staged mesh summaries  |
| motion ingest      | rig normalization                         | canonical rig cache, per-rig worker pools           |
| character/fashion  | multi-component indexing                  | component fan-out with merge barrier                |
| environment ingest | graph extraction                          | scene graph preprocessor, path-analysis workers     |

Do not raise concurrency during memory pressure or while broad verification jobs
are running on the same host.

---

## Backup, Restore, And Legal Hold

Back up:

- canonical reference tables;
- media-specific indexes and index-version metadata;
- object storage manifests and checksums;
- consent, license, public-domain, and rightsholder evidence;
- deduplication and merge history;
- ingestion job results;
- legal hold, takedown, and erasure records.

Restore drill:

1. Restore the database to a staging cluster.
2. Restore object storage manifests and verify checksums.
3. Rebuild or attach vector indexes.
4. Run reference lookup smoke tests for every media type.
5. Run known infringement and known original validation scans.
6. Confirm legal-hold records are still immutable.
7. Confirm revoked-consent references are not eligible.

Legal hold:

- Freeze deletion and mutation of the held reference records and scan evidence.
- Disable scan eligibility if counsel requests removal from publication gates.
- Record who placed the hold, the matter id, scope, and release criteria.
- Keep ordinary erasure workflows from deleting held evidence until counsel
  releases the hold.

---

## Takedown, Erasure, And Disablement

Use disablement when a record should no longer influence scans but must remain
available for audit. Use erasure or anonymization only when legal and privacy
review confirms it is allowed.

Required steps:

1. Change `eligibilityStatus` to `disabled` or `erasure_pending`.
2. Remove the record from adapter-serving indexes.
3. Preserve audit records, unless erasure legally requires deletion and no legal
   hold applies.
4. Rebuild affected index shards.
5. Run smoke scans confirming the disabled reference no longer matches.
6. Record the reason: claim, license expired, consent revoked, duplicate,
   public-domain correction, privacy request, or legal instruction.

Voice, likeness, and biometric records must be disabled immediately on consent
revocation unless a legal hold overrides deletion while still removing
production scan eligibility.

---

## Release-Gate Eligibility Checklist

Before marking a reference eligible:

- [ ] Canonical metadata is complete.
- [ ] Source terms permit the indexing use.
- [ ] Rights, license, public-domain, or claim status is recorded.
- [ ] Consent metadata is present for voice, likeness, biometric, or persona
      references.
- [ ] At least one detection index relevant to the media type is complete.
- [ ] Deduplication has run and merge candidates are resolved or linked.
- [ ] Protectability notes are present for high-risk or ambiguous material.
- [ ] Provenance includes source, checksum, operator/service, and ingest time.
- [ ] The record is not disabled, erasure-pending, or under scan-exclusion legal
      hold.
- [ ] Smoke scans pass for representative positive and original controls.

---

## Incident Runbook

Use this runbook when a database gap causes a false negative, false positive, or
claim escalation:

1. Freeze the affected reference work and scan response.
2. Identify whether the root cause is missing reference, stale metadata,
   disabled index, bad merge, consent mismatch, or adapter bug.
3. Disable unsafe records or indexes if they can affect publication decisions.
4. Backfill or correct the reference data in `quarantine`.
5. Run deduplication, protectability recompute, and smoke scans.
6. Re-run the affected Universal Shield scan.
7. Update reviewer and compliance reports.
8. Record lessons learned and add a validation case if the failure can recur.
