This guide is for operators who administer the all-media reference database used by the Phase 75 Universal Originality Shield. The database is the authoritative source of reference works, rightsholder metadata, protectability annotations, consent evidence, provenance records, and media-specific detection indexes.
The Music Shield reference database guide remains the detailed runbook for music-specific ingestion. This guide defines the universal operating model for all media types.
Operator Responsibilities#
Reference database administrators are responsible for:
- Accepting or rejecting reference work ingestion requests.
- Verifying rights, license, public-domain, consent, and source terms before a record becomes release-gate eligible.
- Running media-specific ingestion pipelines and index builders.
- Preserving provenance and source catalog evidence for every reference work.
- Running deduplication, merge review, and stale-record cleanup.
- Keeping protectability and review annotations current.
- Monitoring index coverage, ingestion failures, freshness, latency, and storage growth.
- Supporting takedown, erasure, legal hold, and discovery exports.
- Planning scale-out for vector search, object storage, and high-volume backfills.
Do not make a reference work eligible for production scans until metadata, rights status, deduplication status, and at least one relevant detection index are complete.
Canonical Reference Record#
Every reference work should have one canonical record keyed by
referenceWorkId. The record must be media-neutral so compound works can link
multiple media components to the same rights object.
Required fields:
| Field | Purpose |
|---|---|
referenceWorkId |
Stable internal id used by scans and reviews |
mediaTypes |
One or more Universal Shield media types |
title |
Human-readable title or catalog label |
creatorNames |
Creator, artist, author, studio, designer |
rightsholderNames |
Parties with clearance or notice interest |
sourceCatalog |
Provider, internal catalog, claim, or manual load |
sourceRecordIds |
External ids, URLs, ISRC/ISWC, ISBN, SKU, etc. |
rightsStatus |
licensed, public_domain, owned, claim, unknown |
licenseScope |
permitted use, territory, dates, restrictions |
protectabilityNotes |
expression, functional limits, public-domain caveat |
provenance |
source ingest time, operator, checksums, lineage |
eligibilityStatus |
quarantine, indexed, eligible, disabled, held |
retentionClass |
ordinary, consent-bound, legal-hold, delete-by |
Optional fields should include consent ids, public-domain jurisdictions, content credential manifests, version links, duplicate ids, takedown history, and reviewer annotations.
Eligibility State Machine#
Reference records move through a strict lifecycle:
quarantine: metadata was received but cannot be used for release gates.metadata_verified: identity, title, source, and creator fields are valid.rights_verified: license, ownership, public-domain, or claim status is recorded.indexed: one or more media-specific detection indexes are complete.deduped: duplicate and merge candidates have been reviewed.eligible: adapters may use the reference for production release gates.disabled: record is retained but not used in scans.legal_hold: deletion and mutation are suspended by counsel.erasure_pending: deletion or anonymization is queued where legally available.
Only eligible records may influence automated release decisions. quarantine,
disabled, legal_hold, and erasure_pending records may be visible to
operators and counsel but must not silently affect publication gates.
Ingestion Pipelines By Media Type#
Each ingestion job must persist a job id, source, operator or service principal, input checksum, processed count, stored count, skipped count, errors, and eligibility transition.
| Media type | Ingestion sources | Indexes to build |
|---|---|---|
audio |
licensed audio, public-domain recordings | fingerprints, chroma, spectral, embeddings |
music |
MusicBrainz, ISRC/ISWC, MIDI, lyrics, stems | melody, harmony, rhythm, lyric, audio, structure |
voice |
consented voice references, claims, removals | voice embeddings, speaker metadata, consent scope |
image |
licensed images, claim evidence, owned catalogs | perceptual hash, visual embeddings, segmentation |
text |
books, scripts, lyrics, articles, claim packets | n-grams, embeddings, paraphrase maps, language |
video |
clips, film references, storyboards, claims | keyframes, shot graph, audio/text sub-index links |
model_3d |
GLB, glTF, OBJ, FBX, marketplace claims | topology, silhouette, material, UV, embedding indexes |
motion |
mocap clips, animation packs, choreography | skeleton, key pose, timing curve, loop signatures |
character |
character bibles, turnarounds, franchise claims | trait matrix, silhouette, costume, lore embeddings |
fashion |
patterns, tech packs, lookbooks, claims | pattern diff, silhouette, textile print embeddings |
environment |
level layouts, scene graphs, venue claims | map topology, landmarks, path graph, scene embedding |
Audio, Music, And Voice#
- Follow the Music Shield reference database guide for MusicBrainz, AcoustID, MIDI, lyric, audio feature, public-domain, and release-monitoring pipelines.
- Store recording rights separately from composition rights.
- Require explicit consent metadata before voice references become eligible for voice-likeness detection.
- Disable voice references immediately when consent is revoked unless legal hold requires preservation outside scan eligibility.
Image#
- Preserve original dimensions, color profile, crop history, source URL, and content credential manifest when available.
- Build perceptual hash and embedding indexes from normalized images.
- Store segmentation, subject placement, and composition summaries separately from raw media.
- Mark public-domain images by jurisdiction and distinguish work rights from photograph or scan rights.
Text#
- Preserve source edition, translation, language, chapter/scene/page offsets, and license scope.
- Build n-gram, phrase, embedding, and paraphrase indexes.
- Keep public-domain editions separate from modern translations and annotated editions.
- For scripts and lyrics, preserve line ordering because sequence similarity is part of the detection signal.
Video#
- Extract keyframes, shot boundaries, audio tracks, transcripts, captions, and scene metadata.
- Link video references to audio, music, text, character, and image sub-indexes when applicable.
- Store frame hashes and shot graph signatures with timecode ranges.
- Preserve rights for source clip, soundtrack, performers, and embedded works separately.
3D Models#
- Store source format, unit scale, vertex count, mesh names, material names, texture hashes, rig presence, and export tool.
- Build topology, silhouette, material, UV, and normalized embedding indexes.
- Keep marketplace license, purchased asset id, and derivative-work status in metadata.
- Review functional or mechanical parts for protectability before enabling release-gate use.
Motion#
- Normalize skeletons to a canonical rig before indexing.
- Store frame rate, duration, joint hierarchy, key poses, timing curves, and loop signatures.
- Preserve performer, mocap pack, choreography, and consent/license metadata.
- Reindex after retargeting pipeline changes because timing and pose features can drift.
Character#
- Ingest turnarounds, model sheets, costume sheets, character bibles, name variants, catchphrases, props, and lore references.
- Build silhouette, trait, costume, color-blocking, prop, and lore indexes.
- Keep parody, license, internal IP, and franchise claim status separate.
- Link character references to image, 3D, fashion, motion, and text components.
Fashion#
- Store garment category, season, collection, pattern files, textile art, hardware, ornament, and lookbook context.
- Build garment silhouette, pattern, textile print, and tech-pack indexes.
- Distinguish functional garment shapes from protectable pattern art or trade dress annotations.
- Track brand, designer, collection, and license metadata.
Environment#
- Ingest map layouts, scene graphs, prop placement, landmark lists, navigation paths, lighting plans, and venue references.
- Build topology, path graph, landmark, scene embedding, and prop-cluster indexes.
- Store source game, venue, architecture, asset pack, and license metadata.
- Review real-world venue or branded environment rights before eligibility.
Deduplication And Merge Policy#
Run deduplication after every ingestion batch and before an index becomes eligible.
Deduplication evidence by media:
- exact identifiers: ISRC, ISWC, ISBN, catalog id, marketplace id, claim id;
- perceptual hashes: audio, image, video keyframes, texture images;
- vector similarity: audio, visual, text, 3D, motion, character, scene;
- metadata fuzzy match: title, creator, rightsholder, release date, collection;
- component overlap: same source work appearing in several media indexes.
Merge policy:
- Prefer the record with verified rights, richer source identifiers, and complete provenance.
- Preserve every external id and duplicate id as aliases.
- Preserve stricter rights, consent, and retention limits.
- Do not merge public-domain and licensed edition records unless counsel has confirmed the rights object is the same.
- Record merge rationale, operator, timestamp, and confidence.
- Recompute indexes and protectability annotations after merge.
Low-confidence merge candidates should remain separate but linked through a duplicate-candidate relationship for reviewer visibility.
Maintenance Procedures#
Daily#
- Review failed ingestion jobs and high-volume skip reasons.
- Check queue age for reference indexing and release-monitoring feeds.
- Verify scan adapters can read current eligible index versions.
- Confirm no
quarantinerecords are accidentally markedeligible.
Weekly#
- Run deduplication on new and changed records.
- Review missing rights, public-domain, and consent metadata.
- Recompute stale protectability annotations for high-traffic catalogs.
- Sample review false positives and false negatives for database gaps.
- Check object storage integrity and media hash drift.
Monthly#
- Refresh external catalog metadata.
- Rotate stale source credentials.
- Run restore drills in a staging clone.
- Review vector index bloat, slow queries, and partition growth.
- Audit disabled, legal-hold, and erasure-pending records.
- Update coverage dashboards by media type and domain.
Before Large Backfills#
- Confirm source terms allow the intended indexing use.
- Create a staging ingest and validate sample scan behavior.
- Set batch size, concurrency, and retry limits.
- Reserve object storage and vector index capacity.
- Take a database backup.
- Run ingestion into
quarantine. - Validate metadata, indexes, deduplication, and rights.
- Promote to
eligibleonly after smoke scans pass.
Monitoring#
Monitor database health, ingestion health, coverage, quality, and compliance separately.
| Metric | Alert threshold |
|---|---|
| ingestion job failure rate | over 5 percent in a rolling hour |
| skipped for missing rights | over 10 percent of a production batch |
| eligible records without index | any record |
| indexed records without rights status | any record |
| p95 reference lookup latency | above release-gate budget |
| vector index recall smoke test | below baseline |
| false negative database gap rate | above policy threshold |
| consent-revoked eligible references | any voice/likeness record |
| stale metadata age | above media-specific refresh interval |
| legal-hold mutation attempts | any attempt |
| storage growth | above forecast by 20 percent |
Dashboards should break down metrics by media type, source catalog, ingestion worker, index version, and eligibility status.
Scaling#
Use these defaults before optimizing:
- Partition large tables by media type and index version.
- Keep ingestion workers separate from query-serving replicas.
- Use object storage for raw media and store only hashes, pointers, and derived features in the database.
- Use read replicas for reviewer lookup and analytics dashboards.
- Build new vector indexes side by side, then atomically switch adapter read versions.
- Run backfills at lower priority than publication-gate scans.
- Cap worker concurrency per media type instead of using one global queue.
Scaling guidance by workload:
| Workload | Primary bottleneck | Scaling action |
|---|---|---|
| audio/music ingest | provider rate limits, CPU feature extract | provider-aware queues, CPU workers, batch writes |
| image ingest | embedding GPU, object storage bandwidth | GPU worker pool, thumbnail cache, staged writes |
| text ingest | tokenization and embedding throughput | shard by language and corpus, cache normalized text |
| video ingest | decode/keyframe extraction | transcode queue, frame sampling, derived artifacts |
| 3D ingest | mesh normalization and topology analysis | CPU workers, file size caps, staged mesh summaries |
| motion ingest | rig normalization | canonical rig cache, per-rig worker pools |
| character/fashion | multi-component indexing | component fan-out with merge barrier |
| environment ingest | graph extraction | scene graph preprocessor, path-analysis workers |
Do not raise concurrency during memory pressure or while broad verification jobs are running on the same host.
Backup, Restore, And Legal Hold#
Back up:
- canonical reference tables;
- media-specific indexes and index-version metadata;
- object storage manifests and checksums;
- consent, license, public-domain, and rightsholder evidence;
- deduplication and merge history;
- ingestion job results;
- legal hold, takedown, and erasure records.
Restore drill:
- Restore the database to a staging cluster.
- Restore object storage manifests and verify checksums.
- Rebuild or attach vector indexes.
- Run reference lookup smoke tests for every media type.
- Run known infringement and known original validation scans.
- Confirm legal-hold records are still immutable.
- Confirm revoked-consent references are not eligible.
Legal hold:
- Freeze deletion and mutation of the held reference records and scan evidence.
- Disable scan eligibility if counsel requests removal from publication gates.
- Record who placed the hold, the matter id, scope, and release criteria.
- Keep ordinary erasure workflows from deleting held evidence until counsel releases the hold.
Takedown, Erasure, And Disablement#
Use disablement when a record should no longer influence scans but must remain available for audit. Use erasure or anonymization only when legal and privacy review confirms it is allowed.
Required steps:
- Change
eligibilityStatustodisabledorerasure_pending. - Remove the record from adapter-serving indexes.
- Preserve audit records, unless erasure legally requires deletion and no legal hold applies.
- Rebuild affected index shards.
- Run smoke scans confirming the disabled reference no longer matches.
- Record the reason: claim, license expired, consent revoked, duplicate, public-domain correction, privacy request, or legal instruction.
Voice, likeness, and biometric records must be disabled immediately on consent revocation unless a legal hold overrides deletion while still removing production scan eligibility.
Release-Gate Eligibility Checklist#
Before marking a reference eligible:
- Canonical metadata is complete.
- Source terms permit the indexing use.
- Rights, license, public-domain, or claim status is recorded.
- Consent metadata is present for voice, likeness, biometric, or persona references.
- At least one detection index relevant to the media type is complete.
- Deduplication has run and merge candidates are resolved or linked.
- Protectability notes are present for high-risk or ambiguous material.
- Provenance includes source, checksum, operator/service, and ingest time.
- The record is not disabled, erasure-pending, or under scan-exclusion legal hold.
- Smoke scans pass for representative positive and original controls.
Incident Runbook#
Use this runbook when a database gap causes a false negative, false positive, or claim escalation:
- Freeze the affected reference work and scan response.
- Identify whether the root cause is missing reference, stale metadata, disabled index, bad merge, consent mismatch, or adapter bug.
- Disable unsafe records or indexes if they can affect publication decisions.
- Backfill or correct the reference data in
quarantine. - Run deduplication, protectability recompute, and smoke scans.
- Re-run the affected Universal Shield scan.
- Update reviewer and compliance reports.
- Record lessons learned and add a validation case if the failure can recur.