Eve library

Themis Universal Shield Reference Database Administration Guide

Guide

Revision 1

On this page

This guide is for operators who administer the all-media reference database used by the Phase 75 Universal Originality Shield. The database is the authoritative source of reference works, rightsholder metadata, protectability annotations, consent evidence, provenance records, and media-specific detection indexes.

The Music Shield reference database guide remains the detailed runbook for music-specific ingestion. This guide defines the universal operating model for all media types.


Operator Responsibilities#

Reference database administrators are responsible for:

  • Accepting or rejecting reference work ingestion requests.
  • Verifying rights, license, public-domain, consent, and source terms before a record becomes release-gate eligible.
  • Running media-specific ingestion pipelines and index builders.
  • Preserving provenance and source catalog evidence for every reference work.
  • Running deduplication, merge review, and stale-record cleanup.
  • Keeping protectability and review annotations current.
  • Monitoring index coverage, ingestion failures, freshness, latency, and storage growth.
  • Supporting takedown, erasure, legal hold, and discovery exports.
  • Planning scale-out for vector search, object storage, and high-volume backfills.

Do not make a reference work eligible for production scans until metadata, rights status, deduplication status, and at least one relevant detection index are complete.


Canonical Reference Record#

Every reference work should have one canonical record keyed by referenceWorkId. The record must be media-neutral so compound works can link multiple media components to the same rights object.

Required fields:

Field Purpose
referenceWorkId Stable internal id used by scans and reviews
mediaTypes One or more Universal Shield media types
title Human-readable title or catalog label
creatorNames Creator, artist, author, studio, designer
rightsholderNames Parties with clearance or notice interest
sourceCatalog Provider, internal catalog, claim, or manual load
sourceRecordIds External ids, URLs, ISRC/ISWC, ISBN, SKU, etc.
rightsStatus licensed, public_domain, owned, claim, unknown
licenseScope permitted use, territory, dates, restrictions
protectabilityNotes expression, functional limits, public-domain caveat
provenance source ingest time, operator, checksums, lineage
eligibilityStatus quarantine, indexed, eligible, disabled, held
retentionClass ordinary, consent-bound, legal-hold, delete-by

Optional fields should include consent ids, public-domain jurisdictions, content credential manifests, version links, duplicate ids, takedown history, and reviewer annotations.


Eligibility State Machine#

Reference records move through a strict lifecycle:

  1. quarantine: metadata was received but cannot be used for release gates.
  2. metadata_verified: identity, title, source, and creator fields are valid.
  3. rights_verified: license, ownership, public-domain, or claim status is recorded.
  4. indexed: one or more media-specific detection indexes are complete.
  5. deduped: duplicate and merge candidates have been reviewed.
  6. eligible: adapters may use the reference for production release gates.
  7. disabled: record is retained but not used in scans.
  8. legal_hold: deletion and mutation are suspended by counsel.
  9. erasure_pending: deletion or anonymization is queued where legally available.

Only eligible records may influence automated release decisions. quarantine, disabled, legal_hold, and erasure_pending records may be visible to operators and counsel but must not silently affect publication gates.


Ingestion Pipelines By Media Type#

Each ingestion job must persist a job id, source, operator or service principal, input checksum, processed count, stored count, skipped count, errors, and eligibility transition.

Media type Ingestion sources Indexes to build
audio licensed audio, public-domain recordings fingerprints, chroma, spectral, embeddings
music MusicBrainz, ISRC/ISWC, MIDI, lyrics, stems melody, harmony, rhythm, lyric, audio, structure
voice consented voice references, claims, removals voice embeddings, speaker metadata, consent scope
image licensed images, claim evidence, owned catalogs perceptual hash, visual embeddings, segmentation
text books, scripts, lyrics, articles, claim packets n-grams, embeddings, paraphrase maps, language
video clips, film references, storyboards, claims keyframes, shot graph, audio/text sub-index links
model_3d GLB, glTF, OBJ, FBX, marketplace claims topology, silhouette, material, UV, embedding indexes
motion mocap clips, animation packs, choreography skeleton, key pose, timing curve, loop signatures
character character bibles, turnarounds, franchise claims trait matrix, silhouette, costume, lore embeddings
fashion patterns, tech packs, lookbooks, claims pattern diff, silhouette, textile print embeddings
environment level layouts, scene graphs, venue claims map topology, landmarks, path graph, scene embedding

Audio, Music, And Voice#

  • Follow the Music Shield reference database guide for MusicBrainz, AcoustID, MIDI, lyric, audio feature, public-domain, and release-monitoring pipelines.
  • Store recording rights separately from composition rights.
  • Require explicit consent metadata before voice references become eligible for voice-likeness detection.
  • Disable voice references immediately when consent is revoked unless legal hold requires preservation outside scan eligibility.

Image#

  • Preserve original dimensions, color profile, crop history, source URL, and content credential manifest when available.
  • Build perceptual hash and embedding indexes from normalized images.
  • Store segmentation, subject placement, and composition summaries separately from raw media.
  • Mark public-domain images by jurisdiction and distinguish work rights from photograph or scan rights.

Text#

  • Preserve source edition, translation, language, chapter/scene/page offsets, and license scope.
  • Build n-gram, phrase, embedding, and paraphrase indexes.
  • Keep public-domain editions separate from modern translations and annotated editions.
  • For scripts and lyrics, preserve line ordering because sequence similarity is part of the detection signal.

Video#

  • Extract keyframes, shot boundaries, audio tracks, transcripts, captions, and scene metadata.
  • Link video references to audio, music, text, character, and image sub-indexes when applicable.
  • Store frame hashes and shot graph signatures with timecode ranges.
  • Preserve rights for source clip, soundtrack, performers, and embedded works separately.

3D Models#

  • Store source format, unit scale, vertex count, mesh names, material names, texture hashes, rig presence, and export tool.
  • Build topology, silhouette, material, UV, and normalized embedding indexes.
  • Keep marketplace license, purchased asset id, and derivative-work status in metadata.
  • Review functional or mechanical parts for protectability before enabling release-gate use.

Motion#

  • Normalize skeletons to a canonical rig before indexing.
  • Store frame rate, duration, joint hierarchy, key poses, timing curves, and loop signatures.
  • Preserve performer, mocap pack, choreography, and consent/license metadata.
  • Reindex after retargeting pipeline changes because timing and pose features can drift.

Character#

  • Ingest turnarounds, model sheets, costume sheets, character bibles, name variants, catchphrases, props, and lore references.
  • Build silhouette, trait, costume, color-blocking, prop, and lore indexes.
  • Keep parody, license, internal IP, and franchise claim status separate.
  • Link character references to image, 3D, fashion, motion, and text components.

Fashion#

  • Store garment category, season, collection, pattern files, textile art, hardware, ornament, and lookbook context.
  • Build garment silhouette, pattern, textile print, and tech-pack indexes.
  • Distinguish functional garment shapes from protectable pattern art or trade dress annotations.
  • Track brand, designer, collection, and license metadata.

Environment#

  • Ingest map layouts, scene graphs, prop placement, landmark lists, navigation paths, lighting plans, and venue references.
  • Build topology, path graph, landmark, scene embedding, and prop-cluster indexes.
  • Store source game, venue, architecture, asset pack, and license metadata.
  • Review real-world venue or branded environment rights before eligibility.

Deduplication And Merge Policy#

Run deduplication after every ingestion batch and before an index becomes eligible.

Deduplication evidence by media:

  • exact identifiers: ISRC, ISWC, ISBN, catalog id, marketplace id, claim id;
  • perceptual hashes: audio, image, video keyframes, texture images;
  • vector similarity: audio, visual, text, 3D, motion, character, scene;
  • metadata fuzzy match: title, creator, rightsholder, release date, collection;
  • component overlap: same source work appearing in several media indexes.

Merge policy:

  1. Prefer the record with verified rights, richer source identifiers, and complete provenance.
  2. Preserve every external id and duplicate id as aliases.
  3. Preserve stricter rights, consent, and retention limits.
  4. Do not merge public-domain and licensed edition records unless counsel has confirmed the rights object is the same.
  5. Record merge rationale, operator, timestamp, and confidence.
  6. Recompute indexes and protectability annotations after merge.

Low-confidence merge candidates should remain separate but linked through a duplicate-candidate relationship for reviewer visibility.


Maintenance Procedures#

Daily#

  • Review failed ingestion jobs and high-volume skip reasons.
  • Check queue age for reference indexing and release-monitoring feeds.
  • Verify scan adapters can read current eligible index versions.
  • Confirm no quarantine records are accidentally marked eligible.

Weekly#

  • Run deduplication on new and changed records.
  • Review missing rights, public-domain, and consent metadata.
  • Recompute stale protectability annotations for high-traffic catalogs.
  • Sample review false positives and false negatives for database gaps.
  • Check object storage integrity and media hash drift.

Monthly#

  • Refresh external catalog metadata.
  • Rotate stale source credentials.
  • Run restore drills in a staging clone.
  • Review vector index bloat, slow queries, and partition growth.
  • Audit disabled, legal-hold, and erasure-pending records.
  • Update coverage dashboards by media type and domain.

Before Large Backfills#

  1. Confirm source terms allow the intended indexing use.
  2. Create a staging ingest and validate sample scan behavior.
  3. Set batch size, concurrency, and retry limits.
  4. Reserve object storage and vector index capacity.
  5. Take a database backup.
  6. Run ingestion into quarantine.
  7. Validate metadata, indexes, deduplication, and rights.
  8. Promote to eligible only after smoke scans pass.

Monitoring#

Monitor database health, ingestion health, coverage, quality, and compliance separately.

Metric Alert threshold
ingestion job failure rate over 5 percent in a rolling hour
skipped for missing rights over 10 percent of a production batch
eligible records without index any record
indexed records without rights status any record
p95 reference lookup latency above release-gate budget
vector index recall smoke test below baseline
false negative database gap rate above policy threshold
consent-revoked eligible references any voice/likeness record
stale metadata age above media-specific refresh interval
legal-hold mutation attempts any attempt
storage growth above forecast by 20 percent

Dashboards should break down metrics by media type, source catalog, ingestion worker, index version, and eligibility status.


Scaling#

Use these defaults before optimizing:

  • Partition large tables by media type and index version.
  • Keep ingestion workers separate from query-serving replicas.
  • Use object storage for raw media and store only hashes, pointers, and derived features in the database.
  • Use read replicas for reviewer lookup and analytics dashboards.
  • Build new vector indexes side by side, then atomically switch adapter read versions.
  • Run backfills at lower priority than publication-gate scans.
  • Cap worker concurrency per media type instead of using one global queue.

Scaling guidance by workload:

Workload Primary bottleneck Scaling action
audio/music ingest provider rate limits, CPU feature extract provider-aware queues, CPU workers, batch writes
image ingest embedding GPU, object storage bandwidth GPU worker pool, thumbnail cache, staged writes
text ingest tokenization and embedding throughput shard by language and corpus, cache normalized text
video ingest decode/keyframe extraction transcode queue, frame sampling, derived artifacts
3D ingest mesh normalization and topology analysis CPU workers, file size caps, staged mesh summaries
motion ingest rig normalization canonical rig cache, per-rig worker pools
character/fashion multi-component indexing component fan-out with merge barrier
environment ingest graph extraction scene graph preprocessor, path-analysis workers

Do not raise concurrency during memory pressure or while broad verification jobs are running on the same host.


Back up:

  • canonical reference tables;
  • media-specific indexes and index-version metadata;
  • object storage manifests and checksums;
  • consent, license, public-domain, and rightsholder evidence;
  • deduplication and merge history;
  • ingestion job results;
  • legal hold, takedown, and erasure records.

Restore drill:

  1. Restore the database to a staging cluster.
  2. Restore object storage manifests and verify checksums.
  3. Rebuild or attach vector indexes.
  4. Run reference lookup smoke tests for every media type.
  5. Run known infringement and known original validation scans.
  6. Confirm legal-hold records are still immutable.
  7. Confirm revoked-consent references are not eligible.

Legal hold:

  • Freeze deletion and mutation of the held reference records and scan evidence.
  • Disable scan eligibility if counsel requests removal from publication gates.
  • Record who placed the hold, the matter id, scope, and release criteria.
  • Keep ordinary erasure workflows from deleting held evidence until counsel releases the hold.

Takedown, Erasure, And Disablement#

Use disablement when a record should no longer influence scans but must remain available for audit. Use erasure or anonymization only when legal and privacy review confirms it is allowed.

Required steps:

  1. Change eligibilityStatus to disabled or erasure_pending.
  2. Remove the record from adapter-serving indexes.
  3. Preserve audit records, unless erasure legally requires deletion and no legal hold applies.
  4. Rebuild affected index shards.
  5. Run smoke scans confirming the disabled reference no longer matches.
  6. Record the reason: claim, license expired, consent revoked, duplicate, public-domain correction, privacy request, or legal instruction.

Voice, likeness, and biometric records must be disabled immediately on consent revocation unless a legal hold overrides deletion while still removing production scan eligibility.


Release-Gate Eligibility Checklist#

Before marking a reference eligible:

  • Canonical metadata is complete.
  • Source terms permit the indexing use.
  • Rights, license, public-domain, or claim status is recorded.
  • Consent metadata is present for voice, likeness, biometric, or persona references.
  • At least one detection index relevant to the media type is complete.
  • Deduplication has run and merge candidates are resolved or linked.
  • Protectability notes are present for high-risk or ambiguous material.
  • Provenance includes source, checksum, operator/service, and ingest time.
  • The record is not disabled, erasure-pending, or under scan-exclusion legal hold.
  • Smoke scans pass for representative positive and original controls.

Incident Runbook#

Use this runbook when a database gap causes a false negative, false positive, or claim escalation:

  1. Freeze the affected reference work and scan response.
  2. Identify whether the root cause is missing reference, stale metadata, disabled index, bad merge, consent mismatch, or adapter bug.
  3. Disable unsafe records or indexes if they can affect publication decisions.
  4. Backfill or correct the reference data in quarantine.
  5. Run deduplication, protectability recompute, and smoke scans.
  6. Re-run the affected Universal Shield scan.
  7. Update reviewer and compliance reports.
  8. Record lessons learned and add a validation case if the failure can recur.