Eve library

Themis Music Shield Compose Stack

Other

Revision 1

On this page

This directory contains the local and production-like Docker Compose topology for Music Shield:

  • Shield API server on :8080
  • BullMQ scan workers
  • PostgreSQL with pgvector for reference metadata and embedding tables
  • Redis for scan queue, cache, and perceptual-hash state
  • Qdrant for vector similarity search

Development#

bash
cp infra/themis/music-shield/.env.example infra/themis/music-shield/.env
docker compose --env-file infra/themis/music-shield/.env \
  -f infra/themis/music-shield/docker-compose.yml \
  --profile dev up -d --build

Health check:

bash
curl http://localhost:8080/health

Prometheus metrics:

bash
curl http://localhost:8080/metrics

Authenticated endpoints use x-api-key; the development default is themis-dev-key.

Monitoring#

Run Prometheus and Grafana alongside either runtime profile:

bash
docker compose --env-file infra/themis/music-shield/.env \
  -f infra/themis/music-shield/docker-compose.yml \
  --profile dev --profile monitoring up -d --build

Prometheus is available on http://localhost:9090 by default and scrapes /metrics from the Shield API. Grafana is available on http://localhost:3000; the provisioned dashboard is Themis / Themis Music Shield.

Monitoring assets live under monitoring/:

  • prometheus.yml scrapes the dev and production-like API service names.
  • alert-rules.yml defines alerts for p95 scan latency above 60 seconds, engine health failures, newly detected false negatives, and queue depth above 1000.
  • grafana/dashboards/themis-music-shield.json tracks scan volume, latency percentiles, risk distribution, queue depth, cache hit rate, engine health, and FP/FN rates.

Production-Like#

bash
docker compose --env-file infra/themis/music-shield/.env \
  -f infra/themis/music-shield/docker-compose.yml \
  --profile prod up -d --build

Before using the prod profile, set strong values for:

  • THEMIS_SHIELD_API_KEYS
  • THEMIS_REFERENCE_DB_PASSWORD
  • image tag and registry variables
  • port bindings if the host already runs PostgreSQL, Redis, or Qdrant

Runtime Notes#

The Shield API package is library-first, so the Dockerfile builds the Nx @themis/shield-api target and runs explicit runtime entrypoints:

  • API: libs/themis/shield-api/dist/main.js
  • worker: libs/themis/shield-api/dist/worker.js

Full scans are queued through BullMQ when THEMIS_SHIELD_ASYNC_SCANS=true. POST /scan/quick runs in-process for low-latency smoke testing. The reference database initializes from libs/themis/reference-db/drizzle/0000_themis_reference_db_initial.sql.

Kubernetes#

Kustomize manifests live in infra/themis/music-shield/kubernetes:

  • base/ contains production-oriented API, worker, PostgreSQL/pgvector, Redis, Qdrant, HPA, PVC, ingress, secret-template, network-policy, ServiceMonitor, and PrometheusRule resources.
  • overlays/development/ lowers replica counts and uses the local image tag.
  • overlays/production/ keeps production replica defaults and expands persistent storage requests.

Apply only after replacing base/secret-template.yaml with sealed, external, or cluster-managed secrets:

bash
kubectl apply -k infra/themis/music-shield/kubernetes/overlays/production

The worker HPA expects an external metrics adapter that exposes bullmq_queue_depth with a queue=themis.shield.scan label. The API ServiceMonitor expects Prometheus Operator CRDs in the cluster and scrapes /metrics on the http service port.

Reference Database Seeding#

After PostgreSQL is initialized, run the idempotent reference seed pipeline:

bash
THEMIS_REFERENCE_DATABASE_URL=postgresql://themis_shield:themis_shield_dev@localhost:55432/themis_shield \
  pnpm --filter @themis/reference-db run seed -- \
  --musicbrainz-user-agent=OshunThemisSeeder/1.0 \
  --lakh-midi-json=/data/themis/lakh-midi \
  --public-domain-json=/data/themis/public-domain

The seed command reports JSON progress lines, creates support tables for bundled common phrases, protected drum breaks, and copyright cases, and safely updates existing rows on re-run. The initial production bootstrap is expected to take 4-8 hours depending on MusicBrainz rate limits and local MIDI/audio throughput.