Eve library

Oshun Model Registry

Other

Revision 1

On this page

S3 / MinIO-compatible storage for the ML models loaded by @oshun/ml (V1-P2-0217). Every model is content-addressed by SHA-256 — clients fetch the bytes, compute the hash, and refuse to load when the hash does not match the manifest entry.

Layout#

text
infra/models/
  README.md       — this file
  manifest.json   — model index (id, sha256, size, license, source, …)
  schema.json     — JSON-Schema for `manifest.json`

The bucket itself stores models/<id>.onnx. The url field on each manifest entry resolves against registry.endpoint. The CDN at registry.publicCdn mirrors the bucket for unauthenticated reads.

Verification flow#

ts
import { loadModelFromRegistry } from '@oshun/ml/registry';

const session = await loadModelFromRegistry('movenet-lightning-int8', {
  manifestUrl: 'https://models.oshun.ai/v1/manifest.json',
});

Internally:

  1. fetch manifest.json
  2. find id === 'movenet-lightning-int8'
  3. fetch registry.endpoint + entry.url
  4. compute SHA-256 of the bytes
  5. fail loudly if it does not match entry.sha256
  6. pass bytes + expectedSha256: entry.sha256 to @oshun/ml#loadModel
  7. cache the result by (manifestUrl, id, sha256) in caches.open('oshun-models-v1') (browser) or os.tmpdir() (Node)

Bucket layout#

text
oshun-models/
  manifest.json
  models/
    movenet-lightning-int8.onnx
    movenet-thunder-fp16.onnx
    yolov8n-fp16.onnx
    whisper-base-en-int8.onnx
    silero-vad-v5.onnx
    ecapa-tdnn-fp16.onnx
    bge-reranker-base-int8.onnx

A nightly job re-runs the export pipeline against the upstream sources and refreshes the SHA-256 fields whenever an upstream model is patched. The manifest is the source of truth — clients never hard-code hashes.

Local development#

The dev docker-compose.dev.yml includes a MinIO instance the model registry can target with minioCompat: true:

bash
docker compose -f docker/docker-compose.dev.yml up -d minio
mc alias set local http://127.0.0.1:9000 minio minio12345
mc cp -r models/ local/oshun-models/models/
mc cp manifest.json local/oshun-models/manifest.json

Then point the loader at the local endpoint:

ts
const session = await loadModelFromRegistry('movenet-lightning-int8', {
  manifestUrl: 'http://localhost:9000/oshun-models/manifest.json',
});

Hash discipline#

sha256 in manifest.json is the canonical truth. Treat it as immutable per (id, version): a re-quantisation that produces different bytes must be a new id (or, if the consumer can adopt silently, a bumped manifest version and a synchronised client bump). Never edit a hash in place — that breaks the integrity contract for any client that cached the old artefact.

License audit#

Every entry includes license. The build refuses to publish a manifest where any entry's license is not in the SPDX-approved list. At V1 the allow-list is: apache-2.0, mit, bsd-3-clause, agpl-3.0 (compile-time gated; downstream consumers must understand the AGPL implications of YOLOv8).