S3 / MinIO-compatible storage for the ML models loaded by @oshun/ml
(V1-P2-0217). Every model is content-addressed by SHA-256 — clients fetch the
bytes, compute the hash, and refuse to load when the hash does not match the
manifest entry.
Layout#
infra/models/
README.md — this file
manifest.json — model index (id, sha256, size, license, source, …)
schema.json — JSON-Schema for `manifest.json`
The bucket itself stores models/<id>.onnx. The url field on each manifest
entry resolves against registry.endpoint. The CDN at registry.publicCdn
mirrors the bucket for unauthenticated reads.
Verification flow#
import { loadModelFromRegistry } from '@oshun/ml/registry';
const session = await loadModelFromRegistry('movenet-lightning-int8', {
manifestUrl: 'https://models.oshun.ai/v1/manifest.json',
});
Internally:
- fetch manifest.json
- find
id === 'movenet-lightning-int8' - fetch
registry.endpoint + entry.url - compute SHA-256 of the bytes
- fail loudly if it does not match
entry.sha256 - pass bytes +
expectedSha256: entry.sha256to@oshun/ml#loadModel - cache the result by
(manifestUrl, id, sha256)incaches.open('oshun-models-v1')(browser) oros.tmpdir()(Node)
Bucket layout#
oshun-models/
manifest.json
models/
movenet-lightning-int8.onnx
movenet-thunder-fp16.onnx
yolov8n-fp16.onnx
whisper-base-en-int8.onnx
silero-vad-v5.onnx
ecapa-tdnn-fp16.onnx
bge-reranker-base-int8.onnx
A nightly job re-runs the export pipeline against the upstream sources and refreshes the SHA-256 fields whenever an upstream model is patched. The manifest is the source of truth — clients never hard-code hashes.
Local development#
The dev docker-compose.dev.yml includes a MinIO instance the model registry
can target with minioCompat: true:
docker compose -f docker/docker-compose.dev.yml up -d minio
mc alias set local http://127.0.0.1:9000 minio minio12345
mc cp -r models/ local/oshun-models/models/
mc cp manifest.json local/oshun-models/manifest.json
Then point the loader at the local endpoint:
const session = await loadModelFromRegistry('movenet-lightning-int8', {
manifestUrl: 'http://localhost:9000/oshun-models/manifest.json',
});
Hash discipline#
sha256 in manifest.json is the canonical truth. Treat it as immutable per
(id, version): a re-quantisation that produces different bytes must be a new
id (or, if the consumer can adopt silently, a bumped manifest version and a
synchronised client bump). Never edit a hash in place — that breaks the
integrity contract for any client that cached the old artefact.
License audit#
Every entry includes license. The build refuses to publish a manifest where
any entry's license is not in the SPDX-approved list. At V1 the allow-list is:
apache-2.0, mit, bsd-3-clause, agpl-3.0 (compile-time gated; downstream
consumers must understand the AGPL implications of YOLOv8).