Owner library: libs/isis/gaussian-splatting Estimate: ~1–3 weeks
(depending on Niantic spec ambiguities surfaced) Blocks:
E (output format),
I (gallery — splat assets in
gallery need a portable format). Dependencies: none directly; can run in
parallel with block A. Status: not started — picks up the deferred task #3
from the original adoption work.
Overview#
Implement a bit-exact Niantic .spz codec (encoder + decoder) in
libs/isis/gaussian-splatting/src/exporters/. Validate against the Niantic
reference loader by round-tripping a corpus of reference splats.
The current exporter (isis-splat-exporter.ts) writes a proprietary .isplat
format for Unreal/Unity engines and stays as a separate surface. Block D adds
.spz alongside, not in place of.
Why this is needed#
- Niantic's
.spzis the de facto interop format for Gaussian splats. Web viewers (gsplat.js,mkkellogg/GaussianSplats3D), engine plugins (UnrealLumaAIplugin, Unitygsplat-unity), and SDK loaders all consume it. - The Marble provider already returns
.spzas opaque bytes. To produce composed splats (block E) we need to encode.spzfrom in-memory Gaussians, not just decode. - Block I (gallery) needs splats in a format that web viewers can render inline;
.spzis the choice.
Files to create#
libs/isis/gaussian-splatting/src/exporters/spz-exporter.ts— encoder.libs/isis/gaussian-splatting/src/exporters/spz-exporter.test.tslibs/isis/gaussian-splatting/src/exporters/spz-decoder.ts— decoder (needed for round-trip tests and for ingesting Marble's.spzoutput as in-memory Gaussians during the bake step).libs/isis/gaussian-splatting/src/exporters/spz-decoder.test.tslibs/isis/gaussian-splatting/src/exporters/spz-binary-layout.ts— the binary layout constants and quantisation helpers, shared by encoder and decoder.libs/isis/gaussian-splatting/src/exporters/spz-binary-layout.test.ts— exhaustive quantisation round-trip tests for each field.libs/isis/gaussian-splatting/test-fixtures/spz-reference/— directory containing 5–10 reference.spzfiles downloaded from public sources (Niantic samples, viewer demos), plus per-file expected metadata (gaussian count, sh degree, bounds) for assertion targets.
Files to modify#
libs/isis/gaussian-splatting/src/exporters/index.ts— exportSpzExporter,SpzDecoder, types.libs/isis/gaussian-splatting/package.json— add apakodependency for gzip (already available in the workspace pnpm catalog as a transitive, but needs an explicit declaration here).
Niantic .spz spec summary#
Public spec: https://github.com/nianticlabs/spz. The format is gzipped. Once
decompressed, the layout is:
| Offset (uncompressed) | Field | Size | Notes |
|---|---|---|---|
| 0 | magic | u32 LE | 0x5053474e ("NGSP" reversed; Niantic Gaussian Splat) |
| 4 | version | u32 LE | currently 2 |
| 8 | numPoints | u32 LE | gaussian count |
| 12 | shDegree | u8 | 0..3 |
| 13 | fractionalBits | u8 | for fixed-point positions; default 12 |
| 14 | flags | u8 | bit 0: antialiased |
| 15 | reserved | u8 | always 0 |
After the 16-byte header, sections follow in this order:
- Positions:
numPoints * 3 * 24-bit signed fixed-point. Decoded asvalue = signed24 / (1 << fractionalBits)per axis. - Alphas:
numPoints * u8. Decoded asalpha = sigmoid((u8 / 255 - 0.5) * SCALE)whereSCALE = 8.0per the Niantic reference. - Colors:
numPoints * 3 * u8. SH0-derived 8-bit RGB. - Scales:
numPoints * 3 * u8. Decoded asscale = exp((u8 - 64) / 16.0)per axis (log-scale quantisation). - Rotations:
numPoints * 3 * u8. Three components of a unit quaternion (w reconstructed). Decoded asxyz = (u8 - 127.5) / 127.5; w = sqrt(max(0, 1 - x*x - y*y - z*z)). - SH coefficients:
numPoints * shCoefficientsPerGaussian * u8whereshCoefficientsPerGaussian = (degree+1)^2 * 3 - 3(skipping SH0 which was already written as colors). Decoded ascoeff = (u8 - 128) * (1.0 / 128.0).
The whole concatenated buffer is then gzipped.
Note. Confirm every quantisation formula and offset against the Niantic reference loader before declaring done. The exact constants (
SCALE = 8.0for alpha,(u8 - 64) / 16.0for scale,(u8 - 127.5) / 127.5for rotation) are sourced from public references but must be verified on a known-good.spzround trip.
Public API surface#
// spz-exporter.ts
export interface SpzExportOptions {
readonly fractionalBits?: number; // default 12
readonly antialiased?: boolean; // default false
readonly shDegree?: 0 | 1 | 2 | 3; // inferred from input if omitted
}
export class SpzExporter {
encode(scene: LoadedScene, options?: SpzExportOptions): Promise<Buffer>;
}
export function createSpzExporter(): SpzExporter;
// spz-decoder.ts
export interface SpzDecodeResult {
readonly scene: LoadedScene;
readonly metadata: {
readonly version: number;
readonly fractionalBits: number;
readonly antialiased: boolean;
readonly shDegree: number;
};
}
export class SpzDecoder {
decode(buffer: Buffer): Promise<SpzDecodeResult>;
}
export function createSpzDecoder(): SpzDecoder;
LoadedScene is the existing in-memory representation from
splat-renderer.ts:114-117 (positions, scales, rotations, opacities, colors,
optional SH).
Algorithms#
Encode#
encode(scene, options):
1. validate scene shape (positions length === 3*N, scales === 3*N, etc.)
2. write 16-byte header:
magic = 0x5053474e
version = 2
numPoints = scene.gaussianCount
shDegree = options.shDegree ?? inferShDegree(scene)
fractionalBits = options.fractionalBits ?? 12
flags = options.antialiased ? 0x01 : 0x00
reserved = 0
3. write positions section (signed 24-bit fixed-point per axis)
for each i in 0..N:
x_q = round(scene.positions[3i + 0] * (1 << fractionalBits))
check fits in [-2^23, 2^23 - 1]; throw on overflow
writeInt24LE(buffer, offset, x_q); offset += 3
(repeat for y, z)
4. write alphas (u8 per point) — invert sigmoid + scale to u8
5. write colors (u8 RGB per point)
6. write scales (u8 per axis) — log-scale quantise
7. write rotations (u8 xyz per point) — drop w, sign-bit from rotation[3]
8. write SH coefficients (u8 per coefficient)
9. gzip the entire buffer (level 6, default)
10. return gzipped buffer
Decode#
decode(gzippedBuffer):
1. ungzip
2. read header (16 bytes); validate magic
3. allocate Float32Arrays for positions/scales/rotations/colors/opacities/sh
4. read positions section, dequantise signed-24 → float
5. read alphas, dequantise u8 → sigmoid space
6. read colors, dequantise u8 → [0, 1]
7. read scales, dequantise u8 → exp(.)
8. read rotations, dequantise u8 → unit quaternion (reconstruct w)
9. read SH if shDegree > 0
10. return { scene: LoadedScene, metadata }
inferShDegree#
inferShDegree(scene):
if !scene.sphericalHarmonics: return 0
shPerGaussian = scene.sphericalHarmonics.length / scene.gaussianCount
// shPerGaussian = (deg+1)^2 * 3 - 3 → solve for deg
// deg = sqrt((shPerGaussian + 3) / 3) - 1
return ROUND_AND_VALIDATE_INT(...) ?? throw new Error('SH length invalid')
Test plan#
Unit (spz-binary-layout.test.ts):
- Quantisation/dequantisation round-trips for each field type (position fixed-point at fractionalBits ∈ {8, 10, 12, 14}, scale log-quantisation, rotation unit-quaternion sign-bit reconstruction). Each assert known expected u8/u24 outputs against synthetic input values.
- Position overflow at fractionalBits=12 throws (input value 2^11 + ε).
- Rotation w-reconstruction handles edge case
xyzexactly on unit sphere (w = 0).
Unit (spz-exporter.test.ts):
- Encode 100-Gaussian synthetic scene → round-trip via
SpzDecoder→ every field within quantisation tolerance. - Encode the same scene at
fractionalBits ∈ {10, 12, 14}produces different file sizes (lower = smaller, expected ratios asserted). - Encode with
shDegree: 0produces a file with no SH section. - Encode with
antialiased: truesets bit 0 of flags byte (verified by ungzipping and reading byte 14).
Reference corpus (spz-decoder.test.ts):
- For each file in
test-fixtures/spz-reference/:- Decode → assert
metadata.version === 2,metadata.fractionalBits === <expected>,scene.gaussianCount === <expected>(per-file sidecar JSON of expected values). - Re-encode → bit-exact match against the original (gzipped) bytes. Asserts our encoder produces output indistinguishable from Niantic's.
- Decode → assert
- This is the definition of done for the codec: byte-for-byte match.
Adversarial:
- Decoder rejects bad magic (mid-byte corruption).
- Decoder rejects inconsistent header (numPoints * fields > buffer length).
- Encoder rejects scene with mismatched array lengths.
- Encoder rejects scene with SH length not matching
shDegreeformula.
Acceptance criteria#
npx vitest run src/exporters/clean.npx tsc --noEmitclean fromlibs/isis/gaussian-splatting.- Stub-indicator scan clean.
- Bit-exact encode round-trip for every reference fixture in
test-fixtures/spz-reference/. - A composer-produced
.spz(synthetic scene) loads in themkkellogg/GaussianSplats3Dweb viewer without errors. Manual verification step; recorded in the PR description with a screenshot. - The decoder correctly ingests a Marble-produced
.spz(real one downloaded from a Marble run) and reproduces the same Gaussian count and bounds Marble reported via metadata.
Open questions#
- Reference corpus sourcing. Need 5–10 known-good
.spzfiles. Sources: Niantic's published demos, World Labs Marble outputs, public splat galleries. License audit needed for each — the test fixtures may not redistribute. Alternative: synthesise reference splats with the encoder, then assert decoder handles them correctly. Less rigorous but copyright-safe. - Quantisation constants verification. The exact alpha/scale/rotation decode formulas are sourced from public references but not from the Niantic source directly (license precludes lifting code). v1 assumes the constants are correct; if visual QA shows artifacts, re-check against the Niantic header/source comments before changing math.
pakovs Nodezlib. Both work;pakois browser-friendly if we ever ship a worker version. Recommended:pako(already in workspace catalog).- Streaming encode/decode. For large splats (>50M Gaussians) the buffer-based API allocates a lot. v1 ships buffer-based; streaming is a v2 nice-to-have.