Owner library: libs/isis/gaussian-splatting Estimate: ~3–4 weeks
Blocks: F (lighting needs baked Gaussians).
Dependencies: block D (.spz codec), block B (composer surface that calls
into the bake), G for production GPU
dispatch (block E ships an in-process baseline first). Status: not started
Overview#
Convert each foreground mesh into Gaussians and merge them into the Marble splat
to produce a single composed .spz artefact. Two paths shipped:
- Surface-sampling baseline — Poisson-disc sample the mesh surface, one Gaussian per sample. Fast, deterministic, ~30s per mesh on CPU.
- Hybrid (sampling + differentiable refinement) — surface-sampling warm start, then 100–500 steps of differentiable splat fitting against PBR mesh renders. Higher fidelity, ~2–5min per mesh on GPU.
Plus a splat-merging algorithm: concat Gaussian buffers, depth-sort, occlusion-cull, dedupe, recompute bounds.
Why this is needed#
The composer's "transform-only" output (block B) leaves a bag of files for the
consumer to assemble at runtime — every viewer/engine needs custom glue. Block E
produces a single .spz that loads correctly in any conformant splat viewer
with no custom code.
This is also the prerequisite for lighting consistency (block F) — to apply environment lighting to the meshes, we need the meshes as PBR-shaded samples first.
Files to create#
Bake module structure#
libs/isis/gaussian-splatting/src/bake/— new subdirectory.libs/isis/gaussian-splatting/src/bake/index.ts— public re-exports.libs/isis/gaussian-splatting/src/bake/types.ts—BakeRequest,BakeResult,BakeMethod,BakedGaussians.
Surface sampling#
libs/isis/gaussian-splatting/src/bake/surface-sampling/poisson-sampler.ts— Poisson-disc sampling on a triangle mesh (uniform, area-weighted).libs/isis/gaussian-splatting/src/bake/surface-sampling/poisson-sampler.test.tslibs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.ts— given a sample point + local triangle + texture sample, fit a single Gaussian (position, oriented disk-shape from surface normal, color).libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.test.tslibs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.ts— the orchestrator: parse GLB → sample → fit Gaussians → returnBakedGaussians.libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.test.ts
Differentiable refinement (hybrid path)#
libs/isis/gaussian-splatting/src/bake/differentiable/render-from-views.ts— given mesh + camera poses, render PBR images using a CPU/GPU rasteriser. v1 uses@gltf-transform/functionsfor headless render; v1.5 useswgpuvia the existinglibs/isis/3d-inference-localRust binding.libs/isis/gaussian-splatting/src/bake/differentiable/render-from-views.test.tslibs/isis/gaussian-splatting/src/bake/differentiable/refine-gaussians.ts— given initial Gaussians + target renders + cameras, run K steps of gradient descent on Gaussian parameters minimising L2 + SSIM loss.libs/isis/gaussian-splatting/src/bake/differentiable/refine-gaussians.test.tslibs/isis/gaussian-splatting/src/bake/differentiable/hybrid-baker.ts— the orchestrator: surface-sample for warm start, then refine.libs/isis/gaussian-splatting/src/bake/differentiable/hybrid-baker.test.ts
Splat merging#
libs/isis/gaussian-splatting/src/bake/merge/concat-splats.ts— naive Float32Array concatenation across all input scenes.libs/isis/gaussian-splatting/src/bake/merge/depth-sort.ts— re-order Gaussians by depth from a canonical view (front-camera) for streaming- friendly load order.libs/isis/gaussian-splatting/src/bake/merge/occlusion-cull.ts— prune Gaussians from the baked mesh that are entirely behind opaque environment Gaussians. Voxel-grid based.libs/isis/gaussian-splatting/src/bake/merge/dedupe.ts— collapse Gaussians within ε spatial distance with similar color/scale.libs/isis/gaussian-splatting/src/bake/merge/merge-splats.ts— the orchestrator: concat → sort → cull → dedupe → recompute bounds → return.libs/isis/gaussian-splatting/src/bake/merge/merge-splats.test.ts- One
*.test.tsper merge submodule.
Composer integration#
libs/isis/scene-from-image-composer/src/phases/bake-phase.ts(new in composer) — replaces the transform-onlyalign-phaseoutput with a baked composed splat. Calls into@isis/gaussian-splatting/bake.libs/isis/scene-from-image-composer/src/phases/bake-phase.test.ts
Files to modify#
libs/isis/gaussian-splatting/src/index.ts— re-export the bake module.libs/isis/scene-from-image-composer/src/composer.ts:- Insert
bake-phaseafteralign-phaseand beforeaudio-phase. - Conditional on
composeOptions.outputMode === 'baked'(default).composeOptions.outputMode === 'transform-only'keeps block B's behavior for callers that need the unbaked artefacts.
- Insert
libs/isis/scene-from-image-composer/src/types.ts:- Add
outputMode: 'baked' | 'transform-only'toSceneFromImageRequest. - Add
bakeOptions?: BakeOptionstoSceneFromImageRequest.
- Add
Public API surface#
// bake/types.ts
export type BakeMethod = 'surface-sampling' | 'hybrid';
export interface BakeRequest {
readonly mesh: Buffer; // GLB bytes
readonly transform: Float32Array; // 16-element column-major 4×4
readonly method: BakeMethod;
readonly targetGaussianCount?: number; // default 5_000 per mesh
readonly refinementSteps?: number; // default 200 (hybrid only)
readonly cameraCount?: number; // default 16 (hybrid only)
}
export interface BakedGaussians {
readonly positions: Float32Array; // 3 * N
readonly scales: Float32Array; // 3 * N
readonly rotations: Float32Array; // 4 * N (quaternion x,y,z,w)
readonly opacities: Float32Array; // N
readonly colors: Float32Array; // 3 * N (RGB 0..1)
readonly sphericalHarmonics?: Float32Array; // optional
readonly count: number;
}
export interface BakeResult {
readonly baked: BakedGaussians;
readonly metrics: { readonly durationMs: number; readonly steps?: number };
}
// bake/index.ts
export async function bakeMesh(request: BakeRequest): Promise<BakeResult>;
// merge/merge-splats.ts
export interface MergeOptions {
readonly enableDepthSort?: boolean; // default true
readonly enableOcclusionCull?: boolean; // default true
readonly enableDedupe?: boolean; // default true
readonly dedupeEpsilon?: number; // default 0.001 (1mm)
readonly canonicalView?: {
readonly position: [number, number, number];
readonly target: [number, number, number];
};
}
export function mergeSplats(
scenes: readonly LoadedScene[],
options?: MergeOptions
): LoadedScene;
Algorithms#
Surface-sampling baker#
sample(mesh, count):
triangles = parseGLB(mesh).triangles
totalArea = sum(triangle.area for triangle in triangles)
// CDF over triangles weighted by area
cdf = buildCDF(triangles, weight: t -> t.area)
samples = []
for i in 0..count:
triangle = sampleFromCDF(cdf, rand())
barycentric = uniformBarycentricSample(rand2())
point = barycentric.eval(triangle)
normal = triangle.normal()
color = sampleTexture(triangle.uv.eval(barycentric), mesh.diffuseTexture)
samples.append({ point, normal, color, triangle })
return samples
fitGaussianFromSample(sample, neighborhood):
position = sample.point
// Scale derived from local surface area density
scale = [
sqrt(neighborhood.areaPerSample), // tangent X
sqrt(neighborhood.areaPerSample), // tangent Y
0.0001 // very thin in normal direction (oriented disk)
]
// Rotation aligns Z-axis with surface normal
rotation = quaternionFromZAxisAlignment(sample.normal)
opacity = 1.0
color = sample.color
return { position, scale, rotation, opacity, color }
Hybrid baker#
hybridBake(mesh, transform, request):
baked = surfaceSamplingBake(mesh, transform, request.targetGaussianCount)
cameras = generateCamerasOnSphere(
request.cameraCount, focusPoint = transform.translation, radius = 2 * meshDiameter
)
targetRenders = renderMeshFromViews(mesh, transform, cameras)
for step in 0..request.refinementSteps:
rendered = differentiableSplatRender(baked, cameras)
loss = sum(L2(rendered[i], targetRenders[i]) + 0.5 * SSIM_loss(rendered[i], targetRenders[i]) for i)
grads = backprop(loss, baked)
baked = applyAdam(baked, grads, lr = 0.005)
return baked
For block E v1, the differentiable renderer is CPU (slow, but deterministic and testable). Block G migrates the heavy work to GPU workers via job-envelope; block E keeps the CPU path as a reference implementation for tests.
Splat merging — occlusion cull#
occlusionCull(scenes):
// Build voxel grid over union bounds.
voxelGrid = new VoxelGrid(unionBounds, voxelSize = 0.05) // 5cm cells
// Mark voxels containing high-opacity environment Gaussians.
for gaussian in environmentScene:
if gaussian.opacity > 0.5:
voxelGrid.markOccluded(gaussian.position)
// Drop baked-mesh Gaussians whose voxel cell is occluded.
filtered = []
for gaussian in bakedMeshScene:
if !voxelGrid.isOccluded(gaussian.position):
filtered.append(gaussian)
return filtered
Splat merging — dedupe#
dedupe(gaussians, epsilon):
// Spatial hash over epsilon-grid.
hash = new SpatialHash(cellSize = epsilon)
out = []
for g in gaussians:
cell = hash.cell(g.position)
if !cell.has(approximately g):
cell.add(g)
out.append(g)
return out
Test plan#
Unit (poisson-sampler.test.ts):
- Sample 1000 points on a unit cube → all points lie within bounds.
- Sample on a unit cube → samples are area-weighted (each face gets ~166 points within ±10%).
- Same input + seed → identical sample list (determinism).
Unit (sample-to-gaussian.test.ts):
- Sample with known normal
[0, 1, 0]produces rotation that maps Z-axis to[0, 1, 0](quaternion verified to ε). - Sample on a 1m² triangle with area-per-sample = 0.001 produces tangent scales
≈
sqrt(0.001) ≈ 0.0316.
Unit (sampling-baker.test.ts):
- Bake a unit cube GLB at identity transform with
targetGaussianCount: 1000→ output has ≥ 950 Gaussians (some small triangles may get rounded down), ≤ 1050. All positions inside[-0.51, 0.51]^3(cube + epsilon).
Unit (render-from-views.test.ts):
- Render unit cube from canonical front view → output image's centre pixel is the cube's diffuse colour.
Unit (refine-gaussians.test.ts):
- Synthetic test: start with 100 Gaussians whose positions are perturbed from a
target arrangement; refinement reduces position MSE by
50% in 100 steps.
Unit (hybrid-baker.test.ts):
- Bake a textured unit cube via hybrid path with 50 steps → compare against surface-sampling-only output → hybrid has lower L2 error against ground-truth render of the cube. ε absolute, asserted.
Unit (merge-splats.test.ts):
- Concat-only: two scenes of 100 Gaussians → merged scene has 200.
- Depth sort: 5 Gaussians at depths
[3, 1, 4, 1, 5]from canonical view → output order corresponds to depths[1, 1, 3, 4, 5]. - Occlusion cull: 100 baked Gaussians fully inside a 200-Gaussian high-opacity environment box → all 100 baked Gaussians culled.
- Dedupe: 50 Gaussians at the same position → output has 1.
- Bounds recompute: union of two scenes with bounds
[-1,1]^3and[0,2]^3→ output bounds[-1,2]^3.
Integration (bake-phase.test.ts in composer):
- Composer with 1 mesh, surface-sampling method, transform-only env (skip Marble
call) → produces a baked
LoadedScenewithcount > 0and positions correctly transformed into world space. - Same with hybrid method (10 refinement steps for test speed) → produces a
baked scene; metrics record
steps: 10. - Composer with
outputMode: 'transform-only'→ bake phase skipped; manifest unchanged from block B behavior.
Adversarial:
- Bake degenerate mesh (all triangles zero area) → throws
BakeError('mesh has no surface area'). - Merge with empty input array → throws.
- Merge two scenes with mismatched SH degrees → throws (or upcasts the smaller — ADR-style decision; recommend throw).
Acceptance criteria#
- All new tests pass.
npx tsc --noEmitclean from bothlibs/isis/gaussian-splattingandlibs/isis/scene-from-image-composer.- Stub-indicator scan clean.
- A composer end-to-end run with 2 placed meshes (cube + sphere) on a synthetic
env splat (10k Gaussians) produces a baked
.spzthat loads in themkkellogg/GaussianSplats3Dweb viewer, with the meshes visible in the expected positions. Manual visual QA recorded in PR. - Hybrid path with 100 refinement steps produces visibly sharper edges than surface-sampling alone (visual QA).
- Performance: surface-sampling baseline bakes a 50k-triangle GLB in < 60s on CPU. Hybrid path with 200 steps + 16 cameras at 256×256 in < 5min on a single GPU.
Open questions#
- Differentiable rasteriser choice. Three options: pure-TS WebGPU (slow, no
Node port), Rust binding via
gsplat/brush(libs/isis/3d-inference-localis the existing seam), Python subprocess vianerfstudio(heavy dependency). Recommended: Rust binding. Fallback: CPU reference implementation (current spec) for tests; opt into Rust path viabakeOptions.useNativeRefinerflag. - Splat merge: do we keep separate scene IDs after merge? For attribution +
per-mesh editing it would be useful to retain a
gaussianGroupIdsarray (Uint16Arrayof lengthcount). Adds 2 bytes per Gaussian to the in-memory representation but doesn't ship into.spz(no extension point). Recommended: yes, in-memory only. - Occlusion cull threshold. Voxel size = 5cm is reasonable for metres-scale
scenes but blunt for fine detail. Recommended: configurable via
MergeOptions.occlusionVoxelSize. - Dedupe colour tolerance. Spatial dedupe alone may merge visually distinct
Gaussians at the same point. Add
dedupeColorEpsilonand require both spatial and color match. Default:0.05in normalised RGB.