Eve library

Block E · Mesh-to-Gaussians Bake Pipeline

Other

Revision 1

On this page

Owner library: libs/isis/gaussian-splatting Estimate: ~3–4 weeks Blocks: F (lighting needs baked Gaussians). Dependencies: block D (.spz codec), block B (composer surface that calls into the bake), G for production GPU dispatch (block E ships an in-process baseline first). Status: not started

Overview#

Convert each foreground mesh into Gaussians and merge them into the Marble splat to produce a single composed .spz artefact. Two paths shipped:

  1. Surface-sampling baseline — Poisson-disc sample the mesh surface, one Gaussian per sample. Fast, deterministic, ~30s per mesh on CPU.
  2. Hybrid (sampling + differentiable refinement) — surface-sampling warm start, then 100–500 steps of differentiable splat fitting against PBR mesh renders. Higher fidelity, ~2–5min per mesh on GPU.

Plus a splat-merging algorithm: concat Gaussian buffers, depth-sort, occlusion-cull, dedupe, recompute bounds.

Why this is needed#

The composer's "transform-only" output (block B) leaves a bag of files for the consumer to assemble at runtime — every viewer/engine needs custom glue. Block E produces a single .spz that loads correctly in any conformant splat viewer with no custom code.

This is also the prerequisite for lighting consistency (block F) — to apply environment lighting to the meshes, we need the meshes as PBR-shaded samples first.

Files to create#

Bake module structure#

  • libs/isis/gaussian-splatting/src/bake/ — new subdirectory.
  • libs/isis/gaussian-splatting/src/bake/index.ts — public re-exports.
  • libs/isis/gaussian-splatting/src/bake/types.ts — BakeRequest, BakeResult, BakeMethod, BakedGaussians.

Surface sampling#

  • libs/isis/gaussian-splatting/src/bake/surface-sampling/poisson-sampler.ts — Poisson-disc sampling on a triangle mesh (uniform, area-weighted).
  • libs/isis/gaussian-splatting/src/bake/surface-sampling/poisson-sampler.test.ts
  • libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.ts — given a sample point + local triangle + texture sample, fit a single Gaussian (position, oriented disk-shape from surface normal, color).
  • libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.test.ts
  • libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.ts — the orchestrator: parse GLB → sample → fit Gaussians → return BakedGaussians.
  • libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.test.ts

Differentiable refinement (hybrid path)#

  • libs/isis/gaussian-splatting/src/bake/differentiable/render-from-views.ts — given mesh + camera poses, render PBR images using a CPU/GPU rasteriser. v1 uses @gltf-transform/functions for headless render; v1.5 uses wgpu via the existing libs/isis/3d-inference-local Rust binding.
  • libs/isis/gaussian-splatting/src/bake/differentiable/render-from-views.test.ts
  • libs/isis/gaussian-splatting/src/bake/differentiable/refine-gaussians.ts — given initial Gaussians + target renders + cameras, run K steps of gradient descent on Gaussian parameters minimising L2 + SSIM loss.
  • libs/isis/gaussian-splatting/src/bake/differentiable/refine-gaussians.test.ts
  • libs/isis/gaussian-splatting/src/bake/differentiable/hybrid-baker.ts — the orchestrator: surface-sample for warm start, then refine.
  • libs/isis/gaussian-splatting/src/bake/differentiable/hybrid-baker.test.ts

Splat merging#

  • libs/isis/gaussian-splatting/src/bake/merge/concat-splats.ts — naive Float32Array concatenation across all input scenes.
  • libs/isis/gaussian-splatting/src/bake/merge/depth-sort.ts — re-order Gaussians by depth from a canonical view (front-camera) for streaming- friendly load order.
  • libs/isis/gaussian-splatting/src/bake/merge/occlusion-cull.ts — prune Gaussians from the baked mesh that are entirely behind opaque environment Gaussians. Voxel-grid based.
  • libs/isis/gaussian-splatting/src/bake/merge/dedupe.ts — collapse Gaussians within ε spatial distance with similar color/scale.
  • libs/isis/gaussian-splatting/src/bake/merge/merge-splats.ts — the orchestrator: concat → sort → cull → dedupe → recompute bounds → return.
  • libs/isis/gaussian-splatting/src/bake/merge/merge-splats.test.ts
  • One *.test.ts per merge submodule.

Composer integration#

  • libs/isis/scene-from-image-composer/src/phases/bake-phase.ts (new in composer) — replaces the transform-only align-phase output with a baked composed splat. Calls into @isis/gaussian-splatting/bake.
  • libs/isis/scene-from-image-composer/src/phases/bake-phase.test.ts

Files to modify#

  • libs/isis/gaussian-splatting/src/index.ts — re-export the bake module.
  • libs/isis/scene-from-image-composer/src/composer.ts:
    • Insert bake-phase after align-phase and before audio-phase.
    • Conditional on composeOptions.outputMode === 'baked' (default). composeOptions.outputMode === 'transform-only' keeps block B's behavior for callers that need the unbaked artefacts.
  • libs/isis/scene-from-image-composer/src/types.ts:
    • Add outputMode: 'baked' | 'transform-only' to SceneFromImageRequest.
    • Add bakeOptions?: BakeOptions to SceneFromImageRequest.

Public API surface#

ts
// bake/types.ts

export type BakeMethod = 'surface-sampling' | 'hybrid';

export interface BakeRequest {
  readonly mesh: Buffer; // GLB bytes
  readonly transform: Float32Array; // 16-element column-major 4×4
  readonly method: BakeMethod;
  readonly targetGaussianCount?: number; // default 5_000 per mesh
  readonly refinementSteps?: number; // default 200 (hybrid only)
  readonly cameraCount?: number; // default 16 (hybrid only)
}

export interface BakedGaussians {
  readonly positions: Float32Array; // 3 * N
  readonly scales: Float32Array; // 3 * N
  readonly rotations: Float32Array; // 4 * N (quaternion x,y,z,w)
  readonly opacities: Float32Array; // N
  readonly colors: Float32Array; // 3 * N (RGB 0..1)
  readonly sphericalHarmonics?: Float32Array; // optional
  readonly count: number;
}

export interface BakeResult {
  readonly baked: BakedGaussians;
  readonly metrics: { readonly durationMs: number; readonly steps?: number };
}

// bake/index.ts

export async function bakeMesh(request: BakeRequest): Promise<BakeResult>;

// merge/merge-splats.ts

export interface MergeOptions {
  readonly enableDepthSort?: boolean; // default true
  readonly enableOcclusionCull?: boolean; // default true
  readonly enableDedupe?: boolean; // default true
  readonly dedupeEpsilon?: number; // default 0.001 (1mm)
  readonly canonicalView?: {
    readonly position: [number, number, number];
    readonly target: [number, number, number];
  };
}

export function mergeSplats(
  scenes: readonly LoadedScene[],
  options?: MergeOptions
): LoadedScene;

Algorithms#

Surface-sampling baker#

text
sample(mesh, count):
  triangles = parseGLB(mesh).triangles
  totalArea = sum(triangle.area for triangle in triangles)
  // CDF over triangles weighted by area
  cdf = buildCDF(triangles, weight: t -> t.area)
  samples = []
  for i in 0..count:
    triangle = sampleFromCDF(cdf, rand())
    barycentric = uniformBarycentricSample(rand2())
    point = barycentric.eval(triangle)
    normal = triangle.normal()
    color = sampleTexture(triangle.uv.eval(barycentric), mesh.diffuseTexture)
    samples.append({ point, normal, color, triangle })
  return samples

fitGaussianFromSample(sample, neighborhood):
  position = sample.point
  // Scale derived from local surface area density
  scale = [
    sqrt(neighborhood.areaPerSample),       // tangent X
    sqrt(neighborhood.areaPerSample),       // tangent Y
    0.0001                                  // very thin in normal direction (oriented disk)
  ]
  // Rotation aligns Z-axis with surface normal
  rotation = quaternionFromZAxisAlignment(sample.normal)
  opacity = 1.0
  color = sample.color
  return { position, scale, rotation, opacity, color }

Hybrid baker#

text
hybridBake(mesh, transform, request):
  baked = surfaceSamplingBake(mesh, transform, request.targetGaussianCount)

  cameras = generateCamerasOnSphere(
    request.cameraCount, focusPoint = transform.translation, radius = 2 * meshDiameter
  )
  targetRenders = renderMeshFromViews(mesh, transform, cameras)

  for step in 0..request.refinementSteps:
    rendered = differentiableSplatRender(baked, cameras)
    loss = sum(L2(rendered[i], targetRenders[i]) + 0.5 * SSIM_loss(rendered[i], targetRenders[i]) for i)
    grads = backprop(loss, baked)
    baked = applyAdam(baked, grads, lr = 0.005)

  return baked

For block E v1, the differentiable renderer is CPU (slow, but deterministic and testable). Block G migrates the heavy work to GPU workers via job-envelope; block E keeps the CPU path as a reference implementation for tests.

Splat merging — occlusion cull#

text
occlusionCull(scenes):
  // Build voxel grid over union bounds.
  voxelGrid = new VoxelGrid(unionBounds, voxelSize = 0.05)  // 5cm cells
  // Mark voxels containing high-opacity environment Gaussians.
  for gaussian in environmentScene:
    if gaussian.opacity > 0.5:
      voxelGrid.markOccluded(gaussian.position)
  // Drop baked-mesh Gaussians whose voxel cell is occluded.
  filtered = []
  for gaussian in bakedMeshScene:
    if !voxelGrid.isOccluded(gaussian.position):
      filtered.append(gaussian)
  return filtered

Splat merging — dedupe#

text
dedupe(gaussians, epsilon):
  // Spatial hash over epsilon-grid.
  hash = new SpatialHash(cellSize = epsilon)
  out = []
  for g in gaussians:
    cell = hash.cell(g.position)
    if !cell.has(approximately g):
      cell.add(g)
      out.append(g)
  return out

Test plan#

Unit (poisson-sampler.test.ts):

  • Sample 1000 points on a unit cube → all points lie within bounds.
  • Sample on a unit cube → samples are area-weighted (each face gets ~166 points within ±10%).
  • Same input + seed → identical sample list (determinism).

Unit (sample-to-gaussian.test.ts):

  • Sample with known normal [0, 1, 0] produces rotation that maps Z-axis to [0, 1, 0] (quaternion verified to ε).
  • Sample on a 1m² triangle with area-per-sample = 0.001 produces tangent scales ≈ sqrt(0.001) ≈ 0.0316.

Unit (sampling-baker.test.ts):

  • Bake a unit cube GLB at identity transform with targetGaussianCount: 1000 → output has ≥ 950 Gaussians (some small triangles may get rounded down), ≤ 1050. All positions inside [-0.51, 0.51]^3 (cube + epsilon).

Unit (render-from-views.test.ts):

  • Render unit cube from canonical front view → output image's centre pixel is the cube's diffuse colour.

Unit (refine-gaussians.test.ts):

  • Synthetic test: start with 100 Gaussians whose positions are perturbed from a target arrangement; refinement reduces position MSE by

    50% in 100 steps.

Unit (hybrid-baker.test.ts):

  • Bake a textured unit cube via hybrid path with 50 steps → compare against surface-sampling-only output → hybrid has lower L2 error against ground-truth render of the cube. ε absolute, asserted.

Unit (merge-splats.test.ts):

  • Concat-only: two scenes of 100 Gaussians → merged scene has 200.
  • Depth sort: 5 Gaussians at depths [3, 1, 4, 1, 5] from canonical view → output order corresponds to depths [1, 1, 3, 4, 5].
  • Occlusion cull: 100 baked Gaussians fully inside a 200-Gaussian high-opacity environment box → all 100 baked Gaussians culled.
  • Dedupe: 50 Gaussians at the same position → output has 1.
  • Bounds recompute: union of two scenes with bounds [-1,1]^3 and [0,2]^3 → output bounds [-1,2]^3.

Integration (bake-phase.test.ts in composer):

  • Composer with 1 mesh, surface-sampling method, transform-only env (skip Marble call) → produces a baked LoadedScene with count > 0 and positions correctly transformed into world space.
  • Same with hybrid method (10 refinement steps for test speed) → produces a baked scene; metrics record steps: 10.
  • Composer with outputMode: 'transform-only' → bake phase skipped; manifest unchanged from block B behavior.

Adversarial:

  • Bake degenerate mesh (all triangles zero area) → throws BakeError('mesh has no surface area').
  • Merge with empty input array → throws.
  • Merge two scenes with mismatched SH degrees → throws (or upcasts the smaller — ADR-style decision; recommend throw).

Acceptance criteria#

  • All new tests pass.
  • npx tsc --noEmit clean from both libs/isis/gaussian-splatting and libs/isis/scene-from-image-composer.
  • Stub-indicator scan clean.
  • A composer end-to-end run with 2 placed meshes (cube + sphere) on a synthetic env splat (10k Gaussians) produces a baked .spz that loads in the mkkellogg/GaussianSplats3D web viewer, with the meshes visible in the expected positions. Manual visual QA recorded in PR.
  • Hybrid path with 100 refinement steps produces visibly sharper edges than surface-sampling alone (visual QA).
  • Performance: surface-sampling baseline bakes a 50k-triangle GLB in < 60s on CPU. Hybrid path with 200 steps + 16 cameras at 256×256 in < 5min on a single GPU.

Open questions#

  • Differentiable rasteriser choice. Three options: pure-TS WebGPU (slow, no Node port), Rust binding via gsplat/brush (libs/isis/3d-inference-local is the existing seam), Python subprocess via nerfstudio (heavy dependency). Recommended: Rust binding. Fallback: CPU reference implementation (current spec) for tests; opt into Rust path via bakeOptions.useNativeRefiner flag.
  • Splat merge: do we keep separate scene IDs after merge? For attribution + per-mesh editing it would be useful to retain a gaussianGroupIds array (Uint16Array of length count). Adds 2 bytes per Gaussian to the in-memory representation but doesn't ship into .spz (no extension point). Recommended: yes, in-memory only.
  • Occlusion cull threshold. Voxel size = 5cm is reasonable for metres-scale scenes but blunt for fine detail. Recommended: configurable via MergeOptions.occlusionVoxelSize.
  • Dedupe colour tolerance. Spatial dedupe alone may merge visually distinct Gaussians at the same point. Add dedupeColorEpsilon and require both spatial and color match. Default: 0.05 in normalised RGB.