# Block E · Mesh-to-Gaussians Bake Pipeline

**Owner library:** `libs/isis/gaussian-splatting` **Estimate:** ~3–4 weeks
**Blocks:** [F](./F-lighting-consistency.md) (lighting needs baked Gaussians).
**Dependencies:** block D (`.spz` codec), block B (composer surface that calls
into the bake), [G](./G-job-envelope-and-gpu-dispatch.md) for production GPU
dispatch (block E ships an in-process baseline first). **Status:** not started

## Overview

Convert each foreground mesh into Gaussians and merge them into the Marble splat
to produce a single composed `.spz` artefact. Two paths shipped:

1. **Surface-sampling baseline** — Poisson-disc sample the mesh surface, one
   Gaussian per sample. Fast, deterministic, ~30s per mesh on CPU.
2. **Hybrid (sampling + differentiable refinement)** — surface-sampling warm
   start, then 100–500 steps of differentiable splat fitting against PBR mesh
   renders. Higher fidelity, ~2–5min per mesh on GPU.

Plus a splat-merging algorithm: concat Gaussian buffers, depth-sort,
occlusion-cull, dedupe, recompute bounds.

## Why this is needed

The composer's "transform-only" output (block B) leaves a bag of files for the
consumer to assemble at runtime — every viewer/engine needs custom glue. Block E
produces a single `.spz` that loads correctly in any conformant splat viewer
with no custom code.

This is also the prerequisite for lighting consistency (block F) — to apply
environment lighting to the meshes, we need the meshes as PBR-shaded samples
first.

## Files to create

### Bake module structure

- `libs/isis/gaussian-splatting/src/bake/` — new subdirectory.
- `libs/isis/gaussian-splatting/src/bake/index.ts` — public re-exports.
- `libs/isis/gaussian-splatting/src/bake/types.ts` — `BakeRequest`,
  `BakeResult`, `BakeMethod`, `BakedGaussians`.

### Surface sampling

- `libs/isis/gaussian-splatting/src/bake/surface-sampling/poisson-sampler.ts` —
  Poisson-disc sampling on a triangle mesh (uniform, area-weighted).
- `libs/isis/gaussian-splatting/src/bake/surface-sampling/poisson-sampler.test.ts`
- `libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.ts`
  — given a sample point + local triangle + texture sample, fit a single
  Gaussian (position, oriented disk-shape from surface normal, color).
- `libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.test.ts`
- `libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.ts` —
  the orchestrator: parse GLB → sample → fit Gaussians → return
  `BakedGaussians`.
- `libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.test.ts`

### Differentiable refinement (hybrid path)

- `libs/isis/gaussian-splatting/src/bake/differentiable/render-from-views.ts` —
  given mesh + camera poses, render PBR images using a CPU/GPU rasteriser. v1
  uses `@gltf-transform/functions` for headless render; v1.5 uses `wgpu` via the
  existing `libs/isis/3d-inference-local` Rust binding.
- `libs/isis/gaussian-splatting/src/bake/differentiable/render-from-views.test.ts`
- `libs/isis/gaussian-splatting/src/bake/differentiable/refine-gaussians.ts` —
  given initial Gaussians + target renders + cameras, run K steps of gradient
  descent on Gaussian parameters minimising L2 + SSIM loss.
- `libs/isis/gaussian-splatting/src/bake/differentiable/refine-gaussians.test.ts`
- `libs/isis/gaussian-splatting/src/bake/differentiable/hybrid-baker.ts` — the
  orchestrator: surface-sample for warm start, then refine.
- `libs/isis/gaussian-splatting/src/bake/differentiable/hybrid-baker.test.ts`

### Splat merging

- `libs/isis/gaussian-splatting/src/bake/merge/concat-splats.ts` — naive
  Float32Array concatenation across all input scenes.
- `libs/isis/gaussian-splatting/src/bake/merge/depth-sort.ts` — re-order
  Gaussians by depth from a canonical view (front-camera) for streaming-
  friendly load order.
- `libs/isis/gaussian-splatting/src/bake/merge/occlusion-cull.ts` — prune
  Gaussians from the baked mesh that are entirely behind opaque environment
  Gaussians. Voxel-grid based.
- `libs/isis/gaussian-splatting/src/bake/merge/dedupe.ts` — collapse Gaussians
  within ε spatial distance with similar color/scale.
- `libs/isis/gaussian-splatting/src/bake/merge/merge-splats.ts` — the
  orchestrator: concat → sort → cull → dedupe → recompute bounds → return.
- `libs/isis/gaussian-splatting/src/bake/merge/merge-splats.test.ts`
- One `*.test.ts` per merge submodule.

### Composer integration

- `libs/isis/scene-from-image-composer/src/phases/bake-phase.ts` (new in
  composer) — replaces the transform-only `align-phase` output with a baked
  composed splat. Calls into `@isis/gaussian-splatting/bake`.
- `libs/isis/scene-from-image-composer/src/phases/bake-phase.test.ts`

## Files to modify

- `libs/isis/gaussian-splatting/src/index.ts` — re-export the bake module.
- `libs/isis/scene-from-image-composer/src/composer.ts`:
  - Insert `bake-phase` after `align-phase` and before `audio-phase`.
  - Conditional on `composeOptions.outputMode === 'baked'` (default).
    `composeOptions.outputMode === 'transform-only'` keeps block B's behavior
    for callers that need the unbaked artefacts.
- `libs/isis/scene-from-image-composer/src/types.ts`:
  - Add `outputMode: 'baked' | 'transform-only'` to `SceneFromImageRequest`.
  - Add `bakeOptions?: BakeOptions` to `SceneFromImageRequest`.

## Public API surface

```ts
// bake/types.ts

export type BakeMethod = 'surface-sampling' | 'hybrid';

export interface BakeRequest {
  readonly mesh: Buffer; // GLB bytes
  readonly transform: Float32Array; // 16-element column-major 4×4
  readonly method: BakeMethod;
  readonly targetGaussianCount?: number; // default 5_000 per mesh
  readonly refinementSteps?: number; // default 200 (hybrid only)
  readonly cameraCount?: number; // default 16 (hybrid only)
}

export interface BakedGaussians {
  readonly positions: Float32Array; // 3 * N
  readonly scales: Float32Array; // 3 * N
  readonly rotations: Float32Array; // 4 * N (quaternion x,y,z,w)
  readonly opacities: Float32Array; // N
  readonly colors: Float32Array; // 3 * N (RGB 0..1)
  readonly sphericalHarmonics?: Float32Array; // optional
  readonly count: number;
}

export interface BakeResult {
  readonly baked: BakedGaussians;
  readonly metrics: { readonly durationMs: number; readonly steps?: number };
}

// bake/index.ts

export async function bakeMesh(request: BakeRequest): Promise<BakeResult>;

// merge/merge-splats.ts

export interface MergeOptions {
  readonly enableDepthSort?: boolean; // default true
  readonly enableOcclusionCull?: boolean; // default true
  readonly enableDedupe?: boolean; // default true
  readonly dedupeEpsilon?: number; // default 0.001 (1mm)
  readonly canonicalView?: {
    readonly position: [number, number, number];
    readonly target: [number, number, number];
  };
}

export function mergeSplats(
  scenes: readonly LoadedScene[],
  options?: MergeOptions
): LoadedScene;
```

## Algorithms

### Surface-sampling baker

```
sample(mesh, count):
  triangles = parseGLB(mesh).triangles
  totalArea = sum(triangle.area for triangle in triangles)
  // CDF over triangles weighted by area
  cdf = buildCDF(triangles, weight: t -> t.area)
  samples = []
  for i in 0..count:
    triangle = sampleFromCDF(cdf, rand())
    barycentric = uniformBarycentricSample(rand2())
    point = barycentric.eval(triangle)
    normal = triangle.normal()
    color = sampleTexture(triangle.uv.eval(barycentric), mesh.diffuseTexture)
    samples.append({ point, normal, color, triangle })
  return samples

fitGaussianFromSample(sample, neighborhood):
  position = sample.point
  // Scale derived from local surface area density
  scale = [
    sqrt(neighborhood.areaPerSample),       // tangent X
    sqrt(neighborhood.areaPerSample),       // tangent Y
    0.0001                                  // very thin in normal direction (oriented disk)
  ]
  // Rotation aligns Z-axis with surface normal
  rotation = quaternionFromZAxisAlignment(sample.normal)
  opacity = 1.0
  color = sample.color
  return { position, scale, rotation, opacity, color }
```

### Hybrid baker

```
hybridBake(mesh, transform, request):
  baked = surfaceSamplingBake(mesh, transform, request.targetGaussianCount)

  cameras = generateCamerasOnSphere(
    request.cameraCount, focusPoint = transform.translation, radius = 2 * meshDiameter
  )
  targetRenders = renderMeshFromViews(mesh, transform, cameras)

  for step in 0..request.refinementSteps:
    rendered = differentiableSplatRender(baked, cameras)
    loss = sum(L2(rendered[i], targetRenders[i]) + 0.5 * SSIM_loss(rendered[i], targetRenders[i]) for i)
    grads = backprop(loss, baked)
    baked = applyAdam(baked, grads, lr = 0.005)

  return baked
```

For block E v1, the differentiable renderer is **CPU** (slow, but deterministic
and testable). Block G migrates the heavy work to GPU workers via job-envelope;
block E keeps the CPU path as a reference implementation for tests.

### Splat merging — occlusion cull

```
occlusionCull(scenes):
  // Build voxel grid over union bounds.
  voxelGrid = new VoxelGrid(unionBounds, voxelSize = 0.05)  // 5cm cells
  // Mark voxels containing high-opacity environment Gaussians.
  for gaussian in environmentScene:
    if gaussian.opacity > 0.5:
      voxelGrid.markOccluded(gaussian.position)
  // Drop baked-mesh Gaussians whose voxel cell is occluded.
  filtered = []
  for gaussian in bakedMeshScene:
    if !voxelGrid.isOccluded(gaussian.position):
      filtered.append(gaussian)
  return filtered
```

### Splat merging — dedupe

```
dedupe(gaussians, epsilon):
  // Spatial hash over epsilon-grid.
  hash = new SpatialHash(cellSize = epsilon)
  out = []
  for g in gaussians:
    cell = hash.cell(g.position)
    if !cell.has(approximately g):
      cell.add(g)
      out.append(g)
  return out
```

## Test plan

**Unit (`poisson-sampler.test.ts`):**

- Sample 1000 points on a unit cube → all points lie within bounds.
- Sample on a unit cube → samples are area-weighted (each face gets ~166 points
  within ±10%).
- Same input + seed → identical sample list (determinism).

**Unit (`sample-to-gaussian.test.ts`):**

- Sample with known normal `[0, 1, 0]` produces rotation that maps Z-axis to
  `[0, 1, 0]` (quaternion verified to ε).
- Sample on a 1m² triangle with area-per-sample = 0.001 produces tangent scales
  ≈ `sqrt(0.001) ≈ 0.0316`.

**Unit (`sampling-baker.test.ts`):**

- Bake a unit cube GLB at identity transform with `targetGaussianCount: 1000` →
  output has ≥ 950 Gaussians (some small triangles may get rounded down),
  ≤ 1050. All positions inside `[-0.51, 0.51]^3` (cube + epsilon).

**Unit (`render-from-views.test.ts`):**

- Render unit cube from canonical front view → output image's centre pixel is
  the cube's diffuse colour.

**Unit (`refine-gaussians.test.ts`):**

- Synthetic test: start with 100 Gaussians whose positions are perturbed from a
  target arrangement; refinement reduces position MSE by
  > 50% in 100 steps.

**Unit (`hybrid-baker.test.ts`):**

- Bake a textured unit cube via hybrid path with 50 steps → compare against
  surface-sampling-only output → hybrid has lower L2 error against ground-truth
  render of the cube. ε absolute, asserted.

**Unit (`merge-splats.test.ts`):**

- Concat-only: two scenes of 100 Gaussians → merged scene has 200.
- Depth sort: 5 Gaussians at depths `[3, 1, 4, 1, 5]` from canonical view →
  output order corresponds to depths `[1, 1, 3, 4, 5]`.
- Occlusion cull: 100 baked Gaussians fully inside a 200-Gaussian high-opacity
  environment box → all 100 baked Gaussians culled.
- Dedupe: 50 Gaussians at the same position → output has 1.
- Bounds recompute: union of two scenes with bounds `[-1,1]^3` and `[0,2]^3` →
  output bounds `[-1,2]^3`.

**Integration (`bake-phase.test.ts` in composer):**

- Composer with 1 mesh, surface-sampling method, transform-only env (skip Marble
  call) → produces a baked `LoadedScene` with `count > 0` and positions
  correctly transformed into world space.
- Same with hybrid method (10 refinement steps for test speed) → produces a
  baked scene; metrics record `steps: 10`.
- Composer with `outputMode: 'transform-only'` → bake phase skipped; manifest
  unchanged from block B behavior.

**Adversarial:**

- Bake degenerate mesh (all triangles zero area) → throws
  `BakeError('mesh has no surface area')`.
- Merge with empty input array → throws.
- Merge two scenes with mismatched SH degrees → throws (or upcasts the smaller —
  ADR-style decision; recommend throw).

## Acceptance criteria

- All new tests pass.
- `npx tsc --noEmit` clean from both `libs/isis/gaussian-splatting` and
  `libs/isis/scene-from-image-composer`.
- Stub-indicator scan clean.
- A composer end-to-end run with 2 placed meshes (cube + sphere) on a synthetic
  env splat (10k Gaussians) produces a baked `.spz` that loads in the
  `mkkellogg/GaussianSplats3D` web viewer, with the meshes visible in the
  expected positions. Manual visual QA recorded in PR.
- Hybrid path with 100 refinement steps produces visibly sharper edges than
  surface-sampling alone (visual QA).
- Performance: surface-sampling baseline bakes a 50k-triangle GLB in < 60s on
  CPU. Hybrid path with 200 steps + 16 cameras at 256×256 in < 5min on a single
  GPU.

## Open questions

- **Differentiable rasteriser choice.** Three options: pure-TS WebGPU (slow, no
  Node port), Rust binding via `gsplat`/`brush` (`libs/isis/3d-inference-local`
  is the existing seam), Python subprocess via `nerfstudio` (heavy dependency).
  Recommended: Rust binding. Fallback: CPU reference implementation (current
  spec) for tests; opt into Rust path via `bakeOptions.useNativeRefiner` flag.
- **Splat merge: do we keep separate scene IDs after merge?** For attribution +
  per-mesh editing it would be useful to retain a `gaussianGroupIds` array
  (`Uint16Array` of length `count`). Adds 2 bytes per Gaussian to the in-memory
  representation but doesn't ship into `.spz` (no extension point). Recommended:
  yes, in-memory only.
- **Occlusion cull threshold.** Voxel size = 5cm is reasonable for metres-scale
  scenes but blunt for fine detail. Recommended: configurable via
  `MergeOptions.occlusionVoxelSize`.
- **Dedupe colour tolerance.** Spatial dedupe alone may merge visually distinct
  Gaussians at the same point. Add `dedupeColorEpsilon` and require both spatial
  and color match. Default: `0.05` in normalised RGB.
