# Block F · Lighting Consistency

**Owner library:** `libs/isis/gaussian-splatting` **Estimate:** ~1–2 weeks
**Blocks:** none structurally; without it composed scenes look obviously
"flat-lit" against the env. **Dependencies:** block E (need baked-Gaussian path
to inject lighting into); block D (need the composed `.spz` to evaluate
against). **Status:** not started

## Overview

Marble bakes radiance (lighting × albedo) into per-Gaussian SH coefficients;
Hunyuan/Tripo/Meshy emit albedo + PBR factors that have never been lit by the
scene. Naively rasterising meshes into Gaussians using their own albedo produces
meshes that look flat against an environment with directional light, ambient
occlusion, and color cast.

Block F adds environment-aware lighting:

1. Sample the Marble splat's SH coefficients at each mesh's placement point to
   derive a local irradiance environment map.
2. Apply that lighting to the mesh's PBR shading at sample time, before fitting
   Gaussians (block E's surface-sampling step consumes lit colour instead of raw
   albedo).
3. Composition coherence gate — a new QA gate in
   `libs/isis/3d-quality-gates/src/delivery/` that scores style consistency
   between baked-mesh regions and environment regions in the final composed
   renders.

## Files to create

### Lighting module

- `libs/isis/gaussian-splatting/src/lighting/` — new subdirectory.
- `libs/isis/gaussian-splatting/src/lighting/index.ts`
- `libs/isis/gaussian-splatting/src/lighting/types.ts` —
  `EnvironmentLightField`, `IrradianceMap`, `LightingEstimateOptions`.
- `libs/isis/gaussian-splatting/src/lighting/sample-sh-at-point.ts` — given a
  `LoadedScene` with SH and a 3D point, sample the SH-encoded irradiance.
- `libs/isis/gaussian-splatting/src/lighting/sample-sh-at-point.test.ts`
- `libs/isis/gaussian-splatting/src/lighting/estimate-environment-light-field.ts`
  — given a `LoadedScene` (env splat) and a target placement point, compute an
  `IrradianceMap` (low-resolution env map, e.g. 32×16 equirect, or 9-coefficient
  SH irradiance representation).
- `libs/isis/gaussian-splatting/src/lighting/estimate-environment-light-field.test.ts`
- `libs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.ts` — given a
  sample point + normal + material params + `IrradianceMap`, compute the lit
  colour at that point.
- `libs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.test.ts`

### Composition coherence gate

- `libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.ts` —
  evaluates a composed scene's renders for style consistency between baked-mesh
  regions and environment regions.
- `libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.test.ts`

## Files to modify

- `libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.ts`:
  - Accept an optional `IrradianceMap` parameter; when supplied, replace raw
    `sampleTexture(...)` colour with `applyPbrShading(...)`.
- `libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.ts`:
  - Accept `lightingContext?: { irradianceMap }`; pass through to
    `sample-to-gaussian`.
- `libs/isis/gaussian-splatting/src/bake/index.ts`:
  - `bakeMesh` accepts a new optional `lightingContext` field.
- `libs/isis/scene-from-image-composer/src/phases/bake-phase.ts`:
  - Before each mesh bake, call
    `estimateEnvironmentLightField(envScene, placementPoint)` and pass the
    resulting `IrradianceMap` into `bakeMesh`.
- `libs/isis/scene-from-image-composer/src/phases/grounding-phase.ts`:
  - Add a _post-bake_ grounding subphase that runs the composition coherence
    gate after bake completes.
- `libs/isis/3d-quality-gates/src/delivery/index.ts`:
  - Re-export the coherence gate.
- `libs/isis/3d-quality-gates/src/index.ts`:
  - Re-export.

## Public API surface

```ts
// lighting/types.ts

/**
 * Low-resolution irradiance representation. Two encodings supported:
 *   - 'sh9': 9 coefficients per channel, 27 floats total. Compact, fast
 *           lookup (single matrix multiply). Good for diffuse-only.
 *   - 'env-map': low-res equirectangular env map. Larger (32*16*3 = 1536
 *           floats), supports specular reflections.
 */
export type IrradianceMap =
  | { readonly encoding: 'sh9'; readonly coefficients: Float32Array /* 27 */ }
  | {
      readonly encoding: 'env-map';
      readonly width: number;
      readonly height: number;
      readonly data: Float32Array;
    };

export interface EnvironmentLightField {
  readonly origin: readonly [number, number, number];
  readonly map: IrradianceMap;
  readonly metadata: {
    readonly sourceGaussianCount: number;
    readonly sampledRadiusMeters: number;
    readonly estimationMethod: 'gaussian-weighted-sh' | 'voxel-aggregate';
  };
}

// lighting/estimate-environment-light-field.ts

export interface LightingEstimateOptions {
  readonly encoding?: 'sh9' | 'env-map'; // default 'sh9'
  readonly samplingRadiusMeters?: number; // default 2.0
  readonly maxSamples?: number; // default 1000
}

export function estimateEnvironmentLightField(
  scene: LoadedScene,
  placementPoint: readonly [number, number, number],
  options?: LightingEstimateOptions
): EnvironmentLightField;

// lighting/apply-pbr-shading.ts

export interface PbrSurfaceSample {
  readonly position: readonly [number, number, number];
  readonly normal: readonly [number, number, number];
  readonly albedo: readonly [number, number, number]; // 0..1
  readonly roughness?: number; // 0..1, default 0.7
  readonly metallic?: number; // 0..1, default 0.0
}

export function applyPbrShading(
  sample: PbrSurfaceSample,
  irradiance: IrradianceMap
): readonly [number, number, number];
```

## Algorithms

### Sample SH at a point

The Marble splat's per-Gaussian SH coefficients encode view-dependent radiance
from each Gaussian. The irradiance at a point is the integral of incoming light
from all directions over the upper hemisphere.

For block F v1 we approximate via gaussian-weighted aggregation of SH:

```
sampleSHAtPoint(scene, point, radius):
  weights = []
  contributions = []
  for each gaussian g in scene where distance(g.position, point) < radius:
    w = exp(-distance(g.position, point)^2 / radius^2) * g.opacity
    sh = g.shCoefficients  // 9 floats per channel for SH degree 2 (27 total)
    weights.append(w)
    contributions.append(sh * w)
  totalWeight = sum(weights)
  irradianceSH = sum(contributions) / totalWeight
  return irradianceSH
```

This is a coarse approximation — it treats each Gaussian's outgoing radiance as
if it were incoming irradiance at the placement point. Good enough for plausible
"tone match" but not physically accurate. Block F1.5 (future work) replaces with
proper light-transport sampling via path tracing if visual QA demands.

### Apply PBR shading

For diffuse-only (no specular):

```
applyPbrShading(sample, irradiance):
  if irradiance.encoding === 'sh9':
    shInWorldFrame = rotateSHByNormal(irradiance.coefficients, sample.normal)
    diffuseIrradiance = evaluateSHDirection(shInWorldFrame, sample.normal)
    return sample.albedo * diffuseIrradiance
  else:
    // env-map path: sample the equirect map at the normal direction
    uv = sphereToEquirectUV(sample.normal)
    diffuseIrradiance = sampleEquirect(irradiance, uv)
    return sample.albedo * diffuseIrradiance
```

Specular contribution (block F v1.5):

```
  // For metals or low-roughness surfaces, also sample env map at the
  // reflection direction `reflect(viewDir, normal)`. Blend by Fresnel.
```

### Composition coherence gate

Distinct from input-grounding-gate — that one compares input image to output
renders. Composition coherence compares **regions within the same output
render**: do the baked-mesh regions look stylistically consistent with the
environment regions?

```
evaluateCoherence(composedRenders, placements):
  for each render in composedRenders:
    // Project each placement's transform into screen space using the
    // render's camera; rasterise to get a mask of pixels covered by
    // baked mesh regions vs environment regions.
    meshRegionMask = rasterizePlacementMasks(render.camera, placements)
    envRegionMask  = NOT meshRegionMask
    // Extract per-region perceptual features via a small CNN/VLM adapter
    meshFeatures = adapter.extractFeatures(render, mask: meshRegionMask)
    envFeatures  = adapter.extractFeatures(render, mask: envRegionMask)
    // Score similarity in a perceptual feature space.
    score = cosineSimilarity(meshFeatures, envFeatures)
  return mean(scores)

decision(score):
  if score > 0.85: 'pass'
  if score > 0.7:  'soft-fail'
  else:             'hard-fail'
```

The adapter (`ThreeDQualityCoherenceFeatureAdapter`) is pluggable — production
uses CLIP or DINO features; tests use a fixture adapter.

## Test plan

**Unit (`sample-sh-at-point.test.ts`):**

- Synthetic scene: 1 Gaussian at origin with SH that encodes constant white
  irradiance. Sample at origin → returns the same SH coefficients.
- Same scene, sample 10m away with `radius = 2` → returns ~zero (Gaussian
  contribution falls off).
- Two Gaussians at ±1m, opposing colors. Sample at midpoint → returns blend.

**Unit (`estimate-environment-light-field.test.ts`):**

- 100-Gaussian env scene → `estimateEnvironmentLightField` returns SH9 with 27
  coefficients; coefficients deterministic for fixed input.
- Same scene encoded as `env-map` → returns 32×16 equirect map; brightest pixel
  direction matches the brightest Gaussian's direction.

**Unit (`apply-pbr-shading.test.ts`):**

- Surface with albedo `[1, 1, 1]`, normal `[0, 1, 0]`, irradiance = uniform
  white SH → returns `[π/π, π/π, π/π] = [1, 1, 1]` (Lambertian irradiance
  integral over hemisphere = π × albedo for uniform white).
- Same surface, albedo `[1, 0, 0]` → returns `[1, 0, 0]`.
- Normal pointing away from primary irradiance direction → diminished output.

**Unit (`composition-coherence-gate.test.ts`):**

- Synthetic render with mesh-region features = env-region features (use fixture
  adapter that returns the same vector for both) → score = 1.0, decision =
  `pass`.
- Adapter returns orthogonal vectors (cosine = 0) → decision = `hard-fail`.
- Adapter returns vectors with cosine = 0.75 → decision = `soft-fail`.

**Integration (`bake-phase.test.ts` extended):**

- Bake unit cube against synthetic env splat with directional light → baked
  Gaussian colours show clear directional shading (top brighter than bottom).
- Compare against baked unit cube with `lightingContext: undefined` (block E
  baseline) → colours uniform; visual diff asserted via mean brightness
  variance.

**Integration (composer end-to-end):**

- Full composer run with 1 mesh + lighting enabled → manifest's
  `lightingMetadata.shCoefficients` field populated; composed `.spz` loads in
  viewer; mesh region visually matches env tone (manual QA).

## Acceptance criteria

- All new tests pass.
- `npx tsc --noEmit` clean.
- Stub-indicator scan clean.
- Visual QA: a unit white cube placed in a known-warm-toned env splat shows warm
  tinting in the composed render (manual screenshot in PR).
- Composition coherence gate evaluates a known-good composed scene
  (env-baked-with-lighting) at score > 0.85 and a known-bad scene (env without
  lighting) at score < 0.7. Validated on a small reference corpus.
- The composer's manifest extension populates
  `metadata.sceneFromImage.lightingMetadata` correctly.

## Open questions

- **SH9 vs env-map default.** SH9 is faster and smaller; env-map supports
  specular. Default for v1: SH9 (the meshes Hunyuan/Tripo emit are predominantly
  diffuse). Switch to env-map per-mesh if material has `metallic > 0.5` or
  `roughness < 0.2`.
- **Light-transport approximation accuracy.** The gaussian-weighted SH
  aggregation is coarse; visual QA may surface artifacts (shadows in wrong
  places, missing colour bleed). v1.5 fallback: path-traced irradiance samples
  via `gsplat`'s differentiable renderer.
- **Coherence adapter sourcing.** Production wants CLIP or DINO features. Both
  have public TF.js / ONNX models we could ship via `onnxruntime-node` (already
  an optional dep of `3d-generation`). Recommended: ship a CLIP-ViT-B/32 adapter
  as the default.
- **Per-mesh vs scene-wide lighting.** Estimating a separate
  `EnvironmentLightField` per mesh placement is more accurate but slower. v1
  estimates per-mesh; v1.5 may share if placements are clustered.
