Eve library

Block F · Lighting Consistency

Other

Revision 1

On this page

Owner library: libs/isis/gaussian-splatting Estimate: ~1–2 weeks Blocks: none structurally; without it composed scenes look obviously "flat-lit" against the env. Dependencies: block E (need baked-Gaussian path to inject lighting into); block D (need the composed .spz to evaluate against). Status: not started

Overview#

Marble bakes radiance (lighting × albedo) into per-Gaussian SH coefficients; Hunyuan/Tripo/Meshy emit albedo + PBR factors that have never been lit by the scene. Naively rasterising meshes into Gaussians using their own albedo produces meshes that look flat against an environment with directional light, ambient occlusion, and color cast.

Block F adds environment-aware lighting:

  1. Sample the Marble splat's SH coefficients at each mesh's placement point to derive a local irradiance environment map.
  2. Apply that lighting to the mesh's PBR shading at sample time, before fitting Gaussians (block E's surface-sampling step consumes lit colour instead of raw albedo).
  3. Composition coherence gate — a new QA gate in libs/isis/3d-quality-gates/src/delivery/ that scores style consistency between baked-mesh regions and environment regions in the final composed renders.

Files to create#

Lighting module#

  • libs/isis/gaussian-splatting/src/lighting/ — new subdirectory.
  • libs/isis/gaussian-splatting/src/lighting/index.ts
  • libs/isis/gaussian-splatting/src/lighting/types.ts — EnvironmentLightField, IrradianceMap, LightingEstimateOptions.
  • libs/isis/gaussian-splatting/src/lighting/sample-sh-at-point.ts — given a LoadedScene with SH and a 3D point, sample the SH-encoded irradiance.
  • libs/isis/gaussian-splatting/src/lighting/sample-sh-at-point.test.ts
  • libs/isis/gaussian-splatting/src/lighting/estimate-environment-light-field.ts — given a LoadedScene (env splat) and a target placement point, compute an IrradianceMap (low-resolution env map, e.g. 32×16 equirect, or 9-coefficient SH irradiance representation).
  • libs/isis/gaussian-splatting/src/lighting/estimate-environment-light-field.test.ts
  • libs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.ts — given a sample point + normal + material params + IrradianceMap, compute the lit colour at that point.
  • libs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.test.ts

Composition coherence gate#

  • libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.ts — evaluates a composed scene's renders for style consistency between baked-mesh regions and environment regions.
  • libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.test.ts

Files to modify#

  • libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.ts:
    • Accept an optional IrradianceMap parameter; when supplied, replace raw sampleTexture(...) colour with applyPbrShading(...).
  • libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.ts:
    • Accept lightingContext?: { irradianceMap }; pass through to sample-to-gaussian.
  • libs/isis/gaussian-splatting/src/bake/index.ts:
    • bakeMesh accepts a new optional lightingContext field.
  • libs/isis/scene-from-image-composer/src/phases/bake-phase.ts:
    • Before each mesh bake, call estimateEnvironmentLightField(envScene, placementPoint) and pass the resulting IrradianceMap into bakeMesh.
  • libs/isis/scene-from-image-composer/src/phases/grounding-phase.ts:
    • Add a post-bake grounding subphase that runs the composition coherence gate after bake completes.
  • libs/isis/3d-quality-gates/src/delivery/index.ts:
    • Re-export the coherence gate.
  • libs/isis/3d-quality-gates/src/index.ts:
    • Re-export.

Public API surface#

ts
// lighting/types.ts

/**
 * Low-resolution irradiance representation. Two encodings supported:
 *   - 'sh9': 9 coefficients per channel, 27 floats total. Compact, fast
 *           lookup (single matrix multiply). Good for diffuse-only.
 *   - 'env-map': low-res equirectangular env map. Larger (32*16*3 = 1536
 *           floats), supports specular reflections.
 */
export type IrradianceMap =
  | { readonly encoding: 'sh9'; readonly coefficients: Float32Array /* 27 */ }
  | {
      readonly encoding: 'env-map';
      readonly width: number;
      readonly height: number;
      readonly data: Float32Array;
    };

export interface EnvironmentLightField {
  readonly origin: readonly [number, number, number];
  readonly map: IrradianceMap;
  readonly metadata: {
    readonly sourceGaussianCount: number;
    readonly sampledRadiusMeters: number;
    readonly estimationMethod: 'gaussian-weighted-sh' | 'voxel-aggregate';
  };
}

// lighting/estimate-environment-light-field.ts

export interface LightingEstimateOptions {
  readonly encoding?: 'sh9' | 'env-map'; // default 'sh9'
  readonly samplingRadiusMeters?: number; // default 2.0
  readonly maxSamples?: number; // default 1000
}

export function estimateEnvironmentLightField(
  scene: LoadedScene,
  placementPoint: readonly [number, number, number],
  options?: LightingEstimateOptions
): EnvironmentLightField;

// lighting/apply-pbr-shading.ts

export interface PbrSurfaceSample {
  readonly position: readonly [number, number, number];
  readonly normal: readonly [number, number, number];
  readonly albedo: readonly [number, number, number]; // 0..1
  readonly roughness?: number; // 0..1, default 0.7
  readonly metallic?: number; // 0..1, default 0.0
}

export function applyPbrShading(
  sample: PbrSurfaceSample,
  irradiance: IrradianceMap
): readonly [number, number, number];

Algorithms#

Sample SH at a point#

The Marble splat's per-Gaussian SH coefficients encode view-dependent radiance from each Gaussian. The irradiance at a point is the integral of incoming light from all directions over the upper hemisphere.

For block F v1 we approximate via gaussian-weighted aggregation of SH:

text
sampleSHAtPoint(scene, point, radius):
  weights = []
  contributions = []
  for each gaussian g in scene where distance(g.position, point) < radius:
    w = exp(-distance(g.position, point)^2 / radius^2) * g.opacity
    sh = g.shCoefficients  // 9 floats per channel for SH degree 2 (27 total)
    weights.append(w)
    contributions.append(sh * w)
  totalWeight = sum(weights)
  irradianceSH = sum(contributions) / totalWeight
  return irradianceSH

This is a coarse approximation — it treats each Gaussian's outgoing radiance as if it were incoming irradiance at the placement point. Good enough for plausible "tone match" but not physically accurate. Block F1.5 (future work) replaces with proper light-transport sampling via path tracing if visual QA demands.

Apply PBR shading#

For diffuse-only (no specular):

text
applyPbrShading(sample, irradiance):
  if irradiance.encoding === 'sh9':
    shInWorldFrame = rotateSHByNormal(irradiance.coefficients, sample.normal)
    diffuseIrradiance = evaluateSHDirection(shInWorldFrame, sample.normal)
    return sample.albedo * diffuseIrradiance
  else:
    // env-map path: sample the equirect map at the normal direction
    uv = sphereToEquirectUV(sample.normal)
    diffuseIrradiance = sampleEquirect(irradiance, uv)
    return sample.albedo * diffuseIrradiance

Specular contribution (block F v1.5):

text
  // For metals or low-roughness surfaces, also sample env map at the
  // reflection direction `reflect(viewDir, normal)`. Blend by Fresnel.

Composition coherence gate#

Distinct from input-grounding-gate — that one compares input image to output renders. Composition coherence compares regions within the same output render: do the baked-mesh regions look stylistically consistent with the environment regions?

text
evaluateCoherence(composedRenders, placements):
  for each render in composedRenders:
    // Project each placement's transform into screen space using the
    // render's camera; rasterise to get a mask of pixels covered by
    // baked mesh regions vs environment regions.
    meshRegionMask = rasterizePlacementMasks(render.camera, placements)
    envRegionMask  = NOT meshRegionMask
    // Extract per-region perceptual features via a small CNN/VLM adapter
    meshFeatures = adapter.extractFeatures(render, mask: meshRegionMask)
    envFeatures  = adapter.extractFeatures(render, mask: envRegionMask)
    // Score similarity in a perceptual feature space.
    score = cosineSimilarity(meshFeatures, envFeatures)
  return mean(scores)

decision(score):
  if score > 0.85: 'pass'
  if score > 0.7:  'soft-fail'
  else:             'hard-fail'

The adapter (ThreeDQualityCoherenceFeatureAdapter) is pluggable — production uses CLIP or DINO features; tests use a fixture adapter.

Test plan#

Unit (sample-sh-at-point.test.ts):

  • Synthetic scene: 1 Gaussian at origin with SH that encodes constant white irradiance. Sample at origin → returns the same SH coefficients.
  • Same scene, sample 10m away with radius = 2 → returns ~zero (Gaussian contribution falls off).
  • Two Gaussians at ±1m, opposing colors. Sample at midpoint → returns blend.

Unit (estimate-environment-light-field.test.ts):

  • 100-Gaussian env scene → estimateEnvironmentLightField returns SH9 with 27 coefficients; coefficients deterministic for fixed input.
  • Same scene encoded as env-map → returns 32×16 equirect map; brightest pixel direction matches the brightest Gaussian's direction.

Unit (apply-pbr-shading.test.ts):

  • Surface with albedo [1, 1, 1], normal [0, 1, 0], irradiance = uniform white SH → returns [π/π, π/π, π/π] = [1, 1, 1] (Lambertian irradiance integral over hemisphere = π × albedo for uniform white).
  • Same surface, albedo [1, 0, 0] → returns [1, 0, 0].
  • Normal pointing away from primary irradiance direction → diminished output.

Unit (composition-coherence-gate.test.ts):

  • Synthetic render with mesh-region features = env-region features (use fixture adapter that returns the same vector for both) → score = 1.0, decision = pass.
  • Adapter returns orthogonal vectors (cosine = 0) → decision = hard-fail.
  • Adapter returns vectors with cosine = 0.75 → decision = soft-fail.

Integration (bake-phase.test.ts extended):

  • Bake unit cube against synthetic env splat with directional light → baked Gaussian colours show clear directional shading (top brighter than bottom).
  • Compare against baked unit cube with lightingContext: undefined (block E baseline) → colours uniform; visual diff asserted via mean brightness variance.

Integration (composer end-to-end):

  • Full composer run with 1 mesh + lighting enabled → manifest's lightingMetadata.shCoefficients field populated; composed .spz loads in viewer; mesh region visually matches env tone (manual QA).

Acceptance criteria#

  • All new tests pass.
  • npx tsc --noEmit clean.
  • Stub-indicator scan clean.
  • Visual QA: a unit white cube placed in a known-warm-toned env splat shows warm tinting in the composed render (manual screenshot in PR).
  • Composition coherence gate evaluates a known-good composed scene (env-baked-with-lighting) at score > 0.85 and a known-bad scene (env without lighting) at score < 0.7. Validated on a small reference corpus.
  • The composer's manifest extension populates metadata.sceneFromImage.lightingMetadata correctly.

Open questions#

  • SH9 vs env-map default. SH9 is faster and smaller; env-map supports specular. Default for v1: SH9 (the meshes Hunyuan/Tripo emit are predominantly diffuse). Switch to env-map per-mesh if material has metallic > 0.5 or roughness < 0.2.
  • Light-transport approximation accuracy. The gaussian-weighted SH aggregation is coarse; visual QA may surface artifacts (shadows in wrong places, missing colour bleed). v1.5 fallback: path-traced irradiance samples via gsplat's differentiable renderer.
  • Coherence adapter sourcing. Production wants CLIP or DINO features. Both have public TF.js / ONNX models we could ship via onnxruntime-node (already an optional dep of 3d-generation). Recommended: ship a CLIP-ViT-B/32 adapter as the default.
  • Per-mesh vs scene-wide lighting. Estimating a separate EnvironmentLightField per mesh placement is more accurate but slower. v1 estimates per-mesh; v1.5 may share if placements are clustered.