Owner library: libs/isis/gaussian-splatting Estimate: ~1–2 weeks
Blocks: none structurally; without it composed scenes look obviously
"flat-lit" against the env. Dependencies: block E (need baked-Gaussian path
to inject lighting into); block D (need the composed .spz to evaluate
against). Status: not started
Overview#
Marble bakes radiance (lighting × albedo) into per-Gaussian SH coefficients; Hunyuan/Tripo/Meshy emit albedo + PBR factors that have never been lit by the scene. Naively rasterising meshes into Gaussians using their own albedo produces meshes that look flat against an environment with directional light, ambient occlusion, and color cast.
Block F adds environment-aware lighting:
- Sample the Marble splat's SH coefficients at each mesh's placement point to derive a local irradiance environment map.
- Apply that lighting to the mesh's PBR shading at sample time, before fitting Gaussians (block E's surface-sampling step consumes lit colour instead of raw albedo).
- Composition coherence gate — a new QA gate in
libs/isis/3d-quality-gates/src/delivery/that scores style consistency between baked-mesh regions and environment regions in the final composed renders.
Files to create#
Lighting module#
libs/isis/gaussian-splatting/src/lighting/— new subdirectory.libs/isis/gaussian-splatting/src/lighting/index.tslibs/isis/gaussian-splatting/src/lighting/types.ts—EnvironmentLightField,IrradianceMap,LightingEstimateOptions.libs/isis/gaussian-splatting/src/lighting/sample-sh-at-point.ts— given aLoadedScenewith SH and a 3D point, sample the SH-encoded irradiance.libs/isis/gaussian-splatting/src/lighting/sample-sh-at-point.test.tslibs/isis/gaussian-splatting/src/lighting/estimate-environment-light-field.ts— given aLoadedScene(env splat) and a target placement point, compute anIrradianceMap(low-resolution env map, e.g. 32×16 equirect, or 9-coefficient SH irradiance representation).libs/isis/gaussian-splatting/src/lighting/estimate-environment-light-field.test.tslibs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.ts— given a sample point + normal + material params +IrradianceMap, compute the lit colour at that point.libs/isis/gaussian-splatting/src/lighting/apply-pbr-shading.test.ts
Composition coherence gate#
libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.ts— evaluates a composed scene's renders for style consistency between baked-mesh regions and environment regions.libs/isis/3d-quality-gates/src/delivery/composition-coherence-gate.test.ts
Files to modify#
libs/isis/gaussian-splatting/src/bake/surface-sampling/sample-to-gaussian.ts:- Accept an optional
IrradianceMapparameter; when supplied, replace rawsampleTexture(...)colour withapplyPbrShading(...).
- Accept an optional
libs/isis/gaussian-splatting/src/bake/surface-sampling/sampling-baker.ts:- Accept
lightingContext?: { irradianceMap }; pass through tosample-to-gaussian.
- Accept
libs/isis/gaussian-splatting/src/bake/index.ts:bakeMeshaccepts a new optionallightingContextfield.
libs/isis/scene-from-image-composer/src/phases/bake-phase.ts:- Before each mesh bake, call
estimateEnvironmentLightField(envScene, placementPoint)and pass the resultingIrradianceMapintobakeMesh.
- Before each mesh bake, call
libs/isis/scene-from-image-composer/src/phases/grounding-phase.ts:- Add a post-bake grounding subphase that runs the composition coherence gate after bake completes.
libs/isis/3d-quality-gates/src/delivery/index.ts:- Re-export the coherence gate.
libs/isis/3d-quality-gates/src/index.ts:- Re-export.
Public API surface#
// lighting/types.ts
/**
* Low-resolution irradiance representation. Two encodings supported:
* - 'sh9': 9 coefficients per channel, 27 floats total. Compact, fast
* lookup (single matrix multiply). Good for diffuse-only.
* - 'env-map': low-res equirectangular env map. Larger (32*16*3 = 1536
* floats), supports specular reflections.
*/
export type IrradianceMap =
| { readonly encoding: 'sh9'; readonly coefficients: Float32Array /* 27 */ }
| {
readonly encoding: 'env-map';
readonly width: number;
readonly height: number;
readonly data: Float32Array;
};
export interface EnvironmentLightField {
readonly origin: readonly [number, number, number];
readonly map: IrradianceMap;
readonly metadata: {
readonly sourceGaussianCount: number;
readonly sampledRadiusMeters: number;
readonly estimationMethod: 'gaussian-weighted-sh' | 'voxel-aggregate';
};
}
// lighting/estimate-environment-light-field.ts
export interface LightingEstimateOptions {
readonly encoding?: 'sh9' | 'env-map'; // default 'sh9'
readonly samplingRadiusMeters?: number; // default 2.0
readonly maxSamples?: number; // default 1000
}
export function estimateEnvironmentLightField(
scene: LoadedScene,
placementPoint: readonly [number, number, number],
options?: LightingEstimateOptions
): EnvironmentLightField;
// lighting/apply-pbr-shading.ts
export interface PbrSurfaceSample {
readonly position: readonly [number, number, number];
readonly normal: readonly [number, number, number];
readonly albedo: readonly [number, number, number]; // 0..1
readonly roughness?: number; // 0..1, default 0.7
readonly metallic?: number; // 0..1, default 0.0
}
export function applyPbrShading(
sample: PbrSurfaceSample,
irradiance: IrradianceMap
): readonly [number, number, number];
Algorithms#
Sample SH at a point#
The Marble splat's per-Gaussian SH coefficients encode view-dependent radiance from each Gaussian. The irradiance at a point is the integral of incoming light from all directions over the upper hemisphere.
For block F v1 we approximate via gaussian-weighted aggregation of SH:
sampleSHAtPoint(scene, point, radius):
weights = []
contributions = []
for each gaussian g in scene where distance(g.position, point) < radius:
w = exp(-distance(g.position, point)^2 / radius^2) * g.opacity
sh = g.shCoefficients // 9 floats per channel for SH degree 2 (27 total)
weights.append(w)
contributions.append(sh * w)
totalWeight = sum(weights)
irradianceSH = sum(contributions) / totalWeight
return irradianceSH
This is a coarse approximation — it treats each Gaussian's outgoing radiance as if it were incoming irradiance at the placement point. Good enough for plausible "tone match" but not physically accurate. Block F1.5 (future work) replaces with proper light-transport sampling via path tracing if visual QA demands.
Apply PBR shading#
For diffuse-only (no specular):
applyPbrShading(sample, irradiance):
if irradiance.encoding === 'sh9':
shInWorldFrame = rotateSHByNormal(irradiance.coefficients, sample.normal)
diffuseIrradiance = evaluateSHDirection(shInWorldFrame, sample.normal)
return sample.albedo * diffuseIrradiance
else:
// env-map path: sample the equirect map at the normal direction
uv = sphereToEquirectUV(sample.normal)
diffuseIrradiance = sampleEquirect(irradiance, uv)
return sample.albedo * diffuseIrradiance
Specular contribution (block F v1.5):
// For metals or low-roughness surfaces, also sample env map at the
// reflection direction `reflect(viewDir, normal)`. Blend by Fresnel.
Composition coherence gate#
Distinct from input-grounding-gate — that one compares input image to output renders. Composition coherence compares regions within the same output render: do the baked-mesh regions look stylistically consistent with the environment regions?
evaluateCoherence(composedRenders, placements):
for each render in composedRenders:
// Project each placement's transform into screen space using the
// render's camera; rasterise to get a mask of pixels covered by
// baked mesh regions vs environment regions.
meshRegionMask = rasterizePlacementMasks(render.camera, placements)
envRegionMask = NOT meshRegionMask
// Extract per-region perceptual features via a small CNN/VLM adapter
meshFeatures = adapter.extractFeatures(render, mask: meshRegionMask)
envFeatures = adapter.extractFeatures(render, mask: envRegionMask)
// Score similarity in a perceptual feature space.
score = cosineSimilarity(meshFeatures, envFeatures)
return mean(scores)
decision(score):
if score > 0.85: 'pass'
if score > 0.7: 'soft-fail'
else: 'hard-fail'
The adapter (ThreeDQualityCoherenceFeatureAdapter) is pluggable — production
uses CLIP or DINO features; tests use a fixture adapter.
Test plan#
Unit (sample-sh-at-point.test.ts):
- Synthetic scene: 1 Gaussian at origin with SH that encodes constant white irradiance. Sample at origin → returns the same SH coefficients.
- Same scene, sample 10m away with
radius = 2→ returns ~zero (Gaussian contribution falls off). - Two Gaussians at ±1m, opposing colors. Sample at midpoint → returns blend.
Unit (estimate-environment-light-field.test.ts):
- 100-Gaussian env scene →
estimateEnvironmentLightFieldreturns SH9 with 27 coefficients; coefficients deterministic for fixed input. - Same scene encoded as
env-map→ returns 32×16 equirect map; brightest pixel direction matches the brightest Gaussian's direction.
Unit (apply-pbr-shading.test.ts):
- Surface with albedo
[1, 1, 1], normal[0, 1, 0], irradiance = uniform white SH → returns[π/π, π/π, π/π] = [1, 1, 1](Lambertian irradiance integral over hemisphere = π × albedo for uniform white). - Same surface, albedo
[1, 0, 0]→ returns[1, 0, 0]. - Normal pointing away from primary irradiance direction → diminished output.
Unit (composition-coherence-gate.test.ts):
- Synthetic render with mesh-region features = env-region features (use fixture
adapter that returns the same vector for both) → score = 1.0, decision =
pass. - Adapter returns orthogonal vectors (cosine = 0) → decision =
hard-fail. - Adapter returns vectors with cosine = 0.75 → decision =
soft-fail.
Integration (bake-phase.test.ts extended):
- Bake unit cube against synthetic env splat with directional light → baked Gaussian colours show clear directional shading (top brighter than bottom).
- Compare against baked unit cube with
lightingContext: undefined(block E baseline) → colours uniform; visual diff asserted via mean brightness variance.
Integration (composer end-to-end):
- Full composer run with 1 mesh + lighting enabled → manifest's
lightingMetadata.shCoefficientsfield populated; composed.spzloads in viewer; mesh region visually matches env tone (manual QA).
Acceptance criteria#
- All new tests pass.
npx tsc --noEmitclean.- Stub-indicator scan clean.
- Visual QA: a unit white cube placed in a known-warm-toned env splat shows warm tinting in the composed render (manual screenshot in PR).
- Composition coherence gate evaluates a known-good composed scene (env-baked-with-lighting) at score > 0.85 and a known-bad scene (env without lighting) at score < 0.7. Validated on a small reference corpus.
- The composer's manifest extension populates
metadata.sceneFromImage.lightingMetadatacorrectly.
Open questions#
- SH9 vs env-map default. SH9 is faster and smaller; env-map supports
specular. Default for v1: SH9 (the meshes Hunyuan/Tripo emit are predominantly
diffuse). Switch to env-map per-mesh if material has
metallic > 0.5orroughness < 0.2. - Light-transport approximation accuracy. The gaussian-weighted SH
aggregation is coarse; visual QA may surface artifacts (shadows in wrong
places, missing colour bleed). v1.5 fallback: path-traced irradiance samples
via
gsplat's differentiable renderer. - Coherence adapter sourcing. Production wants CLIP or DINO features. Both
have public TF.js / ONNX models we could ship via
onnxruntime-node(already an optional dep of3d-generation). Recommended: ship a CLIP-ViT-B/32 adapter as the default. - Per-mesh vs scene-wide lighting. Estimating a separate
EnvironmentLightFieldper mesh placement is more accurate but slower. v1 estimates per-mesh; v1.5 may share if placements are clustered.