Eve library

@isis/ai-texturing

Other

Revision 1

On this page

AI-native PBR texture generation and refinement for Isis 3D pipelines.

71.2.1.2 is now implemented with a dedicated standalone Hunyuan3D-Paint surface for applying PBR textures to arbitrary existing meshes, not just meshes generated by Hunyuan shape pipelines.

Implemented now#

  • standalone Hunyuan3D-Paint texture application on arbitrary obj, ply, or glb meshes
  • image-only, text-only, and image-and-text conditioning modes
  • dedicated text-guided PBR generation with prompt composition and required-map validation for albedo, normal, metallic, and roughness
  • dedicated image-guided PBR generation that applies a reference material photo to a mesh and validates the same required PBR map set
  • dedicated multi-view consistency workflow with profile-driven view counts, coverage analysis, and seam-risk reporting
  • honest texture-resolution delivery from 512 through 8192, with native Hunyuan baking up to 4096 and an explicit post-scale + GLB rebuild path for 8192
  • automatic texture completeness validation over the final baked output, including required-map presence, UV island detection, painted-coverage scoring, and warnings for islands that still contain only background data
  • dedicated re-texturing workflow for existing obj and glb assets that forces no-remesh operation, replaces the baked texture set, and blocks output if geometry or UV layout changed
  • depth-conditioned texture painting workflow that rasterizes real mesh depth views, passes them to an injected depth-ControlNet-style backend, and projects the generated view back into UV space as an albedo atlas with paint coverage reporting
  • lighting-disentangled texture generation that estimates a smooth illumination field over the visible mesh surface, flattens baked shadows/highlights in the generated view, and bakes the neutralized albedo plus illumination map back into UV space
  • iterative multi-view texture painting that reprojects already painted texels into each new canonical view, builds visible-surface-only inpaint masks for the remaining unseen regions, and skips redundant passes when a repeated view has nothing left to fill
  • cross-attention view-consistency enforcement for iterative painting, with geometry-derived overlap planning that supplies reference views, target-attention masks, and pixel correspondences for later passes
  • seam-aware UV-boundary blending that detects duplicated geometric edges mapped to split UV islands, smooths color discontinuities across the seam, and emits a seam mask plus blend telemetry
  • local tangent-space normal-map synthesis that combines projected geometry depth with albedo high-frequency detail and supports both OpenGL and DirectX green-channel conventions
  • local material-property estimation that infers roughness, metallic, and packed metallicRoughness maps from the synthesized albedo and normal detail using canonical metal-color priors plus texture-statistics inference
  • local ambient-occlusion baking that ray-samples the real mesh geometry in UV space and emits an ao map aligned to the baked atlas
  • local displacement-map synthesis that combines projected depth structure with diffuse high-frequency detail to emit a normalized displacement atlas
  • optional UV-island atlas optimization that repacks the existing layout before painting so the same texture resolution yields higher texel utilization
  • text-to-material-graph generation that converts a material description into a typed Substance-style procedural node graph with exposed parameters, layered coating/weathering support, and explicit output-channel bindings
  • material-graph parameter extraction that turns exposed graph inputs into a high-level tweakable control catalog with semantic labels, suggested ranges, UI groups, and output-channel impact
  • material-graph tiling control that rewrites procedural graph sources with managed periodic transforms, exposed repeat/offset/mirror controls, and an honest seamlessness report
  • deterministic material-graph variation generation that can emit curated sets such as 10 weathered concrete variants with different stain distributions, crack densities, and parameter overrides
  • material blending recipes that merge generated procedural materials such as mossy stone via an explicit height-derived cavity mask and rebaked normal/AO outputs
  • environment-oriented material library builder that batch-generates curated urban, forest, desert, sci-fi, and medieval material sets using a mix of base graphs, variations, and recipe entries
  • deterministic Substance Designer .sbs export that converts generated material graphs into XML package documents with compositing nodes, exposed parameter inputs, graph outputs, and stable node/output UIDs
  • OpenUSD-ready MaterialX export that emits both a .mtlx procedural material document and a companion .usda wrapper referencing it through usdMtlx
  • PBR-aware texture super-resolution with channel-specific processing: gamma-aware albedo detail reconstruction, tangent-space normal renormalization, scalar-preserving roughness upscaling, metallic coverage stabilization, and glTF-packed metallic-roughness channel splitting/repacking
  • PBR-aware texture denoising with artifact-adaptive block suppression, microdetail residual reinjection, vector-safe normal cleanup, histogram-preserving roughness filtering, metallic coverage preservation, and glTF-packed metallic-roughness channel splitting/repacking
  • PBR-aware texture detail synthesis with seeded material-specific microdetail generation for pores, grain, scratches, and stone breakup while preserving UV-authored structure, normal safety, and metallic coverage
  • PBR-aware texture compression optimization that preconditions albedo, normal, roughness, metallic, and packed metallic-roughness maps for BC7 / ASTC / ETC2 block encoders, emits codec-ready KTX2 preparation manifests, and reports surrogate compressibility gain plus GPU-memory estimates
  • PBR-aware texture style transfer with reference-driven albedo statistics transfer, structure-aware detail preservation, estimated roughness/metallic fallback from the material photo, and glTF-packed channel support
  • UDIM-aware texture processing that resolves <UDIM> and embedded numeric tile patterns, runs super-resolution / denoising / detail synthesis / style transfer per tile, supports heterogeneous tile resolutions, and harmonizes cross-tile seams for high-detail character and environment layouts
  • AI-based texture baking that distills high-poly surface normals into a low-poly tangent-space normal map through shared-frustum soft rasterization, multi-view confidence fusion, UV-space refinement, and explicit non-raycast coverage / confidence outputs
  • textured obj and glb outputs with albedo / metallic / roughness / metallic-roughness / normal / ambient-occlusion / displacement map passthrough
  • runtime preparation hooks and package-level capability manifest

Usage#

ts
import { Local3DModelRegistry } from '@isis/3d-inference-local';
import { createLocal3DHunyuan3DPaintStandaloneTexturingPipeline } from '@isis/ai-texturing';

const registry = new Local3DModelRegistry({
  rootDir: '/models/registry',
  hub: {},
  cachePolicy: {
    maxDiskUsageGb: 500,
    minFreeDiskGb: 25,
  },
});

const pipeline = createLocal3DHunyuan3DPaintStandaloneTexturingPipeline({
  registry,
  runtimeRoot: '/opt/hunyuan3d-2.1',
});

const result = await pipeline.run({
  mesh: await fs.readFile('/assets/chair.glb'),
  meshFormat: 'glb',
  referenceImage: await fs.readFile('/refs/worn-leather.png'),
  prompt: 'aged brown leather with subtle stitching',
  outputFormat: 'glb',
  textureResolution: 8192,
});

result includes the textured output mesh path, preview image path, texture-map paths, provider-style completion assets, runtime asset locations, warnings, and standalone workflow metadata. When textureResolution is 8192, metadata also records that delivery used a native 4096 Hunyuan bake followed by a 2x post-scale step. The same metadata now also includes textureCompletenessReport, textureCompletenessValid, texturePaintedCoverage, and untexturedIslandCount so downstream workflows can reject or review partially baked outputs instead of silently shipping them.

PBR Texture Denoising#

ts
import { denoiseLocal3DAiTexturingPbrTextures } from '@isis/ai-texturing';

const result = await denoiseLocal3DAiTexturingPbrTextures({
  mapPaths: {
    albedo: '/assets/albedo-blocky.png',
    normal: '/assets/normal-blocky.png',
    metallicRoughness: '/assets/metallic-roughness-blocky.png',
  },
  artifactProfile: 'balanced',
  strength: 0.85,
});

This workflow preserves each source map at its original resolution, rewrites the output as lossless PNG, and returns per-map reports for blockiness reduction, retained detail energy, and mean absolute texture delta so you can review cleanup quality rather than assuming the denoiser was safe.

PBR Texture Detail Synthesis#

ts
import { synthesizeLocal3DAiTexturingPbrTextureDetails } from '@isis/ai-texturing';

const result = await synthesizeLocal3DAiTexturingPbrTextureDetails({
  mapPaths: {
    albedo: '/assets/low-detail-wood-albedo.png',
    normal: '/assets/low-detail-wood-normal.png',
    roughness: '/assets/low-detail-wood-roughness.png',
  },
  detailProfile: 'wood',
  strength: 0.74,
  seed: 42,
});

This workflow preserves each source map at its original resolution, adds deterministic UV-space microdetail using profile-specific procedural synthesis, and returns per-map reports for detail gain, UV-detail preservation, mean absolute delta, and metallic-coverage drift where applicable so you can review whether the added grain, pores, or scratches stayed physically plausible.

PBR Texture Style Transfer#

ts
import { transferLocal3DAiTexturingPbrTextureStyle } from '@isis/ai-texturing';

const result = await transferLocal3DAiTexturingPbrTextureStyle({
  mapPaths: {
    albedo: '/assets/cool-steel-albedo.png',
    normal: '/assets/cool-steel-normal.png',
    metallicRoughness: '/assets/cool-steel-metallic-roughness.png',
  },
  styleReferenceImagePath: '/refs/brushed-copper-material.jpg',
  transferProfile: 'balanced',
  strength: 0.74,
});

This workflow preserves each source map at its original resolution, applies reference-material style using AdaIN-like low-frequency transfer plus content-residual preservation, and derives fallback roughness/metallic style targets from the reference photo when explicit style PBR maps are not available. The result returns per-map reports for style-match improvement, UV-detail preservation, mean absolute delta, and metallic coverage drift so you can verify that the style landed without flattening authored UV detail.

Retexturing#

ts
import { createLocal3DHunyuan3DPaintRetexturingPipeline } from '@isis/ai-texturing';

const pipeline = createLocal3DHunyuan3DPaintRetexturingPipeline({
  registry,
  runtimeRoot: '/opt/hunyuan3d-2.1',
});

const result = await pipeline.run({
  mesh: await readFile('/assets/robot.glb'),
  meshFormat: 'glb',
  referenceImage: await readFile('/refs/anodized-blue-metal.jpg'),
  materialPrompt: 'anodized cobalt blue aluminum with subtle machining marks',
  textureResolution: 4096,
});

This workflow is stricter than generic standalone texturing. It always runs with useRemesh: false, validates the output against the source mesh, and fails if vertex positions, face topology, or UV layout changed. The result metadata includes retextureMode, preservationReport, geometryPreserved, uvLayoutPreserved, replacedTextureMaps, and the same texture completeness report already exposed by the standalone workflow.

Depth-Conditioned Texture Painting#

ts
import { createLocal3DHunyuan3DPaintDepthConditionedTexturePaintingPipeline } from '@isis/ai-texturing';

const pipeline =
  createLocal3DHunyuan3DPaintDepthConditionedTexturePaintingPipeline({
    backend: {
      async generateView({ depthMap, prompt }) {
        return {
          image: await someDepthControlBackend(prompt, depthMap),
          model: 'sd15-depth-controlnet',
        };
      },
    },
  });

const result = await pipeline.run({
  mesh: await readFile('/assets/helmet.obj'),
  meshFormat: 'obj',
  prompt: 'worn painted steel with chipped edges',
  viewpoint: 'front',
  renderWidth: 512,
  renderHeight: 512,
  textureResolution: 1024,
});

This workflow is intentionally honest about its current role in the stack. It performs the real mesh-to-depth rendering and UV reprojection locally, but it expects an injected depth-conditioned diffusion backend for the image synthesis step. The output includes a rendered depth map, the backend-generated view, the projected albedo texture, a paint mask, an occupied-texel mask, and coverage metadata so later multi-view passes can target the remaining unseen texels. After UV projection, the local pipeline also runs seam-aware blending over duplicated UV boundaries and emits a seam-mask texture plus quantitative seam-blending metadata. It also synthesizes a tangent-space normal map from the projected depth atlas and the final albedo detail signal so downstream PBR consumers are not left with a flat material response. The same local stack now also estimates roughness, metallic, and packed metallicRoughness maps from the generated albedo and normal detail so the workflow emits a usable PBR set instead of albedo alone. The same baked atlas workspace now also ray-bakes an ao map from the mesh geometry, so cavity shading tracks real self-occlusion instead of image-space darkening. That same atlas workspace also derives a normalized displacement map by combining the projected depth field with the albedo's high-frequency relief signal so downstream renderers can recover macro and micro surface variation from the generated texture set. When optimizeTextureAtlas is enabled, the pipeline first repacks the mesh's existing UV islands before any rendering or baking work begins, then writes an optimized-mesh artifact plus atlas-usage telemetry alongside the painted maps so the higher-utilization texture set has a matching mesh.

Lighting-Disentangled Texture Generation#

ts
import { createLocal3DHunyuan3DPaintLightingDisentangledTextureGenerationPipeline } from '@isis/ai-texturing';

const pipeline =
  createLocal3DHunyuan3DPaintLightingDisentangledTextureGenerationPipeline({
    backend: {
      async generateView({ prompt, depthMap }) {
        return {
          image: await someDepthControlBackend(prompt, depthMap),
          model: 'sd15-depth-controlnet',
        };
      },
    },
  });

const result = await pipeline.run({
  mesh: await readFile('/assets/helmet.obj'),
  meshFormat: 'obj',
  prompt: 'worn painted steel with chipped edges',
  viewpoint: 'front',
  renderWidth: 512,
  renderHeight: 512,
  textureResolution: 1024,
  strength: 0.95,
});

This workflow starts from the same real depth-conditioned rendering path, but it runs a local illumination-estimation pass over the backend-generated view before UV projection. The visible-surface luminance is box-blurred into a smooth illumination field, then shadow and highlight deltas are compressed so the baked result trends toward flat intrinsic albedo instead of carrying view lighting. The output includes the original generated view, a lighting-neutral view, an illumination-map, the baked albedo texture, and metadata describing the measured shading range plus the applied flattening strength. The final baked albedo then passes through the same seam-aware UV blending stage, producing a seam-mask and seam-refinement telemetry alongside the lighting outputs. A derived tangent-space normal map is also emitted by combining the projected mesh depth field with the lighting-neutral albedo detail. That normal map then feeds local material-property inference, which estimates roughness, metallic, and packed metallicRoughness outputs from the final albedo/normal pair. The same UV-space geometry pass now also emits an ao texture derived from actual mesh occlusion before export. That lighting-neutral atlas then also feeds local displacement synthesis, which blends the projected depth structure with detail extracted from the flattened albedo into a normalized displacement map. The same optional optimizeTextureAtlas prepass is available here too, so lighting-neutral bakes can target a repacked UV layout before any ControlNet rendering or UV projection occurs.

Iterative Multi-View Painting#

ts
import { createLocal3DHunyuan3DPaintIterativeMultiViewTexturePaintingPipeline } from '@isis/ai-texturing';

const pipeline =
  createLocal3DHunyuan3DPaintIterativeMultiViewTexturePaintingPipeline({
    backend: {
      async generateView({ prompt, depthMap, baseView, inpaintMask }) {
        return {
          image: await someDepthAwareInpaintBackend({
            prompt,
            depthMap,
            baseView,
            inpaintMask,
          }),
          model: 'sdxl-depth-inpaint',
        };
      },
    },
  });

const result = await pipeline.run({
  mesh: await readFile('/assets/helmet.obj'),
  meshFormat: 'obj',
  prompt: 'worn painted steel with chipped edges',
  viewSequence: ['front', 'right', 'back', 'left', 'top', 'bottom'],
  renderWidth: 512,
  renderHeight: 512,
  textureResolution: 1024,
});

This workflow extends the single-view depth-conditioned path into a real front-to-back accumulation loop. Each pass reprojects the current atlas into the next camera view, computes an inpaint mask only over visible mesh pixels that still have no painted texel assignment, calls the injected backend with depthMap + baseView + inpaintMask, and projects the generated result back into UV space. The result metadata exposes per-pass missingPixelCount, newlyPaintedTexelCount, cumulativeCoverageRatio, and skippedGeneration flags so downstream orchestration can see exactly where coverage increased and which repeated views became redundant.

By default, the iterative workflow now also enables consistencyMode: 'cross-attention'. Before each later pass, it computes real overlap between the current visible texels and the texels owned by previous passes, selects the strongest prior views, and passes crossAttentionReferences into the backend. Each reference includes the prior generated image, its coverage mask, a target-view attention mask for overlapping pixels, and a dense referencePixelIndexByTargetPixel mapping so a diffusion backend can enforce view consistency using actual UV-mediated correspondences instead of a prompt hint. Once the atlas is complete, the pipeline also performs seam-aware UV boundary blending across duplicated geometric edges and records the resulting seam-mask, seam texel counts, and average blend delta in the final metadata. The same final atlas then feeds local normal-map synthesis so the completed multi-view result ships with a generated tangent-space normal texture instead of albedo alone. The finalized albedo + normal pair also feeds local material-property inference so iterative output now includes generated roughness, metallic, and packed metallicRoughness textures. That shared UV workspace also now bakes an ao texture from geometry, so the final atlas set carries per-texel ambient occlusion without requiring a separate DCC bake. The completed diffuse atlas and accumulated projected depth coverage also feed a local displacement synthesis pass, so the iterative workflow now exports a normalized height field alongside the rest of the PBR atlas set. Enabling optimizeTextureAtlas applies the same pre-pass UV-island repack before the iterative view loop begins, and the returned metadata includes before/after utilization plus texel-density deltas for the optimized layout.

Text To Material Graph#

ts
import {
  applyLocal3DAiTexturingMaterialGraphTilingControl,
  extractLocal3DAiTexturingMaterialGraphParameters,
  generateLocal3DAiTexturingTextToMaterialGraph,
} from '@isis/ai-texturing';

const graphResult = generateLocal3DAiTexturingTextToMaterialGraph({
  prompt: 'weathered painted steel with chipped red paint and rust streaks',
});
const tiledGraphResult = applyLocal3DAiTexturingMaterialGraphTilingControl({
  graph: graphResult.graph,
  targetScale: 3.5,
  repeatX: 4,
  repeatY: 2,
});
const parameterResult = extractLocal3DAiTexturingMaterialGraphParameters({
  graph: tiledGraphResult.graph,
});

This workflow emits a procedural graph description instead of baked textures. The result includes prompt analysis, detected material family, a typed Substance-style node/edge graph, layered material-stack metadata, exposed tweak parameters such as tile_scale, base_color_primary, roughness_amount, wear_amount, and optional coating/contaminant controls, plus explicit output bindings for channels like baseColor, roughness, metallic, normal, height, ao, opacity, transmission, and emissive when the prompt implies them. The internal graph model follows the same node-graph concepts used by MaterialX and Substance Designer so later .sbs and MaterialX export tasks can target a real structured graph rather than a lossy prompt summary. The parameter extractor then walks the graph topology to identify the practical artist-facing controls, groups them into buckets like Surface/Color, Pattern/Scale, and Weathering/Primary, and records which output channels each control actually influences. The tiling-control pass can then rewrite the graph into a seamless periodic variant by inserting managed transform-2d nodes after every procedural pattern source, exposing repeat/offset/rotation and mirror controls, and returning a report with per-axis coverage plus an estimated seamlessness score for the tiled outputs.

ts
import { generateLocal3DAiTexturingMaterialGraphVariations } from '@isis/ai-texturing';

const variationSet = generateLocal3DAiTexturingMaterialGraphVariations({
  prompt: 'weathered concrete',
  count: 10,
});

The variation generator builds deterministic prompt-backed variants on top of the same graph workflow, then injects real stain overlays and crack tuning into each graph so the output set spans different runoff/blotchy/speckled stain patterns, crack profiles, roughness levels, and tile scales instead of merely relabeling one underlying graph.

ts
import { buildLocal3DAiTexturingMaterialBlendRecipe } from '@isis/ai-texturing';

const recipe = buildLocal3DAiTexturingMaterialBlendRecipe({
  recipeId: 'mossy-stone',
});

The recipe builder composes separately generated base and overlay graphs, imports the overlay graph with isolated parameter namespaces, derives a real blend mask from the base height field plus overlay growth structure, blends the main PBR channels, and regenerates normal plus ao from the merged height so the resulting material graph stays internally coherent.

ts
import { buildLocal3DAiTexturingMaterialLibrary } from '@isis/ai-texturing';

const library = buildLocal3DAiTexturingMaterialLibrary({
  environments: ['urban', 'medieval'],
  variationsPerMaterial: 2,
});

The library builder turns those lower-level graph tools into environment packs: it batches base materials, variation sets, and recipe-driven blends into tagged catalogs that can seed urban streets, forest floors, desert ruins, sci-fi hard-surface scenes, or medieval environments with deterministic graph assets rather than ad hoc prompt lists.

Text-Guided PBR#

ts
import { createLocal3DHunyuan3DPaintTextGuidedPbrPipeline } from '@isis/ai-texturing';

const pipeline = createLocal3DHunyuan3DPaintTextGuidedPbrPipeline({
  registry,
  runtimeRoot: '/opt/hunyuan3d-2.1',
});

const result = await pipeline.run({
  mesh: await readFile('/assets/statue.glb'),
  meshFormat: 'glb',
  materialDescriptor: {
    material: 'weathered bronze',
    finish: 'satin',
    wear: 'edge patina and subtle scratches',
    microDetail: 'fine cast metal grain',
  },
  negativeMaterialPrompt: 'plastic sheen',
  textureResolution: 2048,
});

This workflow is stricter than generic standalone texturing: it composes a material-focused prompt and fails fast if any of the required PBR maps are missing from the result.

Image-Guided PBR#

ts
import { createLocal3DHunyuan3DPaintImageGuidedPbrPipeline } from '@isis/ai-texturing';

const pipeline = createLocal3DHunyuan3DPaintImageGuidedPbrPipeline({
  registry,
  runtimeRoot: '/opt/hunyuan3d-2.1',
});

const result = await pipeline.run({
  mesh: await readFile('/assets/chair.glb'),
  meshFormat: 'glb',
  referenceImage: await readFile('/refs/distressed-leather.jpg'),
  materialHint: 'retain the stitched leather look and dark creases',
  textureResolution: 2048,
});

This workflow requires a real reference image and fails fast if the returned map set is missing albedo, normal, metallic, or roughness.

Multi-View Consistency#

ts
import { createLocal3DHunyuan3DPaintMultiViewConsistencyPipeline } from '@isis/ai-texturing';

const pipeline = createLocal3DHunyuan3DPaintMultiViewConsistencyPipeline({
  registry,
  runtimeRoot: '/opt/hunyuan3d-2.1',
});

const result = await pipeline.run({
  mesh: await readFile('/assets/vase.glb'),
  meshFormat: 'glb',
  prompt: 'hand-painted ceramic glaze with subtle edge wear',
  consistencyProfile: 'maximum',
  textureResolution: 2048,
});

This workflow still uses the real Hunyuan multiview bake path, but it adds a coverage report over the selected front/back/side/top/bottom views and reports a computed seam-risk score instead of silently assuming the bake was consistent.

Notes#

  • The upstream Hunyuan3D-Paint release is fundamentally a mesh texturing stage, so this package wraps the existing local runtime instead of reimplementing it.
  • The official quick-start centers on mesh + reference image -> textured mesh; this package also supports text-only by delegating to the existing local wrapper, which uses the same neutral anchor fallback already implemented in @isis/3d-inference-local.