Eve library

@isis/3d-semantic-editing

Other

Revision 1

On this page

Semantic part decomposition, mask management, and localized edit contracts for the Isis 3D toolchain.

This package is the Phase 71 home for:

  • hierarchical part trees spanning asset, assembly, subassembly, part, and material-region scopes
  • instance-aware semantic labels for mirrored and repeated components
  • mesh-space, UV-space, and texture-space editable masks
  • localized edit session state for non-destructive refinement workflows

The initial scaffold intentionally keeps the runtime small while establishing a stable Nx package boundary, public entrypoint, and core type system for the follow-on 71.11.* tasks.

Semantic Part Segmentation Pipeline#

Local3DSemanticPartSegmentationPipeline provides the first semantic decomposition pass for:

  • characters
  • props
  • vehicles
  • architecture
  • furniture
  • creatures

The current implementation is intentionally deterministic and local:

  • it reuses @isis/universal-rigging mesh summarization to infer a stable semantic frame from extents and symmetry
  • it applies PartNet-inspired family templates over mesh-space face centroids to produce hierarchical part trees and per-part face/vertex assignments
  • it emits confidence scores, normalized bounds, and warnings when required family-specific parts cannot be isolated cleanly
  • it now derives instance-aware labels for mirrored and repeated components so later localized-edit workflows can distinguish front-left-wheel, rear-right-wheel, left-arm, or repeated indexed parts instead of only the coarse semantic class
ts
import { createLocal3DSemanticPartSegmentationPipeline } from '@isis/3d-semantic-editing';

const pipeline = createLocal3DSemanticPartSegmentationPipeline();

const result = pipeline.run({
  familyHint: 'vehicle',
  mesh,
});

console.log(result.family);
console.log(result.segments.map((segment) => segment.id));
console.log(result.assembly.rootId);

Segmentation Confidence Scoring#

scoreLocal3DSemanticSegmentationConfidence grades a segmentation pass before localized editing or metadata export. It reuses the emitted per-part confidence, coverage, connectivity, hierarchy completeness, and mirrored-part balance to produce:

  • per-segment confidence scores with issue codes and readiness tiers
  • aggregate region scores over the semantic assembly tree
  • low-confidence highlighted regions with face/vertex membership and normalized bounds
  • manual correction recommendations for missing required parts, fragmented regions, and oversized fallback buckets
ts
import {
  createLocal3DSemanticPartSegmentationPipeline,
  scoreLocal3DSemanticSegmentationConfidence,
} from '@isis/3d-semantic-editing';

const pipeline = createLocal3DSemanticPartSegmentationPipeline();
const segmentation = pipeline.run({
  familyHint: 'vehicle',
  mesh,
});

const confidenceReport =
  scoreLocal3DSemanticSegmentationConfidence(segmentation);

console.log(confidenceReport.overallConfidence);
console.log(confidenceReport.highlightedRegions.map((region) => region.label));
console.log(
  confidenceReport.manualCorrectionRecommendations[0]?.recommendedActions
);

Hierarchical Part Trees#

createLocal3DHierarchicalPartTree turns the flat assembly index emitted by segmentation into one normalized traversal model for downstream editing.

It provides:

  • normalized levels across asset -> assembly -> subassembly -> part -> material-region
  • stable depth-first ordering and ancestry paths for edit UIs and exporters
  • descendant part/material lookup without recomputing subtree membership
  • optional explicit or implicit material-region leaves attached beneath parts
ts
import {
  createLocal3DHierarchicalPartTree,
  materializeLocal3DHierarchicalPartAssembly,
} from '@isis/3d-semantic-editing';

const tree = createLocal3DHierarchicalPartTree({
  assembly: segmentation.assembly,
  segments: segmentation.segments,
  includeImplicitMaterialRegions: true,
});

console.log(tree.nodes[tree.rootId].descendantPartIds);
console.log(tree.stats.levelCounts['material-region']);

const normalizedAssembly = materializeLocal3DHierarchicalPartAssembly(tree);

Editable Part Masks#

createLocal3DEditablePartMasks generates one aligned mask bundle for the same selected part across:

  • mesh space: sparse face and vertex membership
  • UV space: exact atlas coverage for the selected faces
  • texture space: editable and protected texel masks with configurable dilation

When the caller does not provide UVs, the helper can auto-generate them through the repo’s UV unwrap pipeline before rasterizing atlas masks.

ts
import { createLocal3DEditablePartMasks } from '@isis/3d-semantic-editing';

const masks = createLocal3DEditablePartMasks({
  mesh,
  segment,
  uvCoordinates,
  textureResolution: 1024,
});

console.log(masks.meshSpace.selectedFaceCount);
console.log(masks.uvSpace.selectedIslandIds);
console.log(masks.textureSpace.editableCoverageRatio);

Connector And Joint Detection#

Local3DConnectorJointDetector inspects semantic parts plus mesh-space contact geometry to infer localized edit anchors such as:

  • hinges for doors, lids, visors, limbs, and other line-style articulation
  • sockets and drawer-style guided insertions
  • weld seams and hard-surface mating surfaces for rigid joins
  • articulation pivots for wheel- and rotor-like rotational components

It returns both connector hypotheses and downstream joint hypotheses so later editing tasks can constrain replacements, opening motions, weld preservation, and snap targets with the same detection pass.

ts
import { createLocal3DConnectorJointDetector } from '@isis/3d-semantic-editing';

const detector = createLocal3DConnectorJointDetector();
const detection = detector.run({
  mesh,
  tree,
  segments,
});

console.log(detection.connectors.map((connector) => connector.kind));
console.log(detection.joints.map((joint) => joint.type));

Connector Compatibility System#

analyzeLocal3DConnectorCompatibility scores whether two connector sets can join directly, need an adapter, or should be rejected. It covers:

  • sockets and pegs
  • rails and guided slides
  • hinges and articulation pivots
  • mounting points and modular architecture joins
  • weld-seam and mating-surface style structural joins

The helper can also normalize the current detected connector graph or candidate replacement descriptors into one compatibility descriptor format for downstream snap, align, and kitbash workflows.

ts
import {
  analyzeLocal3DConnectorCompatibility,
  createLocal3DConnectorCompatibilityDescriptors,
} from '@isis/3d-semantic-editing';

const source = createLocal3DConnectorCompatibilityDescriptors({
  detectedConnectors: detection.connectors,
});
const target = createLocal3DConnectorCompatibilityDescriptors({
  candidateConnectors,
});

const compatibility = analyzeLocal3DConnectorCompatibility({
  sourceConnectors: source,
  targetConnectors: target,
  mountProfile: 'architecture-modular',
});

console.log(compatibility.bestMatches[0]?.grade);
console.log(compatibility.bestMatches[0]?.recommendedActions);

Snap And Align Constraints#

planLocal3DSnapAndAlignConstraints converts the best connector compatibility match plus optional bounding features and semantic landmarks into an ordered snap plan.

It emits:

  • anchor translation for the selected connector pair
  • axis and surface-normal alignment constraints
  • socket insertion constraints when a peg/socket pair is involved
  • bounding-box and landmark fallback constraints when connectors are missing
ts
import { planLocal3DSnapAndAlignConstraints } from '@isis/3d-semantic-editing';

const snapPlan = planLocal3DSnapAndAlignConstraints({
  sourceName: 'blade tang',
  targetName: 'hilt socket',
  sourceBBox,
  targetBBox,
  compatibility: compatibility.bestMatches,
});

console.log(snapPlan.transform.translation);
console.log(snapPlan.constraints.map((constraint) => constraint.kind));
console.log(snapPlan.warnings);

Semantic Merge Conflict Resolution#

resolveLocal3DSemanticMergeConflicts reconciles multiple localized edit intents before they are committed into one semantic edit session or baked into a final asset.

It normalizes edit candidates into one scope model and then:

  • detects overlap across semantic parts, descendant part scopes, connectors, material regions, mesh patches, and UV islands
  • auto-composes compatible edits such as geometry plus texture, or connector snap followed by merge-weld
  • deterministically picks a winner when priorities, timestamps, or locked scopes provide a safe tie-break
  • escalates unresolved replacement-vs-reshape or channel-overwrite collisions to manual review
ts
import { resolveLocal3DSemanticMergeConflicts } from '@isis/3d-semantic-editing';

const mergePlan = resolveLocal3DSemanticMergeConflicts({
  assembly,
  connectors,
  candidates: [
    {
      id: 'reshape-blade',
      label: 'Reshape blade profile',
      operation: 'geometry-edit',
      domains: ['geometry'],
      scope: { partIds: ['blade'], faceIndices: [0, 1, 2] },
    },
    {
      id: 'repaint-blade',
      label: 'Repaint blade metal',
      operation: 'texture-repaint',
      domains: ['texture'],
      scope: { partIds: ['blade'], materialRegionIds: ['blade__steel'] },
    },
  ],
});

console.log(mergePlan.conflicts[0]?.resolution.action);
console.log(mergePlan.plan.orderedSteps.map((step) => step.editId));

Assembly Validation#

validateLocal3DAssembly provides one deterministic validation pass over a semantic assembly before publish, bake, or handoff.

It combines mesh topology, semantic segmentation, hierarchical material-region ownership, and optional rigging signals to detect:

  • floating parts on detached components
  • likely self-intersections or interior shells
  • inverted normals on closed shells
  • orphan material regions with invalid part ownership
  • rig disconnects caused by detached bound geometry
ts
import { validateLocal3DAssembly } from '@isis/3d-semantic-editing';

const report = validateLocal3DAssembly({
  mesh,
  assembly,
  segments,
  tree,
  rigConfidence,
});

console.log(report.status);
console.log(report.issues.map((issue) => issue.code));
console.log(report.metadata.totalIssueCount);

Reusable Part Library#

InMemoryLocal3DReusablePartLibraryIndex indexes reusable semantic parts for kitbash and replacement workflows and reranks them with semantic-editing-aware signals.

It combines:

  • full-text and semantic retrieval via the shared asset-library search patterns
  • semantic-tag overlap against requested part semantics
  • connector compatibility scoring for direct-fit versus adapter-fit matches
  • project style-profile awareness so donor parts can be filtered toward the active visual language
ts
import { InMemoryLocal3DReusablePartLibraryIndex } from '@isis/3d-semantic-editing';

const index = new InMemoryLocal3DReusablePartLibraryIndex();
await index.upsertPart({
  id: 'wheel-front-left',
  assetId: 'vehicle-kit-1',
  partId: 'front-left-wheel',
  name: 'Front Left Wheel',
  description: 'Reusable armored buggy wheel with tire and hub assembly.',
  family: 'vehicle',
  semanticLabel: 'wheel',
  semanticTags: ['wheel', 'front-left', 'suspension'],
});

const result = await index.searchParts({
  query: 'buggy wheel',
  desiredSemanticTags: ['wheel', 'vehicle'],
});

console.log(result.hits[0]?.entry.id);
console.log(result.hits[0]?.breakdown.connectorCompatibilityScore);

Kitbash Final-Asset Bake#

bakeLocal3DKitbashFinalAsset finalizes a selected kitbash assembly into one shipping-ready asset workflow. It:

  • merges and welds the selected semantic parts
  • repacks UVs when UV coordinates are available
  • emits a texture rebake plan keyed to changed channels and seam risk
  • regenerates a runtime-friendly LOD chain
  • refreshes semantic metadata and can export GLB/USD/BLEND outputs
ts
import { bakeLocal3DKitbashFinalAsset } from '@isis/3d-semantic-editing';

const baked = await bakeLocal3DKitbashFinalAsset({
  assetName: 'hero-sword-final',
  mesh,
  assembly,
  segments,
  selectedPartIds: ['blade', 'handle'],
  uvCoordinates,
  sourceTextures,
});

console.log(baked.textureBake.executionSteps);
console.log(baked.lodChain.levels.map((level) => level.id));
console.log(baked.semanticMetadata.metadata);

Cross-Format Metadata Handoff#

createLocal3DSemanticMetadataHandoffBundle and createLocal3DSemanticMetadataExportAdapters package the semantic tree plus connector/joint results into one canonical payload and map it into:

  • glTF/GLB extras
  • OpenUSD customData
  • Blender scene/object custom properties and text blocks

That gives downstream exporters a format-native place to preserve localized editing metadata while still letting Oshun recover one stable semantic payload after round-tripping.

ts
import {
  createLocal3DSemanticMetadataExportAdapters,
  createLocal3DSemanticMetadataHandoffBundle,
} from '@isis/3d-semantic-editing';

const bundle = createLocal3DSemanticMetadataHandoffBundle({
  tree,
  detection,
});
const adapters = createLocal3DSemanticMetadataExportAdapters(bundle, {
  blendObjectName: 'AssetMesh',
});

console.log(adapters.gltf.sceneExtras);
console.log(adapters.usd.assetCustomData);
console.log(adapters.blend.textBlocks?.['oshun_semantic_editing.json']);

Part Merge And Weld Pipeline#

mergeLocal3DPartsAndWeld fuses multiple semantic parts into one localized merge patch for kitbashing and downstream baking. The workflow:

  • extracts the selected semantic parts into one bounded indexed mesh patch
  • detects weldable seam interfaces from geometry overlap and optional connector metadata
  • collapses seam vertices into deterministic weld clusters
  • runs localized degenerate and non-manifold cleanup after welding
  • emits tangent-repair normals plus hard-edge preservation directives for material boundaries
ts
import { mergeLocal3DPartsAndWeld } from '@isis/3d-semantic-editing';

const merged = mergeLocal3DPartsAndWeld({
  mesh,
  assembly,
  segments,
  tree,
  connectors,
  selectedPartIds: ['left_panel', 'right_panel'],
});

console.log(merged.seamInterfaces[0]?.matchedPairCount);
console.log(merged.weldClusters.map((cluster) => cluster.sourceVertexIndices));
console.log(merged.tangentRepair.instructions);

Before/After Part Diff Visualization#

visualizeLocal3DBeforeAfterPartDiff compares two semantic-edit snapshots and builds review-ready diff cards for each part across:

  • mesh and topology changes
  • texture and UV repaint scope
  • material-region overrides
  • rig-state deltas from skinning, attachments, and rig-confidence reports

Instead of rendering pixels directly, it emits typed overlay layers that UI, review tools, or exporters can present consistently.

ts
import { visualizeLocal3DBeforeAfterPartDiff } from '@isis/3d-semantic-editing';

const diff = visualizeLocal3DBeforeAfterPartDiff({
  before,
  after,
  textureEdit,
  remeshCleanup,
  replacement,
});

console.log(diff.parts[0]?.changedDomains);
console.log(diff.visualization.layers.map((layer) => layer.id));
console.log(diff.warnings);

Iterative Edit Sessions#

createLocal3DIterativeEditSession and the related helpers turn localized edit snapshots into an immutable branch graph for:

  • sequential edit history on the active branch
  • non-destructive variant forks from any prior step
  • rollback that restores an older snapshot without deleting newer history
  • step-to-step comparison through the same before/after diff planner
ts
import {
  commitLocal3DIterativeEditSessionStep,
  compareLocal3DIterativeEditSessionSteps,
  createLocal3DIterativeEditSession,
  forkLocal3DIterativeEditSessionVariant,
  rollbackLocal3DIterativeEditSessionBranch,
} from '@isis/3d-semantic-editing';

const session = createLocal3DIterativeEditSession({
  assetId: 'sword',
  initialSnapshot,
});
const edited = commitLocal3DIterativeEditSessionStep(session, {
  label: 'Blue blade concept',
  snapshot: editedSnapshot,
});
const forked = forkLocal3DIterativeEditSessionVariant(edited, {
  fromStepId: 'main:step-0002',
  branchName: 'variant-brass',
});
const comparison = compareLocal3DIterativeEditSessionSteps(forked, {
  beforeStepId: 'main:step-0001',
  afterStepId: 'main:step-0002',
});

console.log(forked.branches['variant-brass']?.headStepId);
console.log(comparison.diff.visualization.changedPartIds);
console.log(
  rollbackLocal3DIterativeEditSessionBranch(forked, {
    targetStepId: 'main:step-0001',
  }).warnings
);

Domain-Specific Edit Recipes#

planLocal3DDomainSpecificEditRecipe expands high-level intents into composed semantic edit plans using the existing geometry, texture, replacement, and propagation workflows.

Built-in recipes currently cover:

  • replace-wheel
  • change-jacket-fabric
  • convert-to-battle-damage
  • turn-chair-into-stool
ts
import { planLocal3DDomainSpecificEditRecipe } from '@isis/3d-semantic-editing';

const recipePlan = planLocal3DDomainSpecificEditRecipe({
  recipeId: 'replace-wheel',
  mesh,
  assembly,
  segments,
  tree,
  replacementCandidates,
});

console.log(recipePlan.selectedPartIds);
console.log(recipePlan.geometryPlan.plan.scopedPrompt);
console.log(recipePlan.replacementPlan?.selectedCandidate?.candidate.id);

Part Separation Export#

exportLocal3DPartSeparation extracts semantic parts into standalone export packages for downstream DCC tools and marketplace delivery.

For each selected part it now produces:

  • an isolated mesh slice with remapped indices
  • semantic subtree metadata for that part
  • GLB and OpenUSD exports
  • optional BLEND export when a Blender runner is available
  • Blender/Maya DCC bundle descriptors
  • a marketplace package manifest with archive layout
ts
import { exportLocal3DPartSeparation } from '@isis/3d-semantic-editing';

const separated = await exportLocal3DPartSeparation({
  mesh,
  assembly,
  segments,
  tree,
});

console.log(separated.parts.map((part) => part.partId));
console.log(separated.parts[0]?.glb.fileName);
console.log(separated.parts[0]?.marketplacePackage.manifest.archiveLayout);

Localized Geometry Edit Prompting#

planLocal3DLocalizedGeometryEdit turns a natural-language edit like replace blade, thicken handle, or open visor into one deterministic, part-scoped edit plan.

It packages:

  • the selected semantic parts that are allowed to move
  • the protected mesh region that must remain fixed
  • operation classification and edit strategy
  • seam-preservation constraints for neighboring parts
  • connector-aware articulation guidance for hinged or pivoting parts
ts
import { planLocal3DLocalizedGeometryEdit } from '@isis/3d-semantic-editing';

const editPlan = planLocal3DLocalizedGeometryEdit({
  mesh,
  assembly: segmentation.assembly,
  segments: segmentation.segments,
  selectedPartIds: ['visor'],
  instruction: 'open visor',
  connectors: detection.connectors,
  joints: detection.joints,
});

console.log(editPlan.plan.strategy);
console.log(editPlan.plan.scopedPrompt);
console.log(editPlan.protectedRegion.protectedFaceCount);

Localized Texture Repaint And Material Overrides#

planLocal3DLocalizedTextureMaterialEditing builds part-scoped or UV-island-scoped texture workflows over the same semantic selection system.

It produces:

  • a merged editable/protected mask bundle in mesh, UV, and texture space
  • a localized repaint prompt with bleed and seam constraints
  • a material override plan targeting semantic material regions when available
  • protected-region summaries for untouched parts, UV islands, and material areas
ts
import { planLocal3DLocalizedTextureMaterialEditing } from '@isis/3d-semantic-editing';

const textureEdit = planLocal3DLocalizedTextureMaterialEditing({
  mesh,
  segments: segmentation.segments,
  tree,
  selectedPartIds: ['blade'],
  uvCoordinates,
  repaint: {
    prompt: 'repaint the blade with blue enamel and silver wear',
  },
  materialOverride: {
    materialKey: 'painted-steel',
    roughness: 0.34,
    metallic: 0.88,
  },
});

console.log(textureEdit.selection.selectedUvIslandIds);
console.log(textureEdit.repaintPlan?.scopedPrompt);
console.log(textureEdit.materialOverridePlan?.targetMaterialRegionIds);

Protected-Region Edit Locks#

createLocal3DProtectedRegionEditLocks creates one preservation contract for untouched regions before localized edits run.

It locks:

  • protected faces, vertices, and seam-adjacent geometry
  • protected UV islands and texture texels
  • protected material regions and channel assignments
  • protected skin-weight domains and clothing attachment offsets when rig data is available
ts
import { createLocal3DProtectedRegionEditLocks } from '@isis/3d-semantic-editing';

const locks = createLocal3DProtectedRegionEditLocks({
  mesh,
  segments: segmentation.segments,
  tree,
  selectedPartIds: ['blade'],
  uvCoordinates,
  skinWeights,
  clothingBinding,
});

console.log(locks.geometry.protectedPartIds);
console.log(locks.materials.protectedMaterialRegionIds);
console.log(locks.rig.protectedJointIds);

Part Replacement Flow#

planLocal3DPartReplacementFlow coordinates semantic part swaps using:

  • generated candidates
  • marketplace assets
  • project-approved donor parts

Each candidate is ranked against the selected target by semantic label fit, connector compatibility, scale fit, style alignment, rig transfer viability, and license or approval policy. The planner returns a ranked candidate list plus the exact handoff needed for execution:

  • selected candidate and score breakdown
  • connector-aware alignment plan
  • integration plan for seam, material, and rig preservation
  • generated fallback brief when no approved swap candidate is available
ts
import { planLocal3DPartReplacementFlow } from '@isis/3d-semantic-editing';

const replacement = planLocal3DPartReplacementFlow({
  mesh,
  assembly: segmentation.assembly,
  segments: segmentation.segments,
  selectedPartIds: ['blade'],
  candidates,
  connectors: detection.connectors,
  joints: detection.joints,
  tree,
  protectedLocks,
  distributionIntent: 'client-delivery',
});

console.log(replacement.selectedCandidate?.candidate.name);
console.log(replacement.alignmentPlan?.mode);
console.log(replacement.integrationPlan?.executionSteps);

Localized Remesh And Topology Cleanup#

cleanupLocal3DLocalizedRemeshTopology extracts a bounded patch around the edited region, preserves the neighboring seam and boundary anchors, and runs localized topology cleanup using the repo's existing post-pipeline primitives.

It returns:

  • editable and support face sets for the bounded patch
  • local topology and triangle-quality analysis before and after cleanup
  • a cleaned patch mesh ready for reintegration
  • reintegration instructions that keep protected faces and surrounding edge flow stable
ts
import { cleanupLocal3DLocalizedRemeshTopology } from '@isis/3d-semantic-editing';

const cleanup = cleanupLocal3DLocalizedRemeshTopology({
  mesh,
  segments: segmentation.segments,
  selectedPartIds: ['blade'],
  protectedLocks,
  preserveBoundaryRingCount: 1,
  targetTriangleRatio: 0.8,
});

console.log(cleanup.region.seamVertexIndices);
console.log(cleanup.cleanupOperations.map((operation) => operation.id));
console.log(cleanup.metadata.finalPatchTriangleCount);

Masked Geometry Infill#

infillLocal3DMaskedGeometry repairs missing, occluded, or damaged mesh regions discovered during semantic review or localized edit loops.

It can derive the masked region from:

  • explicit face or vertex masks
  • confidence-scoring highlighted regions
  • topology-derived fallback patches when review signals are sparse

The workflow emits:

  • a bounded masked patch plus seam and boundary anchors
  • donor hints from mirrored or nearby semantic parts
  • a deterministic scaffold mesh for infill handoff
  • reintegration steps that keep protected neighboring regions stable
ts
import { infillLocal3DMaskedGeometry } from '@isis/3d-semantic-editing';

const infill = infillLocal3DMaskedGeometry({
  mesh,
  segments: segmentation.segments,
  selectedPartIds: ['blade'],
  defectMode: 'missing',
  maskedFaceIndices: [12, 13, 14],
  protectedLocks,
  remeshCleanup: cleanup,
});

console.log(infill.maskRegion.maskedFaceIndices);
console.log(infill.donorHints);
console.log(infill.integration.strategy);

Smart Edit Propagation#

planLocal3DSmartEditPropagation takes one localized edit and expands it across mirrored or repeated semantic instances such as:

  • left/right limb panels
  • wheel sets
  • repeating windows
  • bolts or other indexed hardware groups

The workflow uses instance metadata from semantic segmentation to emit:

  • propagation targets with confidence scores
  • transformed instructions for mirrored counterparts
  • repeated-instance propagation previews
  • warnings when asymmetry or instance drift makes automatic replay unsafe
ts
import { planLocal3DSmartEditPropagation } from '@isis/3d-semantic-editing';

const propagation = planLocal3DSmartEditPropagation({
  assembly: segmentation.assembly,
  segments: segmentation.segments,
  selectedPartIds: ['front_left_wheel'],
  instruction: 'replace left wheel tire with heavy tread tire',
  localizedEdit: editPlan,
});

console.log(propagation.preview.propagationMode);
console.log(propagation.targets.map((target) => target.targetPartId));
console.log(propagation.targets[0]?.transformedInstruction);