3dgs-articulated-reasoner
3DGS Articulated Object Reasoning & Digital Twin Agent. Reason about articulated objects in 3DGS scenes: extract part structure, infer kinematic constraints, generate interactive digital twins. Use when: articulated object, digital twin, part structure, kinematic chain, URDF, interactive 3DGS, part-aware rendering, articulated manipulation, joint estimation, part segmentation 3DGS, ArtiSplat, ArtiTwinSplat, articulated reconstruction.
jaccen · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add jaccen/Awesome-Gaussian-Skills --skill 3dgs-articulated-reasoner技能说明
name: 3dgs-articulated-reasoner description: "3DGS Articulated Object Reasoning & Digital Twin Agent. Reason about articulated objects in 3DGS scenes: extract part structure, infer kinematic constraints, generate interactive digital twins. Use when: articulated object, digital twin, part structure, kinematic chain, URDF, interactive 3DGS, part-aware rendering, articulated manipulation, joint estimation, part segmentation 3DGS, ArtiSplat, ArtiTwinSplat, articulated reconstruction." license: Apache-2.0 user-invocable: true metadata: version: "0.2.0" author: jaccen tags: ["3dgs", "articulated-object", "digital-twin", "kinematics", "part-aware", "urdf", "interactive-3d"] arguments: [task] when_to_use: - "Reason about articulated objects in a 3DGS scene" - "Build an interactive digital twin from Gaussian splats" - "Extract part structure or segment movable/fixed Gaussians" - "Infer kinematic chains or generate URDF from 3DGS" - "Part-aware rendering or articulated manipulation" - "Estimate joint types and axes from multi-view observation" - "铰接式物体 / 数字孪生 / 部件分割 / 运动链推断 / URDF生成"
3DGS Articulated Reasoner
Reason about articulated objects within 3DGS scenes: extract part structure, infer kinematics, build interactive digital twins.
Capabilities
- Part Structure Extraction: Given a 3DGS scene containing articulated objects (furniture, vehicles, tools), identify part boundaries and segment Gaussians into movable/ fixed groups.
- Kinematic Inference: Estimate joint types (revolute, prismatic, fixed) and axes from multi-view observation or user specification. Output URDF-compatible kinematic tree.
- Part-Aware Compositing: Apply part-aware alpha-compositing that penalizes inter-part penetration at boundaries (inspired by Innovation I-01 in the knowledge base).
- Digital Twin Generation: Produce an interactable digital twin where Agent commands (open, rotate, slide) drive real-time Gaussian deformation.
- Simulation-Ready Export: Export articulated 3DGS scenes to simulation frameworks (MuJoCo, Isaac Sim) with preserved visual fidelity.
Instructions
Step 1: Scene Analysis
When the user provides a 3DGS scene or asks about articulated objects:
- Check
references/3dgs-methods-overview.mdfor relevant methods (ArtiTwinSplat, ArtiSplat, ULF-Loc). - Identify the object category and expected articulation pattern from the knowledge base.
- Determine whether part segmentation is available or needs estimation.
Key references:
- ArtiTwinSplat (arXiv 2026): Articulated digital twin from 3DGS
- ULF-Loc (CVPR 2026): Exposed feature bias at part boundaries
- ArtiSplat: Differentiable physics + rendering for articulated objects
Step 2: Part Segmentation
If the user needs part segmentation:
Input: 3DGS point cloud / rendered views
Method options:
a) Semantic segmentation (LangSplat/Feature 3DGS) → part labels
b) Motion-based segmentation (multi-frame observation) → movable vs fixed
c) User-specified masks (manual annotation on rendered views)
Output: Per-Gaussian part label {0, 1, ..., K}
Validation criteria:
- Part boundaries should align with geometric discontinuities (normal/depth edges)
- Each part should be spatially contiguous (no isolated Gaussians)
- Fixed parts should be the largest spatially connected component
Step 3: Kinematic Estimation
For each part pair with relative motion:
Joint Type Decision Tree:
IF relative motion is rotational around a fixed axis
→ Revolute joint (hinge)
→ Estimate: axis direction, axis point, rotation limits [θ_min, θ_max]
IF relative motion is translational along a fixed axis
→ Prismatic joint (slider)
→ Estimate: slide axis, slide limits [d_min, d_max]
IF no relative motion observed
→ Fixed joint (weld)
Output format: URDF XML string with visual (mesh from TSDF fusion) and collision (simplified convex hull) links.
Step 4: Part-Aware Rendering
When rendering articulated objects for visualization or editing:
# Part-aware alpha-compositing (Innovation I-01)
C(theta) = sum_i T_i * alpha_i * omega_{p(i)}(theta) * c_i
# where omega penalizes penetration:
# omega = 1.0 if no penetration
# omega = exp(-lambda * pen_i) if penetration detected
# pen_i = SDF_violation at Gaussian center
# lambda = 10.0 (default)
This prevents color bleeding at part boundaries that standard alpha-compositing produces.
Step 5: Agent-Driven Interaction
Support natural-language commands for digital twin interaction:
| Command | Action | Implementation |
|---|---|---|
| "Open the drawer" | Translate drawer part along slide axis | Apply d_offset to prismatic joint |
| "Rotate the door 30 degrees" | Rotate door part around hinge | Apply θ_offset to revolute joint |
| "Show the internal structure" | Hide external shell parts | Set alpha=0 for shell Gaussians |
| "Reset pose" | Return all parts to initial configuration | Zero all joint offsets |
Deformation pipeline:
- Load URDF + initial Gaussian positions
- Apply forward kinematics: new_pos = FK(joint_offsets)
- Transform Gaussian positions: G_new.mu = R * (G.mu - joint_origin) + joint_origin + t
- Re-render with part-aware compositing
Step 6: Export & Integration
Export options:
- URDF + Mesh: For MuJoCo / Isaac Sim (convex decomposition via VHACD)
- GLTF with morph targets: For web viewers (Three.js)
- 3DGS-Compatible: Gaussian attributes + joint parametrization for real-time rendering
Reference Data
Articulated Object Methods in Knowledge Base
| Method | Venue | Category | Key Contribution |
|---|---|---|---|
| ArtiTwinSplat | arXiv 2026 | Articulated/Digital Twin | Digital twin via 3DGS |
| ArtiSplat | 2026 | Articulated | Differentiable physics + rendering |
| ULF-Loc | CVPR 2026 | SLAM/Articulated | Feature bias at part boundaries |
| Articul |