3dgs-mcp-renderer
MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering.
jaccen · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add jaccen/Awesome-Gaussian-Skills --skill 3dgs-mcp-rendererAbout this skill
name: 3dgs-mcp-renderer description: "MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering." license: Apache-2.0 metadata: version: "1.1.0" author: jaccen tags: ["mcp", "3dgs", "gaussian-splatting", "rendering", "three.js", "webgpu", "voice", "agent", "interactive", "spec-first", "sculpting", "code-first", "slat", "latent-editing"] disable-model-invocation: true user-invocable: true
3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol
Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.
Architecture
┌─────────────┐ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Voice/Text │────▶│ Agent │────▶│ MCP Server │────▶│ 3DGS Renderer │
│ (Whisper/ │ │ (Claude/ │ │ (Node.js/ │ │ (Three.js/ │
│ Prompt) │ │ TeleClaw) │ │ Python) │ │ WebGPU/HiGS/ │
│ │◀────│ │◀────│ │◀────│ DDF-GS) │
└─────────────┘ └─────────────┘ └──────────────────┘ └──────────────────┘
│ │ │
│ Tool calls │ WebSocket/HTTP │ WebGL/WebGPU/
│ (MCP protocol) │ transport │ HiGS/DDF-GS
Spec-First Sculpting Pipeline (v0.9.0)
Design inspiration: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) spec-first — define quality criteria and component hierarchy before any rendering; (2) stage-gated sculpting — progressive refinement with acceptance checks at each stage.
Why Spec-First for MCP Rendering?
The original MCP pipeline was reactive: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:
- No upfront quality criteria → agent cannot self-assess before rendering
- No stage gates → errors compound across steps (bad camera → bad selection → bad edit)
- No component hierarchy → edits are flat, no part-level control
The fix: Introduce a define_scene_spec tool that runs before any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.
The 6-Stage Sculpting Pipeline
┌─────────────────────────────────────────────────────────────┐
│ SPEC-FIRST SCULPTING │
│ │
│ ┌──────────────┐ │
│ │ define_scene │ ← Object Spec: component hierarchy, │
│ │ _spec │ material system, quality criteria │
│ └──────┬───────┘ │
│ │ │
│ ▼ │
│ Stage 1: blockout → Bounding boxes, camera framing │
│ │ gate: bbox coverage ≥ spec.target_coverage? │
│ ▼ │
│ Stage 2: structural → Part decomposition, hierarchy │
│ │ gate: part count & nesting matches spec? │
│ ▼ │
│ Stage 3: form → Gaussian density/scale/rotation │
│ │ gate: PSNR estimate ≥ spec.min_psnr? │
│ ▼ │
│ Stage 4: material → PBR/SH assignment per part │
│ │ gate: material count per part matches spec? │
│ ▼ │
│ Stage 5: surface → Normal consistency, thin structures │
│ │ gate: normal consistency score ≥ spec.threshold? │
│ ▼ │
│ Stage 6: lighting → Environment, shadows, AO │
│ gate: render quality score ≥ spec.target_score? │
└─────────────────────────────────────────────────────────────┘
Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's blockout → structural → form → material → surface → lighting flow.
Gate Evaluation Protocol
For each stage gate, the agent follows this protocol:
1. Execute stage tool (e.g., sculpt_form with parameters)
2. Call render_frame() to get current visual state
3. Call query_scene(query_type="stats") to get quantitative metrics
4. Compare metrics against spec gate criteria
5. If pass → advance to next stage
6. If fail → adjust parameters and retry (max 3 attempts)
7. If 3 failures → report to user with diagnostic info
Voice-Driven Sculpting Example
Loaded on demand — See mcp-tools-spec.md for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).
Code-First Rendering Philosophy (v0.9.0)
Design inspiration: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.
Traditional 3DGS Export vs Code-First Export
| Aspect | Traditional (.ply/.splat) | Code-First (
Read the complete source ↗