--- name: 3dgs-mcp-renderer description: "MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering." license: Apache-2.0 metadata: version: "1.1.0" author: jaccen tags: ["mcp", "3dgs", "gaussian-splatting", "rendering", "three.js", "webgpu", "voice", "agent", "interactive", "spec-first", "sculpting", "code-first", "slat", "latent-editing"] disable-model-invocation: true user-invocable: true ---

# 3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

## Architecture

``` ┌─────────────┐ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Voice/Text │────▶│ Agent │────▶│ MCP Server │────▶│ 3DGS Renderer │ │ (Whisper/ │ │ (Claude/ │ │ (Node.js/ │ │ (Three.js/ │ │ Prompt) │ │ TeleClaw) │ │ Python) │ │ WebGPU/HiGS/ │ │ │◀────│ │◀────│ │◀────│ DDF-GS) │ └─────────────┘ └─────────────┘ └──────────────────┘ └──────────────────┘ │ │ │ │ Tool calls │ WebSocket/HTTP │ WebGL/WebGPU/ │ (MCP protocol) │ transport │ HiGS/DDF-GS ```

## Spec-First Sculpting Pipeline (v0.9.0)

> **Design inspiration**: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) **spec-first** — define quality criteria and component hierarchy before any rendering; (2) **stage-gated sculpting** — progressive refinement with acceptance checks at each stage.

### Why Spec-First for MCP Rendering?

The original MCP pipeline was **reactive**: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

- No upfront quality criteria → agent cannot self-assess before rendering - No stage gates → errors compound across steps (bad camera → bad selection → bad edit) - No component hierarchy → edits are flat, no part-level control

**The fix**: Introduce a `define_scene_spec` tool that runs *before* any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

### The 6-Stage Sculpting Pipeline

``` ┌─────────────────────────────────────────────────────────────┐ │ SPEC-FIRST SCULPTING │ │ │ │ ┌──────────────┐ │ │ │ define_scene │ ← Object Spec: component hierarchy, │ │ │ _spec │ material system, quality criteria │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ Stage 1: blockout → Bounding boxes, camera framing │ │ │ gate: bbox coverage ≥ spec.target_coverage? │ │ ▼ │ │ Stage 2: structural → Part decomposition, hierarchy │ │ │ gate: part count & nesting matches spec? │ │ ▼ │ │ Stage 3: form → Gaussian density/scale/rotation │ │ │ gate: PSNR estimate ≥ spec.min_psnr? │ │ ▼ │ │ Stage 4: material → PBR/SH assignment per part │ │ │ gate: material count per part matches spec? │ │ ▼ │ │ Stage 5: surface → Normal consistency, thin structures │ │ │ gate: normal consistency score ≥ spec.threshold? │ │ ▼ │ │ Stage 6: lighting → Environment, shadows, AO │ │ gate: render quality score ≥ spec.target_score? │ └─────────────────────────────────────────────────────────────┘ ```

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's `blockout → structural → form → material → surface → lighting` flow.

### Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

``` 1. Execute stage tool (e.g., sculpt_form with parameters) 2. Call render_frame() to get current visual state 3. Call query_scene(query_type="stats") to get quantitative metrics 4. Compare metrics against spec gate criteria 5. If pass → advance to next stage 6. If fail → adjust parameters and retry (max 3 attempts) 7. If 3 failures → report to user with diagnostic info ```

### Voice-Driven Sculpting Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

## Code-First Rendering Philosophy (v0.9.0)

> **Design inspiration**: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.

### Traditional 3DGS Export vs Code-First Export

| Aspect | Traditional (.ply/.splat) | Code-First (.js + .splat) | |--------|--------------------------|--------------------------| | Editability | Binary blob, hard to edit | Source code, any field adjustable | | Version control | Binary diff, no merge | Text diff, git-friendly | | File size | Full Gaussian set (MB-GB) | Code skeleton (KB) + compressed splat data | | Scene composition | Single flat Gaussian cloud | Hierarchical code with part-level control | | Interaction logic | Must be added externally | Embedded in code | | 3DGS data | All in one file | Separate .splat file loaded by code | | Procedural elements | Not supported | Parametric geometry in code (e.g., desk surface = PlaneGeometry) |

### Hybrid: Procedural Code + 3DGS Splatting

The key insight: **not everything needs to be Gaussians**. For a desk scene: - Desk surface → procedural `BoxGeometry` in code (simple, editable, lightweight) - Monitor screen texture → procedural `MeshStandardMaterial` (or 3DGS if view-dependent) - Complex organic objects → 3DGS splatting data (where procedural code can't compete)

> **Loaded on demand** — See [code-first-examples.md](references/code-first-examples.md) for hybrid export code examples.

### When to Use Code-First vs Pure Splat

| Scene Element | Recommendation | Why | |--------------|---------------|-----| | Flat surfaces (walls, floors, desks) | Procedural code | Simple, editable, tiny file size | | Parametric objects (cabinets, shelves) | Procedural code | Adjust dimensions in code | | Organic objects (plants, food, fabric) | 3DGS splat | Can't match quality procedurally | | View-dependent surfaces (screens, mirrors) | 3DGS splat | SH coefficients capture view dependence | | Articulated parts (joints, hinges) | Procedural code | Joint parameters are explicit in code | | Mixed scenes (most real cases) | Hybrid code + splat | Best of both worlds |

### SLAT Connection

The code-first approach connects to SLAT (see `../../references/slat-unified-representation.md`): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. **SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.**

## SLAT Latent Editing (v1.0.0)

> **Theoretical basis**: SLAT (Structured Latent Aggregation Transform) — see `../../references/slat-unified-representation.md`. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.

### Encoding: Scene → Structured Latent

`encode_scene_slatent` voxelizes the active scene into a regular grid (`voxel_size`, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a `slat_id` referencing an in-memory snapshot with an `encode_loss` (reconstruction RMSE), letting the agent judge fidelity before editing.

### Editing in Latent Space

`edit_scene_latent` applies a `LatentEditOp` to voxels matched by a `LatentSelector` (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:

| Op | Fields | Effect | |----|--------|--------| | `translate` | `delta: Vec3` | Move matched voxels (and their Gaussians) by a vector | | `scale` | `factor: number`, `origin: Vec3` | Scale voxel positions relative to an origin | | `rotate` | `angleDeg: number`, `axis: Vec3`, `origin: Vec3` | Rotate voxels around an axis (degrees) | | `recolor` | `color: Vec3`, `mix: number` | Blend matched voxels' colors toward a target | | `opacity` | `opacity: number`, `mode` | Set or scale opacity (mode `set`/`scale`) | | `smooth` | `iterations: number`, `strength: number` | Smooth feature positions/colors by averaging neighbors | | `delete` | `target: "voxel"` | Remove all Gaussians in matched voxels |

**Schema vs core naming**: the MCP JSON schema uses snake_case (`angle_deg`); the internal `LatentEditOp` uses camelCase (`angleDeg`). Handlers convert at the boundary. Library/test callers use camelCase directly.

**Safety gate**: `edit_scene_latent` computes `affected_gaussians`; if this exceeds 10% of the scene, the edit is rejected unless `confirm=true`. This reuses the project-wide 10% safety rule.

**Apply to scene**: with `apply_to_scene=true` (default) the edit is re-decoded and broadcast to the renderer via `modify_gaussians`; with `false` it only updates the in-memory snapshot, so the agent can preview/cancel before committing.

### Decoding: Latent → Scene

Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. `delete` removes the affected Gaussians entirely.

### Voice-Driven SLAT Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).

## Cross-Scene Latent Transfer & Interpolation (v1.1.0)

v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be **transferred** to another scene (target), or the two scenes can be **interpolated** in latent space. Both operations rely on a spatial correspondence built over the voxel grids.

### Correspondence: Voxel Grid Matching

Both operations build a voxel grid over the source scene (cell size = `match_radius`) via `buildVoxelGrid`, then for each target voxel find the nearest source voxel within `match_radius` (`nearestVoxel`, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.

### Transferring a Latent Edit

`transfer_scene_edit` re-applies a `LatentEditOp` from source to target **as a relative change**:

|

--- name: 3dgs-mcp-renderer description: "MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering." license: Apache-2.0 metadata: version: "1.1.0" author: jaccen tags: ["mcp", "3dgs", "gaussian-splatting", "rendering", "three.js", "webgpu", "voice", "agent", "interactive", "spec-first", "sculpting", "code-first", "slat", "latent-editing"] disable-model-invocation: true user-invocable: true ---

# 3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

## Architecture

``` ┌─────────────┐ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Voice/Text │────▶│ Agent │────▶│ MCP Server │────▶│ 3DGS Renderer │ │ (Whisper/ │ │ (Claude/ │ │ (Node.js/ │ │ (Three.js/ │ │ Prompt) │ │ TeleClaw) │ │ Python) │ │ WebGPU/HiGS/ │ │ │◀────│ │◀────│ │◀────│ DDF-GS) │ └─────────────┘ └─────────────┘ └──────────────────┘ └──────────────────┘ │ │ │ │ Tool calls │ WebSocket/HTTP │ WebGL/WebGPU/ │ (MCP protocol) │ transport │ HiGS/DDF-GS ```

## Spec-First Sculpting Pipeline (v0.9.0)

> **Design inspiration**: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) **spec-first** — define quality criteria and component hierarchy before any rendering; (2) **stage-gated sculpting** — progressive refinement with acceptance checks at each stage.

### Why Spec-First for MCP Rendering?

The original MCP pipeline was **reactive**: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

- No upfront quality criteria → agent cannot self-assess before rendering - No stage gates → errors compound across steps (bad camera → bad selection → bad edit) - No component hierarchy → edits are flat, no part-level control

**The fix**: Introduce a `define_scene_spec` tool that runs *before* any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

### The 6-Stage Sculpting Pipeline

``` ┌─────────────────────────────────────────────────────────────┐ │ SPEC-FIRST SCULPTING │ │ │ │ ┌──────────────┐ │ │ │ define_scene │ ← Object Spec: component hierarchy, │ │ │ _spec │ material system, quality criteria │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ Stage 1: blockout → Bounding boxes, camera framing │ │ │ gate: bbox coverage ≥ spec.target_coverage? │ │ ▼ │ │ Stage 2: structural → Part decomposition, hierarchy │ │ │ gate: part count & nesting matches spec? │ │ ▼ │ │ Stage 3: form → Gaussian density/scale/rotation │ │ │ gate: PSNR estimate ≥ spec.min_psnr? │ │ ▼ │ │ Stage 4: material → PBR/SH assignment per part │ │ │ gate: material count per part matches spec? │ │ ▼ │ │ Stage 5: surface → Normal consistency, thin structures │ │ │ gate: normal consistency score ≥ spec.threshold? │ │ ▼ │ │ Stage 6: lighting → Environment, shadows, AO │ │ gate: render quality score ≥ spec.target_score? │ └─────────────────────────────────────────────────────────────┘ ```

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's `blockout → structural → form → material → surface → lighting` flow.

### Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

``` 1. Execute stage tool (e.g., sculpt_form with parameters) 2. Call render_frame() to get current visual state 3. Call query_scene(query_type="stats") to get quantitative metrics 4. Compare metrics against spec gate criteria 5. If pass → advance to next stage 6. If fail → adjust parameters and retry (max 3 attempts) 7. If 3 failures → report to user with diagnostic info ```

### Voice-Driven Sculpting Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

## Code-First Rendering Philosophy (v0.9.0)

> **Design inspiration**: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.

### Traditional 3DGS Export vs Code-First Export

| Aspect | Traditional (.ply/.splat) | Code-First (.js + .splat) | |--------|--------------------------|--------------------------| | Editability | Binary blob, hard to edit | Source code, any field adjustable | | Version control | Binary diff, no merge | Text diff, git-friendly | | File size | Full Gaussian set (MB-GB) | Code skeleton (KB) + compressed splat data | | Scene composition | Single flat Gaussian cloud | Hierarchical code with part-level control | | Interaction logic | Must be added externally | Embedded in code | | 3DGS data | All in one file | Separate .splat file loaded by code | | Procedural elements | Not supported | Parametric geometry in code (e.g., desk surface = PlaneGeometry) |

### Hybrid: Procedural Code + 3DGS Splatting

The key insight: **not everything needs to be Gaussians**. For a desk scene: - Desk surface → procedural `BoxGeometry` in code (simple, editable, lightweight) - Monitor screen texture → procedural `MeshStandardMaterial` (or 3DGS if view-dependent) - Complex organic objects → 3DGS splatting data (where procedural code can't compete)

> **Loaded on demand** — See [code-first-examples.md](references/code-first-examples.md) for hybrid export code examples.

### When to Use Code-First vs Pure Splat

| Scene Element | Recommendation | Why | |--------------|---------------|-----| | Flat surfaces (walls, floors, desks) | Procedural code | Simple, editable, tiny file size | | Parametric objects (cabinets, shelves) | Procedural code | Adjust dimensions in code | | Organic objects (plants, food, fabric) | 3DGS splat | Can't match quality procedurally | | View-dependent surfaces (screens, mirrors) | 3DGS splat | SH coefficients capture view dependence | | Articulated parts (joints, hinges) | Procedural code | Joint parameters are explicit in code | | Mixed scenes (most real cases) | Hybrid code + splat | Best of both worlds |

### SLAT Connection

The code-first approach connects to SLAT (see `../../references/slat-unified-representation.md`): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. **SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.**

## SLAT Latent Editing (v1.0.0)

> **Theoretical basis**: SLAT (Structured Latent Aggregation Transform) — see `../../references/slat-unified-representation.md`. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.

### Encoding: Scene → Structured Latent

`encode_scene_slatent` voxelizes the active scene into a regular grid (`voxel_size`, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a `slat_id` referencing an in-memory snapshot with an `encode_loss` (reconstruction RMSE), letting the agent judge fidelity before editing.

### Editing in Latent Space

`edit_scene_latent` applies a `LatentEditOp` to voxels matched by a `LatentSelector` (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:

| Op | Fields | Effect | |----|--------|--------| | `translate` | `delta: Vec3` | Move matched voxels (and their Gaussians) by a vector | | `scale` | `factor: number`, `origin: Vec3` | Scale voxel positions relative to an origin | | `rotate` | `angleDeg: number`, `axis: Vec3`, `origin: Vec3` | Rotate voxels around an axis (degrees) | | `recolor` | `color: Vec3`, `mix: number` | Blend matched voxels' colors toward a target | | `opacity` | `opacity: number`, `mode` | Set or scale opacity (mode `set`/`scale`) | | `smooth` | `iterations: number`, `strength: number` | Smooth feature positions/colors by averaging neighbors | | `delete` | `target: "voxel"` | Remove all Gaussians in matched voxels |

**Schema vs core naming**: the MCP JSON schema uses snake_case (`angle_deg`); the internal `LatentEditOp` uses camelCase (`angleDeg`). Handlers convert at the boundary. Library/test callers use camelCase directly.

**Safety gate**: `edit_scene_latent` computes `affected_gaussians`; if this exceeds 10% of the scene, the edit is rejected unless `confirm=true`. This reuses the project-wide 10% safety rule.

**Apply to scene**: with `apply_to_scene=true` (default) the edit is re-decoded and broadcast to the renderer via `modify_gaussians`; with `false` it only updates the in-memory snapshot, so the agent can preview/cancel before committing.

### Decoding: Latent → Scene

Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. `delete` removes the affected Gaussians entirely.

### Voice-Driven SLAT Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).

## Cross-Scene Latent Transfer & Interpolation (v1.1.0)

v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be **transferred** to another scene (target), or the two scenes can be **interpolated** in latent space. Both operations rely on a spatial correspondence built over the voxel grids.

### Correspondence: Voxel Grid Matching

Both operations build a voxel grid over the source scene (cell size = `match_radius`) via `buildVoxelGrid`, then for each target voxel find the nearest source voxel within `match_radius` (`nearestVoxel`, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.

### Transferring a Latent Edit

`transfer_scene_edit` re-applies a `LatentEditOp` from source to target **as a relative change**:

|

--- name: 3dgs-mcp-renderer description: "MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering." license: Apache-2.0 metadata: version: "1.1.0" author: jaccen tags: ["mcp", "3dgs", "gaussian-splatting", "rendering", "three.js", "webgpu", "voice", "agent", "interactive", "spec-first", "sculpting", "code-first", "slat", "latent-editing"] disable-model-invocation: true user-invocable: true ---

# 3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

## Architecture

``` ┌─────────────┐ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Voice/Text │────▶│ Agent │────▶│ MCP Server │────▶│ 3DGS Renderer │ │ (Whisper/ │ │ (Claude/ │ │ (Node.js/ │ │ (Three.js/ │ │ Prompt) │ │ TeleClaw) │ │ Python) │ │ WebGPU/HiGS/ │ │ │◀────│ │◀────│ │◀────│ DDF-GS) │ └─────────────┘ └─────────────┘ └──────────────────┘ └──────────────────┘ │ │ │ │ Tool calls │ WebSocket/HTTP │ WebGL/WebGPU/ │ (MCP protocol) │ transport │ HiGS/DDF-GS ```

## Spec-First Sculpting Pipeline (v0.9.0)

> **Design inspiration**: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) **spec-first** — define quality criteria and component hierarchy before any rendering; (2) **stage-gated sculpting** — progressive refinement with acceptance checks at each stage.

### Why Spec-First for MCP Rendering?

The original MCP pipeline was **reactive**: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

- No upfront quality criteria → agent cannot self-assess before rendering - No stage gates → errors compound across steps (bad camera → bad selection → bad edit) - No component hierarchy → edits are flat, no part-level control

**The fix**: Introduce a `define_scene_spec` tool that runs *before* any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

### The 6-Stage Sculpting Pipeline

``` ┌─────────────────────────────────────────────────────────────┐ │ SPEC-FIRST SCULPTING │ │ │ │ ┌──────────────┐ │ │ │ define_scene │ ← Object Spec: component hierarchy, │ │ │ _spec │ material system, quality criteria │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ Stage 1: blockout → Bounding boxes, camera framing │ │ │ gate: bbox coverage ≥ spec.target_coverage? │ │ ▼ │ │ Stage 2: structural → Part decomposition, hierarchy │ │ │ gate: part count & nesting matches spec? │ │ ▼ │ │ Stage 3: form → Gaussian density/scale/rotation │ │ │ gate: PSNR estimate ≥ spec.min_psnr? │ │ ▼ │ │ Stage 4: material → PBR/SH assignment per part │ │ │ gate: material count per part matches spec? │ │ ▼ │ │ Stage 5: surface → Normal consistency, thin structures │ │ │ gate: normal consistency score ≥ spec.threshold? │ │ ▼ │ │ Stage 6: lighting → Environment, shadows, AO │ │ gate: render quality score ≥ spec.target_score? │ └─────────────────────────────────────────────────────────────┘ ```

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's `blockout → structural → form → material → surface → lighting` flow.

### Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

``` 1. Execute stage tool (e.g., sculpt_form with parameters) 2. Call render_frame() to get current visual state 3. Call query_scene(query_type="stats") to get quantitative metrics 4. Compare metrics against spec gate criteria 5. If pass → advance to next stage 6. If fail → adjust parameters and retry (max 3 attempts) 7. If 3 failures → report to user with diagnostic info ```

### Voice-Driven Sculpting Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

## Code-First Rendering Philosophy (v0.9.0)

> **Design inspiration**: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.

### Traditional 3DGS Export vs Code-First Export

| Aspect | Traditional (.ply/.splat) | Code-First (.js + .splat) | |--------|--------------------------|--------------------------| | Editability | Binary blob, hard to edit | Source code, any field adjustable | | Version control | Binary diff, no merge | Text diff, git-friendly | | File size | Full Gaussian set (MB-GB) | Code skeleton (KB) + compressed splat data | | Scene composition | Single flat Gaussian cloud | Hierarchical code with part-level control | | Interaction logic | Must be added externally | Embedded in code | | 3DGS data | All in one file | Separate .splat file loaded by code | | Procedural elements | Not supported | Parametric geometry in code (e.g., desk surface = PlaneGeometry) |

### Hybrid: Procedural Code + 3DGS Splatting

The key insight: **not everything needs to be Gaussians**. For a desk scene: - Desk surface → procedural `BoxGeometry` in code (simple, editable, lightweight) - Monitor screen texture → procedural `MeshStandardMaterial` (or 3DGS if view-dependent) - Complex organic objects → 3DGS splatting data (where procedural code can't compete)

> **Loaded on demand** — See [code-first-examples.md](references/code-first-examples.md) for hybrid export code examples.

### When to Use Code-First vs Pure Splat

| Scene Element | Recommendation | Why | |--------------|---------------|-----| | Flat surfaces (walls, floors, desks) | Procedural code | Simple, editable, tiny file size | | Parametric objects (cabinets, shelves) | Procedural code | Adjust dimensions in code | | Organic objects (plants, food, fabric) | 3DGS splat | Can't match quality procedurally | | View-dependent surfaces (screens, mirrors) | 3DGS splat | SH coefficients capture view dependence | | Articulated parts (joints, hinges) | Procedural code | Joint parameters are explicit in code | | Mixed scenes (most real cases) | Hybrid code + splat | Best of both worlds |

### SLAT Connection

The code-first approach connects to SLAT (see `../../references/slat-unified-representation.md`): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. **SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.**

## SLAT Latent Editing (v1.0.0)

> **Theoretical basis**: SLAT (Structured Latent Aggregation Transform) — see `../../references/slat-unified-representation.md`. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.

### Encoding: Scene → Structured Latent

`encode_scene_slatent` voxelizes the active scene into a regular grid (`voxel_size`, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a `slat_id` referencing an in-memory snapshot with an `encode_loss` (reconstruction RMSE), letting the agent judge fidelity before editing.

### Editing in Latent Space

`edit_scene_latent` applies a `LatentEditOp` to voxels matched by a `LatentSelector` (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:

| Op | Fields | Effect | |----|--------|--------| | `translate` | `delta: Vec3` | Move matched voxels (and their Gaussians) by a vector | | `scale` | `factor: number`, `origin: Vec3` | Scale voxel positions relative to an origin | | `rotate` | `angleDeg: number`, `axis: Vec3`, `origin: Vec3` | Rotate voxels around an axis (degrees) | | `recolor` | `color: Vec3`, `mix: number` | Blend matched voxels' colors toward a target | | `opacity` | `opacity: number`, `mode` | Set or scale opacity (mode `set`/`scale`) | | `smooth` | `iterations: number`, `strength: number` | Smooth feature positions/colors by averaging neighbors | | `delete` | `target: "voxel"` | Remove all Gaussians in matched voxels |

**Schema vs core naming**: the MCP JSON schema uses snake_case (`angle_deg`); the internal `LatentEditOp` uses camelCase (`angleDeg`). Handlers convert at the boundary. Library/test callers use camelCase directly.

**Safety gate**: `edit_scene_latent` computes `affected_gaussians`; if this exceeds 10% of the scene, the edit is rejected unless `confirm=true`. This reuses the project-wide 10% safety rule.

**Apply to scene**: with `apply_to_scene=true` (default) the edit is re-decoded and broadcast to the renderer via `modify_gaussians`; with `false` it only updates the in-memory snapshot, so the agent can preview/cancel before committing.

### Decoding: Latent → Scene

Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. `delete` removes the affected Gaussians entirely.

### Voice-Driven SLAT Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).

## Cross-Scene Latent Transfer & Interpolation (v1.1.0)

v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be **transferred** to another scene (target), or the two scenes can be **interpolated** in latent space. Both operations rely on a spatial correspondence built over the voxel grids.

### Correspondence: Voxel Grid Matching

Both operations build a voxel grid over the source scene (cell size = `match_radius`) via `buildVoxelGrid`, then for each target voxel find the nearest source voxel within `match_radius` (`nearestVoxel`, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.

### Transferring a Latent Edit

`transfer_scene_edit` re-applies a `LatentEditOp` from source to target **as a relative change**:

|

--- name: 3dgs-mcp-renderer description: "MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering, voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering, agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting, interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering." license: Apache-2.0 metadata: version: "1.1.0" author: jaccen tags: ["mcp", "3dgs", "gaussian-splatting", "rendering", "three.js", "webgpu", "voice", "agent", "interactive", "spec-first", "sculpting", "code-first", "slat", "latent-editing"] disable-model-invocation: true user-invocable: true ---

# 3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

## Architecture

``` ┌─────────────┐ ┌─────────────┐ ┌──────────────────┐ ┌──────────────────┐ │ Voice/Text │────▶│ Agent │────▶│ MCP Server │────▶│ 3DGS Renderer │ │ (Whisper/ │ │ (Claude/ │ │ (Node.js/ │ │ (Three.js/ │ │ Prompt) │ │ TeleClaw) │ │ Python) │ │ WebGPU/HiGS/ │ │ │◀────│ │◀────│ │◀────│ DDF-GS) │ └─────────────┘ └─────────────┘ └──────────────────┘ └──────────────────┘ │ │ │ │ Tool calls │ WebSocket/HTTP │ WebGL/WebGPU/ │ (MCP protocol) │ transport │ HiGS/DDF-GS ```

## Spec-First Sculpting Pipeline (v0.9.0)

> **Design inspiration**: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) **spec-first** — define quality criteria and component hierarchy before any rendering; (2) **stage-gated sculpting** — progressive refinement with acceptance checks at each stage.

### Why Spec-First for MCP Rendering?

The original MCP pipeline was **reactive**: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

- No upfront quality criteria → agent cannot self-assess before rendering - No stage gates → errors compound across steps (bad camera → bad selection → bad edit) - No component hierarchy → edits are flat, no part-level control

**The fix**: Introduce a `define_scene_spec` tool that runs *before* any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

### The 6-Stage Sculpting Pipeline

``` ┌─────────────────────────────────────────────────────────────┐ │ SPEC-FIRST SCULPTING │ │ │ │ ┌──────────────┐ │ │ │ define_scene │ ← Object Spec: component hierarchy, │ │ │ _spec │ material system, quality criteria │ │ └──────┬───────┘ │ │ │ │ │ ▼ │ │ Stage 1: blockout → Bounding boxes, camera framing │ │ │ gate: bbox coverage ≥ spec.target_coverage? │ │ ▼ │ │ Stage 2: structural → Part decomposition, hierarchy │ │ │ gate: part count & nesting matches spec? │ │ ▼ │ │ Stage 3: form → Gaussian density/scale/rotation │ │ │ gate: PSNR estimate ≥ spec.min_psnr? │ │ ▼ │ │ Stage 4: material → PBR/SH assignment per part │ │ │ gate: material count per part matches spec? │ │ ▼ │ │ Stage 5: surface → Normal consistency, thin structures │ │ │ gate: normal consistency score ≥ spec.threshold? │ │ ▼ │ │ Stage 6: lighting → Environment, shadows, AO │ │ gate: render quality score ≥ spec.target_score? │ └─────────────────────────────────────────────────────────────┘ ```

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's `blockout → structural → form → material → surface → lighting` flow.

### Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

``` 1. Execute stage tool (e.g., sculpt_form with parameters) 2. Call render_frame() to get current visual state 3. Call query_scene(query_type="stats") to get quantitative metrics 4. Compare metrics against spec gate criteria 5. If pass → advance to next stage 6. If fail → adjust parameters and retry (max 3 attempts) 7. If 3 failures → report to user with diagnostic info ```

### Voice-Driven Sculpting Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

## Code-First Rendering Philosophy (v0.9.0)

> **Design inspiration**: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.

### Traditional 3DGS Export vs Code-First Export

| Aspect | Traditional (.ply/.splat) | Code-First (.js + .splat) | |--------|--------------------------|--------------------------| | Editability | Binary blob, hard to edit | Source code, any field adjustable | | Version control | Binary diff, no merge | Text diff, git-friendly | | File size | Full Gaussian set (MB-GB) | Code skeleton (KB) + compressed splat data | | Scene composition | Single flat Gaussian cloud | Hierarchical code with part-level control | | Interaction logic | Must be added externally | Embedded in code | | 3DGS data | All in one file | Separate .splat file loaded by code | | Procedural elements | Not supported | Parametric geometry in code (e.g., desk surface = PlaneGeometry) |

### Hybrid: Procedural Code + 3DGS Splatting

The key insight: **not everything needs to be Gaussians**. For a desk scene: - Desk surface → procedural `BoxGeometry` in code (simple, editable, lightweight) - Monitor screen texture → procedural `MeshStandardMaterial` (or 3DGS if view-dependent) - Complex organic objects → 3DGS splatting data (where procedural code can't compete)

> **Loaded on demand** — See [code-first-examples.md](references/code-first-examples.md) for hybrid export code examples.

### When to Use Code-First vs Pure Splat

| Scene Element | Recommendation | Why | |--------------|---------------|-----| | Flat surfaces (walls, floors, desks) | Procedural code | Simple, editable, tiny file size | | Parametric objects (cabinets, shelves) | Procedural code | Adjust dimensions in code | | Organic objects (plants, food, fabric) | 3DGS splat | Can't match quality procedurally | | View-dependent surfaces (screens, mirrors) | 3DGS splat | SH coefficients capture view dependence | | Articulated parts (joints, hinges) | Procedural code | Joint parameters are explicit in code | | Mixed scenes (most real cases) | Hybrid code + splat | Best of both worlds |

### SLAT Connection

The code-first approach connects to SLAT (see `../../references/slat-unified-representation.md`): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. **SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.**

## SLAT Latent Editing (v1.0.0)

> **Theoretical basis**: SLAT (Structured Latent Aggregation Transform) — see `../../references/slat-unified-representation.md`. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.

### Encoding: Scene → Structured Latent

`encode_scene_slatent` voxelizes the active scene into a regular grid (`voxel_size`, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a `slat_id` referencing an in-memory snapshot with an `encode_loss` (reconstruction RMSE), letting the agent judge fidelity before editing.

### Editing in Latent Space

`edit_scene_latent` applies a `LatentEditOp` to voxels matched by a `LatentSelector` (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:

| Op | Fields | Effect | |----|--------|--------| | `translate` | `delta: Vec3` | Move matched voxels (and their Gaussians) by a vector | | `scale` | `factor: number`, `origin: Vec3` | Scale voxel positions relative to an origin | | `rotate` | `angleDeg: number`, `axis: Vec3`, `origin: Vec3` | Rotate voxels around an axis (degrees) | | `recolor` | `color: Vec3`, `mix: number` | Blend matched voxels' colors toward a target | | `opacity` | `opacity: number`, `mode` | Set or scale opacity (mode `set`/`scale`) | | `smooth` | `iterations: number`, `strength: number` | Smooth feature positions/colors by averaging neighbors | | `delete` | `target: "voxel"` | Remove all Gaussians in matched voxels |

**Schema vs core naming**: the MCP JSON schema uses snake_case (`angle_deg`); the internal `LatentEditOp` uses camelCase (`angleDeg`). Handlers convert at the boundary. Library/test callers use camelCase directly.

**Safety gate**: `edit_scene_latent` computes `affected_gaussians`; if this exceeds 10% of the scene, the edit is rejected unless `confirm=true`. This reuses the project-wide 10% safety rule.

**Apply to scene**: with `apply_to_scene=true` (default) the edit is re-decoded and broadcast to the renderer via `modify_gaussians`; with `false` it only updates the in-memory snapshot, so the agent can preview/cancel before committing.

### Decoding: Latent → Scene

Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. `delete` removes the affected Gaussians entirely.

### Voice-Driven SLAT Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).

## Cross-Scene Latent Transfer & Interpolation (v1.1.0)

v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be **transferred** to another scene (target), or the two scenes can be **interpolated** in latent space. Both operations rely on a spatial correspondence built over the voxel grids.

### Correspondence: Voxel Grid Matching

Both operations build a voxel grid over the source scene (cell size = `match_radius`) via `buildVoxelGrid`, then for each target voxel find the nearest source voxel within `match_radius` (`nearestVoxel`, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.

### Transferring a Latent Edit

`transfer_scene_edit` re-applies a `LatentEditOp` from source to target **as a relative change**:

|