perceiving-objects-oneshot
Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Returns a clean not_found output (no exception) when the VLM answers "none" or DINO emits no detections — making this the right skill for clean-all-items loops whose natural termination signal is "no more matching objects in view". Use when a multi-item loop needs a clean no-match exit, or for generic target descriptions on uncluttered scenes with reasonably sized targets.
graph-robots · AI review on file
Installation
Read the source and dependencies before running a command. Copying does not install or execute anything.
npx skills add graph-robots/open-robot-skills --skill perceiving-objects-oneshotAbout this skill
name: perceiving-objects-oneshot description: > Lightweight one-shot 3D object perception. Runs Grounding-DINO broad detection, a SINGLE VLM set-of-marks letter pick over the labeled boxes, SAM3 box segmentation, depth back-projection, and geometry.filter_and_compute_obb. No pairwise tournament, no multi-view safe-gate. Returns a clean not_found output (no exception) when the VLM answers "none" or DINO emits no detections — making this the right skill for clean-all-items loops whose natural termination signal is "no more matching objects in view". Use when a multi-item loop needs a clean no-match exit, or for generic target descriptions on uncluttered scenes with reasonably sized targets. license: MIT compatibility: requires gap>=0.1 metadata: category: perception tags: [perception, dino, vlm, one-shot, set-of-marks] gap: allowed_tools: - robot.get_observation - grounding-dino.detect - vlm.query - sam3.segment_box - geometry.mask_to_world_points - geometry.filter_and_compute_obb exit_conditions: found: Target detected; OBB and mask bound in subgraph outputs. not_found: VLM replied "none" or DINO emitted no detections. In clean-all-items loops route to done; in normal pick-and-place route to abort. produces_outputs: "_obb": OrientedBoundingBox "_mask": Mask "_cloud": PointCloud errors: - "NOT_FOUND: No detection matched the target description." hard_rules: - perception_pipeline_invariants.md#emit-both-obb-and-mask - geometry_calling_conventions.md#obb-field-binding canonical_scripts: - perceive_simple: scripts/perceive_simple.py prompts: vlm_one_shot: prompts/vlm_one_shot.md references: - title: Perception pipeline invariants (emit obb + mask + cloud) path: references/perception_pipeline_invariants.md - title: Geometry tool calling conventions (output field binding) path: references/geometry_calling_conventions.md examples: - title: Canonical perception subgraph (observe → perceive → filter_obb) path: examples/canonical_subgraph.json streaming: false
perceiving-objects-oneshot
Single VLM call over a set-of-marks overlay. Pipeline:
observe → perceive → filter_obb
perceive runs:
grounding-dino.detectwith a broadobject.text prompt- One
vlm.queryshowing the image with letter-labeled boxes: "Which letter is the ? Reply with one letter or 'none'." - On
none: emitfound: Falseso the subgraph exitsnot_found. On a letter:sam3.segment_boxon the chosen box,geometry.mask_to_world_pointsfor the cloud.
When to use
- Clean-all-items / multi-item loops where the cycle needs a clean
"no match" signal to terminate via
target.not_found → done. - Tasks where the target description is generic ("any item on the floor", "the next remaining grocery item") rather than a specific scene-spec id.
- Uncluttered scenes with distinct, reasonably sized targets where the set-of-marks letter pick is reliable.
When NOT to use
- Small / cluttered targets (< 40 px wide) — prefer
perceiving-objectswhose pairwise crop tournament is far more reliable in that regime.
Recommended subgraph state flow
3 states: observe → perceive → filter_obb (mirrors perceiving-objects).
State details:
About
object_namebelow: it is a literal Python string — the natural noun phrase for the object you are perceiving, drawn from this subgraph's description (e.g."alphabet soup","basket","any grocery item on the floor"). It is a constant per subgraph instance, NOT a binding. DO NOT writeRef("in.object_name")or any otherRef(...); the coordinator does not declareobject_nameas a subgraph input. Write the string directly, e.g."object_name": "any grocery item on the floor". The same rule applies toobject_descriptionif you set it.
observe—type: tool,tool: "robot.get_observation",inputs: {}. Connector tool; flat name only.perceive—type: script, filescripts/<sg>/perceive_simple.pyfrom this bundle. Inputs:cameras=Ref("observe.cameras"),object_name="<noun phrase from the subgraph description>", plus any optional literals (object_description,dino_prompt). Returns{found, cloud, mask, score}. When the VLM picks "none" or DINO emits no detections,foundisFalseand the downstreamfilter_obbstep then raises (empty cloud) — caught by the subgraph'son_error: "not_found"exit.filter_obb—type: tool,tool: "geometry.filter_and_compute_obb",inputs={"points": Ref("perceive.cloud")}. Returns{"obb": <OrientedBoundingBox>}.
Wiring the exit (HARD)
Linear perceive → filter_obb → found → END. The filter_obb tool
raises on empty clouds (the not-found path), and the subgraph's
on_error: "not_found" catches that. Do NOT add conditional edges on
perceive — the linear path plus set_on_error is sufficient.
sg.add_node("filter_obb", type="tool",
tool="geometry.filter_and_compute_obb",
inputs={"points": Ref("perceive.cloud")})
sg.add_exit("found")
sg.add_edge("perceive", "filter_obb")
sg.add_edge("filter_obb", "found")
sg.add_edge("found", END)
sg.set_on_error("not_found")
Bind the subgraph outputs (ALL THREE — required, no exceptions):
sg.set_outputs(
target_obb=Ref("filter_obb.obb"),
target_mask=Ref("perceive.mask"),
target_cloud=Ref("perceive.cloud"),
)
Note that geometry.filter_and_compute_obb returns {"obb": ...}, so
the OBB binding walks into the obb field (Ref("filter_obb.obb"),
NOT a bare Ref("filter_obb")). See
references/geometry_calling_conventions.md.