Overview

Make me a sandwich.

A self-balancing robot with dual 7-DOF arms assembles a sandwich end-to-end, driven by pi0.5 VLA policies trained on ~50 demonstrations each.

BracketBot/pi0.5 VLA/2x 7-DOF Arms
Actions100%
  • Pull toaster knobpi0.5

    Learned policy controls the gripper to press and hold the toaster lever.

  • Bread to platepi0.5

    Learned policy grasps bread from the toaster slot and places it on the plate.

  • Apply mayopi0.5

    Learned policy squeezes and spreads mayo across the bread surface.

  • Place lettuceIK

    IK solver computes joint angles to pick and place lettuce onto the sandwich.

Lettuce Detectiondetector.py
1. Finding Green
▸Convert each frame to HSV (hue, saturation, brightness) to separate colour from lighting
▸Green range: hue 35–78, with minimum saturation + brightness thresholds. Upper limit at 78 excludes teal/cyan
▸Cleanup: remove specks, fill small holes, group into blobs. Blobs <1500 px are discarded
▸Split handling: blobs within 40 px merge into one object — a gripper occluding a leaf doesn’t create two detections
2. Head Camera → Table Position
▸HeadModel maps pixel coordinates from the head camera to 3D table-frame positions
▸Detected blob centroid is projected through the model to get the lettuce footprint on the table surface
▸IK solver receives the table-frame coordinates and computes the pick-and-place trajectory
VLA Pipelinepi0.5
Data Collection
▸~50 teleoperated demonstrations per policy, collected via leader-follower arms
▸Home-position reset after every run — arms return to a fixed starting pose
▸Toaster and plates are manually repositioned between episodes for variation
Training
▸Demonstrations are uploaded and fine-tuned on an external GPU server
▸Training completes in <1 hour per policy
Inference
▸pi0.5 is too large to run on the onboard Jetson
▸Inference runs on a remote GPU endpoint called over the network
▸Robot streams wrist camera frames to the server, receives action vectors back in real time
Telemetry
Modelpi0.5
Episodes / Policy~50
Training / Policy<1hr
Learned Policies3
IK Policies1
Arms2x 7-DOF
System
BracketBot
7-DOF Arms + Grippers
Depth + RGB Camera
Balancing Controller
Server
Policy Sequencer
pi0.5 Inference
IK Solver
WebSocket Relay
Web App
3D Visualization
Camera Feeds
Action Map
Status Panel
Pipeline
1Collect
~50 episodes
2Train
<1 hour
3Deploy
On-robot
4Evaluate
Real-world
Home-position resets between episodes
Log
>Training: <1hr per policy. Trained all 3 VLA policies in one session.
>Data collection: home-position resets between episodes. Consistent initial state.
>Episodes per policy: ~50 teleoperated demonstrations.
>TODO: autonomous navigation + mapping. SLAM or visual-nav for delivery.
The Brackey Way
pi0.5Next.jsThree.jsR3FWebSocket