🤖 Robotics Blog
My single-color pick-and-place policy generalized to new shapes and colors, but broke the moment I changed the tray's color. So I built a real sorting task: two trays, two colors, and a policy that has to actually look at the block before deciding where it goes.
My color-sorting ACT policy suddenly started missing the block, reaching to the same spot every time regardless of where the block actually was. Here's how I used the raw recorded training trajectories — replayed directly on the physical arm — to figure out it was a workspace calibration problem, not a model problem.
Four steps to a working pick-and-place policy: 3D printing my own tray and block, collecting a clean 40-episode dataset, training ACT, and then trying to break it to see what generalizes.
Part 2: once the workspace and compute environment stopped fighting me, the next challenge was camera placement and data quality. What the robot actually sees turned out to matter more than I expected, and getting it right is what finally made the first successful pick-and-place happen.
I thought setting up a robot arm and LeRobot would be a weekend project. It took about a month — and almost none of that time was actually about robotics. Part 1: the workspace and compute environment problems nobody warns you about.