Robot Jenga Assistant
ProjectIn this project, my group and I created a ROS2 package to turn an Franka Emika Panda 7DOF arm into a Jenga assistant. Our core goal was to have the robot play alongside a human. The user would select a block and partially remove it from the tower. The robot would then detect this block, remove it the rest of the way, and place it on the top of the tower.
Computer Vision
My main role in this project was to implement the perception pipeline in order to detect where the Jenga tower and the partially removed pieces are. We used an Intel Realsense D435i camera, which provides an RGB-D image, and placed it directly above the center of the tower, looking down. From this viewpoint, there are three groups of objects that exist at distinct distances from the camera: first, the top of the tower; second, any partially removed pieces; and finally, the table.
Upon startup, the CV node performs a scan through the depth frame to locate the top of the tower and the table. Then, it repeatedly scans between these two locations to search for pieces to pull. When one is found, the node broadcasts the location of its centroid with respect to the camera, so that the movement node can start planning to grab the piece. The transformation between camera frame and robot frame is handled by a separate calibration run using April Tags.
Early work scanning to detect the top of the tower, a piece, and the table within a square region surrounding the tower.
The vision node also computes the orientation of the top of the tower on its initial scan. This is necessary so that the pulled pieces can be placed in the correct orientation. When it locates the top of the tower in the depth frame, it applies Canny edge detection and Hough lines on the corresponding portion of the RGB image to determine the angle of the pieces.
Detecting the orientation of the tower top
Motion
The block’s starting location is computed during the CV scans, and the desired ending locations are precomputed based on the initial height/orientation of the tower and the width of the pieces (so as to ensure that the 3 pieces placed per layer are adjacent). We followed a motion plan that first orients the end effector to the piece, then moves in a cartesian path to grab it and pull it slowly straight backwards. Then, we raise the piece up, orient it to the correct ending location, place it slightly offset from the top of the tower (so that the grippers don’t push any other pieces on the level), and gently push it in using the tip of the grippers. All of the trajectories were planned and executed using a custom API we wrote for MoveIt2.
A full turn (2x speed)
Extra Features
We added a feature to “poke” out a block from the side of the tower, using a poker we created wth a flexible silicone tip. This allows the arm to complete an entire turn without any human interaction with the tower. However, the functionality is still very limited as we chose the location of the block to poke ahead of time. Given more time with this project, it would be interesting to set up another camera to the side of the tower to detect pieces to poke automatically.
"Poke" feature (2x speed)
The last feature we added was a command to knock over the tower!
Destroying the tower in a fit of rage
(2x speed)
Acknowledgements
Thanks to my amazing group members: Alyssa Chen, Liz Metzger, and Hang Yin!