Development of a Robot Arm - Visual Perception, Detection, Kinematics and Planning


Robotic Systems Laboratory, University of Michigan


This work was completed for the lab as a team project by a group of 3.

My contribution to this work : My major contributions include development of the Vision system - Calibration, Workspace Reconstruction, Block Color and Size Detection, development of Forward and Inverse Kinematics 

Problem Description

Developing the algorithms and software to enable a robot arm in a workstation equipped with a camera to perform various pick and place tasks

Doing so, involves developing algorithms for

Project overview

In this project we develop the higher level software stack for the ReactorX-200 robot arm to perform various pick and place operations of varying size blocks. The workspace also has a mounted RGB-D camera, using which we developed a vision system to detect the 3D location, color, size of the blocks. The two sub-systems are integrated using a state machine.

Key achievements

Vision System - I

Calibration was performed using checkerboards and Workspace reconstruction using APRIL Tags. The outcome was the generation of intrinsic and extrinsic camera matrices. The calibration process was integrated into the state machine. The used mask and the generated workspace is shown here

Vision System - II

We developed an algorithm that fuses both RGB and HSV colorspace information to robustly detect colors of different blocks (many of which are worn out). This involved a lot of tuning to figure out the bandwidth. Also, the '0' and '1' indicate the size of the block. The position information is passed on to the plannning module.

Kinematics - Highlights

Project outcome

We perform multiple tasks using the developed software, including sorting in the rainbow order, and stacking blocks to heights.