For my final course project at Arizona State University, I created a simulation that used a robot manipulator to autonomously pick and place colored cubes. In the simulation, a Universal Robot 3 (UR3) and an Intel Realsense Depth Camera D435i were used. The program used Python, Robot Operating System (ROS), OpenCV, and the Kivy library for the user interface. I wasn't able to complete the project because of time constraints, but it was still very successful.

This project presented a method for integrating data from onboard proximity sensors with an externally mounted RGBD sensor in order to create a holistic environment representation for a robot manipulator. In order for robots to interact safely and consistently with their environment, they need to be able to sense their surroundings. The main contribution of this work was multisensory integration, which is the fusion of data from different sensors. We used an occupancy grid map to represent information from the vision sensor. This map represented the environment with evenly spaced cubes. Each cube represented a location in space that symbolized the presence of an obstacle. We combined the occupancy grid map with data from the proximity sensors using an algorithm that weighed the influence of each sensor measurement. The weight of each sensor measurement depended on the distance between the measurements and the robot. 

We evaluated our method using the Franka Emika Panda robot arm, onboard proximity sensors, and an external Kinect RGB-D sensor in a Gazebo simulation. We validated our approach with two demonstrations. For the first demonstration, we placed an obstacle outside in the vision sensor’s field of view. For the second demonstration, we placed an obstacle in front of the vision sensor, occluding its view. The goal for the demonstrations was to show the advantages of using both sensor types in an integrated framework.

Although we never published the paper written for this project, there is a final draft, a PowerPoint presentation, and a poster presentation available for review. However, the code is not available. Please reach out if interested.