Projected Augmented-Reality

Augmented Reality is one of the core pillars of the upcoming Industry 4.0 (the next industrial revolution) and is expected to have an enormous impact in the future. This is supported by Meta’s (formerly Facebook) announced plans to unveil their own virtual reality world. Typically, extended reality experiences, like that of Meta and others we have seen, require users to make use of personal headsets. Our project, on the other hand, developed technology for a communal augmented reality experience without the need for personal devices. Our team developed a calibration system to bring this glasses-free augmented reality experience to life. The approach we took involved a projector, a powerful Windows Desktop, a Microsoft Azure Kinect Camera, and a Qualisys Motion Tracking system. Combining these hardware components, we were able to track objects entering the projector’s frustum and accurately display 3D images on moving physical objects.

Our final product entailed a C++ script that utilized OpenCV’s ArUco Marker detection module to estimate positions and scales of markers, a Qualisys Motion Tracking system to track rigid bodies moving around the room, a Unity program to tie all the hardware components together, and necessary documentation for further development of our project.

We found that the best way to approach this problem was to first project a scatter board of 25 ArUco markers on the physical calibration board. With the help of OpenCV and the Azure Kinect Camera, we then panned and scaled one projected ArUco marker to the center of the physical calibration board. Once OpenCV determined the ArUco marker was centered, the calibration was completed. Any graphic could then be accurately projected onto the moving board, giving the user an elegant glasses-free augmented reality experience.