MRRobotics
Project Description
The MR Robotics project explores robot operation in mixed reality (MR) and virtual reality (VR).
Figure 1 illustrates the basic components and structures of our MR mobile robot interaction system. The client-side of the system is an application built with the popular game engine Unity, which can be run on any MR device that supports the OpenXR backend. The MR client can be an OST HMD such as the Microsoft Hololens2, or a VST HMD such as the Oculus Quest2 VR headset with passthrough API. Using OpenXR backend allows our system to be compatible with a wide range of HMDs. Additionally, we use the MRTK, a popular MR development framework for creating immersive user interfaces.
Our system targets Anki Cozmo is an education mobile robot that is easy to set up and develop. The Cozmo robot has four differential wheels and a movable lift. The Cozmo Driver is a Python package that connects the Cozmo SDK with the robot operation system (ROS) to send commands and receive sensory feedback from the robot. We establish communication between the MR HMD client and the robot via the Transmission Control Protocol (TCP) connection over the local network using the ROS-TCP connection package provided by the Unity game engine. The proposed system creates a simple and easy-to-scale design space for developing HRI techniques in MR and could be used for both VST HMD and OST HMD.
[Figure 1] System Architecture Image
Mixed Reality for Robotics (Demo)
Repository: https://github.com/keli95566/VAMCozmo
Contact person:
Publication: “Towards Robust Exocentric Mobile Robot Tele-Operation in Mixed Reality”, Ke Li, Reinhard Bacher, Wim Leemans, Frank Steinicke; Proceedings of ACM SIGCHI Conference on Human-Robot Interaction (HRI) Workshop on Virtual, Augmented and Mixed Reality for Human-Robot Interaction 2022