Human Robotics Virtual Reality Experimental Platform

This virtual reality experimental platform is developed using UE for human-robot interaction research, such as studying human behaviour in or while interacting with autonomous vehicles. Core logic is implemented in C++, with example usages for a haul truck maneouvering environment provided through Blueprints.

Features

• Experimental flow control framework that handles the preparation, start, repetition, and end of experiments, as well as data logging during the experiments., see the GameMode class for details.

• Multi-modal sensor data acquisition system with a defined sensor interface and a sensor registration mechanism, see SensorSystem for details. The integrated sensors includes:

  Modality	Sensor	Note
  Gaze	VIVE Full Face Tracker	Based on VIVE OpenXR plugin
  Electromyography (EMG)	Delsys Trigno EMG	Up to 2kHz sampling rate, realized through Trigno Control Utility and dedicated UE plugin, see DelsysTrignoEMG plugin folder
  Distance radar	Virutal distance radar	Virtual sensor
  Motion Capture	XSens	IMU-based system, data saving in progress

• User information and experimental data saving system, see UserSystem and DataloggerSystem for details.

• Compatible VR headset:

  Headset	XR Plugin
  HTC VIVE XR Elite	VIVE OpenXR

• VR environments:

  • Mining haul truck maneouvering tasks for driver attention prediction research

    

• Performance:

  • NVIDIA's Deep Learning Super Sampling (DLSS) is used for improving VR performance.

Documentation

To be prepared.

Quick Demo

1. Install Unreal Engine 5.4.4.
2. Install NVIDIA DLSS 4.0 UE plugin.
3. For a quick demo,
   1. Open and run "LaunchMap" in the root folder.
   2. Under "Select Experiment", select "TruckTestMap".
   3. Under "User Setting", add new user information.
   4. Click "Start Experiment" and go.