LLM-Based VR Activity Recognition

This project aims to recognize human activities through analyzing sensor data with artificial intelligence (AI) technologies.

Collecting sensor data from VR headsets & controllers and develop machine learning models to classify human activities

This project will also involve leveraging large language models (LLMs) for sensor data interpretation.

Week 1: []

Studied paper "Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display"

Studied paper "FitCoach: Virtual fitness coach empowered by wearable mobile devices"

Week 2: []

Started setting up necessary programs on our devices

• Android Studio, Arctic Fox
• MATLAB

Worked on slides template to use for future presentations. Also started to look at previous data and experiment around with MATLAB features for visualization.

Week 3: ​Slides

Studying papers related to activity recognition using motion sensor

• Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
• Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

Studying the concept of Inertial Measurement Unit (IMU)

• IMU sensor includes both accelerometer and gyroscope
• Accelerometer and gyroscope record the linear acceleration and angular velocity of the device respectively, which can be used for classification

Studying programming using Java/Javascript in Android Studio

• Exploring APIs of the IMU sensor from VR headsets

Started basic tests with Oculus Quest 1 and graphed them to visualize the data

• turn head left and right (in one recording)

Accelerometer X Axis:

Gyroscope X Axis:

Week 4: ​Slides

More in depth analysis of papers

• Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
• Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

In depth analysis in the inner workings of an IMU sensor

Accelerometer

• Takes variance in capacitor charge to determine linear acceleration

• Measures linear acceleration for 3 axes: x, y, and z

Gyroscope

• Small silicon mass vibrating at high Hertz.

• Coriolis effect applies, deflecting the silicon mass perpendicularly

• Measures angular velocity around 3 axes: pitch, roll, and yaw

Additionally, we tried graphing one of our experimental trials in a 3d graph because we thought it would make visualization of the actions easier.

Drifting was a major issue when graphing and we still haven't fully fixed it.

Week 5: ​Slides

This week we collected training data and testing data for model recognition, a total of 250 samples (50 per) for the motions below:

• Turn head left
• Turn head right
• Nodding
• Stand up
• Walk

---

The data we collected all had a 2-second interval with faulty and incomplete data. We were unable to determine what caused this error, but every sample had this error. All data after this 2-second interval was accurate, so we decided to trim out the 2 seconds of faulty data and only take the last 5000 (effectively 5 seconds) samples. We also used a sliding window moving average with a window size of 75 to smooth out the noise.

Before segmentation:

Zoomed in of the boxed section

After segmentation:

After segmentation and smoothing:

Week 6: ​Slides

Used SVM to find features to input to the LLM. Using said features, we created a prompt that would allow the LLM to understand and categorize the raw data in the .txt files. Then, we compared the results of LLM vs SVM, achieving an accuracy of ~98% vs ~94%, respectively. This is probably due to the inherit nature of how LLMs and SVMs work, as SVMs are more black and white, categorizing based on only the data inputted as training, while LLM is blurs the lines, allowing for a more free interpretation of the small variations in the same dataset, allowing it to interpret the variations in the same movement correctly, unlike the SVM.

badoobeeeee

---

Questions or contributions? Please contact us.

[1] Zhao, Jingbo, et al. "Real-time recognition of in-place body actions and head gestures using only a head-mounted display." 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 2023.

[2] Guo, Xiaonan, Jian Liu, and Yingying Chen. "FitCoach: Virtual fitness coach empowered by wearable mobile devices." IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 2017.