====== LLM-Based VR Activity Recognition ======

This project aims to recognize human activities through analyzing sensor data with artificial intelligence (AI) technologies. 

Collecting sensor data from VR headsets & controllers and develop machine learning models to classify human activities

This project will also involve leveraging large language models (LLMs) for sensor data interpretation.

=== Week 1: [] 

Studied paper "Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display"

Studied paper "FitCoach: Virtual fitness coach empowered by wearable mobile devices"

=== Week 2: [] 
Started setting up necessary programs on our devices 
 - Android Studio, Arctic Fox
 - MATLAB

Worked on slides template to use for future presentations.  
Also started to look at previous data and experiment around with MATLAB features for visualization.


=== Week 3: [[https://docs.google.com/presentation/d/10RiVvwob7MiTjVPdoz9zFv8Z5Y2O55E6WgtBuDEMQMU/edit?slide=id.g3704141428e_2_180#slide=id.g3704141428e_2_180|Slides]] 

Studying papers related to activity recognition using motion sensor 
 - Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
 - Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

Studying the concept of Inertial Measurement Unit (IMU)
 - IMU sensor includes both accelerometer and gyroscope
 - Accelerometer and gyroscope record the linear acceleration and angular velocity of the device respectively, which can be used for classification

Studying programming using Java/Javascript in Android Studio
 - Exploring APIs of the IMU sensor from VR headsets

Started basic tests with Oculus Quest 1 and graphed them to visualize the data
 - turn head left and right (in one recording)

Accelerometer X Axis:

[[Image(Week 3 Acc X-Axis.png, width=400, height=200)]]

Gyroscope X Axis:

[[Image(Week 3 Gyro X-Axis.png, width=400, height=200)]]

=== Week 4: [[https://docs.google.com/presentation/d/1KR095rUYToCB6YsFtAxhQfv84Mugkq0Iz3CxqZKLWIs/edit?slide=id.g36300db7477_0_174#slide=id.g36300db7477_0_174|Slides]] 

More in depth analysis of papers
 - Motion sensor from VR: Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display [1]
 - Motion sensor from wearable devices: FitCoach: Virtual fitness coach empowered by wearable mobile devices [2]

In depth analysis in the inner workings of an IMU sensor

Accelerometer
 - Takes variance in capacitor charge to determine linear acceleration

 - Measures linear acceleration for 3 axes: x, y, and z 

Gyroscope
 - Small silicon mass vibrating at high Hertz.  

 - Coriolis effect applies, deflecting the silicon mass perpendicularly

 - Measures angular velocity around 3 axes: pitch, roll, and yaw

Additionally, we tried graphing one of our experimental trials in a 3d graph because we thought it would make visualization of the actions easier. 

[[Image(3d-head-spin-graph.png, height=350, width=700)]]

Drifting was a major issue when graphing and we still haven't fully fixed it.
=== Week 5: [[https://docs.google.com/presentation/d/1e0RMR1sGx5zReXR4hQsTyly3uDP29hHNLIj4iPaGZzA/edit?slide=id.g36334886556_0_343#slide=id.g36334886556_0_343|Slides]] 


This week we collected training data and testing data for model recognition, a total of 250 samples (50 per) for the motions below:
 - Turn head left 
 - Turn head right
 - Nodding
 - Stand up
 - Walk
[[Image(stand.gif, width=250, height=400)]][[Image(walk.gif, width=250, height=400)]][[Image(nod.gif, width=250, height=400)]][[Image(turnright.gif, width=250, height=400)]][[Image(turnleft.gif, width=250, height=400)]]
---- 

The data we collected all had a 2-second interval with faulty and incomplete data. We were unable to determine what caused this error, but every sample had this error. All data after this 2-second interval was accurate, so we decided to trim out the 2 seconds of faulty data and only take the last 5000 (effectively 5 seconds) samples.  We also used a sliding window moving average with a window size of 75 to smooth out the noise.

Before segmentation:

[[Image(Screenshot 2025-07-23 105427.png​, width=500, height=250)]]

Zoomed in of the boxed section

[[Image(Screenshot 2025-07-23 103349.png​​, width=500, height=127)]]

After segmentation:

[[Image(Screenshot 2025-07-23 110334.png​, width=500, height=250)]]

After segmentation and smoothing:

[[Image(Screenshot 2025-07-24 102937.png​, width=500, height=300)]]

Questions or contributions? Please contact us.

[1] Zhao, Jingbo, et al. "Real-time recognition of in-place body actions and head gestures using only a head-mounted display." 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). IEEE, 2023.

[2] Guo, Xiaonan, Jian Liu, and Yingying Chen. "FitCoach: Virtual fitness coach empowered by wearable mobile devices." IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 2017.