Version 15 (modified by 16 months ago) ( diff ) | ,
---|
Robotic IoT Smartspace Testbed
Robotic IoT SmartSpace Testbed
WINLAB Summer Internship 2023
Group Members: Jeremy Hui, Katrina Celario, Julia Rodriguez, Laura Liu, Jose Rubio, Michael Cai
Project Objective
Based on the future research section, there seem to be two main goals: adding a web-based reservation system for remote access to the robot and automating the activity labeling process using the natural language descriptions of the data provided in video format. For the remote reservation/experimentation features, we need to create a user-friendly webpage that not only makes it easier for the user to execute commands, but also keeps them from changing things that they shouldn’t have access to. The remote access to the LoCoBot can be achieved through ROS/SSH as long as all machines involved are connected to the same network (in our case this could be a VPN).
The video auto-labeling can be done using neural networks (ex: CNN and LSTM) in an encoder-decoder architecture for both feature extraction and the language model. For activities that cannot be classified within a specific amount of certainty, the auto-labeling tool could save the time stamp and/or video clip and notify the user that it requires manual labeling. After being fully trained, the network would simply choose the label with the highest probability and could possibly mark that data as “uncertain”.
- Web-based remote reservation/experimentation
→ remote access to robot through ROS/SSH → create a user-friendly webpage that not only makes it easier for the user to execute commands, but also keeps them from accessing/changing things that they shouldn’t
- Automatic labeling using video/captioning tools
→ Neural network models for captioning involve two main elements:
→ Feature Extraction
→ Language Model
Progress Overview
WEEK ONE
- Familiarized on project topics by reading relevant research papers
- Set up ROS (Robotic Operating System) on Ubuntu distro
- Ran elementary python scripts on robot (talker/listener)
- Interacted with robot at CoRE
WEEK TWO
- Started to build the website
- Learned the aspects of a Raspberry pi by starting with a blank SD:
→ Installed Raspberry Pi operating system and ROS
→ Connected to a ZeroTier network
→ Downloaded packages necessary for the data collecting python scripts
WEEK THREE
- Explored the possibility of including Unity in the project
→ Created 3D avatar in Unity which mimics live webcam feed
- Researched necessary equipment needed for experiments
- Started to clone original SD card that contained all python scripts, packages, and ROS onto blank SD cards
→ The cloned cards were inserted into 25 Raspberry Pi
→ Each pi was given a unique name
- Made significant progress on the website aspect
WEEK FOUR
- Measured dimensions of WINLAB
WEEK FIVE
- Automated process of connecting & checking wifi connection (pinging)
- Created a tunnel between R-pi and data base
- Successfully created backend email sender & appointment form on website
- Added foundation for interactive grid on website
WEEK SIX
- Continued exploring Unity
→ Began connecting Unity/ROS - SLAM for robot navigation
- Set up remote desktop on orbit node for Unity
WEEK SEVEN
- Set up PTP on Pi's
→ Pi's do not have hardware timestamping → No IEEE 1588 , must aim Pi at boundary server → Solution: use software emulation → Build kernel that allows ptp4l to run
- Looked into using Raspberry Pi cameras for camera data
→ Successfully captured videos and viewed them using VLC
- Created the coordinate system for the test room
→ Discussed the predetermined activities and based layout on them & outlet placement
→ Turn on Light → Walk in/out of room
Attachments (19)
-
Screenshot 2023-07-05 124600.png
(264.1 KB
) - added by 16 months ago.
Digital Twin Week 3
-
Screenshot 2023-07-05 124811.png
(46.3 KB
) - added by 16 months ago.
Website Week 3
- winlab.png (575.9 KB ) - added by 16 months ago.
- winlab (1).png (104.5 KB ) - added by 16 months ago.
- raspberry-pi-camera-pinout-camera-2.png (133.9 KB ) - added by 16 months ago.
- Preliminary Floorplan - IoT Testbed.png (30.6 KB ) - added by 16 months ago.
-
Maestro.jpg
(554.3 KB
) - added by 16 months ago.
Maestro
-
Raspberry Pi 3 B+.jpeg
(104.3 KB
) - added by 16 months ago.
r-pi
-
Raspberry_Pi_Camera_Module_3_2.jpeg
(213.3 KB
) - added by 16 months ago.
camera
-
website_test.png
(221.3 KB
) - added by 16 months ago.
simple website
-
remote_desktop.png
(169.1 KB
) - added by 15 months ago.
desktop
- Point_Cloud_Unity - SampleScene - Windows, Mac, Linux - Unity 2022.3.4f1_ _DX11_ 2023-07-24 13-38-49.mp4 (26.1 MB ) - added by 15 months ago.
-
Screen Shot 2023-08-01 at 2.21.47 PM.png
(327.5 KB
) - added by 15 months ago.
database
- Screenshot 2023-07-31 160018-min.png (2.4 MB ) - added by 15 months ago.
- ptp.png (106.2 KB ) - added by 15 months ago.
- Screenshot 2023-08-07 144659.png (96.2 KB ) - added by 15 months ago.
- Screenshot 2023-08-07 145140.png (160.9 KB ) - added by 15 months ago.
- Screenshot 2023-08-07 145732.png (220.1 KB ) - added by 15 months ago.
- Screenshot 2023-08-07 145902.png (562.9 KB ) - added by 15 months ago.