Self Driving Vehicular Project

Team: Christopher Lee, Michelle Gutwein, Thierry Antoine, Varun Chirravuri, and Anjali Kapilavai
Advisors: Ivan Seskar and Jennifer Shane
Project Description & Goals:
Build and train miniature autonomous cars to drive in a miniature city.
Technologies: ROS (Robot Operating System), Pytorch

---

Week 1

Read through the ​manual provided to us.

​Week 1 slides

Week 2

Learned what ROS is and how to use it
Also got to know the car, RASCAL

​Week 2 slides

Week 3

Read through all of the code in the ​Gitlab
Learned how to record and edit data on the web display

​Week 3 slides

Week 4

Learned how to train a model
Trained and tested the model on simple paths

​Week 4 slides

Week 5

Trained model on making right and left turns in the city environment
Started training on turns at the intersection
Created a new model called gps which will allow users to give the car commands on the direction to turn at an intersection before running the car

​Week 5 slides

Week 6

Collected new data on making wider right and left turns in the city environment
Continued training on turns at the intersection
Updated the documentation to be more clear

​Week 6 slides

Week 7

Collected lots of data on left turns at the intersection
Figured out how to free up space on the car since it was full
Connected the car to intersection wifi to be able to connect though orbit rather then Zerotier

​Week 7 slides

Week 8

Tested the car to see if it would stop at intersections. It didn't.
Found out that in addition to automatically labeling lookahead point angles, we also needed to manually label intersection frames from the data
Now the car stops at most intersections
Recorded more data at and around the main intersection to fine tune this
Started work on creating a mapping system of the intersection area by measuring the length and width of the lanes

​Week 8 slides

Week 9

Collected data on the entire city at all intersections
Labeled data correctly by highlighting frames with the intersection in them
Tested the car with this new data - it still wasn't stopping at all the intersections and seemed to have trouble making some of the left and right turns
Printed out Aruco markers to put on the ceiling and made a virtual map of the city for the GPS system

​Week 9 slides

Connecting to Rascal

Through Zerotier:

1. Sign into Zerotier under the account rascalstonesdc-at-gmail.com password: si2022sdc
   
2. Connect to the network: rascalstonesdc's 1st network (ID: d3ecf5726d1a9fcd)
   
3. Now you can open a terminal and connect to the IP address of the rascal using the command: ssh rascal@10.244.114.238
   
4. In a separate terminal, enter ssh rascaltrain@10.244.176.240 to connect to the rascal server
   
5. If you get an error message that says: access denied public key then connect through ORBIT (instructions below) to add your public key to the /.ssh file titled authorized_keys
   
6. Now you should be able to connect to the server through Zerotier
   

Through ORBIT:

1. Make a reservation on your orbit account under the intersection bed
   
2. Open a new terminal and type ssh <your username>@console.intersection.orbit-lab.org
   
3. You should see the 'welcome to ORBIT-LAB' sign
   
4. Enter ssh rascaltrain@srv1 for the rascal server
   

Collecting Data

To record data:

1. In the car's terminal, run the command: roslaunch rascal pure_pursuit.launch to launch all the nodes
   
2. In a web page type 10.244.114.238:5000 to open the web display where you can record data
   
3. Press bagrecorder/enable on the web display to start recording
   
4. Use the joystick or keys on the keyboard (wasd) to move the car around
   
5. Press bagrecorder again to stop recording
   
6. When finished, press ctrl+c in the terminal to save the recorded session
   
7. You can record multiple segments within a single session, but note that it's better to keep sessions short— record a few segments, save the session, and then start a new one. This helps prevent data loss in case the car malfunctions or the Wi-Fi connection drops.
   
8. Go to /opt/upcar/dataProcessing and type python bag2AllData.py in the terminal to access all recorded sessions
   
9. Select the number that you want to download (should be the most recent one)
   
10. Open a new terminal and connect to the rascal server
    
11. Open /upcar/dataProcessing on the server and type ./copyFromCar.sh which will get the files from the car and bring them over to the server
    
12. It will ask you to select a host. Choose 1) rascal@172.24.114.238 which is the car's IP address
    
13. If it asks for a password it's always si2022sdc
    
14. It will ask you which sessions you want to copy over (enter the indices of the sessions you want to copy)
    
15. In /sessions change the file name to something relevant to keep track of your data (using mv)

Editing and labeling the Data

1. In the server's terminal, type the command: roslaunch rascal sim.launch which will launch the web dislpay for the server where you can edit the desired session
   
2. In a web page type 10.244.176.240:5000 to open the web display where you can edit data for training (it's best to use the server's web display instead of the car's)
   
3. In the web display you can click replayer/pause-play to run through all of the data segments
   
4. Use ​Key Binds to make editing easier and faster
   

Deleting bad segments of data:

When portions of the recorded data are messy or irrelevant for training, they can be excluded to improve model performance

1. To jump between different segments that the web display automatically divides the data into, click replayer/nextSeg or replayer/prevSeg
2. When you get to a segment you want to delete, click the desired starting point on the orange path and hit replayer/editMode
   
3. Select the endpoint of the segment you want to delete by clicking the orange path at that point
   
4. Select replayer/editMode again to highlight the part that's going to be deleted
   
   • You can use the replayer/jumpStart and replayer/jumpEnd buttons to go to the start or end of a segment
     
   • You can use the replayer/forward and replayer/backward to go back and forth between frames in a given segment to help get exact data highlighted
     
5. Click replayer/deleteSelection to delete the highlighted segment
   

Labeling intersection data:

For intersection data you MUST label the data manually at each segment where an intersection occurs

1. Highlight the frames of data that has multiple valid paths (left, right, and straight) the same way you would for deleting segments
   
2. Click replayer/setLvalid, replayer/setRvalid, replayer/setSvalid to indicate which paths are possible at that intersection. This will be used later when training.
   

IMPORTANT: Once you are finished editing you MUST save the data by running the data_loader/smoothen command

Training Rascal

We used the fisheye and discrete_waypoint model to train on

1. If you go to /sessions in the server terminal and type ls your chosen files should be listed there (it will have the term labeled or smooth in the name)
   
2. Go back to /upcar and copy your data to /rascaltraining/data: cp dataProcessing/sessions/<your file name here> rascaltraining/data
   
3. If you go into /rascaltraining/data you should see your file listed
   
4. Go back to /rascaltraining and type the command: python trainModel.py -m fisheye —use_labels 1.0 -s labels when using fisheye. Type the command: python trainModel.py -m discrete_waypoint —center_width 30 —num_bins 3 —use_labels 1.0 -s labels when using discrete_waypoint
   
5. Now your model is being trained
   
6. To stop training press enter
   
7. Remember to stop training when the validation loss doesn't seem to be decreasing by a significant amount anymore to avoid overfitting
   
8. To check what number model you just created, go to /savedModels and type ls (yours will be the most recent number)
9. To push your newly made model to gitlab, type git add -f <your model number here>
   
10. Next type the command git commit -m "<give it a name>"
    
11. Next type the command git push origin main
    
12. Finally, back in the car's terminal go to /savedModels and type git pull origin main
    

You're now ready to start testing!

Testing Rascal

To test rascal:

1. Go to /upcar in the car's terminal
   
2. Enter ./launch_with_model.sh pure_pursuit
   
3. Open up the web display
   
4. Run these commands in the web display:
   
   1. ml/imgDisp (set to true)
      
   2. ml/0/enable (click enable)
      

Watch the car move on it's own!
Make sure it doesn't crash!