wiki:DSC/zdc_framework

Context Navigation

Version 32 (modified by trappe, 10 years ago) ( diff )
—

Match Execution Framework (Post Preliminary)

(Last Modified: Monday 8/12/2013 11:45PM)

Objective

Note: This document represents an intermediate framework that can be used by teams to test their code. There are several notable changes that will occur in the future, but require further development. Specifically: (1) The framework consists of one diagonal of nodes, providing one competitive or one cooperative arena. The Final event will involve two separate diagonals running in pipeline, and the two diagonals will essentially be identical. Currently, only one diagonal is available. (2) The current set of nodes being used is a slightly older generation of ORBIT node than was used during the Preliminary event. An improved generation of nodes is being constructed and will be installed on ORBIT at a later time, replacing the current diagonal and populating the second diagonal. (3) The framework does not include instructions for code submission for the scrimmages or Final events. Information regarding code submission will be provided at a later time.

For the most part, the framework instructions adhere to the previous framework instructions, with the major change being the inclusion of new topologies for running competitive tournaments.

Use an OEDL script to execute either a competitive-style or cooperative-style game for both Preliminary matches of the DARPA Spectrum Challenge. Even though the use of the framework is illustrated by using 2 house bots (or 3 in the case of a cooperative game), in the actual competition, the teams will use their own radio implementations. An additional purpose of the framework is to provide validation to the images that teams submit for the Preliminary matches: teams must use the script in this framework to validate their image before submission. Validation is required for both competitive and cooperative style matches.

The OEDL script uses three pairs of nodes: dsc-teamA, dsc-teamB, dsc-teamC (the actual topology descriptions are system:topo:dsc-team{A,B,C}). The competitive matches will use teamA and teamC nodes, while cooperative matches will use teamA, teamB and teamC nodes (contestant radios will be only on teamA nodes for image validation, while house bots will be using teamB and teamC nodes); configurations should be considered to be fixed and will be changed only in the case of hardware failures.

Hardware / Software Resources Used

Four grid nodes with USRP N210s (two for each team) for competitive match and six nodes with USRP N210s for cooperative match (two for each team).
dsc-wbot1.ndz,dsc-wbot2.ndz,dsc-wbot3.ndz: house bot disk images with the prerequisite software to configure the USRPs and execute a house radio bot.
team-image.ndz: disk image that contains the team's radio. Please note that it is this image file that will be validated, and that once the image has been loaded for validation, no modification to the radio designs within the image prior to completion of the validation process are allowed.
Scoring packet server: This is the server that resides on the infrastructure machine and is used as the source of data packets as well as the sink for teams to submit received packets for scoring.
The OEDL experiment script to execute the matches and perform image validation.

Teams must ensure that their images contain /root/dsc-code_tx and /root/dsc-code_rx. Only these names will be used for testing and scoring. Further, these executables/scripts must be able to receive as input three parameters:

-f 1700000000 : option for specifying the center frequency (in this example 1.7GHz)
-s idb2 : option for specifying the packet server host name (in this example idb2)
—mode COMP or —mode COOP : option that specifies whether the match style is a competitive or cooperative match.

Please note that idb2 and 1700000000 are provided merely for illustration purposes and that the teams must allow for the values passed to be entered by the evaluators. Finally, no modifications to this execution format is allowed, and no additional fields will be used during evaluation.

Teams should also implement their radios such that packets are drawn from port 5123, while packets are delivered to port 5125 on the packet server specified with the -s argument. An illustration of this can be seen in the Python code in the table below:

Transmitter

Receiver

class dsc_pkt_src(object):
    def __init__(self, server, port=5123 ):
        self.pkt_size = 1440 # 1440 bytes of data  
        self.pkt_server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.pkt_server_socket.connect((server,port))
        self.MESSAGE = struct.pack('!l', self.pkt_size)
        self.pkt_server_socket.send(self.MESSAGE)

    def read(self):
           try:
              data = self.pkt_server_socket.recv(self.pkt_size)
           except socket.error:
              print "Connection to packet server closed"
              return ''
           self.pkt_server_socket.send(self.MESSAGE)
        return data

class dsc_pkt_sink(object):    
      def __init__(self, server, port=5125):       
          self.pkt_sink_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
          self.pkt_sink_socket.connect((server,port))

      def send(self, payload):
          try:
             self.pkt_sink_socket.recv(4)
             self.pkt_sink_socket.send(payload)
          except socket.error:
             print "Connection to packet sink closed"

Teams are also required to label their image with the image name they intend to use for submission, which should be stored in file named /.orbit_image on the root of the filesystem before node is saved. For example, if the team intends to submit an image with image name orbit-bot.ndz the /.orbit_image file should contain orbit-bot.

Packet Server

The packet server application running on the infrastructure machine has

a packet source at port 5123 which serves fixed size packets (1440 bytes). Clients can request packets from the server with the following handshaking process after a TCP socket connection has been established:
1. client sends server the request with packet size as 4 byte word (ignored for the matches).
2. server sends client a single data packet (fixed at 1440 bytes)

The packet source sends out the first packet to each team only after it receives packet requests from all the teams (2 teams for competitive match and 3 for co-operative matches) and the start signal from the experiment script. It closes the transmit socket on the first packet request after the end-of-file is reached or at the 180 second timeout. It will report the score via an email to whoever has a grid reservation (if at least one packet is received correctly).

a packet sink at port 5125 which receive packets from client and verifies that it matches one of the sent packets. Clients can send data to server after a connection has been established:
1. server sends client a ready packet (4 byte word).
2. client sends server data for scoring.

a control interface used by the experiment script to prepare the packet server for a match by setting its mode, duration, etc., and to send a start signal so the packet source can send packets to the teams.

Image Validation and Submission

For the image to be considered for the tournament it must be successfully validated in both the Competitive and Cooperative modes. Different images for different tournament modes will not be accepted. Successful execution of the system:exp:dsc-match script (i.e. with at least one error-free packet received by the scoring server) is considered validation of the image for that match type. Please note that the email sent by the scoring server to contestants (as well as the ORBIT team) contains all the information about images that were used for the match. It is essential that teams validate their final image without any modifications. The best way to ensure that is to simply save the final image, re-image the nodes and execute the scripts (without any further login/access to the nodes). This guarantees that the image checksum will be the same as what gets reported in the validation email. You may run the validation script as often as desired and on multiple images. If the team successfully validates multiple images, the ORBIT team needs to be informed of the actual image to be used, otherwise the last successfully validated image (i.e. passing both types of matches) will be used.

Each team is also advised that the file /.orbit_image should contain the final image name (or team name) in order to facilitate proper reporting by the framework.

Setting Up the Arena

To get started, first make a reservation on the Orbit Scheduler for using the Grid.
After logging into grid console, make sure all nodes are turned off (in the example below, turn off all the nodes in the grid):
```
username@console.grid:~$ omf tell -a offh -t system:topo:all
```
Verify state of node before continuing. Make sure nodes are in the POWEROFF state.
```
username@console.grid:~$ omf stat
```

Image the two nodes to be used for the first team's radios (team A nodes):

username@console.grid:~$ omf load -i dsc-wbot1.ndz -t system:topo:dsc-teamA

In the case of the validation matches these two nodes will receive the competitor's image (and hence replace the house bot image name with the team's image name).

Image the two nodes to be used for the second team's radios (in this case team C nodes):
```
username@console.grid:~$ omf load -i dsc-wbot2.ndz -t system:topo:dsc-teamC
```

FOR COOPERATIVE MATCHES: image the third pair of nodes to be used for the third team's radios (in this case team B nodes):
```
username@console.grid:~$ omf load -i dsc-wbot3.ndz -t system:topo:dsc-teamB
```

Turn nodes back on and verify they are in POWERON state

username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamA
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamC

FOR COOPERATIVE MATCHES: turn on the third pair of nodes

username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamB

Running the Validation/Experiment Script

The experiment script will execute /root/dsc-code_tx and /root/dsc-code_rx on the source and sink respectively (please note that there are no suffixes), for both (all three) pairs of nodes. Teams must ensure that their images contain /root/dsc-code_tx and /root/dsc-code_rx. No modifications to this execution format is allowed. The sender module draws packets from a packet server, while the sink module delivers packets to the same packet server.

Use the system:exp:dsc-match OEDL script to run the match. The script will start the packet server, start the transmitter and receiver for all teams and release the packet source, wait for 180 seconds and then terminate the match.

For competitive matches, the command line is:

username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COMP

Competitive match output

The output of the competitive match should look similar to the following:

username@console.grid:~# omf exec system:exp:dsc-match -- --mode COMP

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-13t11.22.10.060-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 220 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-match
 INFO property.freq: freq = 600000000 (Fixnum)
 INFO property.server: server = "idb2" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 180 (Fixnum)
 INFO property.mode: mode = "COMP" (String)
 INFO property.team1: team1 = "dsc-teamA" (String)
 INFO property.team2: team2 = "dsc-teamC" (String)
 INFO property.team3: team3 = "dsc-teamB" (String)
 INFO Topology: Loaded topology 'system:topo:dsc-teamA'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamC'.
 INFO exp: Connecting to packet server
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 10s....
 INFO exp: Sending signal to packet server to start serving packets
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 180s....
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 12817440 Received 2776320 in 180 seconds
 INFO exp: Team 2: Sent 21457440 Received 16045920 in 180 seconds
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler:
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler:
default_slice-2013-08-13t11.22.10.060-04.00 EXP.DONE Running for 206 sec
 INFO run: Experiment default_slice-2013-08-13t11.22.10.060-04.00 finished after 3:26

username@console.grid:~#

For cooperative matches, the command line is:

username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COOP

Cooperative match output

The output of the cooperative match should look similar to the following:

username@console.grid:~# omf exec system:exp:dsc-match -- --mode COOP

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-13t11.37.26.574-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 220 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-match
 INFO property.freq: freq = 600000000 (Fixnum)
 INFO property.server: server = "idb2" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 180 (Fixnum)
 INFO property.mode: mode = "COOP" (String)
 INFO property.team1: team1 = "dsc-teamA" (String)
 INFO property.team2: team2 = "dsc-teamC" (String)
 INFO property.team3: team3 = "dsc-teamB" (String)
 INFO Topology: Loaded topology 'system:topo:dsc-teamA'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamC'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamB'.
 INFO exp: Connecting to packet server
 INFO stdlib: Waiting for nodes (Up/Down/Total): 3/3/6 - (still down: node19-20.grid.orbit-lab.org,node19-19.grid.orbit-lab.org,node2-1.grid.orbit-lab.org) [0 sec.]
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 10s....
 INFO exp: Sending signal to packet server to start serving packets
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 180s....
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 12817440 Received 2759040 in 180 seconds
 INFO exp: Team 2: Sent 21453120 Received 16178400 in 180 seconds
 INFO exp: Team 3: Sent 5850720 Received 1208160 in 180 seconds
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler:
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler:
default_slice-2013-08-13t11.37.26.574-04.00 EXP.DONE Running for 213 sec
 INFO run: Experiment default_slice-2013-08-13t11.37.26.574-04.00 finished after 3:33

username@console.grid:~#

After each run, the team will receive a notification email ONLY if at least one packet was successful received by the packet server. The team image will be considered for the tournament evaluation if it was successfully verified (i.e. notification email was received) for both competitive and cooperative matches and if image filename and MD5 checksum matches the ones in the email.

Competitive match notification email

Subject: DSC competitive match score for team: prasanthi

Please find the scoring report for the competitive match created at Tue Aug 13 11:26:00 -0400 2013:

Team A:  Exp. Run Time = 180000, RX bytes = 2776320, TX bytes = 12817440, Sucess Rate = 21.6605
 Image: dsc-wbot1.ndz (Owner: root, Created: Thu Jul 18 12:18:35 -0400 2013, Md5: d783eb15b8e12fb95cd03985fd75c1a6 )
 Receiver = node2-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:15:39

Team C:  Exp. Run Time = 180000, RX bytes = 16045920, TX bytes = 21457440, Sucess Rate = 74.7802
 Image: dsc-wbot2.ndz (Owner: root, Created: Thu Jul 18 12:18:55 -0400 2013, Md5: 5ed3d3b94535d56b6964ad0e58a4e1db )
 Receiver = node1-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:09:01

Cooperative match notification email

Subject: DSC cooperative match score for team: prasanthi

Please find the scoring report for the cooperative match created at Tue Aug 13 11:45:13 -0400 2013:

Team B:  Exp. Run Time = 180000, RX bytes = 1208160, TX bytes = 5850720, Sucess Rate = 20.6498
 Image: dsc-wbot3.ndz (Owner: root, Created: Sat Jul 20 01:15:39 -0400 2013, Md5: fbfae44bcbcaec70c0e22759a8e94aa8 )
 Receiver = node1-2.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:10:14

Team A:  Exp. Run Time = 180000, RX bytes = 2759040, TX bytes = 12817440, Sucess Rate = 21.5257
 Image: dsc-wbot1.ndz (Owner: root, Created: Thu Jul 18 12:18:35 -0400 2013, Md5: d783eb15b8e12fb95cd03985fd75c1a6 )
 Receiver = node2-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:15:39

Team C:  Exp. Run Time = 180000, RX bytes = 16176960, TX bytes = 21453120, Sucess Rate = 75.4061
 Image: dsc-wbot2.ndz (Owner: root, Created: Thu Jul 18 12:18:55 -0400 2013, Md5: 5ed3d3b94535d56b6964ad0e58a4e1db )
 Receiver = node1-1.grid.orbit-lab.org, Imaged by: nilanjan, Imaged on: 2013-08-13 at  11:09:01

Turn the nodes off

username@console.grid:~$ omf tell -a offh -t system:topo:all

Running Matches on Sandboxes

In order to facilitate code development, all three sandboxes (sb2,sb3 and sb7) are available and have a simplified version of the OEDL script and packet server. Please do note that, since they have only one pair of nodes, these environments are really not capable of generating interference, but are rather meant for code testing. Further, sandboxes cannot provide validation of an image. Appropriately, the actual match script is named dsc-test and the sequence of commands to execute it is somewhat simplified and consists of:

Loading the image:
```
  omf load -i team-image-name.ndz
```
Turning the nodes on:
```
  omf tell -a on
```

Executing the match:

  omf exec system:exp:dsc-test -- --mode COMP

Competitive match output on the sandbox

The output of the 30 second competitive match should look similar to the following:

console.sb2:~# omf exec system:exp:dsc-test -- --mode COMP --runtime 30

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t20.55.32.533-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-test
 INFO property.freq: freq = 650000000 (Fixnum)
 INFO property.server: server = "10.12.0.10" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 30 (Fixnum)
 INFO property.mode: mode = "COMP" (String)
 INFO exp: Connecting to packet server
 INFO stdlib: Waiting for nodes (Up/Down/Total): 1/1/2 - (still down: node1-1.sb2.orbit-lab.org) [0 sec.]
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 30s....
 INFO exp: Getting the score
 INFO exp: Sent 2161440 Received 2131200 in 30 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-08-12t20.55.32.533-04.00 finished after 0:52

  omf exec system:exp:dsc-test -- --mode COOP

Cooperative match output on the sandbox