= Match Execution Framework (Post Preliminary) = 
'''Last modified 12/12/2013'''

[[TOC(DSC*,depth=2)]]

== Objective ==
This document represents an intermediate framework that can be used by teams to test their code.  The framework does not involve any validation of code for submission for the scrimmages or Final events. As a result, emails describing results of matches will not be sent to users, and users should directly look at screen output to see their scores. Information regarding code submission will be provided at a later time. A later release of the framework will be include modifications related to code submission and validation. 

For the most part, the framework instructions adhere to the previous framework instructions, with the major change being the inclusion of new topologies for running competitive tournaments. 

Use an OEDL script to execute either a competitive-style or cooperative-style game for both Preliminary matches of the DARPA Spectrum Challenge. Even though the use of the framework is illustrated by using 2 house bots (or 3 in the case of a cooperative game), in the actual competition, the teams will use their own radio implementations. Later, an additional purpose of the framework will be to provide validation to the images that teams submit for the Preliminary matches: teams will have to use the script in this framework to validate their '''image''' before submission.  Validation is required for both competitive and cooperative style matches. 

The OEDL scripts use two (primary and secondary) sets of five pairs of nodes: dsc-teamAp, dsc-teamBp, dsc-teamCp, dsc-teamXp, and dsc-teamYp on the main diagonal and dsc-teamAs, dsc-teamBs, dsc-teamCs, dsc-teamXs, and dsc-teamYs on the secondary diagonal (the actual topology descriptions are '''system:topo:dsc-team{Ap,Bp,Cp,Xp,Yp,As,Bs,Cs,Ds}'''). For this version of the framework, the competitive matches will use teamX{p,s} and teamY{p,s} nodes, while cooperative matches will use teamA{p,s}, teamB{p,s} and teamC{p,s} nodes.  

== Hardware / Software Resources Used ==
 1. Four grid nodes with USRP N210s (two for each team) for competitive match and six nodes with USRP N210s for cooperative match (two for each team).
 2. ''dsc-wbot1.ndz,dsc-wbot2.ndz,dsc-wbot3.ndz'': house bot disk images with the prerequisite software to configure the USRPs and execute a house radio bot. 
 3. ''team-image.ndz'': disk image that contains the team's radio.   
 4. Scoring packet server: This is the server that resides on the infrastructure machine and is used as the source of data packets as well as the sink for teams to submit received packets for scoring.
 5. The OEDL experiment script to execute the matches and perform image validation. 

Teams must ensure that their images contain '''/root/dsc-code_tx''' and '''/root/dsc-code_rx'''. Only these names will be used for testing and scoring. Further, these executables/scripts must be able to receive as input three parameters: 
 * '''-f 1700000000''' : option for specifying the center frequency (in this example 1.7GHz)
 * '''-s idb2''' : option for specifying the packet server host name (in this example idb2)
 * '''--mode COMP''' or '''--mode COOP''' : option that specifies whether the match style is a competitive or cooperative match.  

Please note that idb2 and 1700000000 are provided merely for illustration purposes and that the teams must allow for the values passed to be entered by the evaluators. Finally, no modifications to this execution format is allowed, and no additional fields will be used during evaluation.

Teams should also implement their radios such that packets are drawn from port 5123, while packets are delivered to port 5125 on the packet server specified with the '''-s''' argument. An illustration of this can be seen in the Python code in the table below:

||= Transmitter =||= Receiver =||
|--------------------------------------------------------
{{{#!td style="border: 0px; font-size: 90%"
  {{{
class dsc_pkt_src(object):
    def __init__(self, server, port=5123 ):
        self.pkt_size = 1440 # 1440 bytes of data  
        self.pkt_server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.pkt_server_socket.connect((server,port))
        self.MESSAGE = struct.pack('!l', self.pkt_size)
        self.pkt_server_socket.send(self.MESSAGE)

    def read(self):
           try:
              data = self.pkt_server_socket.recv(self.pkt_size)
           except socket.error:
              print "Connection to packet server closed"
              return ''
           self.pkt_server_socket.send(self.MESSAGE)
        return data
  }}}
}}}
{{{#!td style="border: 0px; font-size: 90%"
  {{{
class dsc_pkt_sink(object):    
      def __init__(self, server, port=5125):       
          self.pkt_sink_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
          self.pkt_sink_socket.connect((server,port))

      def send(self, payload):
          try:
             self.pkt_sink_socket.recv(4)
             self.pkt_sink_socket.send(payload)
          except socket.error:
             print "Connection to packet sink closed"
  }}}
}}}

Teams are also required to label their image with the image name they intend to use for submission, which should be stored in file named '''/.orbit_image''' on the root of the filesystem before node is saved. For example, if the team intends to submit an image with image name '''orbit-bot.ndz''' the /.orbit_image file should contain '''orbit-bot'''.

=== Packet Server ===
The packet server application running on the infrastructure machine has 
* a packet source at port 5123 which serves fixed size packets (1440 bytes). Clients can request packets from the server with the following handshaking process after a TCP socket connection has been established:
    1. client sends server the request with packet size as 4 byte word ('''ignored''' for the matches).
    2. server sends client a single data packet (fixed at '''1440''' bytes)

    The packet source sends out the first packet to each team only after it receives packet requests from all the teams (2 teams for competitive match and 3 for co-operative matches) and the start signal from the experiment script. It closes the transmit socket on the first packet request after the end-of-file is reached or at the 180 second timeout. It will report the score via an email to whoever has a grid reservation (if at least one packet is received correctly).
 * a packet sink at port 5125 which receive packets from client and verifies that it matches one of the sent packets. Clients can send data to server after a connection has been established:
    1. server sends client a ready packet (4 byte word).
    2. client sends server data for scoring. 
   
* a control interface used by the experiment script to prepare the packet server for a match by setting its mode, duration, etc., and to send a start signal so the packet source can send packets to the teams.


== Setting Up the Arena for COMPETITIVE ==
 * To get started, first make a reservation on the [https://www.orbit-lab.org/loginService/ControlPanel Orbit Scheduler] for using the Grid. 
 * After logging into grid console, make sure all nodes are turned off (in the example below, turn off all the nodes in the grid):
{{{
username@console.grid:~$ omf tell -a offh -t system:topo:all
}}}
 * Verify state of node before continuing. Make sure nodes are in the POWEROFF state.
{{{
username@console.grid:~$ omf stat
}}}

 * Image the two nodes to be used for the first team's radios (team X nodes in the primary arena): 
{{{
username@console.grid:~$ omf load -i dsc-wbot1.ndz -t system:topo:dsc-teamXp
}}}

In the case of the validation matches these two nodes will receive the competitor's image (and hence replace the house bot image name with the team's image name). 

 * Image the two nodes to be used for the second team's radios (in this case team Y nodes in the primary arena): 
{{{
username@console.grid:~$ omf load -i dsc-wbot2.ndz -t system:topo:dsc-teamYp
}}}


 * Turn nodes back on and verify they are in POWERON state
{{{
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamXp
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamYp
}}}


== Setting Up the Arena for COOPERATIVE ==
 * To get started, first make a reservation on the [https://www.orbit-lab.org/loginService/ControlPanel Orbit Scheduler] for using the Grid. 
 * After logging into grid console, make sure all nodes are turned off (in the example below, turn off all the nodes in the grid):
{{{
username@console.grid:~$ omf tell -a offh -t system:topo:all
}}}
 * Verify state of node before continuing. Make sure nodes are in the POWEROFF state.
{{{
username@console.grid:~$ omf stat
}}}

 * Image the two nodes to be used for the first team's radios (team A nodes in the secondary arena): 
{{{
username@console.grid:~$ omf load -i dsc-wbot1.ndz -t system:topo:dsc-teamAs
}}}

In the case of the validation matches these two nodes will receive the competitor's image (and hence replace the house bot image name with the team's image name). 

 * Image the two nodes to be used for the second team's radios (in this case team C nodes in the secondary arena): 
{{{
username@console.grid:~$ omf load -i dsc-wbot2.ndz -t system:topo:dsc-teamCs
}}}

 * Image the third pair of nodes to be used for the third team's radios (in this case team B nodes in the secondary arena): 
{{{
username@console.grid:~$ omf load -i dsc-wbot3.ndz -t system:topo:dsc-teamBs
}}}

 * Turn nodes back on and verify they are in POWERON state
{{{
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamAs
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamCs
username@console.grid:~$ omf tell -a on -t system:topo:dsc-teamBs
}}}


== Running the !Validation/Experiment Script ==

The experiment script will execute '''/root/dsc-code_tx''' and '''/root/dsc-code_rx''' on the source and sink respectively (please note that there are no suffixes), for both (all three) pairs of nodes. Teams must ensure that their images contain '''/root/dsc-code_tx''' and '''/root/dsc-code_rx'''. No modifications to this execution format is allowed. The sender module draws packets from a packet server, while the sink module delivers packets to the same packet server.  

Use the '''system:exp:dsc-match''' OEDL script to run the match. The script will start the packet server, start the transmitter and receiver for all teams and release the packet source, wait for 180 seconds and then terminate the match.
[[BR]] [[BR]]
 * For competitive matches, the command line is ('''please note the modified option that specifies whether the match will use the primary or secondary arena'''):
{{{
username@console.grid:~$ omf-5.4 exec system:exp:dsc-match -- --mode COMP --arena PRI
}}}
Please note that the command line above is an expanded version of the command line used for the Preliminary round. 

     [[CollapsibleStart(Competitive match output)]]
     The output of the competitive match should look similar to the following:
{{{
username@console.grid:~# omf-5.4 exec system:exp:dsc-match -- --mode COMP --arena PRI

INFO NodeHandler: OMF Experiment Controller 5.4 (git 8f9e1c0)
 INFO NodeHandler: Reading configuration file /etc/omf-expctl-5.4/services.yaml
 INFO NodeHandler: Add domain http - http://internal1.orbit-lab.org:5054/
 INFO NodeHandler: Add domain http - http://repository1.orbit-lab.org:5054/
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-11-21t14.55.06.072-05.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-match
 INFO property.freq: freq = 600000000 (Fixnum)
 INFO property.server: server = "idb2" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 180 (Fixnum)
 INFO property.mode: mode = "COMP" (String)
 INFO property.team1: team1 = "dsc-teamXp" (String)
 INFO property.team2: team2 = "dsc-teamYp" (String)
 INFO property.team3: team3 = "dsc-teamCp" (String)
 INFO Topology: Loaded topology 'system:topo:dsc-teamXp'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamYp'.
 INFO exp: Connecting to idb2 at 5128
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 10s....
INFO exp: Sending signal to packet server to start serving packets
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 180s....
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 8801 Received 2503 in 176741 seconds
 INFO exp: Team 2: Sent 14801 Received 0 in 0 seconds
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler:
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler:
 INFO EXPERIMENT_DONE: Event triggered. Starting the associated tasks.
 INFO run: Experiment default_slice-2013-11-21t14.55.06.072-05.00 finished after 3:54

username@console.grid:~#
}}}
     [[CollapsibleEnd]] 

'''Please note''' the scores that are reported in the above match:
{{{
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 8801 Received 2503 in 176741 seconds
 INFO exp: Team 2: Sent 14801 Received 0 in 0 seconds
}}}

You should recognize that the scores are very low! You should try to '''understand''' why the scores are so low for this match and incorporate the knowledge you learn into your own radio design! 

 * For cooperative matches, the command line is (with the same extended parameter specifying primary or secondary arena for the match):
{{{
username@console.grid:~$ omf exec system:exp:dsc-match -- --mode COOP --arena SEC
}}}

     [[CollapsibleStart(Cooperative match output)]] 
     The output of the cooperative match should look similar to the following:
{{{
username@console.grid:~$ omf-5.4 exec system:exp:dsc-match -- --mode COOP --arena SEC --team1 dsc-teamAs --team2 dsc-teamBs --team3 dsc-teamCs --runtime 30

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 8f9e1c0)
 INFO NodeHandler: Reading configuration file /etc/omf-expctl-5.4/services.yaml
 INFO NodeHandler: Add domain http - http://internal1.orbit-lab.org:5054/
 INFO NodeHandler: Add domain http - http://repository1.orbit-lab.org:5054/
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2014-01-10t13.01.26.951-05.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-match
 INFO property.freq: freq = 600000000 (Fixnum)
 INFO property.server: server = "idb2" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 30 (Fixnum)
 INFO property.mode: mode = "COOP" (String)
 INFO property.arena: arena = "SEC" (String)
 INFO property.team1: team1 = "dsc-teamAs" (String)
 INFO property.team2: team2 = "dsc-teamBs" (String)
 INFO property.team3: team3 = "dsc-teamCs" (String)
 INFO Topology: Loaded topology 'system:topo:dsc-teamAs'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamBs'.
 INFO Topology: Loaded topology 'system:topo:dsc-teamCs'.
 INFO exp: Connecting to idb2 at 5128
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 10s....
 INFO exp: Sending signal to packet server to start serving packets
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 30s....
 INFO exp: Getting the score
 INFO exp: Team 1: Sent 1501 Received 297 in 28975 seconds
 INFO exp: Team 2: Sent 2058 Received 429 in 29209 seconds
 INFO exp: Team 3: Sent 2501 Received 792 in 29995 seconds
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler:
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler:
 INFO EXPERIMENT_DONE: Event triggered. Starting the associated tasks.
 INFO run: Experiment default_slice-2014-01-10t13.01.26.951-05.00 finished after 1:25
username@console.grid:~#

}}}
     [[CollapsibleEnd]] 


Turn the nodes off
{{{
username@console.grid:~$ omf tell -a offh -t system:topo:all
}}}


== Running Matches on Sandboxes ==

In order to facilitate code development, all three sandboxes (sb2,sb3 and sb7) are available and have a simplified version of the OEDL script and packet server. Please do note that, since they have only one pair of nodes, these environments are really not capable of generating interference, but are rather meant for code testing. Further, sandboxes cannot provide validation of an image. Appropriately, the actual match script is named dsc-test and the sequence of commands to execute it is somewhat simplified and consists of:
 
  * Loading the image:
{{{
  omf load -i team-image-name.ndz
}}}
  * Turning the nodes on:
{{{
  omf tell -a on
}}}
  * Executing the ''match'':
{{{
  omf exec system:exp:dsc-test -- --mode COMP
}}}
      [[CollapsibleStart(Competitive match output on the sandbox)]]
      The output of the 30 second competitive match should look similar to the following:
{{{
console.sb2:~# omf exec system:exp:dsc-test -- --mode COMP --runtime 30

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t20.55.32.533-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-test
 INFO property.freq: freq = 650000000 (Fixnum)
 INFO property.server: server = "10.12.0.10" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 30 (Fixnum)
 INFO property.mode: mode = "COMP" (String)
 INFO exp: Connecting to packet server
 INFO stdlib: Waiting for nodes (Up/Down/Total): 1/1/2 - (still down: node1-1.sb2.orbit-lab.org) [0 sec.]
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 30s....
 INFO exp: Getting the score
 INFO exp: Sent 2161440 Received 2131200 in 30 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-08-12t20.55.32.533-04.00 finished after 0:52
}}}
      [[CollapsibleEnd]] 
  or
{{{
  omf exec system:exp:dsc-test -- --mode COOP
}}}
      [[CollapsibleStart(Cooperative match output on the sandbox)]]
      The output of the 30 second competitive match should look similar to the following:
{{{ 
console.sb2:~# omf exec system:exp:dsc-test -- --mode COOP --runtime 30

 INFO NodeHandler: OMF Experiment Controller 5.4 (git 5385228)
 INFO NodeHandler: Slice ID: default_slice (default)
 INFO NodeHandler: Experiment ID: default_slice-2013-08-12t21.00.56.411-04.00
 INFO NodeHandler: Message authentication is disabled
 INFO Experiment: load system:exp:stdlib
 INFO property.resetDelay: resetDelay = 230 (Fixnum)
 INFO property.resetTries: resetTries = 1 (Fixnum)
 INFO Experiment: load system:exp:eventlib
 INFO Experiment: load system:exp:winlib
 INFO Experiment: load system:exp:dsc-test
 INFO property.freq: freq = 650000000 (Fixnum)
 INFO property.server: server = "10.12.0.10" (String)
 INFO property.port: port = 5123 (Fixnum)
 INFO property.runtime: runtime = 30 (Fixnum)
 INFO property.mode: mode = "COOP" (String)
 INFO exp: Connecting to packet server
 INFO ALL_UP_AND_INSTALLED: Event triggered. Starting the associated tasks.
 INFO exp: Preparing packet server
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO exp: Start rx and tx applications
 INFO exp: Request from Experiment Script: Wait for 5s....
 INFO exp: Running the experiment
 INFO exp: Request from Experiment Script: Wait for 30s....
 INFO exp: Getting the score
 INFO exp: Sent 2161440 Received 1990080 in 30 ms
 INFO exp: Stopping the applications
 INFO exp: Request from Experiment Script: Wait for 2s....
 INFO NodeHandler: 
 INFO NodeHandler: Shutting down experiment, please wait...
 INFO NodeHandler: 
 INFO run: Experiment default_slice-2013-08-12t21.00.56.411-04.00 finished after 0:52
}}}
      [[CollapsibleEnd]] 

Please note that the sandboxes use different frequencies in order to minimize the interference, are not going to produce the scoring email (the score is printed on the screen only) and that your code must support the scoring server hostname option (since the scoring server is running on respective consoles of each of the sandboxes).  

NOTES: