wiki:Internal/OpenFlow/QuantaSetup

Context Navigation

Version 16 (modified by akoshibe, 13 years ago) ( diff )
—

jump to bottom

The Quanta LB9A/u-boot primer

This page is meant to be a "handbook" of sorts that give you a general picture and a HOW-TO of the various technicalities associated with configuring the Quanta LB9A, a switch that, when set up properly, becomes a "purely OpenFlow" switch that must be connected to a controller to function as a network device.

Index

…more like quick links.

I Prep work - some things to keep around and in mind when working with these switches
II LB9A facilities - the bootloader, firmware and their modes, hardware, and how to get to them
III A u-boot Primer - using the bootloader to configure and troubleshoot
IV OpenFlow Switching - setting up OpenFlow mode - flashing the switch, configuring through u-boot and grid services

I. Preparations

The following things are good to have on hand.

1.1 References

Flashing the firmware

http://openflowswitch.org/wk/index.php/LB4GImageSetup
http://www.openflow.org/wk/index.php/IndigoQuickStartLB4G (possibly more up-to-date)

Configuring the switch

http://www.openflow.org/wk/index.php/IndigoConfiguration - synopsis of system files, variables
http://www.openflowswitch.org/wk/index.php/IndigoReleaseNotes - the actual HOWTO for turning the box into an OpenFlow switch

1.2 Prerequisites

The things listed here are mostly for connecting to the switch via serial. The actual steps are covered later on.

A PC running some *nix with a serial port
RS232 to RJ45 null cable (a Cisco flat cable actually works)
packages on PC: ckermit, openssh

II LB9A facilities

The Quanta switch is basically a Linux box with a bunch of ports. Out of the box, the Quanta comes with two versions of customized Linux, one pre-installed on the on-board flash, and the other, on the removable cf card. The flash and cf card are the two primary forms of memory found on the switch that we really need to care about. One can chose which image to load through u-boot, the second-stage boot loader provided with the switch. In either case, the Linux base for the Quanta is comprised of the kernel image (uImage), the filesystem (uInitrd2m), and device tree (LB9A.dtb). The primary differences between the two images are:

The image on the flash underlies a diagnostics test interface
The image on the cf card underlies XorPlus, a switching firmware that allows the Quanta to be configured as a regular switch
The image on th cf card also allows access to a older version of the OpenFlow interface, along with the system shell

What follows next is a overview of the two images and u-boot. Note, usage of u-boot is covered in greater detail in section 3.

2.1 The cf card image

This is the image loaded by default when the switch is powered on. It can also be loaded from u-boot with the command run cfcard_bootcmd.

After a good chunk of it loads, you are given three choices:

    Please choose which to start: Pica8 XorPlus, OpenFlow, or System shell:
        (Will choose default entry if no input in 10 seconds.)
        [1]  Pica8 XorPlus   * default
        [2]  OpenFlow
        [3]  System shell

As seen in the above "screenshot", XorPlus will be automatically loaded after 10 seconds, giving you the standard switch CLI. XorPlus is a full-fledged switch configuration interface; Documentation for using XorPlus can be downloaded from the Pica8 website.

The second choice, "OpenFlow", gives you an outdated version of the Indigo interface, the "pure OpenFlow" switch firmware. The interface is essentially a standard Linux shell, plus OpenFlow facilities such as of-bcm-cli.

The last choice, "System shell", lets you access the base system using the ash shell. This last mode is useful if you need to access the cf card to test out a new image, or to check configuration scripts for XorPlus or OpenFlow.

2.2 The flash image

This image can be loaded form the u-boot prompt with the command run flash_bootcmd. As far as I can tell, the sole purpose of this image is to provide a diagnostics test facility. The following prompt shows up after the standard boot-up procedures:

     Diagnostic Test Main Menu v1.3 (LB9A - Linux)

[1  ]  File Management
[2  ]  Board Information
[3  ]  Diagnostic Test
[4  ]  Manufacturing Test Mode Set
[5  ]  Test Error Log File Management
[6  ]  PING
[7  ]  Reset
[8  ]  For Vibration Test      

 Enter your choice:

There isn't much I can tell you about this interface, except that option [1 ] doesn't do anything, [7 ] reboots the switch, and [8 ] might be unsafe to do on a hard, inclined surface¹.

2.3 u-boot

u-boot is the second-stage boot loader that comes with the switch. It has enough smarts to allow you to flash the switch and inspect/configure various boot options, and in the case of OpenFlow, set parameters that the switch will use to connect to the controller.

To enter the u-boot prompt, you have to interrupt the boot process when you see the u-boot banner.The easiest way to do this is to have a serial connection to the switch so you can see it boot. This can be set up using a RJ-45 to RS-232 cable. A Cisco cable will also work for this. Connect the RJ-45 end to the port labeled "Con" on a switch, and the other end to the PC. Next, start a session from a shell with kermit as root:

kermit -l /dev/ttyS0 
set carrier-watch off
set baud 115200
connect

Then power on/reboot the switch. If you see the U-Boot banner, you can stop the boot process by hitting any key. This should lead you to the '⇒' u-boot prompt:

U-Boot 1.3.0 (Sep  9 2010 - 16:00:13)

CPU:   8541, Version: 1.1, (0x80720011)
Core:  E500, Version: 2.0, (0x80200020)
Clock Configuration:
       CPU: 825 MHz, CCB: 330 MHz,

...

Net:   TSEC0, TSEC1
IDE:   Bus 0: OK 
  Device 0: Model: CF Card Firm: Ver2.35 Ser#: 14EF0707040600187736
            Type: Hard Disk
            Capacity: 1923.9 MB = 1.8 GB (3940272 x 512)
Hit any key to stop autoboot:  0 
=>

u-boot is configured by creating and modifying environment variables. These can be anything from a single variable to a multi-line script of many variables, parameters, and flags. Several commands that are good to know for working with u-boot are:

? - list available commands
help - get more information about a command
printenv - list current environment variables
setenv - make changes to / create new environment variables
saveenv - save changes to environment variables
copy - use tftp to copy files to flash
imls - list images in flash
flinfo - print flash memory contents

This list is by no means exhaustive, even with respect to the commands that we mention throughout this page. Which brings us to the next section, the u-boot primer.

III A u-boot Primer

This section is a HOW-TO for using u-boot with respect to the Quanta, e.g. variables, scripts, and memory maps unique to this switch. OpenFlow related u-boot stuff can be found in the next section.

3.1 Environment variables

As mentioned in the last section, u-boot is configured through environment variables. The command printenv lets you see all of them:

=> printenv
flash_bootcmd=setenv bootargs root=/dev/ram console=ttyS0,$baudrate; bootm ffd00000 ff000000 ffee0000
cfcard_bootcmd=setenv bootargs root=/dev/ram console=ttyS0,$baudrate; ext2load ide 0:1 0x1000000 /uImage;ext2load ide 0:1 0x2000000 /uInitrd2m;ext2load ide 0:1 0x400000 /LB9A.dtb;bootm 1000000 2000000 400000
bootdelay=5
baudrate=115200
loads_echo=1
ipaddr=192.168.2.1
serverip=192.168.2.12
rootpath=/nfsroot
gatewayip=192.168.2.254
netmask=255.255.255.0
hostname=LB9A_X
bootfile=eldk-quanta
loadaddr=4000000
ethact=TSEC0
bootcmd=run cfcard_bootcmd
ethaddr=60:eb:69:21:5b:e3
eth1addr=60:eb:69:21:5b:e4
stdin=serial
stdout=serial
stderr=serial

Environment size: 644/8188 bytes

Each entry is a line of the form [variable name]=[parameters]. the parameters can be a combination of u-boot commands and other variables in a way similar to a simple shell script.

The general case for these variables can be summarized as follows:

syntax is [name]=[parameters], where [parameter] is some string
$[name] returns the value of the separately defined variable, [name]
semicolon ( delimits individual commands

For example, note the first two entries in the "screen shot" above:

<entry 1> flash_bootcmd=setenv bootargs root=/dev/ram console=ttyS0,$baudrate; bootm ffd00000 ff000000 ffee0000
<entry 2> cfcard_bootcmd=setenv bootargs root=/dev/ram console=ttyS0,$baudrate; ext2load ide 0:1 0x1000000 /uImage;ext2load ide 0:1 0x2000000/uInitrd2m;ext2load ide 0:1 0x400000 /LB9A.dtb;bootm 1000000 2000000 400000

they were part of the run commands mentioned earlier. The parameters to these variables create another variable called "bootargs" using command setenv. What follows are the parameters for this new variable (which are also environment variables):

root=/dev/ram - probably specifies where the root filesystem should be loaded²
console=ttyS0 - set output to /dev/ttyS0
$baudrate - use the parameters specified in the variable "baudrate," which we can see above is set to 115200 (baud)

The parameters that follow the semicolon after $baudrate differ a bit for the two variables. For the cfcard_bootcmd entry the ext2load commands specifies where to fetch the kernel, ramdisk, and device tree files from, respectively. The specific syntax for each ext2load command is:

ext2load [interface] [memory address]

What is really important here are the memory location values. For the cf card, the values should be:

0x1000000 for kernel image
0x2000000 for filesystem
0x400000 for the device tree

The same follows for the flash image, with ffd00000, ff00000, and ffee0000 being the address locations where the kernel, ramdisk, and device tree files always begin, respectively. The current version of u-boot has some tools for inspecting flash memory. These will be discussed in the troubleshooting section.

Both script variables end with the bootm command, which loads the program files from the memory locations given by ext2load in the cf card or implied in the flash.

3.2 modifying the environment variables list

setenv allows you to modify and create environment variables. In general:

setenv [name], where [name] is new, creates a new variable with that name.
setenv [name], where [name] already exists, wipes out its previous parameters.
setenv [name] [parameters] sets/overwrites parameters for the variable [name] with [parameters].

Long parameters. For variables with whitespaces in its parameters (e.g. "script variables" like flash_bootcmd), the parameters have to be surrounded by single quotes to let u-boot know that everything is part of a single script. For example, to create and configure the variable dn_boot, we'd type this at the command line:

setenv dn_boot 'dhcp; nfs ff000000 $nfsip:$rootpath/uInitrd2m; setenv bootargs root=/dev/ram console=ttyS0,$baudrate DEV_ADDR=$ipaddr ETH0_IP=$ipaddr $extra_boot_args rw; bootm ffd00000 ff000000 ffee0000'

Note the single quotes before dhcp and after ffee0000. Another important thing about long parameters is that they cannot contain newlines; The whole command must be typed in a single line.

3.3 Some important variables

http://www.openflow.org/wk/index.php/IndigoConfiguration - OpenFlow specific variables

3.4 troubleshooting

IV OpenFlow Switching

This section provides example uses of the u-Boot facilities for manipulating the OpenFlow properties of the LB9A. We will also introduce the ORBIT Grid Service facilities that can be used to configure the switch without dealing with the gory details of u-boot (given the switch has the proper firmware, that is).

4.1 Installation

Here we go over how to flash the switch with the proper firmware. Caution is needed so as not to brick the switch. Several steps might reiterate things from the previous sections, since this section was actually written way before the other sections.

4.1.1 Connecting to the switch

A serial connection can be set up between the switch and a machine using a RJ-45 to RS-232 cable. Surprisingly, a Cisco cable will work for this. Connect the RJ-45 end to the port labeled "Con" on the switch, and the other end to the PC. Next, start a session from a shell with kermit as root:

kermit -l /dev/ttyS0 
set carrier-watch off
set baud 115200
connect

Fire up the switch. If you see the U-Boot banner, you can stop the boot process (hit ctrl-c) so you can copy/burn the images onto flash.

4.2 updating u-boot

This step is comprised of the following steps:

change network settings of switch
update u-boot
upload kernel, root filesystem, and device tree to flash
modify boot parameters

The switch for SB9 will be images via nfs with console.sb9 as the server. This is to prevent users from bricking the switch by playing with u-boot.

4.2.1 change network settings

As explained in the u-Boot primer, the network settings are hard-wired into the switch via environment variables. The default settings statically set the switch's IP to a 192.168.X.X address block, which you might want to change depending on your network setup.The four variables of our interest are:

ipaddr (switch's static ip, default 192.168.1.2)
gatewayip (what the switch sees as default gw on its route table)
serverip (who to talk to for services e.g. tftp)
netmask (self-explanatory)

These variables should correspond to those for the network you're attaching the switch to. In our case, we set the switch's IP to 10.50.0.40, and cut out the hassle of setting the gateway IP by just directly pointing the switch to think the tftp server is the gateway:

=> setenv gatewayip 10.50.0.42
=> setenv ipaddr 10.50.0.40
=> setenv netmask 255.255.0.0
=> setenv serverip 10.50.0.42
=> saveenv

4.2.2 updating u-boot/copying files to flash

Once the networking variables are set, you can start copying the necessary files to the switch using copy. copy uses tftp to fetch the images:

=> copy -b tftp://10.50.0.42/u-boot.bin
=> copy -k tftp://10.50.0.42/uImage
=> copy -r tftp://10.50.0.42/uInitrd2m
=> copy -d tftp://10.50.0.42/LB9A.dtb

The first line updates u-boot. The next three fetch the kernel image, root filesystem, and device tree, respectively.

4.2.3 modifying boot parameters

This section covers booting the switch using a manually loaded root filesystem from on-board flash. To use NFS, refer to the next section, 2.2.4.

In addition, you want to modify flash_bootcmd so that it tells the linux kernel to mount the root filesystem read-write:

=> setenv flash_bootcmd 'setenv bootargs root=/dev/ram console=ttyS0,$baudrate rw DEV_ADDR=x.x.x.x; bootm ffd00000 ff000000 ffee0000'
=> saveenv

Several boot parameters such as the DEV_ADDR variable used by the OF startup scripts can also be set here. The details of can be found in IndigoConfiguration.

Also, make sure that the variable bootcmd is set to run flash_bootcmd. If not, reset with setenv and save this change as well:

=> setenv bootcmd 'run flash_bootcmd'
=> setenv bootfile eldk-quanta
=> saveenv

and load the image by issuing the command boot. if all goes well, you should be at the Linux shell prompt once the switch starts up.

Note: The bootcmd parameter will be the proper value if the switch is restarted after u-boot is updated (a.k.a. after the 1st copy command). This, however, resets all of your networking variables, so you'd have to re-do the steps in 2.2.1. Basically, it is up to the tastes of whoever is re-imaging the switches to either manually set the bootcmd variable or reset all of the network parameters after reboot.

4.2.4 Using NFS with DHCP.

Reference articles:

Here, we assume that the switch will be assigned an IP address via DHCP. It will use the assigned IP address for 1) booting via NFS, and once booted into Linux, 2) the value assigned to DEV_ADDR and the management interface eth0. It is also assumed that you have a working NFS server with the following line in /etc/exports:

/config   10.19.0.0/255.255.0.0(rw,sync,insecure,no_subtree_check)

DHCP setup in u-boot:

=> setenv ipaddr
=> setenv bootargs
=> setenv dhcp_boot dhcp
=> setenv autoload no

The first two commands blank out any previously assigned IP addresses and boot arguments. The next two enable DHCP. You can test if DHCP works via the command dhcp:

=> dhcp
Speed: 1000, full duplex
BOOTP broadcast 1
Got error 4
DHCP client bound to address 10.50.250.6

Set the startup script variable:

=> setenv dn_boot 'dhcp; nfs ff000000 $nfsip:$rootpath/uInitrd2m; \
setenv bootargs root=/dev/ram console=ttyS0,$baudrate DEV_ADDR=$ipaddr \
ETH0_IP=$ipaddr $extra_boot_args rw; bootm ffd00000 ff000000 ffee0000'

dhcp invokes the DHCP client we configured earlier, which sets $ipaddr. Next, the necessary image(s) are loaded via NFS. Then, bootargs are set so that the OS knows where to locate the root FS (in /dev/ram). 'rw' set mount parameters for the root FS. Both DEV_ADDR and ETH0_IP are variables used by Linux once the system boots into it; They are both set to the DHCP assigned IP address. $extra_boot_args can be set to additional variables using "setenv extra_boot_args '<VARS>'" (It's essentially a dummy variable). Finally, bootm is used to load the linux kernel, filesystem, and device tree stored on flash.

Set additional variables. Make sure all the variables required by the startup script are defined. You may also want the set the bootcmd to boot using the new script.
```
=> setenv nfsip 10.19.0.10
=> setenv hostname sw-sb9-da
=> setenv netmask 255.255.0.0
=> setenv rootpath /config
```

Save your settings with saveenv.

4.3 Configuration

There are two basic ways that you can go about configuring the OpenFlow behavior of the LB9A - 1) setting OpenFlow parameters via u-boot, and 2) through the ORBIT Grid Services, though at the time that this is being written, the latter is somewhat rough around the edges. Some things to note about the OpenFlow mode LB9A:

eth0 and eth1 as given by ifconfig -a are eth1 and eth2 on the faceplate, respectively. The interfaces can be manipulated as with any Linux box using ifconfig.
The ssh server (dropbear) is enabled by default. Telnet can be started by invoking telnetd. In either case the default username is root, and password is OpenFlow.

4.3.1 Using the OpenFlow CLI (OF-BCM)

OF-BCM is the OpenFlow command line supplied with the image. It can be started manually:

# /sbin/of-bcm-cli
Broadcom Command Monitor: Copyright (c) 1998-2008 Broadcom Corporation
Release: sdk-xgs-robo-5.6.6 built 20100905 (Sun Sep  5 23:01:38 2010)
Stanford OpenFlow Monitor
Platform: LB9A_MPC8541
OS: Unix (Posix)
PCI unit 0: Dev 0xb538, Rev 0x11, Chip BCM56538_B0, Driver BCM56634_B0
rc: unit 0

...

OF-BCM.0>

4.3.2 Key files

The OpenFlow components of the switch rely on the following:

/etc/rc.sh - system startup script: bring up lo, mounts filesystems, loads kernel modules, ect
/etc/system_config - system initialization scrit
/etc/system_config - system configs unique to each switch
/rc.soc, /etc/config.bcm - starts OpenFlow components
/etc/find-env - defines many of the system variables

4.3.3 Grid Services

The ORBIT grid services allow you to query and configure various components of the testbed(s) through a browser from gw.orbit-lab.org. Here our focus is on the network services.

!addOpenFlow

This allows you to set u-Boot environment variables without going into the gory details of u-Boot. For example, to boot a switch into OpenFlow mode, issue the following command at the navigation bar on your browser from external2 (make sure you have X11 tunneling):

http://www.orbit-lab.org/network/addOpenFlow?switch=172.16.20.8&ip=172.16.0.14&port=6633&dpid=001010201608

This points the 8th Grid data plane switch to kvm-big, the Big Switch controller at 172.16.0.14. After thinking for a bit, the browser should return:

dpid = 001010201608
ofip = 172.16.0.14
ofp = 6633

If you get a complain that the service is busy, make sure no-body is using the serial connection to the switch e.g. using telnet to s-nr2.

—

^{1. I honestly don't know what it does.}
^{2. These are educated guesses through observation, and by no means guaranteed to be what is actually happening. If anyone has an actual explanation I'd be very happy to learn about it.}

Attachments (1)

mHT9200-06-gc.pdf (59.6 KB ) - added by akoshibe 13 years ago. instructions on setting u-boot IP

Download all attachments as: .zip

Note: See TracWiki for help on using the wiki.

Download in other formats:

Plain Text