PXE Image

Build Instructions for PXE Images

Build Setup

The current build package for making pxe images lives on:

In there is a make master make file with all the parameters to build the kernel / initramfs image pair. It will also build a pxelinux.bin from sources. This version of pxelinux.bin doesn't need to match with kernel / initramfs pair.

in the Makefiel the values of the following variables control what kernel directory the make process will consult to build a kernel:


In the /root/pxe/omf/pxe/src directory should be a directory named linux-VERSIONNUMBERS. In this directory is the config file for the kernel being built.

Additionally ins the same source dir is a busbox-VERSION directory with a .config file that controls what features are built into busybox.

Building in NTP clock updates

  1. Even though the version of the kernel source tree is specified in the make file, the build process does not use the kernel config file that was downloaded from the mirror. Instead it uses the file named
    in the directory:
  2. the same is true for the busy box, I think it uses the file:
  3. Finally the startup script rcS.actual will be copied to the init-ram-fs and run as part of the pxe preOS sequence. We recently modified that file to have it poll NTP and correct the system clock. We added the following lines:
    /usr/sbin/ntpd -d -n -q -p consolec
    /sbin/hwclock --directisa -w

Bumping the Kernel Version

Made a few modification for the most current version based on kernel 3.11.1

  1. Copied a running ubuntu config from an ubuntu 13.10 install to serve as sane default. It does throw one question about vesa support which I said yes to, but mostly complies. Happily.
  2. After some testing we decided to remove R8169 (realtek) support and only enumerate the intel interfaces (this will cause some other boards to pxe/boot off eth0).
  3. Made specially modified pxelinux.cfg file that boots off of eth0 for the newest (Gen4 asus) nodes

Usage Notes


PXE has a 'changelog' in the top directory. The following extracts the version number from the first line in the changelog - enjoy:

   	24  	VER = $(shell grep -o -E '[0-9]+\.[0-9.-]+' -m 1 changelog)  

The following line copies a directory from one place to the other, but without ".svn" directories and emacs temporary files (*~)

   	119  	        tar -C static --exclude=.svn --exclude='*~' -cf - . | (cd $(RDISK); tar -xf -) 

The image target should now exist on its own (the incuded software is a sub target). I just added a few 'sudo' commands to allow it to compile from a user account.

The main addition is the apt target:

   	141  	apt: dirs kernel image  
  	142 	        @echo Building APT V$(VER)  
  	143 	        rm -rf $(BUILD_APT_DIR)  
  	144 	        install -d $(BUILD_APT_DIR)  
  	145 	        cp -rp debian $(BUILD_APT_DIR)  
  	146 	        rm -rf $(BUILD_APT_DIR)/debian/.svn       
  	147 	        mv $(BUILD_APT_DIR)/debian/Makefile $(BUILD_APT_DIR)  
  	148 	        cp changelog $(BUILD_APT_DIR)/debian  
  	149 	        cp ../copyright $(BUILD_APT_DIR)/debian  
  	150 	        if [ -r README.debian ] ; then \  
  	151 	                cp README.debian $(BUILD_APT_DIR)/debian; \  
  	152 	        fi  
  	153 	        env O_VER=$(VER) O_KERNEL_V=$(KERNEL_VERSION) $(MAKE) -C $(BUILD_APT_DIR) deb  
  	154 	#       $(MAKE) DESTDIR=$(BUILD_APT_DIR)/debian/$(APT_NAME) install  
  	156 	apt-install: apt  
  	157 	        scp $(BUILD_APT_DIR)/../*.deb $(REPOSITORY):$(REPOSITORY_ROOT)/binary  
  	158 	        ssh $(REPOSITORY) sudo $(REPOSITORY_ROOT)/  

The BUILD_APT_DIR variable was defined above, but it's in build. 143 and 144 create new apt directory. I added a new top level directory 'debian' to hold all the debian related files. This directory is copied into the apt build area (145, 146 - Should really be using something similar to 119). The 'debian' directory contains the Makefile which should really be a level up in the build area. Debian apt's also need a change log which we copy into the build area (148) as well. Same goes for the copyright file which we take from the nodehandler proper (149), and the README.debian, if it exists (150-152). 153 is a bit tricky. Basically we need to know the version of this apt, as well as the kernel version for the 'install' target in 'debian/Makefile'. However, we are calling the 'deb' target, which calls some dpkg util, which ultimately uses the 'debian/rules' file to call the 'install' target of 'debian/Makefile' - still following? In short, we put the two versions into environment variables "env O_VER=… make …' before calling the first make which the 'install' target can retrieve and use (remember a non-initialized Makefile variable takes it's value from an environment variable of the same name if it exists).


	echo "INSTALL $(O_VER) $(O_KERNEL_V)"
	install -d $(TFTP_DIR)
	install -m666 ../initrd-orbit-pxe-$(O_VER).img $(TFTP_DIR)
	install -d $(CFG_DIR)
	sed -e s/@KERNEL_VER@/$(O_KERNEL_V)/ ../../config/default.orbit | sed -e s/@VER@/$(O_VER)/ > $(CFG_DIR)/default.orbit-$(O_VER)
	install -m755 ../../config/makeLink $(CFG_DIR)

The O_VER and O_KERNEL_V come form the environment variables defined in 153 above. This apt installs the image we built into /tftpboot, as well as 'default.orbit-XXX' and 'makeLink' in /tftpboot/pxelinux.cfg. The 'default.orbit-XXX' file is created from the template stored in the config directory.


LABEL boel
APPEND vga=extended root=/dev/ram0 rw load_ramdisk=1 prompt_ramdisk=1 ramdisk_size=32768 init=/vmlinuz initrd=initrd-orbit-pxe-@VER@.img console=ttyS0,9600
DISPLAY message.txt

Default Pxe Issues

As part of the normal (non-pxe) boot process the nodes must pass though the pxe phase of the bios and "fail". The pxe client on all of the nodes should contact the tftp server and request a file named with the node ip adress in hex. It will then proceed to remove the trailing character until it runs out of characters. Once that process fails, the node will look for a file named default (no extention).

The current default used the localboot pxe directive (part of the standard) to instruct the nodes to fail over to disk. The file (now named default.localboot) has the following contents:

default harddisk

label harddisk
  localboot 0

A recent test with some of the newest mother boards however failed. The pxeclient did not honor the locaboot 0 option (or rather they locked up when given this option). A work around documented here (stored as a pdf for future use) recommended trying the syslinux chainloader. This required a new default file with the contents:

default harddisk

label harddisk
KERNEL chain.c32 
APPEND hd0 0

Service Requirements

To bootstrap the testbed nodes, in fact any server in the system, we use PXE to have the nodes load an image which contains a nodeagent and a frisbee client.

PXE requires the following support:

  1. A DHCP server pointing to an TFTP server
  2. A TFTP server proving the following:
    1. A network bootstrap program (NBP)
    2. A file containing instructions for the NBP
    3. A boot image
  3. A PXE grid service

Ideally the later referenced "orbit-pxe-server" apt package should include dependencies to all required packages and should setup everything. For instance, it should depend on gridservices and should then automatically instantiate the PXE grid service.

DHCP Server

To support PXE we need a DHCP server. This server does not necessarily need to be instantiated at the same machine as the TFTP server, but it's configuration needs to point to it.

Installing a DHCP server on Debian is as simple as:

% apt-get install dhcp

To support PXE, we need to add to /etc/dhcpd.conf something like:

option domain-name "";

default-lease-time 600;
max-lease-time 7200;

allow booting;

# The next paragraph needs to be modified to fit your case
subnet netmask {
  option broadcast-address;
# the gateway address which can be different
# (access to the internet for instance)
  option routers;
# indicate the dns you want to use
  option domain-name-servers;

group {
# The address of the TFTP server that hosts the PXE image.
 host tftpclient {
# tftp client hardware address
  hardware ethernet  00:10:DC:27:6C:15;

# The PXE boot loader that instructs the client to load a script corresponding to its IP address.
# The script defines the kernel and memory based image to download and boot.  Think of it as a 
# network aware grub/lilo type boot loader.
  filename "/tftpboot/pxelinux.bin";

After you have edited the dhcpd configuration file, restart it with "/etc/init.d/dhcpd restart".

Installing a TFTP server

Debian makes that easy. We prefer atftpd since its the only tftp server that has verbose output. It also gives the ability to control the number of threads which handle transfers. With 400+ simultaneous clients, a good tftp server is required.

% apt-get install atftd

Installing NBP — pxelinux

'pxelinux.bin' is a file found in the syslinux package. Syslinux The nitty gritty of how it works, what to do to make a machine netboot, and all of the other specifics can be found at Syslinux.

Installing an PXE image

The PXE client gets the address of the TFTP server through DHCP. Taking it's IP address it attempts to download a specific file containing further instructions from the TFPT server. The name of the file is created from the PXE client's IP address. For Orbit, the instructions are identical for every node and are stored in "/tftpboot/pxelinux.cfg/default.orbit-version". This file instructs the node to fetch the image "/tftpboot/initrd-orbit-pxe-version.img" from the TFTP server and boot into it.

Both files will be installed through:

% apt-get install orbit-pxe-server

Installing the PXE Grid Service

Every service on Orbit which can be controlled by the experimenter requires a grid service component. The PXE grid service must be installed on the TFTP server.

% apt-get install gridservices
% cd /etc/gridservices/enabled
% ln -s ../available/pxe .
% /etc/init.d/gridservices restart
Last modified 5 years ago Last modified on Jun 10, 2016, 10:27:23 AM
Note: See TracWiki for help on using the wiki.