MNE Scripts User's Guide

 

Quick Instructions

MNE Scripts Introduction

The MNE scripts are a set of bash scripts for Linux that are used to automate IPv4 and IPv6 Mobile Ad-hoc Networking (MANET) laboratory tests using NRL's Mobile Network Emulator, thought they can also be used to start a live field test if the configuration files are changed accordingly. The scripts are quite flexible, allowing many different types of tests to be run, and the design is also extensible, so you can add your own features, though it helps if you know a little bash scripting.

These scripts are designed to emulate a MANET using one Linux machine (laptop or desktop) for each node in your network (i.e., to create a 10-node MANET, you need 10 machines). Each of these machines should ideally have two interfaces: one hard-wired interface with static addresses for backchannel traffic (commands, MNE backchannel control info, etc.), and one wired or wireless interface for test traffic (MGEN data, routing protocol packets, etc.). Two interfaces are not necessarily required, but the scripts have not been thoroughly tested using only one interface, and it is highly recommend that you pick up some cheap PCMCIA ethernet cards or something.

These scripts are also designed for a network where one machine is considered to be a special node, which we designate "XCom". XCom acts as sort of a master control node for the MNE scripts. XCom is the machine you use to start tests, gather results, etc. This machine will have the capability to login automatically (without typing in a password) via SSH to the other machines in the network. It can also be nice to use XCom as a gateway to the internet, or to a larger network (default gateway for the other nodes), though that is not required, and other machines can act as gateways as well. Our "manyto1" MGEN scripts are also designed to send data to XCom.

The MNE Scripts are designed to be installed and run as root. Many of the programs that you need to run require root priviliges anyway, so it's just as well. We don't support running the scripts under a non-root account, and we don't have any suggestions on getting that to work. The scripts are also designed such that XCom must have automatic (no password required) SSH access to each machine in the network, so that it can start tests and run commands via SSH. We have a script that will take care of this for you, which is explained in the next section.

There are many useful little scripts included with the MNE Scripts that are not covered directly in this guide. Running most scripts with "-h" should give you a usage message, and maybe a brief description of what the script does, but it is not supported on every script. There are some where you may need to look into them to understand what they do, hopefully not too many though.

Installing the MNE Scripts

Get Linux Running and Configured

The first thing you need to do is install Linux on all your machines. If you are interested in using these scripts, we're going to assume you're already pretty familiar with the Linux installation and setup process, and that you are able to properly install and configure your network interfaces, X-Windows, etc.. The scripts were designed to run on Fedora Core 6 (Linux 2.6.x kernel), but the distribution of Linux you use shouldn't matter much. The place it will likely make the biggest difference is when you are doing autoconfiguration, as some of the network setup scripts vary from distribution to distribution (and even within different versions of Red Hat/Fedora). Things that you should install during OS installation include IPv6 support (only if you're intersted in IPv6 testing), iptables, ip6tables (again, only for IPv6 testing), the bash shell, ssh, sshd, perl, tcpdump/libpcap, and gnuplot. You may also want to install radvd and/or dhcp(6)d/dhc(6)relay if you're interested in autoconfiguration.

After initial installation, you should disable any unneeded services (man chkconfig) and set up your network the way you want it for each machine (hostnames, addresses, default gateways, static routes, nameservers, etc.). Note that test channel IP addresses should be set up with an all 1s netmask (/32, or single-host network) for IPv4 and as a /128 for IPv6 for proper MANET routing protocol operation. This will keep your nodes from talking to one another via the test channel, though once you have your nodelist set up, there is a setStaticRoutes command that will restore this functionality. Some of the more advanced network configuration can wait until you can make use of the scpAll and cmdAll to copy files to other machines, and run commands remotely. You will still need some basic IP address configuration for your backchannel, however. At this point, you should also generate an SSH key at each node by issuing the command "ssh-keygen -t dsa" and accepting the defaults by hitting enter at each prompt, and start the SSHD server if it is not already started.

A Note on Network Device Order

In our experience, network devices (PCMCIA, USB, and hard-wired ethernets) sometimes come up in a completely random order on boot due to the modules being loaded in different orders. Putting entries in /etc/modprobe.conf does not solve the problem. The solution we found for this problem is to use UDEV to assign interface names, by creating the file /etc/udev/rules.d/25-ifnames.rules:

KERNEL=="eth*", DRIVER=="tg3", NAME="eth0"
KERNEL=="eth*", DRIVER=="tulip", NAME="eth1"
KERNEL=="eth*", DRIVER=="pegasus", NAME="eth2"

These UDEV rules tell it to assign device names based on driver names (the driver names may be different for your particular machine setup). It is unclear what would happen if we had two devices using the same driver. These rules can also be specified using their MAC addresses, but then the rules must be updated when the card is switched out with an identical one. An example rule for specifying using MAC addresses is:

KERNEL=="eth*", SYSFS{address}=="00:12:34:fe:dc:ba", NAME="eth0"

Configure the MNE Scripts

We will set things up on XCom first, and then copy the scripts, executables, etc. to all the other machines later. Go ahead and download and extract the mne-scripts package on XCom from here. Choose whatever directory you want to put these in (/home/mne for instance) and untar the package there. We will refer to this directory as $MNEHOME, and we will now set up an environment variable for that. In the $MNEHOME/setup/etc directory, you will find several different files, and a directory called "profile.d", which contains the script "mne_scripts.sh". Edit $MNEHOME/setup/etc/profile.d/mne_scripts.sh, and change the declaration of MNEHOME to match your $MNEHOME directory. For instance, if your scripts are in /home/mne, the MNEHOME line should read "MNEHOME=/home/mne". You should not need to change anything else in mne_scripts.sh. Copy mne_scripts.sh to your /etc/profile.d directory. Then edit your /etc/bashrc file so that it always sources mne_scripts.sh (put ". /etc/profile.d/mne_scripts.sh" at the bottom of the file). An example bashrc file is already in $MNEHOME/setup/etc. Once you have edited your /etc/bashrc, copy it to the $MNEHOME/setup/etc directory. Then logout and log back in (or close the console and open a new console). These changes should now have taken effect, and $MNEHOME should be an environment variable, and all your tools, routing protocols, and many of the MNE Scripts should now be in your path. You can issue the command "which cmdAll" to make sure this worked correctly.

From XCom, run the "honorMe" script for the backchannel IP address of every machine in your network (including XCom's own backchannel address). As an example, if you have 2 machines with backchannel IP addresses of 192.168.10.1 and 192.168.10.2, you would run "honorMe 192.168.10.1" and "honorMe 192.168.10.2". You will need to answer "yes" to the SSH warning, and then type in the root password twice for each machine. This script will set up automatic ssh capability from XCom to each node in the network (including itself), which will allow you to run tests without having to type in passwords. Please note that the honorMe script may not work if you are not running as root.

You should now edit $MNEHOME/setup/nodelist to add hostname, test channel IP address, test channel IPv6 address, back channel IP address, test channel MAC address, and test and back channel interface names) for every node in the network. Some of this info is optional, please see the comments in the file for further information on file syntax, etc. Also, please note that "node 0" (that will make more sense after seeing the file) *MUST* be your "XCom" machine (see comments in introduction for what "XCom" means). Once you are done editing the nodelist, you should be able to use cmdAll and scpAll to make network and OS configuration easier across all nodes. You can test your setup by running "cmdAll 'hostname'". This should SSH into each machine in your nodelist (via the back channel) and run the command "hostname", which should return each node's hostname (which should match the hostname listed in the nodelist).

In addition to the nodelist, the other main file you need to edit is $MNEHOME/setup/mne.cfg. mne.cfg includes several default parameters for the scripts -- things such as map coordinates, multicast addresses and ports, NTP time servers (for syncing up via ntpdate), etc. The file is well-commented and should be mostly self-explanatory. genCmapCfg and genSdtCfg (in $MNEHOME/setup) are two scripts that can be run after setting up the nodelist to generate configuration files for CMap and SDT, which make visualization much nicer looking than by default.

If you have not enabled forwarding on your machine already, you should do this. See $MNEHOME/setup/etc/sysctl.conf for an example. Set this up correctly for your machine and copy it to /etc. The other two files in $MNEHOME/setup/etc are for autoconfiguration. dhcpd.conf is the DHCP server configuration file; change that to meet your network settings if you're interested in testing DHCP autoconfiguration in the MNE. sysctl.sam is another sysctl file with different settings from sysctl.conf that allow IPv6 stateless autoconfiguration to take place (SAM="Stateless Autoconfiguration for MANET"). You may need to change that file slightly to match your interfaces if you're interested in trying distributed stateless autoconfiguration. All these files should be copied from $MNEHOME/setup/etc to /etc.

Note: there are also a few other things you may wish to edit, but they are not as major and are not in $MNEHOME/setup. In $MNEHOME/routing_protocols/config_files, there are some configuration files for the routing protocols that you may need to edit to add or remove HNA declarations, etc. You may also need to edit and/or link to some files which use the individual hostname, including those in $MNEHOME/motion_scenarios/fixed/ and $MNEHOME/motion_scenarios/wp/. Additionally, some MGEN scripts in $MNEHOME/mgen_scripts/ (e.g. Multicast) require a scriptname.$HOSTNAME file to be created.

Now that MNE script configuration is complete and all nodes have had their network interfaces configured, we will copy the $MNEHOME directory to the other nodes. Change to the directory one above $MNEHOME (i.e., if $MNEHOME is /home/mne, change to /home), and use "scpAll" to copy the entire directory to every node in your network (i.e., if $MNEHOME is /home/mne, cd to /home: "cd /home", and then do "scpAll -r mne"). It may take a while to copy the entire directory over to all the other machines. Once it has finished (check "ps ax" and make sure), you can similarly use scpAll to copy bashrc and mne_scripts.sh to everyone. scpAll only works with files in the current working directory, or with whole directories in the CWD with "-r", thus to do this, you would need to do:

cd /etc; scpAll bashrc sysctl.conf <other files in /etc you wish to copy>
cd profile.d; scpAll mne_scripts.sh

You will also want to run "cmdAll 'sysctl -p'" to have the changes to /etc/sysctl.conf take effect on all nodes. As one last step, some MGEN scripts in $MNEHOME/mgen_scripts/ (e.g. Multicast) require a different symbolic link to scriptname.$HOSTNAME file to be created at each node. This can be fixed by running:

cmdAllQ 'cd $MNEHOME/mgen_scripts; rm -f Multicast.mgen; ln -s \
Multicast/Multicast.mgen.$HOSTNAME Multicast.mgen'
cmdAllQ 'cd $MNEHOME/mgen_scripts; rm -f MulticastMM.mgen; ln -s \
MulticastMM/MulticastMM.mgen.$HOSTNAME MulticastMM.mgen'
cmdAllQ 'cd $MNEHOME/mgen_scripts; rm -f MulticastRamp.mgen; ln -s \
MulticastRamp/MulticastRamp.mgen.$HOSTNAME MulticastRamp.mgen'
(etc.)

Download, Compile, and Install Tools

Assuming all your machines have the same OS and setup, you should not need to compile all these tools on all machines. You should be able to just compile them on XCom and then copy them to the other machines (we'll do that later though). To use the emulator and these scripts, you need to download and compile several tools. These include the NRL Mobile Network Emulator (MNE), MGEN, TRPR, CMap and SDT. NRL's Random Movement Generator (RMG) is also recommended, if you are interested in creating your own random NS2 mobility files for use in the emulator. Other tools and protocols you may be interested in compiling and using, including NRL SMF, NORM, IVOX, and gpsLogger are listed on the Code 5520 Products Page. Each of these tools should be compiled and copied to the $MNEHOME/tools directory. You may occasionally need to install non-default OS/development packages to get some things to compile (e.g. for NRL-SMF, you should run "yum install libpcap-devel" to get the libpcap-devel libraries).

You should also download and compile the routing protocol(s) of your choice. The MNE scripts include support for NRLOLSRD (IPv4/IPv6), NOLSRD (older NRL-modified version of INRIA's code, IPv4 only), CRCOLSR6D (CRC-modified version of NOLSRD, IPv6 only), and UOLSRD. These should be put in the $MNEHOME/routing_protocols directory when they have been successfully compiled. You can add support for any other routing protocol (including AODV), by putting it in this directory and adding a supporting script (see the "extending the scripts" section).

Once you are done compiling everything, you should copy the binaries to your other machines:

cd $MNEHOME/routing_protocols; scpAll *
cd $MNEHOME/tools; scpAll *

Running a Test

Tests can be run with a single command from your XCom machine, startAll. This command will send a command to all the machines in the network, starting them with the same parameters. The syntax for the startAll command is:

startAll <motion_model> <scenario> [options]

The motion model can be one of several that we have already implemented: waypoints (wp), NS2 (ns2) files, and BonnMotion NS2 files or any NS2 files where the node numbering begins at zero instead of one (bm). Another model is the "real" model, which plays back gpsLogger data from a real live test. The "fixedpos" model runs a scenario where all nodes have fixed GPS positions (no motion), while the "fixed" model is another mode with no motion, which just does static blocking without running the MNE, and thus there are no position reports or a sense of "range".

The scenario refers to the type of traffic MGEN produces. There are several models included with the scripts, most of which are "many-to-1" scenarios, where all nodes send data to XCom. The name of any script in $MNEHOME/mgen_scripts (minus the ".mgen") can be used as the scenario name.

startAll has many optional parameters to allow you to start many different kinds of tests. One parameter is the "-file <filename>" parameter, which can change the default file used by the wp, ns2/bm, fixed and fixedpos motion models. Another option is used to control whether the test is an IPv4 or an IPv6 test. This is controlled with "-ipv4" and "-ipv6", or simply "-4" and "-6". The default is IPv4, but that default, along with all defaults for all these optional parameters, can be changed by editing the execParser script in the $MNEHOME/startscripts/common directory. "-protocol <routing_protocol_param_set_name>" can be used to change the routing protocol being run. The routing_protocol_aram_set_name is the name of a script in $MNEHOME/routing_protocols/param_sets that actually starts the routing protocol. These have been made into scripts so that they can each add routes or delete routes or do whatever special thing that needs to be done for it to run properly. Another option is the "-quick", or simply "-q" option. This controls whether nodes are started one at a time, or all at once. You can use "startAll -h" to see the many various options. For example, "startAll wp m1 -6" would start a many-to-one ("m1") IPv6 test ("-ipv6") using the waypoints model ("wp") and the routing protocol would be the default one, NRLOLSR.

You may want to edit the mgen scripts for your specific testbed. Most of the scripts are 30 minutes long, with three different loading levels: low, medium, and high. High is designed to highly congest a 2Mbps wireless network. You may need to adjust the numbers to fit the specifications of your testbed. For instance, the current scripts were designed for 10 nodes. If you have 25 nodes, you’re probably going to saturate the network at the medium loading level if you don’t change the scripts. By the same token, if you’ve only got 3 nodes, you’re probably never going to even begin to congest the network. Also, if you are using a wired test channel than wireless, it may be very difficult to test congestion conditions in the emulator.

Autoconfiguration Tests

Autoconfiguration is currently supported in IPv4 via DHCP (give startAll the "-dhcp" option). This will start the dhcpd server on XCom, and all other nodes will have their network configuration altered for their test interface, and that interface will be restarted to pick up a new address. The scripts were designed to use the dhclient, dhcrelay, and dhcpd implementations that come with Fedora Core 6. Unfortunately, these implementations use LPF, "Linux Packet Filtering" I believe, which means that they sniff traffic, and thus get packets before they are filtered out by the MNE/iptables, which results in a ton of extra packets (and incorrect operation). To fix this, we downloaded the latest version of DHCP from ISC, and change includes/cf/linux.h to change the line "define USE_LPF" to "define DO_NOT_USE_LPF" (or anything but "define USE_LPF") and recompile. Unfortunately, the version of dhcp that ships with Fedora Core 6 also uses the directory var/lib/dhcpd/ for dhcpd lease files, and /var/lib/dhclient/ for dhclient lease files instead of /var/state/dhcp, which ISC uses for both by default. So we also went through the source/makefiles/etc. and replaced references to that location whereever they appeared: includes/cf/linux.h and Makefile.conf. For Makefile.conf, it is unclear which directory should be used, so we chose /var/lib/dhcpd/. After these changes, you should be able to compile, and replace the various DHCP executables that were installed by default:

./configure
make
cd work.linux-2.2
cp relay/dhcrelay server/dhcpd /usr/sbin
cp client/dhclint /sbin
cd /usr/sbin
scpAll dhcpd dhcrelay
cd /sbin
scpAll dhclient

Fedora Core 6 now includes an implementation of DHCP for IPv6 that supports dhcrelay. Unfortunately, it does not allow a node to be configured as a /128, which is required for correct operation of MANET routing protocols. There are other IPv6 DHCP implementation that support dhcrelay, but at the time of writing, we have not tested any of these to see if they support this mode of operation. I have also included a few scripts I wrote that allow stateless autoconfiguration of a MANET using distributed RADV servers. It unfortunately has the same problem as IPv6 DHCP, in that I cannot get nodes to configure themselves as /128. If works if you configure your test interfaces as /64s, but even then, it doesn't work consistently for some reason, due to Linux not always picking up an address, but it should work fine, in theory, and it does work fine sometimes. I've included it so you can play with it if you want.

Viewing and Analyzing Results

Real-Time Results

For a real-time visualization of the network, you can issue the command "cmapAuto real" (assuming you have set CMap up correctly). For real-time plots of rate, latency, etc., you can use "plotAll" (or any of the other "plot*" commands). This may be done from XCom, but it is highly recommended that you not do this. If you wish to view CMap or TRPR data in real time, we suggest getting a separate machine and mounting XCom’s hard drive via NFS, otherwise the programs may use enough CPU to degrade your results. We have seen the programs cause periodic latency spikes and generally poor performance. Neither process uses that much CPU time, but they may use just enough at just the wrong times to throw the results off, so we advise that you avoid doing this.

Postanalysis

When a test has completed, you can use the "stopAll" script (from XCom) to stop MGEN, MNE, OLSR, etc. on all nodes. You can then use "getDataAll <testname>" to get the data from all nodes. This script will create a directory $MNEHOME/results/<testname>, and then SCP all data files from each node to a $MNEHOME/results/<testname>/<hostname> directory. In a many-to-one test, only XCom will get MGEN data, so you don't necessarily need to do this, but it can still be useful for analyzing OLSR logs to see what went wrong, or to see the network from the perspective of another node. Of course, for many-to-many tests, you need to get everyone's data so that you have all the MGEN data that was sent around the network. You can postanalyze the results for a many-to-many test using "postMany" from the $MNEHOME/results/<testname> directory. For a many-to-one test (and test for that matter), you can change to the directory where the files are (e.g. $MNEHOME/results/<testname>/xcom) and run "postThis". Either of these scripts will create rate and latency plots, which you can view using "gplot <gp_file(s)>". You can also play back a visualization of node positions, etc. using "cmapAuto <speed>" ("cmapAuto 3" would play back a 3x realtime). "clearAll" will clear the data from all nodes before starting the next test.

Extending the Scripts

Directory Structure

Adding More Testing Capabilities

Being bash shell scripts, it is easy to add new features and testing capabilities, though it probably helps a lot if you understand bash scripting. I recommend O'Reilly's Learning the bash Shell, by Newham & Rosenblatt, if you want to learn. There are some useful tutorials on the web as well.

There are a few things you can do without knowing a lot of bash scripting though. If you are familiar with the MGEN script syntax (see here), you can add new MGEN traffic models in $MNEHOME/mgen_scripts. You can add new motion model files (NS2, waypoints, gps, etc.) in the appropriate directory in $MNEHOME/motion_models. Please see the MNE documentation (that document is slightly outdated as far as commandline parameters in terms of IPv4/IPv6, but other than that it's good) for details on the file formats, and/or look at the examples included with the scripts.

Support for new routing protocols, or new parameter sets for routing protocols (i.e., you can have one version of nrlolsr with 0.5 second hellos, another with 0.25 second hellos and full link state, another with 2 second hellos, etc.) can also be added quite easily in $MNEHOME/routing_protocols/param_sets. These scripts should be set as executable, and they can be as simple as a one-line script that starts the protocol with a fixed set of parameters, or they can be complex to the point of accessing environment variables during the start process, in order to support IPv4 and IPv6 and read the test interface, set up appropriate routes, etc. See $MNEHOME/routing_protocols/param_sets/README for further details, and a list of environment variables that you may find useful in your script. Also note that if your protocol has the ability to support an HNA configuration file, that should ideally go in $MNEHOME/routing_protocols/hna_files.

Most other extensions to these scripts probably require you to know at least the basics of bash scripting (if/then/fi, etc.). Most other things you would be interested in adding should probably be done in the $MNEHOME/startscripts/common directory. Adding new commandline parameters should be done in execParser, and don't forget to export the variable you save, or it won't work. If you want to add new motion model typess (as opposed to adding new files for motion models that already exist, which is done by adding a file to the appropriate directory in $MNEHOME/motion_models), that can be done in the execMotion script. If you want to add different autoconfiguration methods, or start a Quality of Service (QoS) engine, or anything else that should be done before starting the routing protocol, you should do that in execPreRouting. Similarly, things that should be done after the routing protocol has started, like starting other network services such as multicast flooding, network services, etc., should be done in execPostRouting. New traffic types, applications, etc. should be added in execMgen, though you could also put them in execPostRouting.


Any problems or issues with these scripts should be directed here.