OneLab - Future Internet Testbeds

nepi-ng - B - wireless

Prerequisites

For this series of experiments, we make the same assumptions as in the previous series. In a nutshell, we expect you have a valid reservation, and the 2 nodes fit01and fit02 are properly loaded with the default image, and turned on of course.

Please visit this page to review how to reach that point, if necessary.

Objectives

In this series we will start playing with the wireless interfaces. Of course a minimal background is assumed in terms of dealing with wireless interfaces under linux; further information can be found at the following locations :

nepi-ng

Now as far as nepi-ng is concerned, in the previous series we have seen ways to run remotely simple commands. In this section we will see simple means to come up with more complex logic, simply by using shell scripts that are pushed remotely under the hood before they are triggered. This is what the RunString and RunScript classes are about. Let us see them in action right away.

Objective

We are going to run the exact same experiment as in the previous run A5, that is to say a simple ping triggered on fit01 towards fit02, but this time on one of the wireless interfaces.

b1

What changes then, as compared with our previous experiment, is that we cannot anymore simply run the predefined convenience command turn-on-data, and we are going to have to put a little more work in this step.

We are going to configure a small ad hoc network between both nodes, using their Atheros WiFi card, and this is the purpose of the bash shell script named turn_on_wireless_script; as you will see, this script uses some hard-wired settings like the channel and SSID, but you can easily tweak the code so as to make this a parameter.

New features

The B1 code below exhibits the use of a new class, RunString, which is a very convenient variation around the Run class.

Instead of remotely invoking a command that is supposed to be available there already, like we have done so far when e.g. invoking ping or even turn-on-data, RunString allows you to invoke a command that we provide as a python variable.

This means that we can write our own shell snippet, in charge of creating a small ad-hoc network, and embed this script right inside our nepi-ng script; see the turn_on_wireless_script variable in the code below.

See also how the init_node_01 job now uses RunString to pass it python variables as arguments. For example this is how turn_on_wireless_script gets passed the right IP address for each node.

As you will see in the sample output below, it takes some time for the IP connectivity to be established after the driver is configured.

The code

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False
wireless_driver="ath9k"
wireless_interface="atheros"

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

####################
# This is our own brewed script for setting up a wifi network
# it run on the remote machine - either sender or receiver
# and is in charge of initializing a small ad-hoc network
#
# Thanks to the RunString class, we can just define this as
# a python string, and pass it arguments from python variables
#

turn_on_wireless_script = """#!/bin/bash

# we expect the following arguments
# * wireless driver name (iwlwifi or ath9k)
# * wireless interface name (intel or atheros)
# * IP-address/mask for that interface
# * the wifi network name to join
# * the wifi frequency to use

driver=$1; shift
ifname=$1; shift
ipaddr_mask=$1; shift
netname=$1; shift
freq=$1;   shift

# load the r2lab utilities - code can be found here:
# https://github.com/fit-r2lab/r2lab-embedded/blob/master/shell/nodes.sh
source /etc/profile.d/nodes.sh

# make sure to use the latest code on the node
git-pull-r2lab

turn-off-wireless

echo loading module $driver
modprobe $driver

# some time for udev to trigger its rules
sleep 1

echo configuring interface $ifname
# make sure to wipe down everything first so we can run again and again
ip address flush dev $ifname
ip link set $ifname down
# configure wireless
iw dev $ifname set type ibss
ip link set $ifname up
# set to ad-hoc mode
iw dev $ifname ibss join $netname $freq
ip address add $ipaddr_mask dev $ifname
"""

##########
# setting up the wireless interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = RunString(
        # first argument is a string containing
        # the script to be run remotely
        turn_on_wireless_script,
        # and now its arguments
        wireless_driver, wireless_interface, "10.0.0.1/24", "foobar", 2412,
#        verbose=True,
    ),
    required = check_lease,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = RunString(
        turn_on_wireless_script,
        wireless_driver, wireless_interface, "10.0.0.2/24", "foobar", 2412,
    ),
    required = check_lease,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    command = Run(
        'ping', '-c', '20', '10.0.0.2', '-I', wireless_interface,
#        verbose=True,
    ),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()

# give details if it failed
ok or scheduler.debrief()

success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("B1.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output


fit02:From https://github.com/fit-r2lab/r2lab-embedded
fit02:   e496bfd..b8050d8  master     -> origin/master
fit02: * [new branch]      oai-5g-latest -> origin/oai-5g-latest
fit02: * [new branch]      using-library -> origin/using-library
fit01:From https://github.com/fit-r2lab/r2lab-embedded
fit01:   e496bfd..b8050d8  master     -> origin/master
fit01: * [new branch]      oai-5g-latest -> origin/oai-5g-latest
fit01: * [new branch]      using-library -> origin/using-library
fit02:Already on 'master'
fit01:Already on 'master'
fit02:From https://github.com/fit-r2lab/r2lab-embedded
fit02: * branch            master     -> FETCH_HEAD
fit02:turn-off-wireless: shutting down device intel
fit02:turn-off-wireless: removing driver iwlwifi
fit01:From https://github.com/fit-r2lab/r2lab-embedded
fit01: * branch            master     -> FETCH_HEAD
fit01:turn-off-wireless: shutting down device intel
fit01:turn-off-wireless: removing driver iwlwifi
fit02:turn-off-wireless: shutting down device atheros
fit02:turn-off-wireless: removing driver ath9k
fit01:turn-off-wireless: shutting down device atheros
fit01:turn-off-wireless: removing driver ath9k
faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
fit02:========== Updating /root/r2lab-embedded for branch master
fit02:HEAD is now at e496bfd bugs fixed
fit02:Fetching origin
fit01:========== Updating /root/r2lab-embedded for branch master
fit01:HEAD is now at e496bfd bugs fixed
fit01:Fetching origin
fit02:Your branch is behind 'origin/master' by 29 commits, and can be fast-forwarded.
fit02:  (use "git pull" to update your local branch)
fit01:Your branch is behind 'origin/master' by 29 commits, and can be fast-forwarded.
fit01:  (use "git pull" to update your local branch)
fit02:Updating e496bfd..b8050d8
fit02:Fast-forward
fit02: backgrounds/moto-e-4g.jpg | Bin 0 -> 102057 bytes
fit02: imaging/all-images.sh     |  15 ++++-
fit02: nightly/nightly.py        |  22 +++++---
fit02: shell/faraday.sh          |   6 +-
fit02: shell/imaging.sh          |   5 +-
fit02: shell/nodes.sh            |   4 +-
fit02: shell/oai-common.sh       |  18 ++++++
fit02: shell/oai-enb.sh          | 138 ++++++++++++++++++++++++++++++----------------
fit02: shell/oai-gw.sh           |   5 +-
fit02: 9 files changed, 145 insertions(+), 68 deletions(-)
fit02: create mode 100644 backgrounds/moto-e-4g.jpg
fit01:Updating e496bfd..b8050d8
fit01:Fast-forward
fit01: backgrounds/moto-e-4g.jpg | Bin 0 -> 102057 bytes
fit01: imaging/all-images.sh     |  15 ++++-
fit01: nightly/nightly.py        |  22 +++++---
fit01: shell/faraday.sh          |   6 +-
fit01: shell/imaging.sh          |   5 +-
fit01: shell/nodes.sh            |   4 +-
fit01: shell/oai-common.sh       |  18 ++++++
fit01: shell/oai-enb.sh          | 138 ++++++++++++++++++++++++++++++----------------
fit01: shell/oai-gw.sh           |   5 +-
fit01: 9 files changed, 145 insertions(+), 68 deletions(-)
fit01: create mode 100644 backgrounds/moto-e-4g.jpg
fit02:loading module ath9k
fit01:loading module ath9k
fit02:configuring interface atheros
fit01:configuring interface atheros
fit01:PING 10.0.0.2 (10.0.0.2) from 10.0.0.1 atheros: 56(84) bytes of data.
fit01:From 10.0.0.1 icmp_seq=1 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=2 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=3 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=4 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=5 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=6 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=7 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=8 Destination Host Unreachable
fit01:From 10.0.0.1 icmp_seq=9 Destination Host Unreachable
fit01:64 bytes from 10.0.0.2: icmp_seq=10 ttl=64 time=4.55 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=11 ttl=64 time=2.53 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=12 ttl=64 time=2.47 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=13 ttl=64 time=2.54 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=14 ttl=64 time=2.52 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=15 ttl=64 time=2.52 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=16 ttl=64 time=2.46 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=17 ttl=64 time=2.52 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=18 ttl=64 time=1.69 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=19 ttl=64 time=1.73 ms
fit01:64 bytes from 10.0.0.2: icmp_seq=20 ttl=64 time=1.73 ms
fit01:
fit01:--- 10.0.0.2 ping statistics ---
fit01:20 packets transmitted, 11 received, +9 errors, 45% packet loss, time 19063ms
fit01:rtt min/avg/max/mdev = 1.696/2.483/4.555/0.742 ms, pipe 3

          

Next

Let us see some variants on that theme in the next tab

Objective

In this new variant, we are going to illustrate a few convenient tricks:

  • we add a new option to the script so that one can choose, right on the command line, whether we want to use the intel or the atheros WiFi card; this simply relies on the standard argparse module that we have already used for other options, the only new thing being the use of the choices keyword, so that only the 2 supported driver names can be used;

  • more interestingly, we can remove the wireless_interface variable from that script altogether, thanks to a shell utilily available on all nodes and named wait-for-interface-on-driver. This shell function returns the name of the network interface associated with a given driver name, so it is enough to load, say, module ath9k and let this function figure out that the actual interface name is atheros;

  • in much the same way, we will remove the need to pass an IP address to the turn_on_wireless_script, by computing this address based on the node rank in r2lab, using another convenience tool named r2lab-ip; there actually are 2 similar functions available on each node

    • r2lab-ip: will return the number of the current node, under 1 or 2 digits; for example on node 8, this returns 8; this is suitable to build IP addresses;

    • r2lab-id: will return the number of the current node, but always in 2 digits; for example on node 8, this returns 08; this is suitable to build hostnames; so for example you would do

#

my_data_hostname="data$(r2lab-id)"
echo $my_data_hostname
data08
my_wireless_ip_address="10.0.0.$(r2lab-ip)"
echo my_wireless_ip_address
10.0.0.8
  • finally, this new script displays the outputs of the ssh commands with a slightly different format, in that every line will now receive a timestamp in addition to the hostname. This is achieved through the use of the TimeColonFormatter() class; we create one instance of this class for each instance of SshNode.

So for example we would issue lines like

14-37-28:fit01:turn-off-wireless: shutting down device intel

instead of just

fit01:turn-off-wireless: shutting down device intel

About that last point, note that other types of formatters are available, that will let you store output in a given directory, and in files named after each individual hostname. See this page for more info on other available formatters.

The code

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString
from apssh import TimeColonFormatter

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-d", "--driver", default='ath9k',
                    choices = ['iwlwifi', 'ath9k'],
                    help="specify which driver to use")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh
wireless_driver = args.driver

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh,
                  formatter = TimeColonFormatter())

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

####################
# This is our own brewed script for setting up a wifi network
# it run on the remote machine - either sender or receiver
# and is in charge of initializing a small ad-hoc network
#
# Thanks to the RunString class, we can just define this as
# a python string, and pass it arguments from python variables
#

turn_on_wireless_script = """#!/bin/bash

# we expect the following arguments
# * wireless driver name (iwlwifi or ath9k)
# * the wifi network name to join
# * the wifi frequency to use

driver=$1; shift
netname=$1; shift
freq=$1;   shift

# load the r2lab utilities - code can be found here:
# https://github.com/fit-r2lab/r2lab-embedded/blob/master/shell/nodes.sh
source /etc/profile.d/nodes.sh

# make sure to use the latest code on the node
git-pull-r2lab

turn-off-wireless

# local IP address to use is computed on the 10.0.0.0/24
# subnet and based on current node number (using r2lab-ip)
ipaddr_mask=10.0.0.$(r2lab-ip)/24

echo loading module $driver
modprobe $driver

# some time for udev to trigger its rules
sleep 1

# use r2lab tools to figure out the interface name
ifname=$(wait-for-interface-on-driver $driver)

echo configuring interface $ifname
# make sure to wipe down everything first so we can run again and again
ip address flush dev $ifname
ip link set $ifname down
# configure wireless
iw dev $ifname set type ibss
ip link set $ifname up
# set to ad-hoc mode
iw dev $ifname ibss join $netname $freq
ip address add $ipaddr_mask dev $ifname
"""

##########
# setting up the wireless interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = RunString(
        turn_on_wireless_script,
        wireless_driver, "foobar", 2412,
        # setting a remote_name allows to
        # improve the graphical rendering
        remote_name = 'turn-on-wireless'
#        verbose=True,
    ),
    required = check_lease,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = RunString(
        turn_on_wireless_script,
        wireless_driver, "foobar", 2412,
        remote_name = 'turn-on-wireless'
    ),
    required = check_lease,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    command = Run(
        'ping', '-c', '20', '10.0.0.2',
#        verbose=True,
    ),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()

# give details if it failed
ok or scheduler.debrief()

success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("B2.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output


11-00-55:fit01:Already on 'master'
11-00-55:fit02:Already on 'master'
11-00-55:fit01:From https://github.com/fit-r2lab/r2lab-embedded
11-00-55:fit01: * branch            master     -> FETCH_HEAD
11-00-55:fit01:turn-off-wireless: driver iwlwifi not used
11-00-55:fit02:From https://github.com/fit-r2lab/r2lab-embedded
11-00-55:fit02: * branch            master     -> FETCH_HEAD
11-00-55:fit01:turn-off-wireless: shutting down device atheros
11-00-55:fit02:turn-off-wireless: driver iwlwifi not used
11-00-55:fit02:turn-off-wireless: shutting down device atheros
11-00-55:fit01:turn-off-wireless: removing driver ath9k
11-00-55:fit02:turn-off-wireless: removing driver ath9k
11-00-56:fit01:Using id=01 and fitid=fit01 - from hostname
11-00-56:fit02:Using id=02 and fitid=fit02 - from hostname
11-00-58:fit01:Using device atheros
11-00-58:fit02:Using device atheros
11-00-53:faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
11-00-54:fit01:========== Updating /root/r2lab-embedded for branch master
11-00-54:fit01:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
11-00-54:fit01:Fetching origin
11-00-54:fit02:========== Updating /root/r2lab-embedded for branch master
11-00-54:fit02:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
11-00-54:fit02:Fetching origin
11-00-55:fit01:Your branch is up-to-date with 'origin/master'.
11-00-55:fit02:Your branch is up-to-date with 'origin/master'.
11-00-55:fit01:Already up-to-date.
11-00-55:fit02:Already up-to-date.
11-00-56:fit01:loading module ath9k
11-00-56:fit02:loading module ath9k
11-00-58:fit01:configuring interface atheros
11-00-58:fit02:configuring interface atheros
11-01-01:fit01:PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
11-01-01:fit01:From 10.0.0.1 icmp_seq=1 Destination Host Unreachable
11-01-01:fit01:From 10.0.0.1 icmp_seq=2 Destination Host Unreachable
11-01-01:fit01:From 10.0.0.1 icmp_seq=3 Destination Host Unreachable
11-01-04:fit01:From 10.0.0.1 icmp_seq=4 Destination Host Unreachable
11-01-04:fit01:From 10.0.0.1 icmp_seq=5 Destination Host Unreachable
11-01-04:fit01:From 10.0.0.1 icmp_seq=6 Destination Host Unreachable
11-01-06:fit01:64 bytes from 10.0.0.2: icmp_seq=7 ttl=64 time=2004 ms
11-01-06:fit01:64 bytes from 10.0.0.2: icmp_seq=8 ttl=64 time=1006 ms
11-01-06:fit01:64 bytes from 10.0.0.2: icmp_seq=9 ttl=64 time=6.51 ms
11-01-07:fit01:64 bytes from 10.0.0.2: icmp_seq=10 ttl=64 time=1.33 ms
11-01-08:fit01:64 bytes from 10.0.0.2: icmp_seq=11 ttl=64 time=1.34 ms
11-01-09:fit01:64 bytes from 10.0.0.2: icmp_seq=12 ttl=64 time=1.34 ms
11-01-10:fit01:64 bytes from 10.0.0.2: icmp_seq=13 ttl=64 time=1.35 ms
11-01-11:fit01:64 bytes from 10.0.0.2: icmp_seq=14 ttl=64 time=1.34 ms
11-01-12:fit01:64 bytes from 10.0.0.2: icmp_seq=15 ttl=64 time=1.34 ms
11-01-13:fit01:64 bytes from 10.0.0.2: icmp_seq=16 ttl=64 time=1.34 ms
11-01-14:fit01:64 bytes from 10.0.0.2: icmp_seq=17 ttl=64 time=1.34 ms
11-01-15:fit01:64 bytes from 10.0.0.2: icmp_seq=18 ttl=64 time=1.39 ms
11-01-16:fit01:64 bytes from 10.0.0.2: icmp_seq=19 ttl=64 time=1.22 ms
11-01-17:fit01:64 bytes from 10.0.0.2: icmp_seq=20 ttl=64 time=1.28 ms
11-01-17:fit01:
11-01-17:fit01:--- 10.0.0.2 ping statistics ---
11-01-17:fit01:20 packets transmitted, 14 received, +6 errors, 30% packet loss, time 19015ms
11-01-17:fit01:rtt min/avg/max/mdev = 1.228/216.582/2004.461/558.972 ms, pipe 3

          

Next

In the next variant, we are going to see how we can better esimate the time it takes for the ad hoc network to actually come up.

Objective

In this variant, we will see how we can even design our experiment so that all the shell code goes in a single file, working as a companion for the python script that will only deal with overall logic.

Our pretext for doing so is that we would like to better understand the startup sequence observed in the previous runs B1 and B2. And for doing that it is not good enough to just run ping like we did, but instead we would need run something a little more elaborate - that we'll write in shell.

As much as it is convenient to have the ability to insert shell script right in the python code, when things become more complex, the shell code tends to become in the way.

So, two features are at work in the following code

  • Using RunScript instead of RunString as a way to define commands allows us to separate the shell script in a separate file.

  • Also you can note at the end of the shell script, a very simple trick that lets you group any number of functions in a shell script, and call each individual function by just stating its name and arguments.

In other words, it means that if you write the following in a file named myscript.sh:

#!/bin/bash
function foo () {
    echo "in function foo"
    for arg in "$@"; do echo "foo arg: " $arg; done
}
    
function bar () {
    echo "in function bar"
    for arg in "$@"; do echo "bar arg: " $arg; done
}

"$@"

and then you invoke myscript.sh foo 1 2 "3 4" you will get this ouput

$ myscript.sh foo 1 2 "3 4"
in function foo
foo arg:  1
foo arg:  2
foo arg:  3 4

The code

Make sure you download both files in the same location before trying to run the python script.

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString, RunScript
from apssh import TimeColonFormatter

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-d", "--driver", default='ath9k',
                    choices = ['iwlwifi', 'ath9k'],
                    help="specify which driver to use")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh
wireless_driver = args.driver

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh,
                  formatter = TimeColonFormatter())

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the shell script has gone into B3-wireless.sh
####################

##########
# setting up the wireless interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    # RunString is replaced with RunScript
    command = RunScript(
        # first argument is the local filename
        # where to find the script to run remotely
        "B3-wireless.sh",
        # then its arguments
        "init-ad-hoc-network", wireless_driver, "foobar", 2412,
        # no need to set remote_name with RunScript
    ),
    required = check_lease,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = RunScript(
        "B3-wireless.sh",
        "init-ad-hoc-network", wireless_driver, "foobar", 2412,
        # setting a label to gain space in the graphical output
        label="ditto",
    ),
    required = check_lease,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    command = RunScript(
        "B3-wireless.sh", "my-ping", '10.0.0.2', 20,
#        verbose=True,
    ),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()

# give details if it failed
ok or scheduler.debrief()

success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("B3.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/bin/bash

####################
# This is our own brewed script for setting up a wifi network
# it run on the remote machine - either sender or receiver
# and is in charge of initializing a small ad-hoc network
#
# Thanks to the RunString class, we can just define this as
# a python string, and pass it arguments from python variables
#


# we expect the following arguments
# * wireless driver name (iwlwifi or ath9k)
# * the wifi network name to join
# * the wifi frequency to use

function init-ad-hoc-network (){
    driver=$1; shift
    netname=$1; shift
    freq=$1;   shift

    # load the r2lab utilities - code can be found here:
    # https://github.com/fit-r2lab/r2lab-embedded/blob/master/shell/nodes.sh
    source /etc/profile.d/nodes.sh

    # make sure to use the latest code on the node
    git-pull-r2lab

    turn-off-wireless
    
    ipaddr_mask=10.0.0.$(r2lab-ip)/24

    echo loading module $driver
    modprobe $driver
    
    # some time for udev to trigger its rules
    sleep 1
    
    ifname=$(wait-for-interface-on-driver $driver)

    echo configuring interface $ifname
    # make sure to wipe down everything first so we can run again and again
    ip address flush dev $ifname
    ip link set $ifname down
    # configure wireless
    iw dev $ifname set type ibss
    ip link set $ifname up
    # set to ad-hoc mode
    iw dev $ifname ibss join $netname $freq
    ip address add $ipaddr_mask dev $ifname

    ### addition - would be cool to come up with something along these lines that
    # works on both cards
    # a recipe from Naoufal for Intel
    # modprobe iwlwifi
    # iwconfig wlan2 mode ad-hoc
    # ip addr add 10.0.0.41/16 dev wlan2
    # ip link set wlan2 up
    # iwconfig wlan2 essid mesh channel 1
    
}

function my-ping (){
    dest=$1; shift
    maxwait=$1; shift
    
    start=$(date +%s)

    while true; do
	# allow for one second timeout only; try one packet
	if ping -w 1 -c 1 $dest >& /dev/null; then
	    end=$(date +%s)
	    duration=$(($end - $start))
	    echo "$(hostname) -> $dest: SUCCESS after ${duration}s"
	    return 0
	else
	    echo "$dest not reachable"
	    end=$(date +%s)
	    duration=$(($end - $start))
	    if [ "$duration" -ge "$maxwait" ]; then
		echo "$(hostname) -> $dest: FAILURE after ${duration}s"
		return 1
	    fi
	fi
    done
}

########################################
# just a wrapper so we can call the individual functions. so e.g.
# B3-wireless.sh tracable-ping 10.0.0.2 20
# results in calling tracable-ping 10.0.0.2 20

"$@"

Sample output


11-01-22:fit01:Already on 'master'
11-01-22:fit02:Already on 'master'
11-01-22:fit01:From https://github.com/fit-r2lab/r2lab-embedded
11-01-22:fit01: * branch            master     -> FETCH_HEAD
11-01-22:fit01:turn-off-wireless: driver iwlwifi not used
11-01-22:fit01:turn-off-wireless: shutting down device atheros
11-01-22:fit01:turn-off-wireless: removing driver ath9k
11-01-22:fit02:From https://github.com/fit-r2lab/r2lab-embedded
11-01-22:fit02: * branch            master     -> FETCH_HEAD
11-01-22:fit02:turn-off-wireless: driver iwlwifi not used
11-01-22:fit02:turn-off-wireless: shutting down device atheros
11-01-22:fit02:turn-off-wireless: removing driver ath9k
11-01-22:fit02:Using id=02 and fitid=fit02 - from hostname
11-01-22:fit01:Using id=01 and fitid=fit01 - from hostname
11-01-25:fit02:Using device atheros
11-01-25:fit01:Using device atheros
11-01-20:faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
11-01-21:fit01:========== Updating /root/r2lab-embedded for branch master
11-01-21:fit01:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
11-01-21:fit01:Fetching origin
11-01-21:fit02:========== Updating /root/r2lab-embedded for branch master
11-01-21:fit02:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
11-01-21:fit02:Fetching origin
11-01-22:fit01:Your branch is up-to-date with 'origin/master'.
11-01-22:fit02:Your branch is up-to-date with 'origin/master'.
11-01-22:fit01:Already up-to-date.
11-01-22:fit02:Already up-to-date.
11-01-22:fit02:loading module ath9k
11-01-22:fit01:loading module ath9k
11-01-25:fit02:configuring interface atheros
11-01-25:fit01:configuring interface atheros
11-01-26:fit01:10.0.0.2 not reachable
11-01-27:fit01:10.0.0.2 not reachable
11-01-28:fit01:10.0.0.2 not reachable
11-01-29:fit01:10.0.0.2 not reachable
11-01-30:fit01:10.0.0.2 not reachable
11-01-31:fit01:10.0.0.2 not reachable
11-01-32:fit01:10.0.0.2 not reachable
11-01-33:fit01:10.0.0.2 not reachable
11-01-34:fit01:10.0.0.2 not reachable
11-01-34:fit01:fit01 -> 10.0.0.2: SUCCESS after 9s

          

Next

Let us proceed on this series with an example that adds a cyclic task to this scenario.

Objective

In this example, we will just for fun add an infinite cyclic task in the scheduler. Here we will simply write a TICK mark every second, but this technique is most useful for consuming events in a message queue, or any other similar approach.

The trick is to use the plain Job class from asynciojobs, which expects a regular asyncio coroutine object, that we implement as infinite_clock(). We just need to define the associated job clock_job with forever = True, which tells the scheduler that this job never ends, so that it knows that there is no point in waiting for it to complete.

More on graphical views

Note in this example that, with respect to the graphical representation of schedulers:

  • each job needs a specific label to be displayed in such a graph - as well as in list() and debrief() by the way

  • the system does its best to provide a meaningful label, but

  • you are always free to define your own label on a given job, like we do here with the infinite clock job.

The code

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler
from asynciojobs import Job

from apssh import SshNode, SshJob
from apssh import Run, RunString, RunScript
from apssh import TimeColonFormatter

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-d", "--driver", default='ath9k',
                    choices = ['iwlwifi', 'ath9k'],
                    help="specify which driver to use")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh
wireless_driver = args.driver

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh,
                  formatter = TimeColonFormatter())

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

##########
# setting up the wireless interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = RunScript(
        "B3-wireless.sh",
        "init-ad-hoc-network", wireless_driver, "foobar", 2412,
    ),
    required = check_lease,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = RunScript(
        "B3-wireless.sh",
        "init-ad-hoc-network", wireless_driver, "foobar", 2412,
        label = "ditto",
    ),
    required = check_lease,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    command = RunScript(
        "B3-wireless.sh", "my-ping", '10.0.0.2', 20,
#        verbose=True,
    ),
    scheduler = scheduler,
)

########
# for the fun of it, let's add a job that runs forever and writes
# current time every second
import time
import asyncio

async def infinite_clock():
    while True:
        print("--- TICK - {}".format(time.strftime("%H:%M:%S")))
        await asyncio.sleep(1)

# a forever job is not expected to end, instead
# it gets killed when the rest of the flock is done with
clock_job = Job(
    infinite_clock(),
    forever=True,
    scheduler = scheduler,
    # for the illustrated graph
    label = "infinite clock",
)

##########
# run the scheduler
ok = scheduler.orchestrate()

# give details if it failed
ok or scheduler.debrief()

success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("B4.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output


11-01-39:fit01:Already on 'master'
11-01-39:fit02:Already on 'master'
11-01-39:fit01:From https://github.com/fit-r2lab/r2lab-embedded
11-01-39:fit01: * branch            master     -> FETCH_HEAD
11-01-39:fit01:turn-off-wireless: driver iwlwifi not used
11-01-39:fit01:turn-off-wireless: shutting down device atheros
11-01-40:fit01:turn-off-wireless: removing driver ath9k
11-01-40:fit02:From https://github.com/fit-r2lab/r2lab-embedded
11-01-40:fit02: * branch            master     -> FETCH_HEAD
11-01-40:fit02:turn-off-wireless: driver iwlwifi not used
11-01-40:fit02:turn-off-wireless: shutting down device atheros
11-01-40:fit02:turn-off-wireless: removing driver ath9k
11-01-40:fit01:Using id=01 and fitid=fit01 - from hostname
11-01-40:fit02:Using id=02 and fitid=fit02 - from hostname
11-01-42:fit01:Using device atheros
11-01-42:fit02:Using device atheros
--- TICK - 11:01:35
--- TICK - 11:01:36
--- TICK - 11:01:37
11-01-37:faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
--- TICK - 11:01:38
11-01-38:fit02:========== Updating /root/r2lab-embedded for branch master
11-01-38:fit02:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
11-01-38:fit02:Fetching origin
11-01-38:fit01:========== Updating /root/r2lab-embedded for branch master
11-01-38:fit01:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
11-01-38:fit01:Fetching origin
--- TICK - 11:01:39
11-01-39:fit01:Your branch is up-to-date with 'origin/master'.
11-01-39:fit02:Your branch is up-to-date with 'origin/master'.
11-01-39:fit01:Already up-to-date.
11-01-40:fit02:Already up-to-date.
--- TICK - 11:01:40
11-01-40:fit01:loading module ath9k
11-01-40:fit02:loading module ath9k
--- TICK - 11:01:41
--- TICK - 11:01:42
11-01-42:fit01:configuring interface atheros
11-01-42:fit02:configuring interface atheros
--- TICK - 11:01:43
11-01-44:fit01:10.0.0.2 not reachable
--- TICK - 11:01:44
11-01-45:fit01:10.0.0.2 not reachable
--- TICK - 11:01:45
11-01-46:fit01:10.0.0.2 not reachable
--- TICK - 11:01:46
11-01-47:fit01:10.0.0.2 not reachable
--- TICK - 11:01:47
11-01-48:fit01:10.0.0.2 not reachable
--- TICK - 11:01:48
11-01-49:fit01:10.0.0.2 not reachable
--- TICK - 11:01:49
11-01-50:fit01:10.0.0.2 not reachable
--- TICK - 11:01:50
11-01-51:fit01:10.0.0.2 not reachable
11-01-51:fit01:fit01 -> 10.0.0.2: SUCCESS after 8s

          

Next

In the next example, we will see a convenience class named Watch, that comes in handy in most experiments as it allows to keep track of elapsed time since the beginning of an experiment.

Objective

In this final example in the series, we will take the chance to illustrate a convenience class named Watch from asynciojobs, that allows to keep track of elapsed time since some reference point in time, generally the beginning of the experiment.

So here, we create a Watch instance, and use it when the clock is ticking, instead of showing plain wall clock time.

The code

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler
from asynciojobs import Job
from asynciojobs import Watch

from apssh import SshNode, SshJob
from apssh import Run, RunString, RunScript
from apssh import TimeColonFormatter

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-d", "--driver", default='ath9k',
                    choices = ['iwlwifi', 'ath9k'],
                    help="specify which driver to use")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh
wireless_driver = args.driver

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh,
                  formatter = TimeColonFormatter())

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh,
                formatter = TimeColonFormatter())

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

##########
# setting up the wireless interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = RunScript(
        "B3-wireless.sh",
        "init-ad-hoc-network", wireless_driver, "foobar", 2412,
    ),
    required = check_lease,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = RunScript(
        "B3-wireless.sh",
        "init-ad-hoc-network", wireless_driver, "foobar", 2412,
        label = "ditto",
    ),
    required = check_lease,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    command = RunScript(
        "B3-wireless.sh", "my-ping", '10.0.0.2', 20,
#        verbose=True,
    ),
    scheduler = scheduler,
)

########
# for the fun of it, let's add a job that runs forever and writes
# current time every second
import time
import asyncio


async def infinite_clock(watch):
    while True:
        print("--- TICK - {}".format(watch.elapsed()))
        await asyncio.sleep(1)

# create a Watch instance for keeping track of elapsed time
watch = Watch()

# a forever job is not expected to end, instead
# it gets killed when the rest of the flock is done with
clock_job = Job(
    infinite_clock(watch),
    forever=True,
    scheduler = scheduler,
    # for the illustrated graph
    label = "infinite stopwatch",
)

##########
# run the scheduler
ok = scheduler.orchestrate()

# give details if it failed
ok or scheduler.debrief()

success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("B5.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output


12-35-47:fit01:Already on 'master'
12-35-47:fit02:Already on 'master'
12-35-48:fit01:From https://github.com/fit-r2lab/r2lab-embedded
12-35-48:fit01: * branch            master     -> FETCH_HEAD
12-35-48:fit02:From https://github.com/fit-r2lab/r2lab-embedded
12-35-48:fit02: * branch            master     -> FETCH_HEAD
12-35-48:fit01:turn-off-wireless: driver iwlwifi not used
12-35-48:fit01:turn-off-wireless: shutting down device atheros
12-35-48:fit02:turn-off-wireless: driver iwlwifi not used
12-35-48:fit02:turn-off-wireless: shutting down device atheros
12-35-48:fit01:turn-off-wireless: removing driver ath9k
12-35-48:fit02:turn-off-wireless: removing driver ath9k
12-35-48:fit02:Using id=02 and fitid=fit02 - from hostname
12-35-48:fit01:Using id=01 and fitid=fit01 - from hostname
12-35-50:fit02:Using device atheros
12-35-50:fit01:Using device atheros
000.000  
--- TICK - 000.000
--- TICK - 001.005
12-35-46:faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
12-35-46:fit01:========== Updating /root/r2lab-embedded for branch master
12-35-46:fit01:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
12-35-46:fit01:Fetching origin
12-35-46:fit02:========== Updating /root/r2lab-embedded for branch master
12-35-46:fit02:HEAD is now at b8050d8 Merge branch 'master' of github.com:fit-r2lab/r2lab-embedded
12-35-46:fit02:Fetching origin
--- TICK - 002.008
12-35-47:fit01:Your branch is up-to-date with 'origin/master'.
12-35-47:fit02:Your branch is up-to-date with 'origin/master'.
--- TICK - 003.011
12-35-48:fit01:Already up-to-date.
12-35-48:fit02:Already up-to-date.
12-35-48:fit02:loading module ath9k
12-35-48:fit01:loading module ath9k
--- TICK - 004.016
--- TICK - 005.017
12-35-50:fit02:configuring interface atheros
12-35-50:fit01:configuring interface atheros
--- TICK - 006.018
12-35-51:fit01:10.0.0.2 not reachable
--- TICK - 007.020
12-35-52:fit01:10.0.0.2 not reachable
--- TICK - 008.020
12-35-53:fit01:10.0.0.2 not reachable
--- TICK - 009.025
12-35-54:fit01:10.0.0.2 not reachable
--- TICK - 010.027
12-35-55:fit01:10.0.0.2 not reachable
--- TICK - 011.029
12-35-56:fit01:10.0.0.2 not reachable
--- TICK - 012.030
12-35-57:fit01:10.0.0.2 not reachable
--- TICK - 013.034
12-35-58:fit01:10.0.0.2 not reachable
--- TICK - 014.035
12-35-59:fit01:10.0.0.2 not reachable
12-35-59:fit01:fit01 -> 10.0.0.2: SUCCESS after 9s

          

Next

It is now time to wrap up this series.

We now know how to:

  • have local scripts executed remotely; we have seen that we can either

    • write them as plain shell scripts - using RunScript,
    • or embed them right in python strings - using RunString;
  • obtain remote outputs using alternate formats, using e.g. TimeColonFormatter; see this page for more info on other available formatters;

  • run infinite jobs, that will get properly terminated when all the finite jobs in the scenario are done;

  • use the Watch class to keep track of elapsed time, as opposed to displaying wall clock time.

In the next series of tutorials, we will learn more about transferring files back and forth.