nepi-ng - D - testbed preping

Preparation

At this point, you should have a clear understanding of how to design your experiment using nepi-ng. In this tutorial section, we come back to the initialization and preparation stages, that we had done manually at first - i.e. using the shell tools - and show how to account for these various preparation tasks in a nepi-ng script.

To this end we will improve our script so that it takes care of initializations, and specifically we will add a command line option that, when enabled:

loads the ubuntu image on the nodes that we do use,
and makes sure that all the nodes that we do not use are actually turned off.

It is a rather common need for all experiments to take these aspects into account. We do recommend to make image loading an option though, since performing this for every run vey quickly gets in the way when fine-tuning your experiment.

Please note that the script will not try to get a lease automatically, instead we keep the simple policy of just checking that we have a valid lease.

Like always, we will improve the code by small incremental changes.

Important note on shell tools

Convenience tools are mostly just aliases

Before we dive in this area, we need to stress an important point. In the tutorial on the shell tools, we had seen sessions like this:

rleases
all-off
n 1-5 33,37 ~3
rload -i fedora

It is important to emphasize that the commands involved here are all volatile bash materials like aliases or functions. This means that, as of beginning of Feb. 2018 at least, if you try to run from your laptop something like this:

$ ssh inria_r2lab.tutorial@faraday.inria.fr rleases
bash: rleases: command not found

you get an error, that is to be compared this with:

$ ssh inria_r2lab.tutorial@faraday.inria.fr rhubarbe leases
----- <Leases from PLCAPIproxy@https://r2labapi.inria.fr:443/PLCAPI/ - 25 lease(s)>
  1  < from 02-22 @ 08:00 until 09:00 CET inria_oai
...

The rationale being that there is a need for very short commands in an interactive shell environment. But when invoking commands remotely through ssh, you don't run bash and so the bash-written convenience layer does not come into play.

How to find out long commands

For starters, as a rule of thumb, all convenience commands that are derived from rhubarbe are made up the same way. Just like rleases is an alias for rhubarbe leases, you will find that rbye is an alias for rhubarbe bye.

inria_r2lab.tutorial@faraday:~$ type rload
rload is aliased to `rhubarbe load'
inria_r2lab.tutorial@faraday:~$ type rwait
rwait is aliased to `rhubarbe wait'

As you can see, it is very easy to find out about a given command when logged on faraday, using bash's built-in type command; note that type is much more powerful and accurate than the old-school which thing, that is only able to locate a command along your path, while type knowns about aliases and functions as well.

inria_r2lab.tutorial@faraday:~$ which rwait
inria_r2lab.tutorial@faraday:~$

How to use short commands in scripts

Finally, note that you can expose short commands in your scripts by adding a line Like this in your shell script:

# this would define e.g. the 'rwait' command
source /etc/profile.d/faraday.sh

The same goes of course for convenience tools exposed from the nodes environment; as we have seen on a few occasions before, if you need to use any of these commands in code meant to run on nodes, you can insert this line in your bash script:

# this would define e.g. the `r2lab-id` command
source /etc/profile.d/nodes.sh

All this being said, let us now see in the next tab D1 how to extend the A5 script and turn it into D1 that knows how to load images.

Objective

When working on an experiment, you will find yourself making several reservations, as not everything works as planned the first time.

For this reason it is useful to design your experiment script so that it can be used both:

at the beginning of your reserved timeslot, and in this case you will need the script to load the image of your choice on the nodes,
or the rest of the time, when you just need to re-run everything, but not to necessarily to reload images, given that this is relatively time-consuming.

Starting point

It is the purpose of the present step, to show you how to simply add a command line option that, when provided, triggers image loading.

For this step, we start back from the last code in the A section - namely A5-ping.py, remember:

The code

Every time we will see a piece of code, you can download the raw code with the right-most button.

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
if args.load:
    ready_requirement = SshJob(
        node = faraday,
        commands = [
            Run('rhubarbe load -i ubuntu 1 2'),
            Run('rhubarbe wait 1 2'),
        ],
        required = check_lease,
        scheduler = scheduler,
    )

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D1.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = check_lease,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = check_lease,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    # wait for the 2 init jobs instead
    # check_release is guaranteed to have completed anyway
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()

# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("A5.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
if args.load:
    ready_requirement = SshJob(
        node = faraday,
        commands = [
            Run('rhubarbe load -i ubuntu 1 2'),
            Run('rhubarbe wait 1 2'),
        ],
        required = check_lease,
        scheduler = scheduler,
    )

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D1.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output

Here's the output of the script when run with the -l option.

`$ python3 D1-prep.py -l`


faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
faraday.inria.fr:Found binary frisbeed as /usr/sbin/frisbeed
faraday.inria.fr:Found binary nc as /bin/nc
faraday.inria.fr:08:44:04 - +000s: Selection: fit01 fit02
faraday.inria.fr:08:44:04 - +000s: Loading image /var/lib/rhubarbe-images/ubuntu.ndz
faraday.inria.fr:08:44:04 - +000s: AUTH: checking for a valid lease
faraday.inria.fr:08:44:04 - +000s: AUTH: access granted
faraday.inria.fr:08:44:04 - +000s: fit02 reboot = Sending message 'reset' to CMC reboot02
faraday.inria.fr:08:44:04 - +000s: fit01 reboot = Sending message 'reset' to CMC reboot01
faraday.inria.fr:08:44:06 - +002s: fit02 reboot = idling for 15s
faraday.inria.fr:08:44:06 - +002s: fit01 reboot = idling for 15s
faraday.inria.fr:08:44:22 - +018s: started 
faraday.inria.fr:08:44:22 - +018s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:22 - +018s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:24 - +020s: fit02 frisbee_status = timed out..
faraday.inria.fr:08:44:24 - +020s: fit02 frisbee_status = backing off for 4.03s
faraday.inria.fr:08:44:24 - +020s: fit01 frisbee_status = timed out..
faraday.inria.fr:08:44:24 - +020s: fit01 frisbee_status = backing off for 3.94s
faraday.inria.fr:08:44:28 - +024s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:29 - +024s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:30 - +026s: fit01 frisbee_status = timed out..
faraday.inria.fr:08:44:30 - +026s: fit01 frisbee_status = backing off for 2.21s
faraday.inria.fr:08:44:31 - +026s: fit02 frisbee_status = timed out..
faraday.inria.fr:08:44:31 - +026s: fit02 frisbee_status = backing off for 4.4s
faraday.inria.fr:08:44:33 - +028s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:35 - +030s: fit01 frisbee_status = timed out..
faraday.inria.fr:08:44:35 - +030s: fit01 frisbee_status = backing off for 4.43s
faraday.inria.fr:08:44:35 - +030s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:37 - +032s: fit02 frisbee_status = timed out..
faraday.inria.fr:08:44:37 - +032s: fit02 frisbee_status = backing off for 1.78s
faraday.inria.fr:08:44:39 - +034s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:39 - +034s: fit02 frisbee_status = starting frisbee client
faraday.inria.fr:08:44:39 - +034s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:44:39 - +034s: fit01 frisbee_status = starting frisbee client
faraday.inria.fr:08:45:01 - +057s: fit02 Uploading successful
faraday.inria.fr:08:45:01 - +057s: fit02 reboot = Sending message 'reset' to CMC reboot02
faraday.inria.fr:|                                                   |  0% |0.00s|ETA:  --:--:--
|#                                                   |  3% |1.05s|ETA:  0:00:34
|###                                                 |  6% |1.27s|ETA:  0:00:19
|####                                                |  9% |1.85s|ETA:  0:00:18
|#####                                               | 11% |1.99s|ETA:  0:00:15
|#######                                             | 14% |3.40s|ETA:  0:00:20
|########                                            | 16% |3.57s|ETA:  0:00:18
|##########                                          | 19% |4.67s|ETA:  0:00:19
|###########                                         | 22% |5.11s|ETA:  0:00:17
|#############                                       | 26% |5.46s|ETA:  0:00:15
|###############                                     | 29% |6.19s|ETA:  0:00:15
|################                                    | 32% |7.00s|ETA:  0:00:14
|##################                                  | 36% |7.93s|ETA:  0:00:13
|####################                                | 39% |9.41s|ETA:  0:00:14
|#####################                               | 41% |9.67s|ETA:  0:00:13
|######################                             | 43% |10.40s|ETA:  0:00:13
|#######################                            | 46% |10.88s|ETA:  0:00:12
|#########################                          | 49% |11.28s|ETA:  0:00:11
|###########################                        | 53% |11.73s|ETA:  0:00:10
|#############################                      | 57% |12.29s|ETA:  0:00:09
|##############################                     | 60% |14.43s|ETA:  0:00:09
|################################                   | 63% |15.06s|ETA:  0:00:08
|#################################                  | 65% |15.46s|ETA:  0:00:08
|###################################                | 69% |15.63s|ETA:  0:00:06
|####################################               | 72% |15.77s|ETA:  0:00:06
|#####################################              | 73% |15.96s|ETA:  0:00:05
|#######################################            | 77% |16.60s|ETA:  0:00:04
|#########################################          | 80% |17.28s|ETA:  0:00:04
|###########################################        | 84% |17.57s|ETA:  0:00:03
|############################################       | 88% |19.98s|ETA:  0:00:02
|##############################################     | 91% |20.11s|ETA:  0:00:01
|################################################   | 95% |20.55s|ETA:  0:00:01
|################################################## | 99% |21.00s|ETA:  0:00:00
|################################################## | 99% |22.58s|ETA:  0:00:00
|###################################################|100% |23.05s|Time: 0:00:23
faraday.inria.fr:08:45:02 - +057s: fit01 Uploading successful
faraday.inria.fr:08:45:02 - +057s: fit01 reboot = Sending message 'reset' to CMC reboot01
faraday.inria.fr:08:45:04 - +059s: stopped 
faraday.inria.fr::ssh OK
faraday.inria.fr::ssh OK
fit01:turn-on-data: data network on interface data
fit02:turn-on-data: data network on interface data
fit02:data
fit01:data
fit01:PING data02 (192.168.2.2) from 192.168.2.1 data: 56(84) bytes of data.
fit01:64 bytes from data02 (192.168.2.2): icmp_seq=1 ttl=64 time=0.456 ms
fit01:
fit01:--- data02 ping statistics ---
fit01:1 packets transmitted, 1 received, 0% packet loss, time 0ms
fit01:rtt min/avg/max/mdev = 0.456/0.456/0.456/0.000 ms

We will now see in tab D2 how to also make sure all the rest of the testbed is turned off.

Objectives

In this section we study a variant D2, that is naturally based on D1, and that also makes sure that the unused nodes in the testbed are properly turned off.

Selection in `rhubarbe`

For doing so, we take advantage of a feature in rhubarbe, that is designed to make node selection a smooth process; in particular we use:

the --all (or just -a) option that selects all nodes,
and the ~ operator for removing nodes from the selection.

So for example

inria_r2lab.tutorial@faraday:~$ rhubarbe nodes -a
fit01 fit02 fit03 fit04 fit05 fit06 fit07 fit08 fit09 fit10 fit11 fit12 fit13 fit14 fit15 fit16 fit17 fit18 fit19 fit20 fit21 fit22 fit23 fit24 fit25 fit26 fit27 fit28 fit29 fit30 fit31 fit32 fit33 fit34 fit35 fit36 fit37

selects all 37 nodes, while

inria_r2lab.tutorial@faraday:~$ rhubarbe nodes -a ~1 ~2
fit03 fit04 fit05 fit06 fit07 fit08 fit09 fit10 fit11 fit12 fit13 fit14 fit15 fit16 fit17 fit18 fit19 fit20 fit21 fit22 fit23 fit24 fit25 fit26 fit27 fit28 fit29 fit30 fit31 fit32 fit33 fit34 fit35 fit36 fit37

selects all nodes but nodes number 1 and 2. This is the basic mechanism that we use to invoke the rhubarbe bye (a.k.a. all-off) command to turn off all the nodes that are not involved in our experiment.

requirement(s)

Also you will notice in this code that the required value for a given job may be either a plain job object, or a collection of jobs; in the code above, the ready_requirement variable, that was a single job object in D1-prep.py, is made a list with 2 jobs, and this is how we simply achieve the layout for the new dependency graph.

The code

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu 1 2'),
                Run('rhubarbe wait 1 2'),
                ],
            required = check_lease,
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a ~1 ~2')
            ],
            required = check_lease,
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D2.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
if args.load:
    ready_requirement = SshJob(
        node = faraday,
        commands = [
            Run('rhubarbe load -i ubuntu 1 2'),
            Run('rhubarbe wait 1 2'),
        ],
        required = check_lease,
        scheduler = scheduler,
    )

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D1.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu 1 2'),
                Run('rhubarbe wait 1 2'),
                ],
            required = check_lease,
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a ~1 ~2')
            ],
            required = check_lease,
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D2.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output

No great news here, just observe how all the other nodes, and the phones, get turned off while the 2 target nodes are being imaged.

`$ python3 D2-prep.py -l`


faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
faraday.inria.fr:Found binary frisbeed as /usr/sbin/frisbeed
faraday.inria.fr:Found binary nc as /bin/nc
faraday.inria.fr:08:46:44 - +000s: Selection: fit01 fit02
faraday.inria.fr:08:46:44 - +000s: Loading image /var/lib/rhubarbe-images/ubuntu.ndz
faraday.inria.fr:08:46:44 - +000s: AUTH: checking for a valid lease
faraday.inria.fr:08:46:44 - +000s: AUTH: access granted
faraday.inria.fr:reboot23:ok
faraday.inria.fr:reboot13:ok
faraday.inria.fr:reboot14:ok
faraday.inria.fr:reboot12:ok
faraday.inria.fr:08:46:44 - +000s: fit01 reboot = Sending message 'reset' to CMC reboot01
faraday.inria.fr:08:46:44 - +000s: fit02 reboot = Sending message 'reset' to CMC reboot02
faraday.inria.fr:reboot29:ok
faraday.inria.fr:reboot22:ok
faraday.inria.fr:reboot36:ok
faraday.inria.fr:reboot19:ok
faraday.inria.fr:reboot06:ok
faraday.inria.fr:reboot30:ok
faraday.inria.fr:reboot11:ok
faraday.inria.fr:reboot25:ok
faraday.inria.fr:reboot31:ok
faraday.inria.fr:reboot34:ok
faraday.inria.fr:reboot09:ok
faraday.inria.fr:reboot03:ok
faraday.inria.fr:reboot16:ok
faraday.inria.fr:reboot26:ok
faraday.inria.fr:reboot28:ok
faraday.inria.fr:reboot04:ok
faraday.inria.fr:reboot15:ok
faraday.inria.fr:reboot20:ok
faraday.inria.fr:reboot17:ok
faraday.inria.fr:reboot21:ok
faraday.inria.fr:reboot33:ok
faraday.inria.fr:reboot05:ok
faraday.inria.fr:reboot07:ok
faraday.inria.fr:reboot37:ok
faraday.inria.fr:reboot32:ok
faraday.inria.fr:reboot10:ok
faraday.inria.fr:reboot08:ok
faraday.inria.fr:reboot27:ok
faraday.inria.fr:reboot35:ok
faraday.inria.fr:reboot18:ok
faraday.inria.fr:reboot24:ok
faraday.inria.fr:reboot08:already off
faraday.inria.fr:reboot27:already off
faraday.inria.fr:reboot10:already off
faraday.inria.fr:reboot22:already off
faraday.inria.fr:reboot34:already off
faraday.inria.fr:reboot32:already off
faraday.inria.fr:reboot33:already off
faraday.inria.fr:reboot09:already off
faraday.inria.fr:reboot07:already off
faraday.inria.fr:reboot35:already off
faraday.inria.fr:reboot29:already off
faraday.inria.fr:reboot03:already off
faraday.inria.fr:reboot04:already off
faraday.inria.fr:reboot30:already off
faraday.inria.fr:reboot06:already off
faraday.inria.fr:reboot21:already off
faraday.inria.fr:reboot17:already off
faraday.inria.fr:reboot26:already off
faraday.inria.fr:reboot05:already off
faraday.inria.fr:reboot11:already off
faraday.inria.fr:reboot36:already off
faraday.inria.fr:reboot31:already off
faraday.inria.fr:reboot12:already off
faraday.inria.fr:reboot37:already off
faraday.inria.fr:reboot13:already off
faraday.inria.fr:reboot28:already off
faraday.inria.fr:reboot18:already off
faraday.inria.fr:reboot14:already off
faraday.inria.fr:reboot25:already off
faraday.inria.fr:reboot16:already off
faraday.inria.fr:reboot20:already off
faraday.inria.fr:reboot15:already off
faraday.inria.fr:reboot23:already off
faraday.inria.fr:reboot19:already off
faraday.inria.fr:reboot24:already off
faraday.inria.fr:ssh -i /home/faraday/r2lab-embedded/mac-ssh-keys/macphone tester@macphone1 phone-off
faraday.inria.fr:Turning OFF phone : turning on airplane mode
faraday.inria.fr:08:46:46 - +002s: fit01 reboot = idling for 15s
faraday.inria.fr:08:46:46 - +002s: fit02 reboot = idling for 15s
faraday.inria.fr:Broadcasting: Intent { act=android.intent.action.AIRPLANE_MODE (has extras) }
faraday.inria.fr:Broadcast completed: result=0
faraday.inria.fr:ssh -i /home/faraday/r2lab-embedded/mac-ssh-keys/macphone tester@macphone2 phone-off
faraday.inria.fr:Turning OFF phone : turning on airplane mode
faraday.inria.fr:Broadcasting: Intent { act=android.intent.action.AIRPLANE_MODE (has extras) }
faraday.inria.fr:Broadcast completed: result=0
faraday.inria.fr:08:47:02 - +018s: started 
faraday.inria.fr:08:47:02 - +018s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:02 - +018s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:04 - +020s: fit02 frisbee_status = timed out..
faraday.inria.fr:08:47:04 - +020s: fit02 frisbee_status = backing off for 4.33s
faraday.inria.fr:08:47:04 - +020s: fit01 frisbee_status = timed out..
faraday.inria.fr:08:47:04 - +020s: fit01 frisbee_status = backing off for 4.27s
faraday.inria.fr:08:47:08 - +024s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:08 - +024s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:10 - +026s: fit01 frisbee_status = timed out..
faraday.inria.fr:08:47:10 - +026s: fit01 frisbee_status = backing off for 2.94s
faraday.inria.fr:08:47:10 - +026s: fit02 frisbee_status = timed out..
faraday.inria.fr:08:47:10 - +026s: fit02 frisbee_status = backing off for 1.76s
faraday.inria.fr:08:47:12 - +028s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:13 - +029s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:14 - +030s: fit02 frisbee_status = timed out..
faraday.inria.fr:08:47:14 - +030s: fit02 frisbee_status = backing off for 3.21s
faraday.inria.fr:08:47:15 - +031s: fit01 frisbee_status = timed out..
faraday.inria.fr:08:47:15 - +031s: fit01 frisbee_status = backing off for 2.72s
faraday.inria.fr:08:47:17 - +033s: fit02 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:18 - +034s: fit01 frisbee_status = trying to telnet..
faraday.inria.fr:08:47:18 - +034s: fit01 frisbee_status = starting frisbee client
faraday.inria.fr:08:47:18 - +034s: fit02 frisbee_status = starting frisbee client
faraday.inria.fr:08:47:42 - +058s: fit01 Uploading successful
faraday.inria.fr:08:47:42 - +058s: fit01 reboot = Sending message 'reset' to CMC reboot01
faraday.inria.fr:|                                                   |  0% |0.00s|ETA:  --:--:--
|#                                                   |  3% |1.13s|ETA:  0:00:31
|###                                                 |  6% |1.32s|ETA:  0:00:18
|#####                                               | 10% |2.15s|ETA:  0:00:19
|#######                                             | 14% |2.40s|ETA:  0:00:14
|#########                                           | 17% |3.27s|ETA:  0:00:15
|##########                                          | 21% |3.51s|ETA:  0:00:13
|############                                        | 24% |4.61s|ETA:  0:00:14
|##############                                      | 27% |5.02s|ETA:  0:00:13
|##############                                      | 28% |5.64s|ETA:  0:00:14
|###############                                     | 30% |6.00s|ETA:  0:00:14
|#################                                   | 33% |6.72s|ETA:  0:00:13
|###################                                 | 37% |6.87s|ETA:  0:00:11
|#####################                               | 41% |7.47s|ETA:  0:00:10
|#######################                             | 45% |7.59s|ETA:  0:00:09
|########################                           | 48% |10.62s|ETA:  0:00:11
|##########################                         | 51% |10.74s|ETA:  0:00:10
|###########################                        | 54% |11.23s|ETA:  0:00:09
|#############################                      | 58% |11.37s|ETA:  0:00:08
|###############################                    | 61% |12.10s|ETA:  0:00:07
|################################                   | 64% |12.30s|ETA:  0:00:06
|##################################                 | 67% |13.41s|ETA:  0:00:06
|###################################                | 70% |15.60s|ETA:  0:00:06
|####################################               | 72% |16.11s|ETA:  0:00:06
|######################################             | 75% |16.27s|ETA:  0:00:05
|#######################################            | 78% |16.97s|ETA:  0:00:04
|#########################################          | 81% |17.41s|ETA:  0:00:03
|###########################################        | 84% |18.40s|ETA:  0:00:03
|############################################       | 87% |18.55s|ETA:  0:00:02
|##############################################     | 91% |21.06s|ETA:  0:00:02
|################################################   | 94% |21.23s|ETA:  0:00:01
|#################################################  | 97% |21.81s|ETA:  0:00:00
|################################################## | 99% |22.03s|ETA:  0:00:00
|################################################## | 99% |23.94s|ETA:  0:00:00
|###################################################|100% |24.42s|Time: 0:00:24
faraday.inria.fr:08:47:42 - +058s: fit02 Uploading successful
faraday.inria.fr:08:47:42 - +058s: fit02 reboot = Sending message 'reset' to CMC reboot02
faraday.inria.fr:08:47:44 - +060s: stopped 
faraday.inria.fr::ssh OK
faraday.inria.fr::ssh OK
fit02:turn-on-data: data network on interface data
fit01:turn-on-data: data network on interface data
fit01:data
fit02:data
fit01:PING data02 (192.168.2.2) from 192.168.2.1 data: 56(84) bytes of data.
fit01:64 bytes from data02 (192.168.2.2): icmp_seq=1 ttl=64 time=0.473 ms
fit01:
fit01:--- data02 ping statistics ---
fit01:1 packets transmitted, 1 received, 0% packet loss, time 0ms
fit01:rtt min/avg/max/mdev = 0.473/0.473/0.473/0.000 ms

We will now see in tab D3 how to pass target nodes on the command-line.

Objectives

In this section, we will change the D2 script so that the target nodes can be passed on the command line, instead of using the hardwired nodes 1 and 2.

The `r2lab` python library

We take this opportunity to introduce the r2lab python library, that you need of course to install separately:

pip3 install r2lab

This library is designed to be, and hopefully to stay, very small; in this example we just use convenience functions like r2lab_hostname that lets us compute the hostname for one of the nodes from a variety of inputs, that can be either int or bytes or str.

This mostly is convenient to avoid ending up with hostname like fit1 with a missing 0, so as you can see it is very basic.

A complete reference can be found on readthedocs.io.

The code

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

from r2lab import r2lab_hostname, r2lab_data

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-a", "--node-a", default="fit01",
                    help="packets are sent from this node")
parser.add_argument("-b", "--node-b", default="fit02",
                    help="packets are sent to this node")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

# r2lab_hostname will accept inputs like '1', '01', or 'fit01'
hostnamea = r2lab_hostname(args.node_a)
hostnameb = r2lab_hostname(args.node_b)

nodea = SshNode(gateway = faraday, hostname = hostnamea, username = "root",
                verbose = verbose_ssh)
nodeb = SshNode(gateway = faraday, hostname = hostnameb, username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu', hostnamea, hostnameb),
                Run('rhubarbe wait', hostnamea, hostnameb),
                ],
            required = check_lease,
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a', '~'+hostnamea, '~'+hostnameb)
            ],
            required = check_lease,
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_a = SshJob(
    node = nodea,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_b = SshJob(
    node = nodeb,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the name of receiver nodeb, on the data network, is e.g. data02
data_b = r2lab_data(hostnameb)

ping = SshJob(
    node = nodea,
    required = (init_node_a, init_node_b),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', data_b),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D3.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

node1 = SshNode(gateway = faraday, hostname = "fit01", username = "root",
                verbose = verbose_ssh)
node2 = SshNode(gateway = faraday, hostname = "fit02", username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu 1 2'),
                Run('rhubarbe wait 1 2'),
                ],
            required = check_lease,
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a ~1 ~2')
            ],
            required = check_lease,
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_01 = SshJob(
    node = node1,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_02 = SshJob(
    node = node2,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the command we want to run in node1 is as simple as it gets
ping = SshJob(
    node = node1,
    required = (init_node_01, init_node_02),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', 'data02'),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D2.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

from r2lab import r2lab_hostname, r2lab_data

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-a", "--node-a", default="fit01",
                    help="packets are sent from this node")
parser.add_argument("-b", "--node-b", default="fit02",
                    help="packets are sent to this node")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

# r2lab_hostname will accept inputs like '1', '01', or 'fit01'
hostnamea = r2lab_hostname(args.node_a)
hostnameb = r2lab_hostname(args.node_b)

nodea = SshNode(gateway = faraday, hostname = hostnamea, username = "root",
                verbose = verbose_ssh)
nodeb = SshNode(gateway = faraday, hostname = hostnameb, username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu', hostnamea, hostnameb),
                Run('rhubarbe wait', hostnamea, hostnameb),
                ],
            required = check_lease,
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a', '~'+hostnamea, '~'+hostnameb)
            ],
            required = check_lease,
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_a = SshJob(
    node = nodea,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_b = SshJob(
    node = nodeb,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the name of receiver nodeb, on the data network, is e.g. data02
data_b = r2lab_data(hostnameb)

ping = SshJob(
    node = nodea,
    required = (init_node_a, init_node_b),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', data_b),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D3.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output

It is easy to select other nodes with the -a and -b option.

`$ python3 D3-prep.py --load --node-a 10 --node-b 20`


faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
faraday.inria.fr:Found binary frisbeed as /usr/sbin/frisbeed
faraday.inria.fr:Found binary nc as /bin/nc
faraday.inria.fr:08:54:39 - +000s: Selection: fit10 fit20
faraday.inria.fr:08:54:39 - +000s: Loading image /var/lib/rhubarbe-images/ubuntu.ndz
faraday.inria.fr:08:54:39 - +000s: AUTH: checking for a valid lease
faraday.inria.fr:08:54:39 - +000s: AUTH: access granted
faraday.inria.fr:08:54:39 - +000s: fit10 reboot = Sending message 'reset' to CMC reboot10
faraday.inria.fr:reboot11:ok
faraday.inria.fr:08:54:39 - +000s: fit20 reboot = Sending message 'reset' to CMC reboot20
faraday.inria.fr:reboot12:ok
faraday.inria.fr:reboot23:ok
faraday.inria.fr:reboot24:ok
faraday.inria.fr:reboot13:ok
faraday.inria.fr:reboot15:ok
faraday.inria.fr:reboot27:ok
faraday.inria.fr:reboot18:ok
faraday.inria.fr:reboot14:ok
faraday.inria.fr:reboot34:ok
faraday.inria.fr:reboot17:ok
faraday.inria.fr:reboot25:ok
faraday.inria.fr:reboot19:ok
faraday.inria.fr:reboot31:ok
faraday.inria.fr:reboot21:ok
faraday.inria.fr:reboot22:ok
faraday.inria.fr:reboot16:ok
faraday.inria.fr:reboot36:ok
faraday.inria.fr:reboot01:ok
faraday.inria.fr:reboot30:ok
faraday.inria.fr:reboot37:ok
faraday.inria.fr:reboot29:ok
faraday.inria.fr:reboot28:ok
faraday.inria.fr:reboot33:ok
faraday.inria.fr:reboot26:ok
faraday.inria.fr:reboot32:ok
faraday.inria.fr:reboot35:ok
faraday.inria.fr:reboot05:ok
faraday.inria.fr:reboot04:ok
faraday.inria.fr:reboot03:ok
faraday.inria.fr:reboot08:ok
faraday.inria.fr:reboot06:ok
faraday.inria.fr:reboot09:ok
faraday.inria.fr:reboot07:ok
faraday.inria.fr:reboot02:ok
faraday.inria.fr:reboot09:already off
faraday.inria.fr:reboot15:already off
faraday.inria.fr:reboot37:already off
faraday.inria.fr:reboot35:already off
faraday.inria.fr:reboot21:already off
faraday.inria.fr:reboot05:already off
faraday.inria.fr:reboot16:already off
faraday.inria.fr:reboot07:already off
faraday.inria.fr:reboot29:already off
faraday.inria.fr:reboot12:already off
faraday.inria.fr:reboot03:already off
faraday.inria.fr:reboot30:already off
faraday.inria.fr:reboot11:already off
faraday.inria.fr:reboot28:already off
faraday.inria.fr:reboot33:already off
faraday.inria.fr:reboot36:already off
faraday.inria.fr:reboot32:already off
faraday.inria.fr:reboot14:already off
faraday.inria.fr:reboot17:already off
faraday.inria.fr:reboot24:already off
faraday.inria.fr:reboot02:ok
faraday.inria.fr:reboot01:ok
faraday.inria.fr:reboot19:already off
faraday.inria.fr:reboot06:already off
faraday.inria.fr:reboot27:already off
faraday.inria.fr:reboot13:already off
faraday.inria.fr:reboot25:already off
faraday.inria.fr:reboot26:already off
faraday.inria.fr:reboot31:already off
faraday.inria.fr:reboot04:already off
faraday.inria.fr:reboot08:already off
faraday.inria.fr:reboot34:already off
faraday.inria.fr:reboot22:already off
faraday.inria.fr:reboot18:already off
faraday.inria.fr:reboot23:already off
faraday.inria.fr:ssh -i /home/faraday/r2lab-embedded/mac-ssh-keys/macphone tester@macphone1 phone-off
faraday.inria.fr:Turning OFF phone : turning on airplane mode
faraday.inria.fr:08:54:41 - +002s: fit10 reboot = idling for 15s
faraday.inria.fr:08:54:41 - +002s: fit20 reboot = idling for 15s
faraday.inria.fr:Broadcasting: Intent { act=android.intent.action.AIRPLANE_MODE (has extras) }
faraday.inria.fr:Broadcast completed: result=0
faraday.inria.fr:ssh -i /home/faraday/r2lab-embedded/mac-ssh-keys/macphone tester@macphone2 phone-off
faraday.inria.fr:Turning OFF phone : turning on airplane mode
faraday.inria.fr:Broadcasting: Intent { act=android.intent.action.AIRPLANE_MODE (has extras) }
faraday.inria.fr:Broadcast completed: result=0
faraday.inria.fr:08:54:57 - +018s: started 
faraday.inria.fr:08:54:57 - +018s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:08:54:57 - +018s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:08:54:59 - +020s: fit20 frisbee_status = timed out..
faraday.inria.fr:08:54:59 - +020s: fit20 frisbee_status = backing off for 3.13s
faraday.inria.fr:08:54:59 - +020s: fit10 frisbee_status = timed out..
faraday.inria.fr:08:54:59 - +020s: fit10 frisbee_status = backing off for 4.39s
faraday.inria.fr:08:55:02 - +023s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:08:55:04 - +024s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:08:55:04 - +025s: fit20 frisbee_status = timed out..
faraday.inria.fr:08:55:04 - +025s: fit20 frisbee_status = backing off for 1.84s
faraday.inria.fr:08:55:06 - +026s: fit10 frisbee_status = timed out..
faraday.inria.fr:08:55:06 - +026s: fit10 frisbee_status = backing off for 2.82s
faraday.inria.fr:08:55:06 - +027s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:08:55:08 - +029s: fit20 frisbee_status = timed out..
faraday.inria.fr:08:55:08 - +029s: fit20 frisbee_status = backing off for 1.84s
faraday.inria.fr:08:55:08 - +029s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:08:55:10 - +030s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:08:55:10 - +031s: fit10 frisbee_status = timed out..
faraday.inria.fr:08:55:10 - +031s: fit10 frisbee_status = backing off for 2.96s
faraday.inria.fr:08:55:11 - +031s: fit20 frisbee_status = starting frisbee client
faraday.inria.fr:08:55:13 - +034s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:08:55:14 - +035s: fit10 frisbee_status = starting frisbee client
faraday.inria.fr:08:55:36 - +057s: fit20 Uploading successful
faraday.inria.fr:08:55:36 - +057s: fit20 reboot = Sending message 'reset' to CMC reboot20
faraday.inria.fr:|                                                   |  0% |0.00s|ETA:  --:--:--
|#                                                   |  3% |0.98s|ETA:  0:00:31
|###                                                 |  6% |1.94s|ETA:  0:00:30
|####                                                |  9% |3.31s|ETA:  0:00:31
|######                                              | 12% |4.33s|ETA:  0:00:31
|#######                                             | 14% |5.22s|ETA:  0:00:32
|########                                            | 16% |5.26s|ETA:  0:00:26
|##########                                          | 19% |6.60s|ETA:  0:00:27
|###########                                         | 21% |7.33s|ETA:  0:00:26
|############                                        | 24% |7.99s|ETA:  0:00:24
|##############                                      | 27% |9.25s|ETA:  0:00:24
|################                                    | 31% |9.93s|ETA:  0:00:22
|#################                                  | 34% |10.10s|ETA:  0:00:19
|##################                                 | 37% |10.56s|ETA:  0:00:17
|####################                               | 41% |11.23s|ETA:  0:00:16
|######################                             | 44% |12.39s|ETA:  0:00:15
|########################                           | 48% |12.85s|ETA:  0:00:13
|##########################                         | 51% |13.50s|ETA:  0:00:12
|############################                       | 55% |14.44s|ETA:  0:00:11
|#############################                      | 57% |15.19s|ETA:  0:00:11
|###############################                    | 61% |15.32s|ETA:  0:00:09
|################################                   | 64% |16.10s|ETA:  0:00:08
|##################################                 | 68% |17.01s|ETA:  0:00:07
|####################################               | 72% |17.95s|ETA:  0:00:06
|######################################             | 76% |18.25s|ETA:  0:00:05
|########################################           | 79% |18.58s|ETA:  0:00:04
|#########################################          | 82% |19.87s|ETA:  0:00:04
|##########################################         | 84% |21.20s|ETA:  0:00:04
|############################################       | 87% |21.73s|ETA:  0:00:03
|#############################################      | 90% |22.69s|ETA:  0:00:02
|###############################################    | 93% |23.45s|ETA:  0:00:01
|#################################################  | 96% |24.09s|ETA:  0:00:00
|################################################## | 99% |25.21s|ETA:  0:00:00
|###################################################|100% |27.60s|ETA:  0:00:00
|###################################################|100% |27.60s|Time: 0:00:27
faraday.inria.fr:08:55:39 - +059s: fit10 Uploading successful
faraday.inria.fr:08:55:39 - +059s: fit10 reboot = Sending message 'reset' to CMC reboot10
faraday.inria.fr:08:55:41 - +061s: stopped 
faraday.inria.fr::ssh OK
faraday.inria.fr::ssh OK
fit20:turn-on-data: data network on interface data
fit10:turn-on-data: data network on interface data
fit20:data
fit10:data
fit10:PING data20 (192.168.2.20) from 192.168.2.10 data: 56(84) bytes of data.
fit10:64 bytes from data20 (192.168.2.20): icmp_seq=1 ttl=64 time=0.460 ms
fit10:
fit10:--- data20 ping statistics ---
fit10:1 packets transmitted, 1 received, 0% packet loss, time 0ms
fit10:rtt min/avg/max/mdev = 0.460/0.460/0.460/0.000 ms

In the next example, we will see how to rewrite this example to use nested schedulers, for better modularity.

Objectives

In this section, we will change the D3 script and illustrate a very convenient feature, that allows to create so-called nested schedulers; the idea is to be able to insert a scheduler as a job in a higher-level scheduler. This feature aims in particular at allowing a modular design of schedulers.

In a nutshell, all you need to know is that a Scheduler object can be inserted inside another scheduler, like e.g.:

s_high = Scheduler()
s_high.append(
    Sequence(
        SshJob(...),
        Scheduler(...),  # low-level scheduler
        SshJob()))

Of course in practice, the low-level scheduler is often built separately; for example it could be built by a regular python function.

The other way around, it is also possible to implement a python function that accepts a scheduler in argument, an returns a decorated scheduler that would, for example, check for the lease before running the incoming scheduler.

A more detailed introduction to nested schedulers can be found in the asynciojobs documentation.

So in this example we create a script that behaves exactly like in D3, but using a nested scheduler to provide for more modularity.

The code

Notice:

how nested schedulers are rendered graphically, and
how the reusable check_lease function decorates experiment_scheduler so as to first check for the lease.

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler, Sequence

from apssh import SshNode, SshJob
from apssh import Run, RunString

from r2lab import r2lab_hostname, r2lab_data

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-a", "--node-a", default="fit01",
                    help="packets are sent from this node")
parser.add_argument("-b", "--node-b", default="fit02",
                    help="packets are sent to this node")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

# r2lab_hostname will accept inputs like '1', '01', or 'fit01'
hostnamea = r2lab_hostname(args.node_a)
hostnameb = r2lab_hostname(args.node_b)

nodea = SshNode(gateway = faraday, hostname = hostnamea, username = "root",
                verbose = verbose_ssh)
nodeb = SshNode(gateway = faraday, hostname = hostnameb, username = "root",
                verbose = verbose_ssh)

def check_lease(experiment_scheduler, sshnode):

    """
    re-usable function that acts a bit like a python
    decorator on schedulers.

    Given an experiment described as a scheduler, this function
    returns a higher-level scheduler that first checks for the
    lease, and then proceeds with the experiment.
    """

    check_lease_job = SshJob(
        # checking the lease is done on the gateway
        node = faraday,
        # this means that a failure in any of the commands
        # will cause the scheduler to bail out immediately
        critical = True,
        command = Run("rhubarbe leases --check"),
    )

    return Scheduler(
        Sequence(
            check_lease_job,
            # here we create a nested scheduler
            # by inserting the experiment_scheduler
            # as a regular job in the main scheduler
            experiment_scheduler,
        )
    )


##########
# create an orchestration scheduler
scheduler = Scheduler()


# the job to wait before proceeding
ready_requirement = None
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu', hostnamea, hostnameb),
                Run('rhubarbe wait', hostnamea, hostnameb),
                ],
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a', '~'+hostnamea, '~'+hostnameb)
            ],
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_a = SshJob(
    node = nodea,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_b = SshJob(
    node = nodeb,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the name of receiver nodeb, on the data network, is e.g. data02
data_b = r2lab_data(hostnameb)

ping = SshJob(
    node = nodea,
    required = (init_node_a, init_node_b),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', data_b),
    scheduler = scheduler,
)

scheduler = check_lease(scheduler, faraday)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D4.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler

from apssh import SshNode, SshJob
from apssh import Run, RunString

from r2lab import r2lab_hostname, r2lab_data

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-a", "--node-a", default="fit01",
                    help="packets are sent from this node")
parser.add_argument("-b", "--node-b", default="fit02",
                    help="packets are sent to this node")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

# r2lab_hostname will accept inputs like '1', '01', or 'fit01'
hostnamea = r2lab_hostname(args.node_a)
hostnameb = r2lab_hostname(args.node_b)

nodea = SshNode(gateway = faraday, hostname = hostnamea, username = "root",
                verbose = verbose_ssh)
nodeb = SshNode(gateway = faraday, hostname = hostnameb, username = "root",
                verbose = verbose_ssh)

##########
# create an orchestration scheduler
scheduler = Scheduler()

##########
check_lease = SshJob(
    # checking the lease is done on the gateway
    node = faraday,
    # this means that a failure in any of the commands
    # will cause the scheduler to bail out immediately
    critical = True,
    command = Run("rhubarbe leases --check"),
    scheduler = scheduler,
)

# the job to wait before proceeding
ready_requirement = check_lease
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu', hostnamea, hostnameb),
                Run('rhubarbe wait', hostnamea, hostnameb),
                ],
            required = check_lease,
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a', '~'+hostnamea, '~'+hostnameb)
            ],
            required = check_lease,
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_a = SshJob(
    node = nodea,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_b = SshJob(
    node = nodeb,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the name of receiver nodeb, on the data network, is e.g. data02
data_b = r2lab_data(hostnameb)

ping = SshJob(
    node = nodea,
    required = (init_node_a, init_node_b),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', data_b),
    scheduler = scheduler,
)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D3.dot")

# return something useful to your OS
exit(0 if success else 1)

#!/usr/bin/env python3

from argparse import ArgumentParser

from asynciojobs import Scheduler, Sequence

from apssh import SshNode, SshJob
from apssh import Run, RunString

from r2lab import r2lab_hostname, r2lab_data

##########
gateway_hostname  = 'faraday.inria.fr'
gateway_username  = 'inria_r2lab.tutorial'
verbose_ssh = False

parser = ArgumentParser()
parser.add_argument("-s", "--slice", default=gateway_username,
                    help="specify an alternate slicename, default={}"
                         .format(gateway_username))
parser.add_argument("-l", "--load", default=False, action='store_true',
                    help="load the ubuntu image on nodes before starting")
parser.add_argument("-v", "--verbose-ssh", default=False, action='store_true',
                    help="run ssh in verbose mode")
parser.add_argument("-a", "--node-a", default="fit01",
                    help="packets are sent from this node")
parser.add_argument("-b", "--node-b", default="fit02",
                    help="packets are sent to this node")
args = parser.parse_args()

gateway_username = args.slice
verbose_ssh = args.verbose_ssh

##########
faraday = SshNode(hostname = gateway_hostname, username = gateway_username,
                  verbose = verbose_ssh)

# r2lab_hostname will accept inputs like '1', '01', or 'fit01'
hostnamea = r2lab_hostname(args.node_a)
hostnameb = r2lab_hostname(args.node_b)

nodea = SshNode(gateway = faraday, hostname = hostnamea, username = "root",
                verbose = verbose_ssh)
nodeb = SshNode(gateway = faraday, hostname = hostnameb, username = "root",
                verbose = verbose_ssh)

def check_lease(experiment_scheduler, sshnode):

    """
    re-usable function that acts a bit like a python
    decorator on schedulers.

    Given an experiment described as a scheduler, this function
    returns a higher-level scheduler that first checks for the
    lease, and then proceeds with the experiment.
    """

    check_lease_job = SshJob(
        # checking the lease is done on the gateway
        node = faraday,
        # this means that a failure in any of the commands
        # will cause the scheduler to bail out immediately
        critical = True,
        command = Run("rhubarbe leases --check"),
    )

    return Scheduler(
        Sequence(
            check_lease_job,
            # here we create a nested scheduler
            # by inserting the experiment_scheduler
            # as a regular job in the main scheduler
            experiment_scheduler,
        )
    )


##########
# create an orchestration scheduler
scheduler = Scheduler()


# the job to wait before proceeding
ready_requirement = None
# has the user requested to load images ?
# if so, we just need to wait for 2 jobs to complete instead of 1
if args.load:
    ready_requirement = [
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe load -i ubuntu', hostnamea, hostnameb),
                Run('rhubarbe wait', hostnamea, hostnameb),
                ],
            scheduler = scheduler,
        ),
        SshJob(
            node = faraday,
            commands = [
                Run('rhubarbe bye -a', '~'+hostnamea, '~'+hostnameb)
            ],
            scheduler = scheduler,
        )
    ]

##########
# setting up the data interface on both fit01 and fit02
init_node_a = SshJob(
    node = nodea,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)
init_node_b = SshJob(
    node = nodeb,
    command = Run("turn-on-data"),
    required = ready_requirement,
    scheduler = scheduler,
)

# the name of receiver nodeb, on the data network, is e.g. data02
data_b = r2lab_data(hostnameb)

ping = SshJob(
    node = nodea,
    required = (init_node_a, init_node_b),
    # let's be more specific about what to run
    # we will soon see other things we can do on an ssh connection
    command = Run('ping', '-c1', '-I', 'data', data_b),
    scheduler = scheduler,
)

scheduler = check_lease(scheduler, faraday)

##########
# run the scheduler
ok = scheduler.orchestrate()
# give details if it failed
ok or scheduler.debrief()

# we say this is a success if the ping command succeeded
# the result() of the SshJob is the value that the command
# returns to the OS
# so it's a success if this value is 0
success = ok and ping.result() == 0

# producing a dot file for illustration
scheduler.export_as_dotfile("D4.dot")

# return something useful to your OS
exit(0 if success else 1)

Sample output

`$ python3 D4-prep.py --load --node-a 10 --node-b 20`


faraday.inria.fr:Checking current reservation for inria_r2lab.tutorial : OK
faraday.inria.fr:reboot24:ok
faraday.inria.fr:reboot23:ok
faraday.inria.fr:reboot13:ok
faraday.inria.fr:reboot12:ok
faraday.inria.fr:reboot26:ok
faraday.inria.fr:reboot21:ok
faraday.inria.fr:reboot25:ok
faraday.inria.fr:reboot29:ok
faraday.inria.fr:reboot19:ok
faraday.inria.fr:reboot22:ok
faraday.inria.fr:reboot14:ok
faraday.inria.fr:reboot11:ok
faraday.inria.fr:reboot33:ok
faraday.inria.fr:reboot04:ok
faraday.inria.fr:reboot15:ok
faraday.inria.fr:reboot07:ok
faraday.inria.fr:reboot35:ok
faraday.inria.fr:reboot27:ok
faraday.inria.fr:reboot28:ok
faraday.inria.fr:reboot32:ok
faraday.inria.fr:reboot01:ok
faraday.inria.fr:reboot30:ok
faraday.inria.fr:reboot06:ok
faraday.inria.fr:reboot02:ok
faraday.inria.fr:reboot05:ok
faraday.inria.fr:reboot18:ok
faraday.inria.fr:reboot36:ok
faraday.inria.fr:reboot17:ok
faraday.inria.fr:reboot34:ok
faraday.inria.fr:reboot31:ok
faraday.inria.fr:Found binary frisbeed as /usr/sbin/frisbeed
faraday.inria.fr:reboot16:ok
faraday.inria.fr:Found binary nc as /bin/nc
faraday.inria.fr:reboot03:ok
faraday.inria.fr:reboot37:ok
faraday.inria.fr:reboot08:ok
faraday.inria.fr:reboot09:ok
faraday.inria.fr:12:22:58 - +000s: Selection: fit10 fit20
faraday.inria.fr:12:22:58 - +000s: Loading image /var/lib/rhubarbe-images/ubuntu.ndz
faraday.inria.fr:12:22:58 - +000s: AUTH: checking for a valid lease
faraday.inria.fr:12:22:58 - +000s: AUTH: access granted
faraday.inria.fr:12:22:58 - +000s: fit10 reboot = Sending message 'on' to CMC reboot10
faraday.inria.fr:12:22:58 - +000s: fit20 reboot = Sending message 'on' to CMC reboot20
faraday.inria.fr:reboot33:ok
faraday.inria.fr:reboot32:already off
faraday.inria.fr:reboot27:already off
faraday.inria.fr:reboot19:already off
faraday.inria.fr:reboot31:already off
faraday.inria.fr:reboot04:already off
faraday.inria.fr:reboot37:already off
faraday.inria.fr:reboot26:already off
faraday.inria.fr:reboot06:already off
faraday.inria.fr:reboot25:already off
faraday.inria.fr:reboot21:already off
faraday.inria.fr:reboot18:already off
faraday.inria.fr:reboot22:already off
faraday.inria.fr:reboot36:already off
faraday.inria.fr:reboot11:already off
faraday.inria.fr:reboot12:already off
faraday.inria.fr:reboot03:already off
faraday.inria.fr:reboot34:already off
faraday.inria.fr:reboot29:already off
faraday.inria.fr:reboot02:already off
faraday.inria.fr:reboot16:already off
faraday.inria.fr:reboot24:already off
faraday.inria.fr:reboot15:already off
faraday.inria.fr:reboot28:already off
faraday.inria.fr:reboot14:already off
faraday.inria.fr:reboot35:already off
faraday.inria.fr:reboot05:already off
faraday.inria.fr:reboot08:already off
faraday.inria.fr:reboot09:already off
faraday.inria.fr:reboot13:already off
faraday.inria.fr:reboot17:already off
faraday.inria.fr:12:22:59 - +001s: fit10 reboot = idling for 15s
faraday.inria.fr:12:22:59 - +001s: fit20 reboot = idling for 15s
faraday.inria.fr:reboot01:already off
faraday.inria.fr:reboot30:already off
faraday.inria.fr:reboot07:already off
faraday.inria.fr:reboot23:already off
faraday.inria.fr:ssh -i /home/faraday/r2lab-embedded/mac-ssh-keys/macphone tester@macphone1 phone-off
faraday.inria.fr:Turning OFF phone : turning on airplane mode
faraday.inria.fr:Broadcasting: Intent { act=android.intent.action.AIRPLANE_MODE (has extras) }
faraday.inria.fr:Broadcast completed: result=0
faraday.inria.fr:ssh -i /home/faraday/r2lab-embedded/mac-ssh-keys/macphone tester@macphone2 phone-off
faraday.inria.fr:Turning OFF phone : turning on airplane mode
faraday.inria.fr:Broadcasting: Intent { act=android.intent.action.AIRPLANE_MODE (has extras) }
faraday.inria.fr:Broadcast completed: result=0
faraday.inria.fr:12:23:15 - +017s: started 
faraday.inria.fr:12:23:15 - +017s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:15 - +017s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:17 - +019s: fit20 frisbee_status = timed out..
faraday.inria.fr:12:23:17 - +019s: fit20 frisbee_status = backing off for 1.7s
faraday.inria.fr:12:23:17 - +019s: fit10 frisbee_status = timed out..
faraday.inria.fr:12:23:17 - +019s: fit10 frisbee_status = backing off for 3.97s
faraday.inria.fr:12:23:19 - +020s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:21 - +022s: fit20 frisbee_status = timed out..
faraday.inria.fr:12:23:21 - +022s: fit20 frisbee_status = backing off for 3.93s
faraday.inria.fr:12:23:21 - +023s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:23 - +025s: fit10 frisbee_status = timed out..
faraday.inria.fr:12:23:23 - +025s: fit10 frisbee_status = backing off for 3.38s
faraday.inria.fr:12:23:24 - +026s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:26 - +028s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:26 - +028s: fit20 frisbee_status = timed out..
faraday.inria.fr:12:23:26 - +028s: fit20 frisbee_status = backing off for 1.61s
faraday.inria.fr:12:23:28 - +030s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:28 - +030s: fit10 frisbee_status = timed out..
faraday.inria.fr:12:23:28 - +030s: fit10 frisbee_status = backing off for 1.73s
faraday.inria.fr:12:23:30 - +032s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:30 - +032s: fit20 frisbee_status = timed out..
faraday.inria.fr:12:23:30 - +032s: fit20 frisbee_status = backing off for 3.07s
faraday.inria.fr:12:23:32 - +034s: fit10 frisbee_status = timed out..
faraday.inria.fr:12:23:32 - +034s: fit10 frisbee_status = backing off for 3.29s
faraday.inria.fr:12:23:33 - +035s: fit20 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:33 - +035s: fit20 frisbee_status = starting frisbee client
faraday.inria.fr:12:23:35 - +037s: fit10 frisbee_status = trying to telnet..
faraday.inria.fr:12:23:35 - +037s: fit10 frisbee_status = starting frisbee client
faraday.inria.fr:12:23:57 - +059s: fit20 Uploading successful
faraday.inria.fr:12:23:57 - +059s: fit20 reboot = Sending message 'reset' to CMC reboot20
faraday.inria.fr:|                                                   |  0% |0.00s|ETA:  --:--:--
|                                                    |  1% |0.59s|ETA:  0:00:38
|##                                                  |  5% |1.65s|ETA:  0:00:28
|####                                                |  8% |2.68s|ETA:  0:00:28
|#####                                               | 11% |2.96s|ETA:  0:00:23
|#######                                             | 14% |3.90s|ETA:  0:00:22
|#########                                           | 17% |4.08s|ETA:  0:00:19
|###########                                         | 21% |4.94s|ETA:  0:00:18
|#############                                       | 25% |5.05s|ETA:  0:00:14
|###############                                     | 29% |6.03s|ETA:  0:00:14
|################                                    | 32% |6.32s|ETA:  0:00:13
|#################                                   | 34% |8.02s|ETA:  0:00:15
|###################                                 | 37% |8.20s|ETA:  0:00:13
|#####################                               | 41% |8.92s|ETA:  0:00:12
|######################                             | 44% |10.65s|ETA:  0:00:13
|########################                           | 48% |11.27s|ETA:  0:00:12
|##########################                         | 51% |11.48s|ETA:  0:00:10
|############################                       | 55% |12.19s|ETA:  0:00:09
|#############################                      | 57% |12.35s|ETA:  0:00:09
|##############################                     | 60% |12.98s|ETA:  0:00:08
|################################                   | 63% |14.32s|ETA:  0:00:08
|#################################                  | 66% |15.41s|ETA:  0:00:07
|###################################                | 69% |16.07s|ETA:  0:00:07
|####################################               | 72% |16.85s|ETA:  0:00:06
|######################################             | 75% |17.23s|ETA:  0:00:05
|#######################################            | 78% |17.44s|ETA:  0:00:04
|#########################################          | 81% |18.30s|ETA:  0:00:04
|###########################################        | 85% |19.05s|ETA:  0:00:03
|############################################       | 87% |21.10s|ETA:  0:00:03
|##############################################     | 91% |21.54s|ETA:  0:00:02
|################################################   | 95% |21.67s|ETA:  0:00:01
|#################################################  | 97% |22.32s|ETA:  0:00:00
|################################################## | 99% |23.10s|ETA:  0:00:00
|###################################################|100% |25.80s|ETA:  0:00:00
|###################################################|100% |25.80s|Time: 0:00:25
faraday.inria.fr:12:23:59 - +061s: fit10 Uploading successful
faraday.inria.fr:12:23:59 - +061s: fit10 reboot = Sending message 'reset' to CMC reboot10
faraday.inria.fr:12:24:01 - +063s: stopped 
faraday.inria.fr::ssh OK
faraday.inria.fr::ssh OK
fit10:turn-on-data: data network on interface data
fit20:turn-on-data: data network on interface data
fit10:data
fit20:data
fit10:PING data20 (192.168.2.20) from 192.168.2.10 data: 56(84) bytes of data.
fit10:64 bytes from data20 (192.168.2.20): icmp_seq=1 ttl=64 time=0.446 ms
fit10:
fit10:--- data20 ping statistics ---
fit10:1 packets transmitted, 1 received, 0% packet loss, time 0ms
fit10:rtt min/avg/max/mdev = 0.446/0.446/0.446/0.000 ms

We can now wrap up this D-series.

Summary

You now have a complete toolset at your disposal, and should be able to create nepi-ng scripts for writing a real scale experiment, including features like:

setting up nodes from scratch,
accounting for any node that the previous experimenter may have forgotten to switch off,
passing node numbers on the command-line,
and, last but not least, using nested schedulers to write more modular experiments.

Let us move to the next tutorial for some guidelines and troubleshooting tips.