Setup and basic objects#

Get started with EnOSlib on Grid’5000.

Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html
Instant chat: https://framateam.org/enoslib
Source code: https://gitlab.inria.fr/discovery/enoslib

This is the first notebook in a serie that will let you discover the main features of EnOSlib on Grid’5000.

If you want to actually execute them you’ll need to setup your environment properly. We sum up here the different steps to achieve this process.

Get a Grid’5000 account
- Register using this page. Pay attention to the fact that uploading a SSH key (public part) is mandatory to perform any EnOSlib action from your local machine.
- Make sure the SSH connection is ready. You can follow this tutorial.
Make sure EnOSlib is available in your notebook environment
- Follow the steps here. Using a virtualenv is the way to go, make sure to use one. Also adding the optional jupyter will improve your experience. (pip install enoslib[jupyter])

Testing the import#

[ ]:

import enoslib as en

Resources abstractions#

In this notebook, we won’t execute anything remotely. Instead we’ll just cover some basic abstractions provided by the library. We start with the abstractions of resources (machines and networks that are usually given by an infrastructure)

Host#

An host is anything we can connect to and act on. Most of the time it corresponds to a machine reachable through SSH. The datastructure reflects this.

Usually you don’t instantiate hosts manually, instead they are brought to you by EnOSlib (because most likely they depend on a scheduler decision like OAR on Grid’5000).

[ ]:

bare_host = en.Host("192.168.0.1")
host_with_alias = en.Host("192.168.0.2", alias="one_alias")
host_with_alias_and_username = en.Host("192.168.0.3", alias="one_alias", user="foo")

[ ]:

bare_host

[ ]:

host_with_alias

[ ]:

host_with_alias_and_username

The local machine can be represented by an instance of the LocalHost object. This is a specialization of an Host, the connection to this host will be made using sub-processes (instead of SSH). We can see it in the extra attribute of the LocalHost object. This extra attribute is actually interpreted when a “remote” action is triggered on our hosts.

[ ]:

localhost = en.LocalHost()
localhost

Other types of Hosts are possible. The library has a DockerHost which represents a docker container we want to reach using the docker TCP protocol. One needs to specify where this container is running by passing an host instance.

[ ]:

docker_host = en.DockerHost("alias", "container_name", host_with_alias_and_username)
docker_host

The above extra field suggest that the connection to this docker container will be made through an ssh jump to the remote host hosting the container. This will be done transparently by the library anyway.

Roles#

A common pratice when experimenting, especially with distributed applications, is to form logical group of machines. Indeed, during an experiment your hosts will serve different purposes: some will host the system you are studying while other will install third party tools to inject some load, observe …

A natural way of configuring differently several sets of hosts is to tag them and group them according to their tags.

The Roles datastructure serves this purpose: it lets you group your hosts based on tags. It follow a dict-like interface.

[ ]:

h1 = en.Host("10.0.0.1")
h2 = en.Host("10.0.0.2")
h3 = en.Host("10.0.0.3")
roles = en.Roles()
roles["tag1"] = [h1, h2]
roles["tag2"] = [h3]
roles["tag3"] = [h2, h3]

roles

Network and Networks#

Network and Networks are the same as Host and Roles but for networks:

Network represent a single Network
Networks represent a “Roles” of Network: networks indexed by their tags .

Networks are usually given by an infrastructure and thus you won’t really instantiate Network nor Networks by yourself. More precisely there exists a specific subclass of Network per infrastructure which will be returned automatically by EnOSlib when needed.

Moreover Network datastructure isn’t exposed in EnOSlib at the top level, let’s see however how a DefaultNetwork can look like. A DefaultNetwork is a very common abstraction of a network that allows to represent a basic network with optionnally a pool of free ips/macs address. For instance a subnet or a vlan on Grid5000 are represented by a specific DefaultNetwork.

[ ]:

from enoslib.objects import DefaultNetwork

[ ]:

one_network = DefaultNetwork("192.168.1.0/24")
one_network_with_a_pool_of_ips = DefaultNetwork("192.168.1.0/24", ip_start="192.168.1.10", ip_end="192.168.1.100")

[ ]:

one_network

[ ]:

one_network_with_a_pool_of_ips

[ ]:

# get one free ip
ip_gen = one_network_with_a_pool_of_ips.free_ips
next(ip_gen)

[ ]:

# get another one
next(ip_gen)

Providers (and their configurations)#

EnOSlib uses Providers to … provide resources. Providers let the user get ownership of some resources (for the time of the experiment) in good shape (e.g access granted, network configured …). They transform an abstract Configuration to Roles, Networks :

\(Configuration \xrightarrow{provider} Roles, Networks\)

There are different providers in EnOSlib:

Vbox/KVM to work with locally hosted virtual machines
Openstack/Chameleon to work with bare-metal resources hosted in the Chameleon platform
FiT/IOT lab to work with sensors or low profile machines
Grid’5000 to get bare-metal resources from G5k. There are also some composite providers that sit on top of the Grid’5000 provider
- VmonG5k to work with virtual machines on Grid’5000**
- Distem to work with lxc containers on Grid’5000**

Configurations#

A Provider must be fed with a Configuration. Configuration objects are specific to each provider.

You can build them from a dictionnary (e.g from a yaml/json file) or programmatically. For instance the schema for Grid’5000 is here.

In this section, we’ll only build some configurations (No resource will be reserved on Grid’5000)

[ ]:

import enoslib as en

# An empty configuration isn't really useful but let you see
# some of the default parameters
# Note that by default (empty job_type) the nodes are provisioned
# with the standard Grid'5000 software environment.
conf = en.G5kConf()
conf

[ ]:

# changing the top level options is done by calling the classmethod `from_settings`
en.G5kConf.from_settings(walltime="10:00:00", job_name="my awesome job")

[ ]:

# the canonical way of getting some machines

[ ]:

prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod")
conf = (
    en.G5kConf()
        .add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network)
        .add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network)
        .add_network_conf(prod_network)
        # optional, but do some sanity checks on the configuration
        .finalize()
    )
conf

[ ]:

# Changing to a deploy job.
# The operating system to deploy needs to be selected (debian, ubuntu, centos...)
# See https://www.grid5000.fr/w/Advanced_Kadeploy#Search_an_environment
prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod")
conf = (
    en.G5kConf.from_settings(job_type=["deploy"], env_name="ubuntu2204-min")
        .add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network)
        .add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network)
        .add_network_conf(prod_network)
        # optional, but do some sanity checks on the configuration
        .finalize()
    )
conf

[ ]:

# Using a secondary networks
prod_network = en.G5kNetworkConf(roles=["mynetwork"], site="rennes", type="prod")
kavlan_network = en.G5kNetworkConf(roles=["myprivate"], site="rennes", type="kavlan")
conf = (
    en.G5kConf(job_type=["deploy"], env_name="debian11-nfs")
        .add_machine(cluster="paravance", nodes=3, roles=["role1", "role2"], primary_network=prod_network, secondary_networks=[kavlan_network])
        .add_machine(cluster="parasilo", nodes=3, roles=["role2", "role3"], primary_network=prod_network, secondary_networks=[kavlan_network])
        .add_network_conf(prod_network)
        .add_network_conf(kavlan_network)
        # optional, but do some sanity checks on the configuration
        .finalize()
    )
conf

Discussion and references#

Many configurations options are possible. The documentation will show you some more.
In EnOSlib Roles and Networks don’t really depend on the provider that produced them. In other words you can substitute one provider’s configuration to another one easily without changing the artifact code.

[ ]: