Library objects#

Collections#

These are the base data structures used in EnOSlib. They aren’t meant to be used directly.

Classes:

`ResourcesSet`([iterable])	Wrapper around a Set exposing also some lists' operations.
`RolesDict`([dict])	RolesDict is a way to group resources of the same type together using tags.

class enoslib.collections.ResourcesSet(iterable: Iterable | None = None)#

Wrapper around a Set exposing also some lists’ operations.

Items of the Set must be comparable and hashable.

It’s possible for instance to call append, extend, etc. Indexing comes for instance with some caveats however: resource_set[0] will give you the first resource in the alphabetical order not the first inserted machine as you’d expect with a regular list.

add(value)#: Add an element.

discard(value)#: Remove an element. Do not raise an exception if absent.

class enoslib.collections.RolesDict(dict=None, /, **kwargs)#

RolesDict is a way to group resources of the same type together using tags.

From the user perspective, a RolesDict’s instance roles is a dictionary. To retrieve all the corresponding machine with the tag tag one can use ̀`roles[tag]`.

The value associated with a tag behaves like a set (it’s a py:class:~enoslib.collections.ResourcesSet). So classical set operations are possible, for instance roles[tag1] & roles[tag2] will give you back the set of resources that are both tagged with tag1 and tag2.

This set also accepts some list operations (as it seems that users are more familiar with them). So it’s possible to call append, extend, etc. Indexing comes with some caveats however: roles["tag"][0] will give you the first resource in the alphabetical order associated with the tag ``tag``(not the first inserted machine as you’d expect with a regular list).

inner#: alias of ResourcesSet

Resource abstractions#

These modules are made of the library-level objects. These are provider agnostic objects that can be fed into most of the EnOSlib functions.

Currently, the main abstractions of this module are the Host and the Network. They abstract away the notion of compute servers (something you can access and run some actions on) and networks (something you can get IPs from). Grouping resources is done using the Roles and Networks (plural).

Most likely you’ll interact with Roles and Networks right after calling provider.init(): this is indeed a provider responsibility to turn your abstract resource description into concrete library level objects.

Classes:

`DefaultNetwork`(address[, gateway, dns, ...])	Good enough implementation of Network for most situations.
`Host`(address, alias, user, keyfile, port, ...)	Abstract unit of computation.
`HostsView`([iterable])	A specialization of `ResourcesSet`
`Network`(address)	Base class for the library level network abstraction.
`Networks`([dict])	A specialization of `RolesDict`
`NetworksView`([iterable])	A specialization of `ResourceSet`
`Roles`([dict])	A specialization of `RolesDict`

Good enough implementation of Network for most situations.

Provides pooling for contiguous ips and/or macs. Support IPv4 and IPv6.

Providers must inherit from this class.

Parameters:

address – network address (as in ipaddress.ip_interface)
gateway – (optional) the gateway for this network (as in ipaddress.ip_address)
dns – (optional) the dns address (as in ipaddress.ip_address)
ip_start – (optional) first ip in the ip pool (as in ipaddress.ip_address)
ip_end – (optional) last ip in the ip pool (as in ipaddress.ip_address)
mac_start – (optional) first mac in the mac pool (as in netaddr.EUI)
mac_end – (optional) last mac in the mac pool (as in netaddr.EUI)

class enoslib.objects.Host(address: str, alias: str | None = None, user: str | None = None, keyfile: str | None = None, port: int | None = None, extra: ~typing.Dict = <factory>, net_devices: ~typing.Set[~enoslib.objects.NetDevice] = <factory>)#

Abstract unit of computation.

A Host is anything EnosLib can access (e.g. using SSH) to and run shell commands on. It is an abstraction notion of unit of computation that can be bound to bare-metal machines, virtual machines, or containers.

Note

Internally EnOSlib is using Ansible to connect to the remote hosts. By default, SSH is used but it isn’t the only connection method supported. You can change the connection method to fit your needs by setting the ansible_connection key in the extra field (and other options if needed). Ref: https://docs.ansible.com/ansible/latest/plugins/connection.html

Parameters:

address – host will be reached at this address (using SSH by default).
alias – a human-readable alias
user – user to connect with (e.g. using SSH)
keyfile – keyfile to use to authenticate (e.g. when using SSH)
port – port to connect to (e.g. using SSH)
extra – dictionary of options. Will be passed to Ansible as host_vars. Mutation of this attribute is possible and must be performed using the set_extra() or reset_extra()
net_devices – list of network devices configured on this host. can be synced with sync_info().

Note

In the future we’d like the provider to populate the net_devices to get a consistent initial representation of the hosts.

filter_addresses(networks: Iterable[Network] | None = None, include_unknown: bool = False) → List[IPAddress]#

Get some addresses assigned to this host.

Parameters:

networks – a list of networks to further filter the request If None or [], all the interfaces with at least one network attached will be returned. This doesn’t return interfaces attached to network unknown from EnOSlib.
include_unknown – True iff we want all the interface that are not attached to an EnOSlib network. Ignored if networks is not None.

Returns:

A list of addresses

filter_interfaces(networks: Iterable[Network] | None = None, include_unknown: bool = False) → List[str]#

Get some device interfaces.

Parameters:

networks – a list of networks to further filter the request If None, all the interfaces with at least one network attached will be returned. This doesn’t return interfaces attached to network unknown from EnOSlib.
include_unknown – True iff we want all the interface that are not attached to an EnOSlib network. Ignored if networks is not None.

Returns:

A list of interface names.

get_extra() → Dict#: Get a copy of the extra vars of this host.

reset_extra() → Host#: Recover the extra vars of this host to the original ones.

set_extra(**kwargs) → Host#: Mutate the extra vars of this host.

sync_from_ansible(networks: Networks, host_facts: Dict, clear: bool = True) → Host#

Set the devices based on ansible fact.s

Mutate self, since it add/update the list of network devices Currently the dict must be compatible with the ansible hosts facts.

to_host() → Host#: Copy or coerce to a Host.

class enoslib.objects.HostsView(iterable: Iterable | None = None)#

A specialization of ResourcesSet

for Host.

inner#: alias of Host

class enoslib.objects.Network(address: bytes | int | str | IPv4Network | IPv6Network)#

Base class for the library level network abstraction.

When one calls init on a provider, one takes ownership on nodes and networks. This class reflect one network owned by the user for the experiment lifetime. IPv4 and IPv6 networks can be represented by such object.

Providers must inherit from this class or the DefaultNetwork class which provides a good enough implementation in most cases.

Indeed, currently provenance (which provider created this) is encoded in the __class__ attribute.

class enoslib.objects.Networks(dict=None, /, **kwargs)#

A specialization of RolesDict

for NetworksView.

inner#: alias of NetworksView

class enoslib.objects.NetworksView(iterable: Iterable | None = None)#

A specialization of ResourceSet

for Networks.

inner#: alias of Network

class enoslib.objects.Roles(dict=None, /, **kwargs)#

A specialization of RolesDict

for HostsView.

inner#: alias of HostsView

Local#

Classes:

LocalHost([alias])

Representation of a local machine.

class enoslib.local.LocalHost(alias: str = 'localhost')#

Representation of a local machine.

Parameters:: alias – alias for a local machine must be unique

Docker#

Manage remote docker containers as first class citizens.

A possible workflow would be to start your containers using the method of your choice and build the list of available dockers using the enoslib.docker.get_dockers() function.

A DockerHost is a specialization of a Host and thus can be fed into any Host related operations (play_on, run_command…) [1]. Hosts datastructure in enoslib are tied somehow to Ansible. DockerHost is shaped so that the docker connection plugin can run. So we inject at build time the necessary connection options (ansible_connection=docker, ansible_docker_extra_args="-H <remote_docker>").

Connections to remote docker daemons can be made using different protocols [2].

Using ssh: requires ssh access to remote host but
can go through a bastion host if .ssh/config is configured correctly. Note that the docker client must be available.
Using raw tcp: requires to reach the remote docker daemon (e.g. be inside
g5k). Note that in this case the remote socket must be exposed.

Additionally, the structure is compatible with mitogen and its delegation model [3] which can improve the performance. Note that the facts from the host machines (where the docker daemon runs) needs to be gathered. One way to ensure this is to explicitly gather the facts from such hosts.

Example

"""
Example that makes use of the DockerHost data structure.

This is an advanced example where
- docker containers will be started on g5k machines
- network emulation will be enforced between those docker containers by
reusing |enoslib| api functions.
"""
import logging
from pathlib import Path

import enoslib as en

en.init_logging(level=logging.INFO)
en.check()

job_name = Path(__file__).name


conf = (
    en.G5kConf.from_settings(job_name=job_name, walltime="0:30:00", job_type=[])
    .add_machine(roles=["control"], cluster="ecotype", nodes=2)
    .finalize()
)

provider = en.G5k(conf)

# Get actual resources
roles, networks = provider.init()

# Install docker
registry_opts = dict(type="external", ip="docker-cache.grid5000.fr", port=80)
d = en.Docker(
    agent=roles["control"], bind_var_docker="/tmp/docker", registry_opts=registry_opts
)
d.deploy()

# Start N containers on each G5K host (for a total of 2*N containers)
N = 4
with en.play_on(roles=roles) as p:
    p.raw("modprobe ifb")
    for i in range(N):
        p.docker_container(
            name=f"mydocker-{i}",
            image="ubuntu",
            state="started",
            command="sleep 10d",
            capabilities=["NET_ADMIN"],
        )

# Get all the docker containers running on all remote hosts
dockers = en.get_dockers(roles=roles)

# Build the network contraints to apply on the remote docker
# We assume here the interface name in docker to be eth0
sources = []
for idx, host in enumerate(dockers):
    delay = idx
    print(f"{host.alias} <-> {delay}ms")
    inbound = en.NetemOutConstraint(device="eth0", options=f"delay {delay}ms")
    outbound = en.NetemInConstraint(device="eth0", options=f"delay {delay}ms")
    sources.append(en.NetemInOutSource(host, constraints={inbound, outbound}))

# This requires the Docker client to be installed on the local machine.
# Also, it might not work well because SSH connections are handled by Docker.
# See https://gitlab.inria.fr/discovery/enoslib/-/issues/163 for discussion
with en.play_on(roles=dockers, gather_facts=False) as p:
    # We can't use the 'apt' module because python is not installed in containers
    p.raw("apt update && DEBIAN_FRONTEND=noninteractive apt install -qq -y iproute2")

en.netem(sources)

Classes:

DockerHost(alias, container_name, host[, ...])

A kind of host reachable using docker protocol.

Functions:

get_dockers(roles[, pattern_hosts, ...])

Get remote dockers hosts.

class enoslib.docker.DockerHost(alias: str, container_name: str, host: Host, proto: str | None = None, state: Mapping | None = None)#

A kind of host reachable using docker protocol.

Parameters:

alias – unique name across the deployment
container_name – name of the docker container on the remote host
host – the host where the container can be found
proto – how to connect to the remote host (DockerHost.PROTO_TCP/DockerHost.PROTO_SSH) [Default DockerHost.PROTO_SSH]
state – dict representing the state as returned by docker inspect

classmethod from_state(state: Mapping, host: Host) → DockerHost#: Build a DockerHost from a state json as returned by docker inspect.

enoslib.docker.get_dockers(roles: Roles, pattern_hosts: str = '*', container_name: str = '.*') → List[DockerHost]#

Get remote dockers hosts.

Parameters:

roles – the roles as returned by enoslib.infra.provider.Provider.init()
pattern_hosts – pattern to describe ansible hosts to target. see https://docs.ansible.com/ansible/latest/intro_patterns.html
container_name – name of the containers to look for. Regexp are supported as in filter option of docker inspect.

Returns:

List of DockerHost matching the passed container_name