Provider APIs#

Base Provider Class#

Classes:

Provider(provider_conf[, name])

Base class for the provider.

class enoslib.infra.provider.Provider(provider_conf, name: str | None = None)#

Bases: object

Base class for the provider.

Providers are components that are responsible for acquiring resources on existing testbeds. Concretely, a provider takes a declarative description of the expected resources as argument. It requests the resources on the corresponding testbed, and returns a set of hosts and networks labeled with their roles. In other words, providers help the experimenter deal with the variety of the APIs of testbeds.

Some infrastructure is based on a reservation system (OAR, blazar …). In this case we defer to the decision to start the resource to the underlying scheduler. This scheduler will try to find an available start_time for our configuration in a near future if possible.

In the above situation reservation in advance is possible and can be set in the corresponding configuration, using set_reservation() or init() with the start_time parameter set.

Parameters:

provider_conf (BaseConfiguration) – configuration of the provider. The configuration object is specific to each provider and must follow the provider’s schema
name – a provider can be named. This is used in the context of a multiprovider and all the resources from this provider using this name in roles an networks.

Example

Typical workflow:

# craft a conf (provider specific)
conf = Configuration() ...

provider = Provider(conf)
roles, networks = provider.init()

# release resources
provider.destroy()

Using a context manager

# craft a conf (provider specific)
conf = Configuration() ...

with provider(conf) as (roles, networks):
    # do stuff with roles and networks
    ...
# Resources are automatically released at the end

Methods:

`async_init`(**kwargs)	Partial init: secure the resources to the targeted infrastructure.
`destroy`([wait])	Abstract.
`init`([force_deploy, start_time])	Abstract.
`is_created`()	Is the provider already created.
`offset_walltime`(offset)	Offset the walltime.
`set_reservation`(timestamp)	Change the internal reservation date.
`test_slot`(start_time, end_time)	Test a slot that starts at a given point in time.

async_init(**kwargs)#

Partial init: secure the resources to the targeted infrastructure.

This is primarily used internally by Providers to get the resources from different platforms. As this method actually starts some real resources somewhere, errors may occur (e.g no more available resources, …). It’s up to the provider to indicate if the error is critical or not. For instance an InvalidReservationTime can be raised to indicate the Providers to retry later.

Parameters:

kwargs – keyword arguments. Fit those from init()

Raises:

InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception

abstract destroy(wait: bool = False, **kwargs)#: Abstract. Destroy the resources used for the deployment.

abstract init(force_deploy: bool = False, start_time: int | None = None, **kwargs: Any)#

Abstract. Provides resources and provisions the environment.

This calls the underlying provider and provision resources (machines and networks).

Parameters:

force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.

Returns:

roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see Enos_vagrant

Return type:

(roles, networks) tuple

is_created() → bool#: Is the provider already created.

abstract offset_walltime(offset: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

set_reservation(timestamp: int)#

Change the internal reservation date.

Ignored on platform that aren’t based on a reservation system.

Parameters:: timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.

test_slot(start_time: int, end_time: int) → bool#

Test a slot that starts at a given point in time.

A slot is given by a start_time, a duration and an amount of resources. The two latter are found in the internal configuration.

Parameters:

start_time (timestamp (int)) – Test for a possible slot that starts at start_time (UTC)
end_time – (timestamp (int)): How much time in the future we should look for possible reservation. This is used on some platform to know how much time in the future we look in the planning data. Timestamp is based on UTC.

Returns:

True iff the slot is available

Vagrant#

Vagrant Provider Class#

Classes:

Enos_vagrant(provider_conf[, name])

The provider to use when working with vagrant (local machine).

class enoslib.infra.enos_vagrant.provider.Enos_vagrant(provider_conf, name: str | None = None)#

The provider to use when working with vagrant (local machine).

destroy(wait: bool = False, **kwargs)#: Destroy all vagrant box involved in the deployment.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs) → Tuple[Roles, Networks]#

Reserve and deploys the vagrant boxes.

Parameters:: force_deploy (bool) – True iff new machines should be started

offset_walltime(difference: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

Vagrant Schema#

Vagrant Configuration Schema#

type	object
properties
backend	type	string
	enum	libvirt, virtualbox
box	base image to use (default: generic/debian11)
	type	string
user	SSH user to use (default: root)
	type	string
name_prefix	Prepend this prefix to box names
	type	string
config_extra	Extra config to pass (in vargrant DSL
	type	string
resources	Vagrant Resource
	type	object
	properties
	networks	type	array
		items	Vagrant Network
		uniqueItems	True
	machines	type	array
		items	Vagrant Compute
	additionalProperties	False
additionalProperties	False

Vagrant Network#

type	object
properties
cidr	type	string
roles	type	array
	items	type	string
additionalProperties	False

Vagrant Compute#

type	object
properties
roles	type	array
	items	type	string
number	type	number
name_prefix	type	string
flavour	type	string
	enum	tiny, small, medium, big, large, extra-large
flavour_desc	Vagrant Flavour
additionalProperties	False

Vagrant Flavour#

type	object
properties
core	type	number
mem	type	number
additionalProperties	False

Grid5000 (G5k)#

G5k Provider Class#

Classes:

`G5k`(args, *kwargs)	The provider to use when interacting with Grid'5000.
`G5kBase`(args, *kwargs)	Internal class.
`G5kHost`(fqdn, roles, primary_network[, ...])	A G5k host.
`G5kNetwork`(roles, id, site)	Representation of a reservable network in G5k.
`G5kProdNetwork`(roles, id, site)	Representation of the Production Network in G5k.
`G5kSubnetNetwork`(roles, id, site, subnets)	Representation of a subnet resource of G5k (/16 or /22).
`G5kTunnel`(address, port[, local_port])	A class to initiate a tunnel to a targetted service inside Grid'5000.
`G5kVlanNetwork`(roles, id, site, vlan_id)	Representation of a Vlan resource in G5k.

class enoslib.infra.enos_g5k.provider.G5k(*args, **kwargs)#

The provider to use when interacting with Grid’5000.

Most of the methods are inherited from G5kBase.

reserve()#

Reserve the resources described in the configuration

This support multisite configuration.

class enoslib.infra.enos_g5k.provider.G5kBase(*args, **kwargs)#

Internal class.

Provider dedicated to single site interaction.

jobs#

List of Grid’5000 Job objects managed by this provider

Type:: list

async_init(start_time: int | None = None, **kwargs)#

Reserve but don’t wait.

No node/network information can’t be retrieved at this moment. If a start_time is provided, set the reservation date to it.

destroy(wait: bool = True, **kwargs)#: Destroys the jobs.

Context manager to manage firewall rules

Create a firewall opening when entering
Delete the firewall opening when exiting

Parameters:

hosts – limit the rule to a set of hosts. if None, rules will be applied on all hosts of all underlying jobs.
port – ports to open
src_addr – source addresses to consider
proto – protocol to consider

Create a firewall rules

Reference: https://www.grid5000.fr/w/Reconfigurable_Firewall

Note that port and src_addr and proto are passed to the API calls without any change. It means that accepted values are those accepted by the REST API.

Parameters:

hosts – limit the rule to a set of hosts. if None, rules will be applied on all hosts of all underlying jobs.
port – ports to open
src_addr – source addresses to consider
proto – protocol to consider

fw_delete()#: Delete all existing rules.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs) → Tuple[Roles, Networks]#

Take ownership over some Grid’5000 resources (compute and networks).

The function does the heavy lifting of transforming your abstract resource configuration into concrete resources.

From a high level perspective it works as follow:

First it transforms the configuration of resources into an actual OAR resource selection string (one single reservation per provider instance).
It requests the API to get the corresponding resources (job and optionally deploys a environment)
Those resources are then mapped back to every single item on the configuration.
Finally it applies some more operations (set nodes on vlans, configure secondary interfaces) before returning.

Note

The call to the function is idempotent and the following is ensured:

Existing job(s) (based on the name) will be reloaded.
The mapping between concrete resources and their corresponding roles is fixed across runs. This includes:
- the mapping between machines and roles
- the mapping between networks and roles
- the mapping between network cards and networks
Deployments is performed only on nodes that are not deployed yet (up to three attempts).
At the end machine are reachable using the root account.

Parameters:

force_deploy (bool) – True iff the environment must be redeployed
start_time (timestamp (int)) – Time at which to start the job, by default whenever possible

Raises:

MissingNetworkError – If one network is missing in comparison to what is claimed.
NotEnoughNodesError – If the min constraints can’t be met.
InvalidReservationTime – If the set reservation_date from provider.conf isn’t free
InvalidReservationOld – If the set reservation_date from provider.conf is in the past
InvalidReservationCritical – Any other error that might occur during the reservation is deemed critical

Returns:

Two dictionaries (roles, networks) representing the inventory of resources.

is_created() → bool#: Is the provider already created.

offset_walltime(offset: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

set_reservation(timestamp: int)#

Change the internal reservation date.

Ignored on platform that aren’t based on a reservation system.

Parameters:: timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.

test_slot(start_time: int, end_time: int) → bool#: Test if it is possible to reserve the configuration corresponding to this provider at start_time

static tunnel(address: str, port: int) → Tuple[str, int, SSHTunnelForwarder | None]#

Create a tunnel if necessary between here and there (in G5k).

Parameters:

address (str) – The remote address to reach (assuming inside g5k)
port (int) – The remote port to reach

Returns:

The context manager

class enoslib.infra.enos_g5k.provider.G5kHost(fqdn: str, roles: List[str], primary_network: G5kNetwork, secondary_networks: List[G5kNetwork] | None = None)#

A G5k host.

property apinode: Node#

Get the api Node object.

Returns:: Node object (see python-grid500)

dhcp_networks_command() → str#

Get the command to set up the dhcp an all interfaces.

Returns:: The command as a string.

get_nics(extra_cond: ~typing.Callable[[~typing.Dict], bool] = <function G5kHost.<lambda>>) → List[Tuple[str, str]]#

Get the network interfaces names corresponding to a criteria.

Note

Only the mountable and Ethernet interfaces are returned.
Nic are sorted so that the result is fixed accros run.

Parameters:: extra_cond – predicate over a nic to further filter the results. Here a nic is a dictionary as returned in the API.
Returns:: A list of nics. Each nic is a tuple (legacy name, deterministic name) e.g (“eth0”, “eno1”). Result is sorted to ensure idempotence.

Note

NOTE(msimonin): Since 05/18 nics on g5k nodes have predictable names but the api description keeps the legacy name (device key) and the new predictable name (key name). The legacy names is still used for api request to the vlan endpoint This should be fixed in https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=9272 When its fixed we should be able to only use the new predictable name.

grant_root_access_command() → str#: Get the command to get root access on the node.

mirror_state()#

Make sure the API states are consistent to the Host attributes.

For instance this will set some of the NIC of the nodes in a vlan (POST request on the API). For now the only strategy to map NIC to secondary network is to map them in the same order : nic_i <-> secondary_networks[i]. (where nic_i is not the primary one)

Note that this doesn’t configure the NIC on the node itself (e.g dhcp).

property primary_nic: Tuple[str, str]#

Get the first nic mounted.

On Grid’5000 there’s only one nic mounted by default: this is the primary nic.

Returns:: A tuple of (legacy name, deterministic name) for the network card.

property secondary_nics: List[Tuple[str, str]]#

Get the nics that serves as secondary nics.

Note: only return those eligible to map a secondary network.

Returns:: All the nic that serves to connect the node to a secondary network.

property ssh_address: str#

Get an SSH reachable address for this Host.

This may differ from the fqdn when using vlans.

Returns:: The address as a string.

to_enoslib(address: str = '', user: str = 'root', extra: Dict | None = None) → Host#

Return a generic Host object from a G5kHost object.

We automatically set up a SSH jump configuration if Enoslib is running outside of Grid’5000.

Parameters:

address – host name to use for SSH, defaults to G5kHost.ssh_address
user – SSH user, defaults to “root”
extra – optional dictionary of extra Ansible host variables

Returns: a Host object usable with Ansible.

class enoslib.infra.enos_g5k.provider.G5kNetwork(roles: List[str], id: str, site: str)#

Representation of a reservable network in G5k.

A G5kNetwork wraps the corresponding network resource of the G5k REST API.

Parameters:

roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network

add_host(host: G5kHost)#

Add a host enoslib.infra.enos_g5k.objects.G5kHost.

Currently, this doesn’t attach the node.

Parameters:: host – The host to attach

add_hosts(hosts: List[G5kHost])#

Add some hosts enoslib.infra.enos_g5k.objects.G5kHost.

Currently, this doesn’t attach the node.

Parameters:: hosts – The list host to attach

abstract property apinetwork: RESTObject | None#

Gets the underlying RESTObject representing the network.

Only vlan and prod network have an equivalent entry in the API. None will be returned for subets.

Returns:: The corresponding RESTObject, None otherwise.

abstract attach(fqdns: List[str], device: str)#

Attach the specific devices into this network.

Parameters:

fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.

abstract property cidr: str | None#

Get the network address in cid format.

Returns:: The cidre of the network. None for subnets.

abstract property cidr6: str | None#

Get the ipv6 network address in cidr format.

Returns:: The cidre of the network. None for subnets.

property dns: str | None#: Gets the DNS address for this network.

property dns6: str | None#

Gets the DNS address for this network. IPv6

We fall back to IPv4 for now

abstract property gateway: str | None#

Get the gateway for this network.

Returns:: The gateway address as a string.

abstract property gateway6: str | None#

Get the gateway for this network (IPv6).

Returns:: The gateway address as a string.

abstract to_enos()#

Transform into a provider-agnostic data structure.

For legacy reason we’re still using dicts here…

Returns:: a list of networks, each being a dict.

abstract translate(fqdns: Iterable[str], reverse: bool = False) → List[Tuple[str, str]]#

Gets the DNS names of the passed fqdns in these networks and vice versa.

Parameters:

fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.

Returns:

List of translated names.

abstract translate6(fqdns: Iterable[str], reverse: bool = False) → List[Tuple[str, str]]#

Gets the DNS names (resolved as ipv6) of the passed fqdns in these networks.

Parameters:

fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.

Returns:

List of translated names.

abstract property vlan_id: str | None#

Get the vlan id.

Returns:: The vlan id. None for subnets.

class enoslib.infra.enos_g5k.provider.G5kProdNetwork(roles: List[str], id: str, site: str)#

Representation of the Production Network in G5k.

Note: production network have the “default” uid on the G5K API.

Parameters:

roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network

attach(fqdns: List[str], nic: str)#

Attach the specific devices into this network.

Parameters:

fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.

to_enos() → Tuple[List[str], List]#

Build the generic network type implementation.

A production network in G5k has 2 flavours IPv4 and IPv6 so we generate both generic network type here corresponding to this vlan.

translate(fqdns: Iterable[str], reverse: bool = False) → List[Tuple[str, str]]#

Node in the production network.

node uid == node name

translate6(fqdns: Iterable[str], reverse: bool = False) → List[Tuple[str, str]]#: Translate node name in ipv6 resolvable name.

class enoslib.infra.enos_g5k.provider.G5kSubnetNetwork(roles: List[str], id: str, site: str, subnets: List[str])#

Representation of a subnet resource of G5k (/16 or /22).

Note

Subnets are weird beasts on G5k. Especially /16 networks for which you will be given 64 /22 networks and they aren’t represented in the REST API.

So we encapsulate this in this object

Parameters:

roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network
subnets – the actual subnets (list of cidr) given by OAR.

property apinetwork: None#

Gets the underlying RESTObject representing the network.

Only vlan and prod network have an equivalent entry in the API. None will be returned for subets.

Returns:: The corresponding RESTObject, None otherwise.

attach(fqdns: List[str], nic: str)#

Attach the specific devices into this network.

Parameters:

fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.

property cidr: None#

Not well-defined.

Since subnets might be an aggregation of several smaller ones it’s difficult to know what to return here.

Note that the user will be given with as many Network as small subnets we have. In this case the network address will be well defined.

property cidr6: None#

Not well-defined (and no support for IPv6 now).

Since subnets might be an aggregation of several smaller ones it’s difficult to know what to return here.

Note that the user will be given with as many Network as small subnets we have. In this case the network address will be well defined.

property gateway#

Get the gateway for this network.

Returns:: The gateway address as a string.

property gateway6: None#

Get the gateway for this network (IPv6).

Returns:: The gateway address as a string.

to_enos() → Tuple[List[str], List[G5kEnosSubnetNetwork]]#

Build the generic network type implementation.

A subnet in G5k doesn’t have yet an IPv6 counterpart so we generate a single IPv4 generic representation. A /16 is actually an aggregation of /22 so we emit one generic network per underlying subnet.

translate(fqdns: Iterable[str], reverse: bool = True) → List[Tuple[str, str]]#

Gets the DNS names of the passed fqdns in these networks and vice versa.

Parameters:

fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.

Returns:

List of translated names.

translate6(fqdns: Iterable[str], reverse: bool = True) → List[Tuple[str, str]]#

Gets the DNS names (resolved as ipv6) of the passed fqdns in these networks.

Parameters:

fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.

Returns:

List of translated names.

property vlan_id: None#

Get the vlan id.

Returns:: The vlan id. None for subnets.

class enoslib.infra.enos_g5k.provider.G5kTunnel(address: str, port: int, local_port: int = 0)#

A class to initiate a tunnel to a targetted service inside Grid’5000.

Can be used as a context manager (will close the tunnel automatically). Note that this is a noop when called from inside Grid’5000.

Parameters:

address – The ip address/fqdn of the targetted service
port – The port of the targetted service

close()#

Close the tunnel.

Note that this won’t wait for any connection to finish first.

start() → Tuple[str, int, SSHTunnelForwarder | None]#

Start the tunnel.

Returns:: A tuple composed of the local address , the local port and the tunnel object (if any)

class enoslib.infra.enos_g5k.provider.G5kVlanNetwork(roles: List[str], id: str, site: str, vlan_id: str)#

Representation of a Vlan resource in G5k.

Parameters:

roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network
vlan_id – the vlan id of the vlan

property apinetwork: Vlan#

Gets the underlying RESTObject representing the network.

Only vlan and prod network have an equivalent entry in the API. None will be returned for subets.

Returns:: The corresponding RESTObject, None otherwise.

attach(fqdns: List[str], device: str)#

Attach the specific devices into this network.

Parameters:

fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.

property cidr: str#

Get the network address in cid format.

Returns:: The cidre of the network. None for subnets.

property cidr6: str#

Get the ipv6 network address in cidr format.

Returns:: The cidre of the network. None for subnets.

property gateway: str#

Get the gateway for this network.

Returns:: The gateway address as a string.

property gateway6: str | None#

Get the gateway for this network (IPv6).

Returns:: The gateway address as a string.

to_enos() → Tuple[List[str], List]#

Build the generic network type implementation.

A vlan in G5k has 2 flavours IPv4 and IPv6 so we generate both generic network type here corresponding to this vlan.

translate(fqdns: Iterable[str], reverse: bool = False) → List[Tuple[str, str]]#

Gets the DNS names of the passed fqdns in these networks and vice versa.

Parameters:

fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.

Returns:

List of translated names.

translate6(fqdns: Iterable[str], reverse: bool = False) → List[Tuple[str, str]]#

Gets the DNS names (resolved as ipv6) of the passed fqdns in these networks.

Parameters:

fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.

Returns:

List of translated names.

property vlan_id: str#: Get a vlan id if any.

G5k Schema#

Grid5000 Configuration Schema#

type	object
properties
dhcp	Run dhcp client automatically on kavlan networks (default: True)
	type	boolean
force_deploy	Force systematic redeployment on nodes (deploy only) (default: False)
	type	boolean
env_name	The kadeploy3 environment to use (deploy only)
	type	string
env_version	Kadeploy3 env version to use (deploy only, optional)
	type	integer
	minimum	2016011914
	exclusiveMinimum	True
job_name	Name of the job (default: EnOSlib)
	type	string
job_type	OAR job type (default: []).
	format	job_type
	anyOf	type	string
		type	array
		items	type	string
key	SSH public key to use (default: ~/.ssh/.id_rsa.pub)
	type	string
monitor	Activate on demand metrics (e.g `prom_.*`)
	type	string
oargrid_jobids	Reload from existing job ids
	type	array
	items	Grid5000 JobIds
project	Project / team to use
	type	string
queue	OAR queue to use (default: default)
	type	string
	enum	default, production, testing, besteffort
reservation	reservation date in YYYY-mm-dd HH:MM:SS format
	type	string
	format	reservation
walltime	Job duration (default: 02:00:00)
	type	string
	format	walltime
resources	Grid’5000 Resources
	type	object
	properties
	machines	Description of the servers to reserve
		type	array
		items	oneOf	Grid5000 ComputeCluster
				Grid5000 ComputeServers
	networks	Description of the networks to reserve
		type	array
		items	Grid5000 Network
		uniqueItems	True
	additionalProperties	False
additionalProperties	False

Grid5000 JobIds#

List of tuple (site, jobid) used to reload the jobs from
type	array
items	type	string

Grid5000 ComputeCluster#

Describe a group of machine based on a cluster name
type	object
properties
roles	The concrete resources will be assigned this role
	type	array
	items	type	string
cluster	Which cluster to use
	type	string
nodes	Number of nodes (default: 1)
	type	number
min	Minimal number of nodes to get (default to nodes)
	type	number
reservable_disks	Request access to reservable disks on nodes
	type	boolean
primary_network	Network(id) to use on the primary NIC
	type	string
secondary_networks	Additional networks(ids) to assign
	type	array
	items	type	string
	uniqueItems	True

Grid5000 ComputeServers#

Description of a specific list of servers to get
type	object
properties
roles	The concrete resources will be assigned this role
	type	array
	items	type	string
servers	List of names (e.g [chetemi-1.lille.grid5000.fr])
	type	array
	items	type	string
		format	hostname
	minItems	1
nodes	Number of nodes to get (default: #servers)
	type	number
min	Minimal number of nodes to get (default to nodes)
	type	number
reservable_disks	Request access to reservable disks on nodes
	type	boolean
primary_network	Network to use on this NIC
	type	string
secondary_networks	List of the network to use on the other NICs
	type	array
	items	type	string
	uniqueItems	True

Grid5000 Network#

type	object
properties
id	Id used to identify network in machines
	type	string
type	Type of network to use supported by Grid’5000
	enum	prod, kavlan, kavlan-local, kavlan-global, slash_16, slash_22
roles	The concrete resources will be assigned this role
	type	array
	items	type	string
site	On which site to reserve the network
	type	string

G5k API utils#

This module is composed of helpers functions to deal with the Grid’5000 REST API.

It wraps the python-grid5000 library to provide some usual routines to interact with the platform.

Classes:

`Client`([excluded_sites])	Wrapper of the python-grid5000 client.
`OarNetwork`(site, nature, descriptor)	Create new instance of OarNetwork(site, nature, descriptor)

Functions:

`build_resources`(jobs)	Build the resources from the list of jobs.
`can_start_on_cluster`(nodes_status, number, ...)	Check if #nodes can be started on a given cluster.
`clusters_sites_obj`(clusters)	Get all the corresponding sites of the passed clusters.
`deploy`(site, nodes, config)
`enable_group_storage`(storage_site, ...)	Enable access to a group storage from ips.
`enable_home_for_job`(job, ips)	Enable access to home dir from ips.
`get_all_clusters_sites`()	Get all the cluster of all the sites.
`get_all_sites_obj`()	Return the list of the sites.
`get_api_client`()	Gets the reference to the API client (singleton).
`get_api_username`()	Return username of client
`get_cluster_interfaces`(cluster[, extra_cond])	Get the network interfaces names corresponding to a criteria.
`get_cluster_site`(cluster)	Get the site of a given cluster.
`get_clusters_interfaces`(clusters[, extra_cond])	Returns for each cluster the available cluster interfaces
`get_clusters_sites`(clusters)	Get the corresponding sites of given clusters.
`get_clusters_status`(clusters)	Get the status of the clusters (current and future reservations).
`get_cores`(cluster)	Get the total number of cores in each machine for a given cluster
`get_dns`(site)
`get_ipv6`(site)
`get_memory`(cluster)	Get the memory (in bytes) in each machine for a given cluster
`get_nics`(cluster)	Get the network cards information
`get_node`(site, cluster, uid)
`get_nodes`(cluster)	Get all the nodes of a given cluster.
`get_site_obj`(site)	Get a single site.
`get_subnet_gateway`(site)
`get_threads`(cluster)	Get the total number of threads in each machine for a given cluster
`get_vlan`(site, vlan_id)
`get_vlans`(site)
`grid_deploy`(site, nodes, config)	Deploy and wait for the deployment to be finished.
`grid_destroy_from_ids`(oargrid_jobids[, wait])	Destroy all the jobs with corresponding ids
`grid_destroy_from_name`(job_name[, wait, ...])	Destroy all the jobs with a given name.
`grid_get_or_create_job`(job_name, walltime, ...)
`grid_make_reservation`(job_name, walltime, ...)
`grid_reload_from_ids`(oargrid_jobids)	Reload all running or pending jobs of Grid'5000 from their ids
`grid_reload_jobs_from_ids`(oargrid_jobids)	Reload jobs of Grid'5000 from their ids
`grid_reload_jobs_from_name`(job_name[, ...])	Reload all running or pending jobs of Grid'5000 with a given name.
`job_delete`(job[, wait])
`set_nodes_vlan`(nodes, interface, vlan_id)	Set the interface of the nodes in a specific vlan.
`submit_jobs`(job_specs)	Submit a job
`to_prod_nature`()
`to_subnet_nature`(cidr)
`to_vlan_nature`(vlan_id)
`wait_for_jobs`(jobs)	Waits for all the jobs to be runnning.

class enoslib.infra.enos_g5k.g5k_api_utils.Client(excluded_sites: List | None = None, **kwargs)#

Wrapper of the python-grid5000 client.

It accepts extra parameters to be set in the configuration file.

Constructor.

Parameters:: excluded_sites (list) – sites to forget about when reloading the jobs. The primary use case was to exclude unreachable sites and allow the program to go on.

class enoslib.infra.enos_g5k.g5k_api_utils.OarNetwork(site, nature, descriptor)#

Create new instance of OarNetwork(site, nature, descriptor)

Attributes:

`descriptor`	Alias for field number 2
`nature`	Alias for field number 1
`site`	Alias for field number 0

descriptor#: Alias for field number 2

nature#: Alias for field number 1

site#: Alias for field number 0

enoslib.infra.enos_g5k.g5k_api_utils.build_resources(jobs: Iterable[Job]) → Tuple[List[str], List[OarNetwork]]#

Build the resources from the list of jobs.

Parameters:

jobs (list) – The list of python-grid5000 jobs

Returns:

nodes, networks tuple where

nodes is a list of all the nodes of the various reservations
networks is a list of all the networks of the various reservation

enoslib.infra.enos_g5k.g5k_api_utils.can_start_on_cluster(nodes_status: Mapping, number: int, exact_nodes: List[str], start: float, walltime: int) → bool#

Check if #nodes can be started on a given cluster.

This is intended to give a good enough approximation. This can be used to prefiltered possible reservation dates before submitting them on oar.

Parameters:

nodes_status – a dictionary with all the status of the nodes as returned by the api (cluster status endpoint)
number – number of node in the demand
exact_nodes – the list of the fqdn of the machines to get
start – start time of the job
walltime – walltime of the job

Returns: True iff the job can start

enoslib.infra.enos_g5k.g5k_api_utils.clusters_sites_obj(clusters: Iterable) → Dict[str, Site]#

Get all the corresponding sites of the passed clusters.

Parameters:: clusters (list) – list of string uid of sites (e.g ‘rennes’)
Returns:: dict corresponding to the mapping cluster uid to python-grid5000 site

enoslib.infra.enos_g5k.g5k_api_utils.deploy(site: str, nodes: List[str], config: Dict) → Tuple[List[str], List[str]]#

enoslib.infra.enos_g5k.g5k_api_utils.enable_group_storage(storage_site: str, storage_server: str, storage_name: str, ips: List[str], termination_job: Job)#

Enable access to a group storage from ips.

Parameters:

storage_site – a grid’5000 site. Site where the storage is located
storage_server – a server name Storage server where the storage is located (e.g. ~”storage1”~ or ~”home”~)
storage_name – name of the group storage The name is the one used to identify a Group Storage (this might be your username if you plan to allow your home dir)
ips – list of ips Ips to allow the access from (only IPv4 for now).
termination_job – a job This is used as a termination condition

enoslib.infra.enos_g5k.g5k_api_utils.enable_home_for_job(job: Job, ips: List[str])#

Enable access to home dir from ips.

This allows to mount the home dir corresponding to the site of job from any of the ips provided.

Examples

roles, networks = provider.init()
# get some ips to allow
ips = [str(ip) for ip in networks["role"][0].network]

# get the underlying job
job = provider.jobs[0]
enable_home_for_job(job, ips)

For enabling any group storage, please refer to enable_group_storage()

Parameters:

job – A (running) job. home site and access duration will be inferred from this job
ips – list of IPs Every machine connecting from one of this IPs will be granted an access to the home directory

enoslib.infra.enos_g5k.g5k_api_utils.get_all_clusters_sites() → Dict[str, str]#

Get all the cluster of all the sites.

Returns:: dict corresponding to the mapping cluster uid to python-grid5000 site

enoslib.infra.enos_g5k.g5k_api_utils.get_all_sites_obj() → List[Site]#

Return the list of the sites.

Returns:: list of python-grid5000 sites

enoslib.infra.enos_g5k.g5k_api_utils.get_api_client() → Client#: Gets the reference to the API client (singleton).

enoslib.infra.enos_g5k.g5k_api_utils.get_api_username() → str#

Return username of client

Returns:: client’s username

enoslib.infra.enos_g5k.g5k_api_utils.get_cluster_interfaces(cluster: str, extra_cond=<function <lambda>>) → List[Tuple]#

Get the network interfaces names corresponding to a criteria.

Note that the cluster is passed (not the individual node names), thus it is assumed that all nodes in a cluster have the same interface names same configuration. In addition to extra_cond, only the mountable and Ehernet interfaces are returned.

Parameters:

cluster (str) – the cluster to consider
extra_cond (lambda) – boolean lambda that takes the nic(dict) as parameter

enoslib.infra.enos_g5k.g5k_api_utils.get_cluster_site(cluster: str) → str#

Get the site of a given cluster.

Parameters:: cluster (str) – a Grid’5000 cluster
Returns:: The corresponding site(str)

enoslib.infra.enos_g5k.g5k_api_utils.get_clusters_interfaces(clusters: ~typing.Iterable, extra_cond=<function <lambda>>) → Dict#

Returns for each cluster the available cluster interfaces

Parameters:

clusters (str) – list of the clusters
extra_cond (lambda) – extra predicate to filter network card retrieved from the API. E.g. lambda nic: not nic[‘mounted’] will retrieve all the usable network cards that are not mounted by default.

Returns:

dict of cluster with their associated nic names

Examples

# pseudo code
actual = get_clusters_interfaces(["paravance"])
expected = {"paravance": ["eth0", "eth1"]}
assertDictEquals(expected, actual)

enoslib.infra.enos_g5k.g5k_api_utils.get_clusters_sites(clusters: Iterable[str]) → Dict[str, str]#

Get the corresponding sites of given clusters.

Parameters:: clusters (list) – list of the clusters (str)
Returns:: dict of corresponding to the mapping cluster -> site

enoslib.infra.enos_g5k.g5k_api_utils.get_clusters_status(clusters: Iterable[str]) → Dict#: Get the status of the clusters (current and future reservations).

enoslib.infra.enos_g5k.g5k_api_utils.get_cores(cluster: str) → int#

Get the total number of cores in each machine for a given cluster

Parameters:: cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)

enoslib.infra.enos_g5k.g5k_api_utils.get_dns(site)#

enoslib.infra.enos_g5k.g5k_api_utils.get_ipv6(site)#

enoslib.infra.enos_g5k.g5k_api_utils.get_memory(cluster: str) → int#

Get the memory (in bytes) in each machine for a given cluster

Parameters:: cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)

enoslib.infra.enos_g5k.g5k_api_utils.get_nics(cluster: str)#

Get the network cards information

Parameters:: cluster (str) – Grid’5000 cluster name
Returns:: dict of nic information

enoslib.infra.enos_g5k.g5k_api_utils.get_node(site, cluster, uid) → Node#

enoslib.infra.enos_g5k.g5k_api_utils.get_nodes(cluster: str) → List[Node]#

Get all the nodes of a given cluster.

Parameters:: cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)

enoslib.infra.enos_g5k.g5k_api_utils.get_site_obj(site) → Site#

Get a single site.

Returns:: the python-grid5000 site

enoslib.infra.enos_g5k.g5k_api_utils.get_subnet_gateway(site)#

enoslib.infra.enos_g5k.g5k_api_utils.get_threads(cluster: str) → int#

Get the total number of threads in each machine for a given cluster

Parameters:: cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)

enoslib.infra.enos_g5k.g5k_api_utils.get_vlan(site, vlan_id) → Vlan#

enoslib.infra.enos_g5k.g5k_api_utils.get_vlans(site)#

enoslib.infra.enos_g5k.g5k_api_utils.grid_deploy(site: str, nodes: List[str], config: Dict) → Tuple[List[str], List[str]]#

Deploy and wait for the deployment to be finished.

Parameters:

site (str) – the site
nodes (list) – list of nodes (str) to depoy
config (dict) – option of the deployment (refer to the Grid’5000 API Specifications)

Returns:

tuple of deployed(list), undeployed(list) nodes.

enoslib.infra.enos_g5k.g5k_api_utils.grid_destroy_from_ids(oargrid_jobids: Iterable[Tuple], wait: bool = False)#

Destroy all the jobs with corresponding ids

Parameters:

oargrid_jobids (list) – the (site, oar_job_id) list of tuple identifying the jobs for each site.
wait – True whether we should wait for a status change

enoslib.infra.enos_g5k.g5k_api_utils.grid_destroy_from_name(job_name: str, wait: bool = False, restrict_to: Iterable[str] | None = None)#

Destroy all the jobs with a given name.

Parameters:

job_name (str) – the job name
wait – True whether we should wait for a status change
restrict_to – restrict the action to these sites only. If None is passed, no restriction applies and the action will be taken on all possible sites

enoslib.infra.enos_g5k.g5k_api_utils.grid_get_or_create_job(job_name, walltime, reservation_date, queue, job_type, monitor, project, machines, networks, wait=True, restrict_to: Iterable[str] | None = None) → List[Job]#

enoslib.infra.enos_g5k.g5k_api_utils.grid_make_reservation(job_name: str, walltime, reservation_date, queue, job_type: str | MutableSequence[str], monitor, project, machines: Iterable, networks: Iterable) → List[Job]#

enoslib.infra.enos_g5k.g5k_api_utils.grid_reload_from_ids(oargrid_jobids: Iterable[Tuple]) → List[Job]#

Reload all running or pending jobs of Grid’5000 from their ids

Parameters:: oargrid_jobids (list) – list of (site, oar_jobid) identifying the jobs on each site
Returns:: The list of python-grid5000 jobs retrieved

enoslib.infra.enos_g5k.g5k_api_utils.grid_reload_jobs_from_ids(oargrid_jobids: Iterable[Tuple]) → List[Job]#

Reload jobs of Grid’5000 from their ids

Parameters:: oargrid_jobids (list) – list of (site, oar_jobid) identifying the jobs on each site
Returns:: The list of python-grid5000 jobs retrieved

enoslib.infra.enos_g5k.g5k_api_utils.grid_reload_jobs_from_name(job_name: str, restrict_to: Iterable[str] | None = None) → List[Job]#

Reload all running or pending jobs of Grid’5000 with a given name.

By default, all the sites will be searched for jobs with the name job_name. Using EnOSlib there can be only one job per site with name job_name.

Note that it honors the exluded_sites attribute of the client so the scan can be reduced.

Parameters:

job_name (str) – the job name
restrict_to – restrict the action to these sites only. If None is passed, no restriction applies and the action will be taken on all possible sites

Returns:

The list of the python-grid5000 jobs retrieved.

Raises:

EnosG5kDuplicateJobsError – if there’s several jobs with the same name on a site.

enoslib.infra.enos_g5k.g5k_api_utils.job_delete(job: Job, wait: bool = False)#

enoslib.infra.enos_g5k.g5k_api_utils.set_nodes_vlan(nodes: List[str], interface: str, vlan_id: str)#

Set the interface of the nodes in a specific vlan.

All nodes need to belong to the same Grid’5000 site.

It is assumed that the same interface name is available on all nodes.

Parameters:

nodes (list) – nodes to consider
interface (str) – the network interface to put in the vlan
vlan_id (str) – the id of the vlan

Raises:

EnosG5kInvalidArgument – if not all nodes belong to the same site.
EnosG5kKavlanNodesError – if some nodes couldn’t be added to the VLAN.

enoslib.infra.enos_g5k.g5k_api_utils.submit_jobs(job_specs: List[Tuple]) → List[Job]#

Submit a job

Parameters:: job_specs (dict) – The job specification (see Grid’5000 API reference)

enoslib.infra.enos_g5k.g5k_api_utils.to_prod_nature() → str#

enoslib.infra.enos_g5k.g5k_api_utils.to_subnet_nature(cidr: str) → str#

enoslib.infra.enos_g5k.g5k_api_utils.to_vlan_nature(vlan_id: str) → str#

enoslib.infra.enos_g5k.g5k_api_utils.wait_for_jobs(jobs: Iterable)#

Waits for all the jobs to be runnning.

Parameters:: jobs (list) – list of the python-grid5000 jobs to wait for
Raises:: Exception – if one of the job gets in error state.

Virtual Machines on Grid5000 (VMonG5k)#

VMonG5k Provider Class#

Classes:

`VMonG5k`(args, *kwargs)	The provider to use when deploying virtual machines on Grid'5000.
`VirtualMachine`(name, eui, flavour_desc, pm)	Internal data structure to manipulate virtual machines.

Functions:

`mac_range`(g5k_subnets[, skip, step])	Generator function to get some macs out of G5k subnets
`start_virtualmachines`(provider_conf[, ...])	Starts virtualmachines on G5K.

class enoslib.infra.enos_vmong5k.provider.VMonG5k(*args, **kwargs)#

The provider to use when deploying virtual machines on Grid’5000.

async_init(start_time: int | None = None, force_deploy: bool = False, **kwargs)#

Partial init: secure the resources to the targeted infrastructure.

Parameters:

kwargs – keyword arguments. Fit those from init()

Raises:

InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception

destroy(wait: bool = False, **kwargs)#: Destroy the underlying job.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs) → Tuple[Roles, Networks]#

Abstract. Provides resources and provisions the environment.

This calls the underlying provider and provision resources (machines and networks).

Parameters:

force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.

Returns:

roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see Enos_vagrant

Return type:

(roles, networks) tuple

is_created() → bool#: Is the provider already created.

offset_walltime(difference: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

set_reservation(timestamp: int)#

Change the internal reservation date.

Ignored on platform that aren’t based on a reservation system.

Parameters:: timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.

test_slot(start_time: int, end_time: int) → bool#: Test if it is possible to reserve resources at start_time

undercloud() → Tuple[Roles | None, Networks | None]#: Gets the undercloud information (bare-metal machines).

class enoslib.infra.enos_vmong5k.provider.VirtualMachine(name: str | None, eui: EUI, flavour_desc: Mapping, pm, extra: Dict | None = None, extra_devices='')#: Internal data structure to manipulate virtual machines.

enoslib.infra.enos_vmong5k.provider.mac_range(g5k_subnets: List[G5kEnosSubnetNetwork], skip: int = 0, step: int = 1) → Generator[str, None, None]#

Generator function to get some macs out of G5k subnets

Parameters:

g5k_subnets – a list of g5k subnets (see :py:class::~enoslib.infra.enos_g5k.objects.G5kEnosSubnetNetwork)
skip – skip this amount of macs
step – step as in built-in range method

Returns:

An iterator of mac addresses

enoslib.infra.enos_vmong5k.provider.start_virtualmachines(provider_conf: Configuration, force_deploy: bool = False) → Roles#

Starts virtualmachines on G5K.

This first distributes the virtual machine according to the undercloud attributes of the configuration, assign them IPs and start them. It is idempotent.

Parameters:

provider_conf – This is the abstract description of your overcloud (VMs). Each configuration must have its undercloud attributes filled with the undercloud machines to use. Round Robin strategy to distribute the VMs to the PMs will be used for each configuration. Mac addresses will be generated according to the g5k_subnet parameter.
g5k_subnets – The subnets to use. Each element is a serialization of enoslib.infra.enos_vmong5k.configuration.NetworkConfiguration
skip – number of addresses to skip when distributing them to the virtual machines. This can be useful when starting incrementally the virtual machines to avoid overlaping ip assignments between iterations.
force_deploy (boolean) – controls whether the virtual machines should be restarted from scratch.

Returns:

roles

Examples

import logging
from itertools import islice
from pathlib import Path

import enoslib as en

en.init_logging(level=logging.INFO)
en.check()

job_name = Path(__file__).name

CLUSTER = "parasilo"
SITE = "rennes"


conf = (
    en.G5kConf.from_settings(job_type=[], job_name=job_name)
    .add_network(
        id="not_linked_to_any_machine", type="slash_22", roles=["my_subnet"], site=SITE
    )
    .add_machine(roles=["role1"], cluster=CLUSTER, nodes=1)
    .add_machine(roles=["role2"], cluster=CLUSTER, nodes=1)
)

provider = en.G5k(conf)
roles, networks = provider.init()
roles = en.sync_info(roles, networks)

# Retrieving subnets
subnet = networks["my_subnet"]
logging.info(subnet)

# We describe the VMs types and placement in the following
# We build a VMonG5KConf with some extra fields:
# - undercloud: where the VMs should be placed (round-robin)
# - macs: list of macs to take: on G5k the dhcp is configured to assign specific
#   ip based on the configured mac

n_vms = 16
virt_conf = (
    en.VMonG5kConf.from_settings(image="/grid5000/virt-images/debian11-x64-base.qcow2")
    # Starts some vms on a single role
    # Here that means start the VMs on a single machine
    .add_machine(
        roles=["vms"],
        number=n_vms,
        undercloud=roles["role1"],
        macs=list(islice(subnet[0].free_macs, n_vms))
        # alternative
        # macs=list(islice(en.mac_range(subnet), n_vms))
    )
)

# Start them
vmroles = en.start_virtualmachines(virt_conf)
print(vmroles)
print(networks)

VMonG5k Schema#

VMonG5k schema.
type	object
properties
enable_taktuk	Use TakTuk to distribute the VM image
	type	boolean
force_deploy	Remove and restart all virtual machines
	type	boolean
job_name	Name of the job (default: EnOslib-vmong5k)
	type	string
queue	Grid’5000 queue to use (default: default)
	type	string
	enum	default, production, testing, besteffort
walltime	Job duration (default: 02:00:00)
	type	string
image	Path to the base image on the reserved nodes
	type	string
skip	Skip this number of IPs
	type	number
strategy	Base image strategy (default: cow)
	type	string
	enum	copy, cow
subnet_type	Subnet type to use (default: slash_22)
	type	string
	enum	slash_16, slash_22
working_dir	Remote working directory (default: /tmp/enos_vmong5k)
	type	string
domain_type	Domain type of the guest (default: kvm)
	type	string
reservation	Reservation date %Y-%m-%d %H:%M:%S (Paris Timezone)
	type	string
resources	VMonG5k Resource
	type	object
	properties
	machines	type	array
		items	VMonG5k Compute
	networks	type	array
		items	type	string
	additionalProperties	False
additionalProperties	False

VMonG5k Compute#

VMonG5k Machine description
type	object
properties
roles	EnOSlib’s roles
	type	array
	items	type	string
number	Number of VMs (default: 1)
	type	number
flavour	Predefined flavour (default: (‘tiny’, {‘core’: 1, ‘mem’: 512}))
	type	string
	enum	tiny, small, medium, big, large, extra-large
flavour_desc	Custom flavour description
	VMonG5k Flavour
vcore_type	Type of vcore (default: thread)
	type	string
	enum	thread, core
cluster	Grid’5000 cluster for the undercloud
	type	string
undercloud	List of Host where the VM should be started.
	type	array
	items	type	object
macs	List of MAC addresses to use for the vms
	type	array
	items	type	string
		format	mac
extra_devices	Libvirt XML description for extra devices.
	type	string
additionalProperties	False

VMonG5k Flavour#

Custom flavour for a virtual machine.
type	object
properties
core	number of cores
	type	number
mem	memory size in MB
	type	number
disk	disk size in GB
	type	number
additionalProperties	False

Containers on Grid5000 (Distem)#

Distem Provider Class#

Classes:

Distem(provider_conf[, name])

Use Distem on G5k

class enoslib.infra.enos_distem.provider.Distem(provider_conf, name: str | None = None)#

Use Distem on G5k

async_init(start_time: int | None = None, **kwargs)#

Partial init: secure the resources to the targeted infrastructure.

Parameters:

kwargs – keyword arguments. Fit those from init()

Raises:

InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception

destroy(wait=False)#: Abstract. Destroy the resources used for the deployment.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs)#

Abstract. Provides resources and provisions the environment.

This calls the underlying provider and provision resources (machines and networks).

Parameters:

force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.

Returns:

roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see Enos_vagrant

Return type:

(roles, networks) tuple

is_created()#: Is the provider already created.

offset_walltime(offset: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

set_reservation(timestamp: int)#

Change the internal reservation date.

Ignored on platform that aren’t based on a reservation system.

Parameters:: timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.

test_slot(start_time: int, end_time: int) → bool#: Test if it is possible to reserve the configuration corresponding to this provider at start_time

Distem Schema#

type	object
properties
job_name	The job name (default: EnOslib-distem)
	type	string
queue	Queue to use (default: default)
	type	string
	enum	default, production, testing, besteffort
walltime	Job duration (default: 02:00:00)
	type	string
image	Default base image to use
	type	string
reservation	reservation date in YYYY-mm-dd HH:MM:SS format
	type	string
	format	reservation
force_deploy	Clean the existing containers beforehand (default: False)
	type	boolean
resources	Distem Resource
	type	object
	properties
	machines	type	array
		items	Distem Compute
	networks	type	array
		items	type	string
	additionalProperties	False
additionalProperties	False

Distem Compute#

type	object
properties
roles	The concrete resources will be assigned this role
	type	array
	items	type	string
number	Number of containers (default: 1)
	type	number
flavour	Predefined flavour (default: (‘tiny’, {‘core’: 1, ‘mem’: 512}))
	type	string
	enum	tiny, small, medium, big, large, extra-large
flavour_desc	Distem Flavour
vcore_type	Type of vcore (default: thread)
	type	string
	enum	thread, core
cluster	type	string
undercloud	type	array
	items	type	object
additionalProperties	False

Distem Flavour#

Custom flavour/size for your container
type	object
properties
core	type	number
mem	type	number
additionalProperties	False

FIT/IoT-LAB#

FIT/IoT-LAB Schema#

FIT/IoT-LAB configuration#

type	object
properties
job_name	Name of the job (default: EnOSlib)
	type	string
walltime	Job duration (default: 00:01)
	type	string
	format	walltime
start_time	start time in YYYY-mm-dd HH:MM:SS format
	type	string
	format	start_time
resources	Resource
	type	object
	properties
	machines	oneOf	type	array
			items	FIT/IoT-LAB nodes selected by name
			minItems	1
			type	array
			items	FIT/IoT-LAB boards selected by architecture
			minItems	1
	networks	type	array
		items	FIT/IoT-LAB Network
		uniqueItems	True
	additionalProperties	False
monitoring	FIT/IoT-LAB Monitoring profiles
	type	object
	properties
	profiles	type	array
		items	FIT/IoT-LAB Radio and Consumption profiles
		minItems	1
	additionalProperties	False
additionalProperties	False

FIT/IoT-LAB boards selected by architecture#

type	object
properties
roles	The concrete resources will be assigned this role
	type	array
	items	type	string
archi	Architecture to use
	type	string
site	Site to use
	type	string
number	Number of boards (defaut: 1)
	type	number
image	Firmware to use
	type	string
profile	profile name to use
	type	string

FIT/IoT-LAB nodes selected by name#

type	object
properties
roles	The concrete resources will be assigned this role
	type	array
	items	type	string
hostname	Exact name of physical nodes to use
	type	array
	format	hostname
	items	type	string
	minItems	1
image	Firmware to use
	type	string
profile	profile name to use
	type	string

FIT/IoT-LAB Network#

type	object
properties
description	Network to use
id	type	string
type	enum	prod
roles	type	array
	items	type	string
site	type	string

FIT/IoT-LAB Radio and Consumption profiles#

type	object
properties
name	The profile name
	type	string
archi	Target architecture
	type	string
	enum	m3, a8, custom
radio	type	object
	properties
	mode	type	string
		enum	rssi, sniffer
	num_per_channel	type	integer
		maximum	255
		minimum	0
	period	type	integer
		maximum	65535
		minimum	1
	channels	type	array
		items	type	integer
			maximum	26
			minimum	11
	additionalProperties	False
consumption	type	object
	properties
	current	type	boolean
	power	type	boolean
	voltage	type	boolean
	period	type	integer
		enum	140, 204, 332, 588, 1100, 2116, 4156, 8244
	average	type	integer
		enum	1, 4, 16, 64, 128, 256, 512, 1024
	additionalProperties	False
additionalProperties	False

Sensor#

import socket
from typing import Dict, List, Optional

import iotlabcli.auth
import sshtunnel

from enoslib.api import play_on
from enoslib.infra.enos_iotlab.iotlab_api import IotlabAPI
from enoslib.infra.enos_iotlab.sensor import Sensor
from enoslib.log import getLogger
from enoslib.objects import DefaultNetwork, Host

logger = getLogger(__name__, ["IOTlab"])


def ssh_enabled(network_address: str) -> bool:
    return network_address.startswith("a8") or network_address.startswith("rpi")


class IotlabHost(Host):
    """
    A IoT-LAB host

    IoT-LAB has several boards with different characteristics.
    However, only A8 nodes are able to receive ssh connections
    and run linux commands.
    """

    def __init__(
        self,
        address: str,
        roles: List[str],
        site: str,
        uid: str,
        archi: str,
    ):
        super().__init__(address, user="root")
        # read only attributes
        self.roles = roles
        self.site = site
        self.uid = uid
        self.archi = archi
        self._ssh_address: Optional[str] = None

        if ssh_enabled(self.archi):
            self._ssh_address = f"node-{self.address}"

    @property
    def ssh_address(self) -> Optional[str]:
        """Get an SSH reachable address for this Host.

        Returns:
            str: The address as a string.
        """
        return self._ssh_address

    def __repr__(self) -> str:
        return (
            "<IotlabHost("
            f"roles={self.roles}, "
            f"address={self.address}, "
            f"ssh_address={self.ssh_address}, "
            f"site={self.site}, "
            f"uid={self.uid})>"
        )


class IotlabSensor(Sensor):
    """A IoT-LAB sensor"""

    def __init__(
        self,
        address: str,
        roles: List[str],
        site: str,
        uid: str,
        archi: str,
        image: str,
        iotlab_client: IotlabAPI,
    ):
        alias = address.split(".")[0]
        super().__init__(address, alias)
        # read only attributes
        self.roles: List[str] = roles
        self.site: str = site
        self.uid: str = uid
        self.archi: str = archi
        self.image: str = image
        self.iotlab_client: IotlabAPI = iotlab_client

        self.user, self.passwd = iotlabcli.auth.get_user_credentials()
        self.exp_id = self.iotlab_client.get_job_id()

    def __repr__(self) -> str:
        return (
            "<IotlabSensor("
            f"roles={self.roles}, "
            f"address={self.address}, "
            f"site={self.site}, "
            f"uid={self.uid})>"
            f"image={self.image})>"
        )

    def stop(self):
        """
        Stops this sensor
        """
        self.iotlab_client.send_cmd_node(cmd="stop", nodes=[self.address])

    def start(self):
        """
        Starts this sensor
        """
        self.iotlab_client.send_cmd_node(cmd="start", nodes=[self.address])

    def reset(self):
        """
        Resets this sensor
        """
        self.iotlab_client.send_cmd_node(cmd="reset", nodes=[self.address])

    def to_dict(self) -> Dict:
        d = super().to_dict()
        d.update(
            roles=self.roles,
            site=self.site,
            uid=self.uid,
            archi=self.archi,
            image=self.image,
        )
        return d


class IotlabNetwork(DefaultNetwork):
    """Iotlab network class."""

    def __init__(self, roles: List[str], *args, **kargs):
        super().__init__(*args, **kargs)
        self.roles = roles


class IotlabSerial:
    def __init__(
        self,
        sensor: IotlabSensor,
        serial_port: int = 20000,
        interactive: bool = False,
        timeout: int = 5,
    ):
        """
        Create a serial connection to a sensor in IoT-LAB testbed

        The serial has 2 possibilities:
        1. Interactive: where you can interact with the serial,
        reading and sending commands to it
        2. Logging (default): only for logging purposes, collect the
        output and save it to the file (.iot-lab/<exp_id>/log/<node>-serial.log).
        No command write/read is allowed in this mode.

        More details in open_serial_conn/enable_logging methods.

        Args:
            sensor: Sensor object
            serial_port: serial port to connect to sensor
            interactive: set to true to use write/read methods
            timeout: Timeout for socket connection
        """
        self.sensor = sensor
        self.interactive = interactive
        self.serial_port = serial_port
        self.timeout = timeout
        self._serial_tunnel: Optional[sshtunnel.SSHTunnelForwarder] = None
        self._serial_socket: Optional[socket.socket] = None
        self._filename = f"~/.iot-lab/{self.sensor.exp_id}/log/{self.sensor.alias}_serial.log"  # noqa

    def open_serial_conn(self):
        """
        Opens serial connection to serial node.

        This method creates a SSH tunnel to frontend and a socket
        to the local port created by the SSH tunnel.

        The sshtunnel library will create a thread to do process the forwarded packets.

        Note:
            Remember to call close_serial_conn, in order to stop
            the thread and close the connection properly.
        """

        self._serial_tunnel = sshtunnel.SSHTunnelForwarder(
            self.sensor.site + ".iot-lab.info",
            ssh_username=self.sensor.user,
            ssh_password=self.sensor.passwd,
            remote_bind_address=(self.sensor.address, self.serial_port),
        )

        self._serial_tunnel.start()

        self._serial_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self._serial_socket.connect(("127.0.0.1", self._serial_tunnel.local_bind_port))
        self._serial_socket.settimeout(self.timeout)

        msg = """
 IotlabSensor(%s): Opening SSH tunnel for serial connection: \
 server (%s) remote (%s:%s) local_port: %s"""
        logger.info(
            msg,
            self.sensor.alias,
            self.sensor.site,
            self.sensor.address,
            self.serial_port,
            self._serial_tunnel.local_bind_port,
        )

    def close_serial_conn(self):
        """
        Close serial connection.

        Stops thread created by sshtunnel library.
        """
        if self._serial_socket:
            self._serial_socket.close()
            self._serial_socket = None

        if self._serial_tunnel:
            self._serial_tunnel.stop()
            self._serial_tunnel = None

    def enable_logging_serial(self):
        """
        Enables logging of serial output to a file

        Runs serial_aggregator tool in frontend to write serial output
        to a file.

        Output file is saved in the experiment folder at:
        .iot-lab/<exp_id>/log/<node>-serial.log.

        Note:
            Remember to call disable_logging_serial, in order to stop
            serial_aggregator tool running on the frontend.
        """
        # convert to seconds
        wall_time = self.sensor.iotlab_client.get_walltime()
        if wall_time is not None:
            timeout = wall_time * 60

            with play_on(
                roles=[Host(f"{self.sensor.site}.iot-lab.info", user=self.sensor.user)]
            ) as p:
                cmd = f"screen -dm bash -c 'serial_aggregator -l {self.sensor.site},{self.sensor.alias.replace('-', ',')} > {self._filename} 2>&1'"  # noqa
                p.shell(
                    cmd, task_name="Running serial_aggregator", asynch=timeout, poll=0
                )
        else:
            logger.error("Wall time is None.")

    def disable_logging_serial(self):
        """
        Disables logging of serial output.

        Stops serial_aggregator tool running on the frontend
        """
        with play_on(
            roles=[Host(f"{self.sensor.site}.iot-lab.info", user=self.sensor.user)],
            on_error_continue=True,
        ) as p:
            cmd = "pkill -f 'serial_aggregator -l {},{} > {} 2>&1'".format(
                self.sensor.site,
                self.sensor.alias.replace("-", ","),
                self._filename,
            )
            p.command(cmd, task_name="Killing serial_aggregator")

    def __enter__(self) -> "IotlabSerial":
        if self.interactive:
            self.open_serial_conn()
        else:
            self.enable_logging_serial()

        return self

    def __exit__(self, exc_type, exc_value, exc_traceback):
        if self.interactive:
            self.close_serial_conn()
        else:
            self.disable_logging_serial()

    def write(self, content: str):
        """
        Sends string on serial interface

        Args:
            content: String to be sent
        """
        if not self.interactive:
            logger.error("Not in interactive mode, impossible to write on serial")
            return

        logger.info("IotlabSerial(%s): Writing: %s", self.sensor.alias, content)
        if self._serial_socket is not None:
            self._serial_socket.sendall(content.encode())
        else:
            logger.error("Serial Socket is closed.")

    def read(self, size: int = 1024) -> str:
        """
        Reads string from serial interface

        Args:
            size: Maximum string size to be received

        Returns:
            str: String received
        """
        data = b""

        if not self.interactive:
            logger.error("Not in interactive mode, impossible to read serial")
            return data.decode()

        if self._serial_socket is not None:
            try:
                data = self._serial_socket.recv(size)
            except socket.timeout:
                pass
        return data.decode()


class IotlabSniffer:
    def __init__(self, sensor: IotlabSensor, timeout: int = -1):
        """
        Create a sniffer to a sensor in IoT-LAB testbed

        Runs the sniffer_aggregator tool in the frontend node
        to collect radio packets.

        Pcap is saved in the experiment folder at:
        .iot-lab/<exp_id>/sniffer/<node>.pcap.

        Args:
            sensor: Sensor object
            timeout: Timeout for sniffer_aggregator command (-1 will run
            until the experiment was finished (walltime cfg))
        """
        self.sensor = sensor
        self.timeout = timeout
        wall_time = self.sensor.iotlab_client.get_walltime()
        if timeout == -1 and wall_time is not None:
            # convert to seconds
            self.timeout = wall_time * 60
        self._filename = (
            f"~/.iot-lab/{self.sensor.exp_id}/sniffer/{self.sensor.alias}.pcap"
        )

    def start_sniffer(self):
        """
        Starts radio sniffing

        Run sniffer_aggregator tool in frontend to capture packets in
        this sniffer node. The sniffer will run for "timeout" seconds,
        or until the stop_sniffer method is called.

        Pcap is saved in the experiment folder at:
        .iot-lab/<exp_id>/sniffer/<node>.pcap.

        Note:
            Remember to call stop_sniffer, in order to stop
            sniffer_aggregator tool running on the frontend.
        """

        with play_on(
            roles=[Host(self.sensor.site + ".iot-lab.info", user=self.sensor.user)]
        ) as p:
            cmd = "sniffer_aggregator -l {},{} -o {}".format(
                self.sensor.site,
                self.sensor.alias.replace("-", ","),
                self._filename,
            )
            p.shell(
                cmd,
                task_name="Running sniffer_aggregator",
                asynch=self.timeout,
                poll=0,
            )

    def stop_sniffer(self):
        """
        Stops radio sniffing

        Kills sniffer_aggregator tool running in the frontend.
        """
        with play_on(
            roles=[Host(f"{self.sensor.site}.iot-lab.info", user=self.sensor.user)],
            on_error_continue=True,
        ) as p:
            cmd = f'pkill -f "{self._filename}"'
            p.command(cmd, task_name="Killing sniffer_aggregator")

    def __enter__(self) -> "IotlabSniffer":
        self.start_sniffer()
        return self

    def __exit__(self, exc_type, exc_value, exc_traceback):
        self.stop_sniffer()

Openstack#

Openstack Provider Class#

Get resources from an Openstack [1] private cloud.

Openstack is a widely adopted Open Source Cloud Software. It powers infrastructure provider such as:

OVH: http://ovh.com
Chameleon Cloud: https://www.chameleoncloud.org/
Gulliver: https://gulliver.saclay.inria.fr/
and many others.

The supported deployment model lets you get a bunch of compute servers, a single private network. One of your server can act as a gateway and will be assigned a floating IP (EnOSlib hosts will be configured accordingly).

Classes:

Openstack(provider_conf[, name])

class enoslib.infra.enos_openstack.provider.Openstack(provider_conf, name: str | None = None)#

destroy(wait: bool = False, **kwargs)#: Abstract. Destroy the resources used for the deployment.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs) → Tuple[Roles, Networks]#

Abstract. Provides resources and provisions the environment.

This calls the underlying provider and provision resources (machines and networks).

Parameters:

force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.

Returns:

roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see Enos_vagrant

Return type:

(roles, networks) tuple

offset_walltime(difference: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

Openstack Schema#

SCHEMA = {
    "type": "object",
    "properties": {
        "resources": {"$ref": "#/resources"},
        # Mandatory keys
        "key_name": {"type": "string"},
        "image": {"type": "string"},
        "user": {"type": "string"},
        "rc_file": {"type": "string"},
        # optional keys
        "allocation_pool": {"$ref": "#/os_allocation_pool"},
        "configure_network": {"type": "boolean"},
        "dns_nameservers": {"type": "array", "items": {"type": "string"}},
        "gateway": {"type": "boolean"},
        "gateway_user": {"type": "string"},
        "network": {"$ref": "#/os_network"},
        "subnet": {"$ref": "#/os_subnet"},
        "prefix": {"type": "string"},
    },
    "additionalProperties": False,
    "required": ["resources", "key_name", "image", "user", "rc_file"],
    "os_allocation_pool": {
        "title": "OSallocationPool",
        "type": "object",
        "properties": {"start": {"type": "string"}, "end": {"type": "string"}},
        "required": ["start", "end"],
        "additionalProperties": False,
    },
    "os_network": {
        "title": "OSNetwork",
        "type": "object",
        "properties": {"name": {"type": "string"}},
        "required": ["name"],
        "additionalProperties": False,
    },
    "os_subnet": {
        "title": "OSSubnet",
        "type": "object",
        "properties": {"name": {"type": "string"}, "cidr": {"type": "string"}},
        "required": ["name", "cidr"],
        "additionalProperties": False,
    },
    "resources": {
        "title": "Resource",
        "type": "object",
        "properties": {
            "machines": {"type": "array", "items": {"$ref": "#/machine"}},
            "networks": {"type": "array", "items": {"type": "string"}},
        },
        "additionalProperties": False,
        "required": ["machines", "networks"],
    },
    "machine": {
        "title": "Compute",
        "type": "object",
        "properties": {
            "roles": {"type": "array", "items": {"type": "string"}},
            "flavour": {"type": "string"},
            "number": {"type": "number"},
        },
        "required": ["roles", "number", "flavour"],
    },
}

Chameleon#

Chameleon(kvm) Provider Class#

Classes:

Chameleonkvm(provider_conf[, name])

class enoslib.infra.enos_chameleonkvm.provider.Chameleonkvm(provider_conf, name: str | None = None)#

destroy(wait: bool = False, **kwargs)#: Abstract. Destroy the resources used for the deployment.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs) → Tuple[Roles, Networks]#

Abstract. Provides resources and provisions the environment.

This calls the underlying provider and provision resources (machines and networks).

Parameters:

force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.

Returns:

roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see Enos_vagrant

Return type:

(roles, networks) tuple

offset_walltime(offset: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

Chameleon(kvm) Schema#

SCHEMA = {
    "type": "object",
    "properties": {
        "resources": {"$ref": "#/resources"},
        "key_name": {"type": "string"},
        # everything is optional
        "image": {"type": "string"},
        "user": {"type": "string"},
        "allocation_pool": {"$ref": "#/os_allocation_pool"},
        "configure_network": {"type": "boolean"},
        "dns_nameservers": {"type": "array", "items": {"type": "string"}},
        "gateway": {"type": "boolean"},
        "gateway_user": {"type": "string"},
        "network": {"$ref": "#/os_network"},
        "subnet": {"$ref": "#/os_subnet"},
        "prefix": {"type": "string"},
    },
    "additionalProperties": True,
    "required": ["resources", "key_name"],
    "os_allocation_pool": {
        "title": "OSallocationPool",
        "type": "object",
        "properties": {"start": {"type": "string"}, "end": {"type": "string"}},
        "required": ["start", "end"],
        "additionalProperties": False,
    },
    "os_network": {
        "title": "OSNetwork",
        "type": "object",
        "properties": {"name": {"type": "string"}},
        "required": ["name"],
        "additionalProperties": False,
    },
    "os_subnet": {
        "title": "OSSubnet",
        "type": "object",
        "properties": {"name": {"type": "string"}, "cidr": {"type": "string"}},
        "required": ["name"],
        "additionalProperties": False,
    },
    "resources": {
        "title": "Resource",
        "type": "object",
        "properties": {
            "machines": {"type": "array", "items": {"$ref": "#/machine"}},
            "networks": {"type": "array", "items": {"type": "string"}},
        },
        "additionalProperties": False,
        "required": ["machines", "networks"],
    },
    "machine": {
        "title": "Compute",
        "type": "object",
        "properties": {
            "roles": {"type": "array", "items": {"type": "string"}},
            "flavour": {"type": "string"},
            "number": {"type": "number"},
        },
        "required": ["roles", "number", "flavour"],
    },
}

Chameleon(bare metal) Provider Class#

Classes:

Chameleonbaremetal(provider_conf[, name])

class enoslib.infra.enos_chameleonbaremetal.provider.Chameleonbaremetal(provider_conf, name: str | None = None)#

destroy(wait: bool = False, **kwargs)#: Abstract. Destroy the resources used for the deployment.

init(force_deploy: bool = False, start_time: int | None = None, **kwargs)#

Abstract. Provides resources and provisions the environment.

This calls the underlying provider and provision resources (machines and networks).

Parameters:

force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.

Returns:

roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see Enos_vagrant

Return type:

(roles, networks) tuple

offset_walltime(offset: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

set_reservation(timestamp: int)#

Change the internal reservation date.

Ignored on platform that aren’t based on a reservation system.

Parameters:: timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.

Chameleon(bare metal) schema#

SCHEMA = {
    "type": "object",
    "properties": {
        "resources": {"$ref": "#/resources"},
        "key_name": {"type": "string"},
        # everything is optional
        "image": {"type": "string"},
        "user": {"type": "string"},
        "allocation_pool": {"$ref": "#/os_allocation_pool"},
        "configure_network": {"type": "boolean"},
        "dns_nameservers": {"type": "array", "items": {"type": "string"}},
        "gateway": {"type": "boolean"},
        "gateway_user": {"type": "string"},
        "network": {"$ref": "#/os_network"},
        "subnet": {"$ref": "#/os_subnet"},
        "prefix": {"type": "string"},
    },
    "additionalProperties": True,
    "required": ["resources", "key_name"],
    "os_allocation_pool": {
        "title": "OSallocationPool",
        "type": "object",
        "properties": {"start": {"type": "string"}, "end": {"type": "string"}},
        "required": ["start", "end"],
        "additionalProperties": False,
    },
    "os_network": {
        "title": "OSNetwork",
        "type": "object",
        "properties": {"name": {"type": "string"}},
        "required": ["name"],
        "additionalProperties": False,
    },
    "os_subnet": {
        "title": "OSSubnet",
        "type": "object",
        "properties": {"name": {"type": "string"}, "cidr": {"type": "string"}},
        "required": ["name"],
        "additionalProperties": False,
    },
    "resources": {
        "title": "Resource",
        "type": "object",
        "properties": {
            "machines": {"type": "array", "items": {"$ref": "#/machine"}},
            "networks": {"type": "array", "items": {"type": "string"}},
        },
        "additionalProperties": False,
        "required": ["machines", "networks"],
    },
    "machine": {
        "title": "Compute",
        "type": "object",
        "properties": {
            "roles": {"type": "array", "items": {"type": "string"}},
            "flavour": {"type": "string"},
            "number": {"type": "number"},
        },
        "required": ["roles", "number", "flavour"],
    },
}

Providers Class#

Classes:

`NoneConfiguration`()
`Providers`(providers)	A provider that syncs resources of different infrastructures.

Functions:

`find_slot`(providers, time_window, start_time)	Search for a time slot at which all of the provider in "providers" are able to reserve their configurations time_window is how long in the future are we willing to be looking for start_time is when we start trying to look for a slot, by default a minute after the function is called
`find_slot_and_start`(providers, start_time, ...)	Try to find a common time slot for all the Provider in providers to start and then start them
`start_provider_within_bounds`(provider, ...)	Adjust provider walltime and reservation_date to fit into a slot

class enoslib.infra.providers.NoneConfiguration#

Bases: object

Methods:

finalize()

finalize()#

class enoslib.infra.providers.Providers(providers: List[Provider])#

Bases: Provider

A provider that syncs resources of different infrastructures.

Parameters:: providers – List of Provider instances that you wish to use

Methods:

`async_init`([force_deploy, start_time, ...])	Partial init: secure the resources to the targeted infrastructure.
`destroy`([wait])	Abstract.
`init`([force_deploy, start_time, time_window])	The provider to use when you want to sync multiple providers.
`is_created`()	Is the provider already created.
`offset_walltime`(offset)	Offset the walltime.
`set_reservation`(timestamp)	Change the internal reservation date.
`test_slot`(start_time, end_time)	Test a slot that starts at a given point in time.

async_init(force_deploy: bool = False, start_time: int | None = None, time_window: int | None = None, **kwargs)#

Partial init: secure the resources to the targeted infrastructure.

Parameters:

kwargs – keyword arguments. Fit those from init()

Raises:

InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception

destroy(wait: bool = True, **kwargs)#: Abstract. Destroy the resources used for the deployment.

init(force_deploy: bool = False, start_time: int | None = None, time_window: int | None = None, **kwargs) → Tuple[Roles, Networks]#

The provider to use when you want to sync multiple providers.

This will call init on each provider after finding a common possible reservation date for each one of them. It uses find_slot() and start_provider_within_bounds() internally.

Idempotency: ideally calling this function twice should reload existing reservations on each platform. However, the current behaviour might differ from this specification but we’ll be happy to get your feedback on this.

Parameters:

time_window – duration in seconds How long in the future are you willing to look for a possible start time
start_time – timestamp in seconds The first start_time you will test, incremented after each try (5 minutes increment)

Returns:

Providers’ roles and networks similar to init() return value.

Raises:

NoSlotError – If no common slot can be found

is_created() → bool#: Is the provider already created.

offset_walltime(offset: int)#

Offset the walltime.

Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.

Raises:: NegativeWalltime – the walltime is negative

set_reservation(timestamp: int)#

Change the internal reservation date.

Ignored on platform that aren’t based on a reservation system.

Parameters:: timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.

test_slot(start_time: int, end_time: int) → bool#

Test a slot that starts at a given point in time.

A slot is given by a start_time, a duration and an amount of resources. The two latter are found in the internal configuration.

Parameters:

start_time (timestamp (int)) – Test for a possible slot that starts at start_time (UTC)
end_time – (timestamp (int)): How much time in the future we should look for possible reservation. This is used on some platform to know how much time in the future we look in the planning data. Timestamp is based on UTC.

Returns:

True iff the slot is available

enoslib.infra.providers.find_slot(providers: Sequence[Provider], time_window: int, start_time: int) → int#

Search for a time slot at which all of the provider in “providers” are able to reserve their configurations time_window is how long in the future are we willing to be looking for start_time is when we start trying to look for a slot, by default a minute after the function is called

Parameters:

providers – A list of providers
time_window – How long in the future are you willing to look for for a start time Must be positive.
start_time – The first start_time you will test, incremented after each try (5 minutes increment). Must be positive.

Raises:

NoSlotError – If no compatible slot can be found for all provided providers

enoslib.infra.providers.find_slot_and_start(providers: Sequence[Provider], start_time: int, time_window: int, **kwargs)#

Try to find a common time slot for all the Provider in providers to start and then start them

This search for a reservation date that will fit all provider in providers and the call do_init with them on that found reservation date If this fail it will try to raise an error indicating a next possible slot if possible

Raises:

InvalidReservationTime – Happens if one of the provider cannot be initialized
with reservation_timestamp as reservation date. Will provide an indication –
for a potential next possible time slot –

enoslib.infra.providers.start_provider_within_bounds(provider: Provider, start_time: int, **kwargs)#

Adjust provider walltime and reservation_date to fit into a slot

Mutate the reservation/walltime attributes until finding a slot within [start_time, start_time + provider.walltime] where the provider can be started.

The slot found is guaranteed to:

not exceed the right bound (a negative walltime would raise an error) in the current implementation, we chose to work with a fixed end time.
and start in the future: start_time might be in the past, and reserving resources in the past is weird (and not allowed in some testbeds). We do our best to fiddle with the start_time to make sure it’s in the future. If not we retry with an even further start_time.

Finding the slot depends on a loop on error strategy which is stopped when the retry limit is hit or the walltime become to small. In both cases NoSlotError is raised. Otherwise the same errors as init() can be raised (except InvalidReservationTooOld which can be caught internally)

Raises:

InvalidReservationTime – If a provider object cannot be initialized anymore, due to an update to it’s related platform status since we first fetched it
NoSlotError – If a Providers object cannot be initialized at its given start_time or if a provider fails to be initialized after too many retries.