Provider APIs#
Base Provider Class#
Classes:
|
Base class for the provider. |
- class enoslib.infra.provider.Provider(provider_conf, name: str | None = None)#
Bases:
object
Base class for the provider.
Providers are components that are responsible for acquiring resources on existing testbeds. Concretely, a provider takes a declarative description of the expected resources as argument. It requests the resources on the corresponding testbed, and returns a set of hosts and networks labeled with their roles. In other words, providers help the experimenter deal with the variety of the APIs of testbeds.
Some infrastructure is based on a reservation system (OAR, blazar …). In this case we defer to the decision to start the resource to the underlying scheduler. This scheduler will try to find an available
start_time
for our configuration in a near future if possible.In the above situation reservation in advance is possible and can be set in the corresponding configuration, using
set_reservation()
orinit()
with thestart_time
parameter set.- Parameters:
provider_conf (BaseConfiguration) – configuration of the provider. The configuration object is specific to each provider and must follow the provider’s schema
name – a provider can be named. This is used in the context of a multiprovider and all the resources from this provider using this name in roles an networks.
Example
Typical workflow:
# craft a conf (provider specific) conf = Configuration() ... provider = Provider(conf) roles, networks = provider.init() # release resources provider.destroy()
Using a context manager
# craft a conf (provider specific) conf = Configuration() ... with provider(conf) as (roles, networks): # do stuff with roles and networks ... # Resources are automatically released at the end
Methods:
async_init
(**kwargs)Partial init: secure the resources to the targeted infrastructure.
destroy
([wait])Abstract.
init
([force_deploy, start_time])Abstract.
Is the provider already created.
offset_walltime
(offset)Offset the walltime.
set_reservation
(timestamp)Change the internal reservation date.
test_slot
(start_time, end_time)Test a slot that starts at a given point in time.
- async_init(**kwargs)#
Partial init: secure the resources to the targeted infrastructure.
This is primarily used internally by
Providers
to get the resources from different platforms. As this method actually starts some real resources somewhere, errors may occur (e.g no more available resources, …). It’s up to the provider to indicate if the error is critical or not. For instance anInvalidReservationTime
can be raised to indicate the Providers to retry later.- Parameters:
kwargs – keyword arguments. Fit those from
init()
- Raises:
InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception
- abstract destroy(wait: bool = False, **kwargs)#
Abstract. Destroy the resources used for the deployment.
- abstract init(force_deploy: bool = False, start_time: int | None = None, **kwargs: Any)#
Abstract. Provides resources and provisions the environment.
This calls the underlying provider and provision resources (machines and networks).
- Parameters:
force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.
- Returns:
roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see
Enos_vagrant
- Return type:
(roles, networks) tuple
- is_created() bool #
Is the provider already created.
- abstract offset_walltime(offset: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
- set_reservation(timestamp: int)#
Change the internal reservation date.
Ignored on platform that aren’t based on a reservation system.
- Parameters:
timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.
- test_slot(start_time: int, end_time: int) bool #
Test a slot that starts at a given point in time.
A slot is given by a start_time, a duration and an amount of resources. The two latter are found in the internal configuration.
- Parameters:
start_time (timestamp (int)) – Test for a possible slot that starts at start_time (UTC)
end_time – (timestamp (int)): How much time in the future we should look for possible reservation. This is used on some platform to know how much time in the future we look in the planning data. Timestamp is based on UTC.
- Returns:
True iff the slot is available
Vagrant#
Vagrant Provider Class#
Classes:
|
The provider to use when working with vagrant (local machine). |
- class enoslib.infra.enos_vagrant.provider.Enos_vagrant(provider_conf, name: str | None = None)#
The provider to use when working with vagrant (local machine).
- destroy(wait: bool = False, **kwargs)#
Destroy all vagrant box involved in the deployment.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs) Tuple[Roles, Networks] #
Reserve and deploys the vagrant boxes.
- Parameters:
force_deploy (bool) – True iff new machines should be started
- offset_walltime(difference: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
Vagrant Schema#
Vagrant Configuration Schema#
type |
object |
||
properties |
|||
|
type |
string |
|
enum |
libvirt, virtualbox |
||
|
base image to use (default: generic/debian11) |
||
type |
string |
||
|
SSH user to use (default: root) |
||
type |
string |
||
|
Prepend this prefix to box names |
||
type |
string |
||
|
Extra config to pass (in vargrant DSL |
||
type |
string |
||
|
Vagrant Resource |
||
type |
object |
||
properties |
|||
|
type |
array |
|
items |
|||
uniqueItems |
True |
||
|
type |
array |
|
items |
|||
additionalProperties |
False |
||
additionalProperties |
False |
Vagrant Network#
type |
object |
||
properties |
|||
|
type |
string |
|
|
type |
array |
|
items |
type |
string |
|
additionalProperties |
False |
Vagrant Compute#
type |
object |
||
properties |
|||
|
type |
array |
|
items |
type |
string |
|
|
type |
number |
|
|
type |
string |
|
|
type |
string |
|
enum |
tiny, small, medium, big, large, extra-large |
||
|
|||
additionalProperties |
False |
Vagrant Flavour#
type |
object |
|
properties |
||
|
type |
number |
|
type |
number |
additionalProperties |
False |
Grid5000 (G5k)#
G5k Provider Class#
Classes:
|
The provider to use when interacting with Grid'5000. |
|
Internal class. |
|
A G5k host. |
|
Representation of a reservable network in G5k. |
|
Representation of the Production Network in G5k. |
|
Representation of a subnet resource of G5k (/16 or /22). |
|
A class to initiate a tunnel to a targetted service inside Grid'5000. |
|
Representation of a Vlan resource in G5k. |
- class enoslib.infra.enos_g5k.provider.G5k(*args, **kwargs)#
The provider to use when interacting with Grid’5000.
Most of the methods are inherited from
G5kBase
.- reserve()#
Reserve the resources described in the configuration
This support multisite configuration.
- class enoslib.infra.enos_g5k.provider.G5kBase(*args, **kwargs)#
Internal class.
Provider dedicated to single site interaction.
- jobs#
List of Grid’5000 Job objects managed by this provider
- Type:
list
- async_init(start_time: int | None = None, **kwargs)#
Reserve but don’t wait.
No node/network information can’t be retrieved at this moment. If a start_time is provided, set the reservation date to it.
- destroy(wait: bool = True, **kwargs)#
Destroys the jobs.
- firewall(hosts: Iterable[Host] | None = None, port: int | List[int] | None = None, src_addr: str | List[str] | None = None, proto: str = 'tcp+udp')#
Context manager to manage firewall rules
Create a firewall opening when entering
Delete the firewall opening when exiting
- Parameters:
hosts – limit the rule to a set of hosts. if None, rules will be applied on all hosts of all underlying jobs.
port – ports to open
src_addr – source addresses to consider
proto – protocol to consider
- fw_create(hosts: Iterable[Host] | None = None, port: int | List[int] | None = None, src_addr: str | List[str] | None = None, proto: str = 'tcp+udp')#
Create a firewall rules
Reference: https://www.grid5000.fr/w/Reconfigurable_Firewall
Note that
port
andsrc_addr
andproto
are passed to the API calls without any change. It means that accepted values are those accepted by the REST API.- Parameters:
hosts – limit the rule to a set of hosts. if None, rules will be applied on all hosts of all underlying jobs.
port – ports to open
src_addr – source addresses to consider
proto – protocol to consider
- fw_delete()#
Delete all existing rules.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs) Tuple[Roles, Networks] #
Take ownership over some Grid’5000 resources (compute and networks).
The function does the heavy lifting of transforming your abstract resource configuration into concrete resources.
From a high level perspective it works as follow:
First it transforms the configuration of resources into an actual OAR resource selection string (one single reservation per provider instance).
It requests the API to get the corresponding resources (job and optionally deploys a environment)
Those resources are then mapped back to every single item on the configuration.
Finally it applies some more operations (set nodes on vlans, configure secondary interfaces) before returning.
Note
The call to the function is idempotent and the following is ensured:
Existing job(s) (based on the name) will be reloaded.
The mapping between concrete resources and their corresponding roles is fixed across runs. This includes:
the mapping between machines and roles
the mapping between networks and roles
the mapping between network cards and networks
Deployments is performed only on nodes that are not deployed yet (up to three attempts).
At the end machine are reachable using the root account.
- Parameters:
force_deploy (bool) – True iff the environment must be redeployed
start_time (timestamp (int)) – Time at which to start the job, by default whenever possible
- Raises:
MissingNetworkError – If one network is missing in comparison to what is claimed.
NotEnoughNodesError – If the min constraints can’t be met.
InvalidReservationTime – If the set reservation_date from provider.conf isn’t free
InvalidReservationOld – If the set reservation_date from provider.conf is in the past
InvalidReservationCritical – Any other error that might occur during the reservation is deemed critical
- Returns:
Two dictionaries (roles, networks) representing the inventory of resources.
- is_created() bool #
Is the provider already created.
- offset_walltime(offset: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
- set_reservation(timestamp: int)#
Change the internal reservation date.
Ignored on platform that aren’t based on a reservation system.
- Parameters:
timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.
- test_slot(start_time: int, end_time: int) bool #
Test if it is possible to reserve the configuration corresponding to this provider at start_time
- static tunnel(address: str, port: int) Tuple[str, int, SSHTunnelForwarder | None] #
Create a tunnel if necessary between here and there (in G5k).
- Parameters:
address (str) – The remote address to reach (assuming inside g5k)
port (int) – The remote port to reach
- Returns:
The context manager
- class enoslib.infra.enos_g5k.provider.G5kHost(fqdn: str, roles: List[str], primary_network: G5kNetwork, secondary_networks: List[G5kNetwork] | None = None)#
A G5k host.
- property apinode: Node#
Get the api Node object.
- Returns:
Node object (see python-grid500)
- dhcp_networks_command() str #
Get the command to set up the dhcp an all interfaces.
- Returns:
The command as a string.
- get_nics(extra_cond: ~typing.Callable[[~typing.Dict], bool] = <function G5kHost.<lambda>>) List[Tuple[str, str]] #
Get the network interfaces names corresponding to a criteria.
Note
Only the mountable and Ethernet interfaces are returned.
Nic are sorted so that the result is fixed accros run.
- Parameters:
extra_cond – predicate over a nic to further filter the results. Here a nic is a dictionary as returned in the API.
- Returns:
A list of nics. Each nic is a tuple (legacy name, deterministic name) e.g (“eth0”, “eno1”). Result is sorted to ensure idempotence.
Note
NOTE(msimonin): Since 05/18 nics on g5k nodes have predictable names but the api description keeps the legacy name (device key) and the new predictable name (key name). The legacy names is still used for api request to the vlan endpoint This should be fixed in https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=9272 When its fixed we should be able to only use the new predictable name.
- grant_root_access_command() str #
Get the command to get root access on the node.
- mirror_state()#
Make sure the API states are consistent to the Host attributes.
For instance this will set some of the NIC of the nodes in a vlan (POST request on the API). For now the only strategy to map NIC to secondary network is to map them in the same order : nic_i <-> secondary_networks[i]. (where nic_i is not the primary one)
Note that this doesn’t configure the NIC on the node itself (e.g dhcp).
- property primary_nic: Tuple[str, str]#
Get the first nic mounted.
On Grid’5000 there’s only one nic mounted by default: this is the primary nic.
- Returns:
A tuple of (legacy name, deterministic name) for the network card.
- property secondary_nics: List[Tuple[str, str]]#
Get the nics that serves as secondary nics.
Note: only return those eligible to map a secondary network.
- Returns:
All the nic that serves to connect the node to a secondary network.
- property ssh_address: str#
Get an SSH reachable address for this Host.
This may differ from the fqdn when using vlans.
- Returns:
The address as a string.
- to_enoslib(address: str = '', user: str = 'root', extra: Dict | None = None) Host #
Return a generic Host object from a G5kHost object.
We automatically set up a SSH jump configuration if Enoslib is running outside of Grid’5000.
- Parameters:
address – host name to use for SSH, defaults to G5kHost.ssh_address
user – SSH user, defaults to “root”
extra – optional dictionary of extra Ansible host variables
Returns: a Host object usable with Ansible.
- class enoslib.infra.enos_g5k.provider.G5kNetwork(roles: List[str], id: str, site: str)#
Representation of a reservable network in G5k.
A G5kNetwork wraps the corresponding network resource of the G5k REST API.
- Parameters:
roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network
- add_host(host: G5kHost)#
Add a host
enoslib.infra.enos_g5k.objects.G5kHost
.Currently, this doesn’t attach the node.
- Parameters:
host – The host to attach
- add_hosts(hosts: List[G5kHost])#
Add some hosts
enoslib.infra.enos_g5k.objects.G5kHost
.Currently, this doesn’t attach the node.
- Parameters:
hosts – The list host to attach
- abstract property apinetwork: RESTObject | None#
Gets the underlying RESTObject representing the network.
Only vlan and prod network have an equivalent entry in the API. None will be returned for subets.
- Returns:
The corresponding RESTObject, None otherwise.
- abstract attach(fqdns: List[str], device: str)#
Attach the specific devices into this network.
- Parameters:
fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.
- abstract property cidr: str | None#
Get the network address in cid format.
- Returns:
The cidre of the network. None for subnets.
- abstract property cidr6: str | None#
Get the ipv6 network address in cidr format.
- Returns:
The cidre of the network. None for subnets.
- property dns: str | None#
Gets the DNS address for this network.
- property dns6: str | None#
Gets the DNS address for this network. IPv6
We fall back to IPv4 for now
- abstract property gateway: str | None#
Get the gateway for this network.
- Returns:
The gateway address as a string.
- abstract property gateway6: str | None#
Get the gateway for this network (IPv6).
- Returns:
The gateway address as a string.
- abstract to_enos()#
Transform into a provider-agnostic data structure.
For legacy reason we’re still using dicts here…
- Returns:
a list of networks, each being a dict.
- abstract translate(fqdns: Iterable[str], reverse: bool = False) List[Tuple[str, str]] #
Gets the DNS names of the passed fqdns in these networks and vice versa.
- Parameters:
fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.
- Returns:
List of translated names.
- abstract translate6(fqdns: Iterable[str], reverse: bool = False) List[Tuple[str, str]] #
Gets the DNS names (resolved as ipv6) of the passed fqdns in these networks.
- Parameters:
fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.
- Returns:
List of translated names.
- abstract property vlan_id: str | None#
Get the vlan id.
- Returns:
The vlan id. None for subnets.
- class enoslib.infra.enos_g5k.provider.G5kProdNetwork(roles: List[str], id: str, site: str)#
Representation of the Production Network in G5k.
Note: production network have the “default” uid on the G5K API.
- Parameters:
roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network
- attach(fqdns: List[str], nic: str)#
Attach the specific devices into this network.
- Parameters:
fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.
- to_enos() Tuple[List[str], List] #
Build the generic network type implementation.
A production network in G5k has 2 flavours IPv4 and IPv6 so we generate both generic network type here corresponding to this vlan.
- translate(fqdns: Iterable[str], reverse: bool = False) List[Tuple[str, str]] #
Node in the production network.
node uid == node name
- translate6(fqdns: Iterable[str], reverse: bool = False) List[Tuple[str, str]] #
Translate node name in ipv6 resolvable name.
- class enoslib.infra.enos_g5k.provider.G5kSubnetNetwork(roles: List[str], id: str, site: str, subnets: List[str])#
Representation of a subnet resource of G5k (/16 or /22).
Note
Subnets are weird beasts on G5k. Especially /16 networks for which you will be given 64 /22 networks and they aren’t represented in the REST API.
So we encapsulate this in this object
- Parameters:
roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network
subnets – the actual subnets (list of cidr) given by OAR.
- property apinetwork: None#
Gets the underlying RESTObject representing the network.
Only vlan and prod network have an equivalent entry in the API. None will be returned for subets.
- Returns:
The corresponding RESTObject, None otherwise.
- attach(fqdns: List[str], nic: str)#
Attach the specific devices into this network.
- Parameters:
fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.
- property cidr: None#
Not well-defined.
Since subnets might be an aggregation of several smaller ones it’s difficult to know what to return here.
Note that the user will be given with as many Network as small subnets we have. In this case the network address will be well defined.
- property cidr6: None#
Not well-defined (and no support for IPv6 now).
Since subnets might be an aggregation of several smaller ones it’s difficult to know what to return here.
Note that the user will be given with as many Network as small subnets we have. In this case the network address will be well defined.
- property gateway#
Get the gateway for this network.
- Returns:
The gateway address as a string.
- property gateway6: None#
Get the gateway for this network (IPv6).
- Returns:
The gateway address as a string.
- to_enos() Tuple[List[str], List[G5kEnosSubnetNetwork]] #
Build the generic network type implementation.
A subnet in G5k doesn’t have yet an IPv6 counterpart so we generate a single IPv4 generic representation. A /16 is actually an aggregation of /22 so we emit one generic network per underlying subnet.
- translate(fqdns: Iterable[str], reverse: bool = True) List[Tuple[str, str]] #
Gets the DNS names of the passed fqdns in these networks and vice versa.
- Parameters:
fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.
- Returns:
List of translated names.
- translate6(fqdns: Iterable[str], reverse: bool = True) List[Tuple[str, str]] #
Gets the DNS names (resolved as ipv6) of the passed fqdns in these networks.
- Parameters:
fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.
- Returns:
List of translated names.
- property vlan_id: None#
Get the vlan id.
- Returns:
The vlan id. None for subnets.
- class enoslib.infra.enos_g5k.provider.G5kTunnel(address: str, port: int, local_port: int = 0)#
A class to initiate a tunnel to a targetted service inside Grid’5000.
Can be used as a context manager (will close the tunnel automatically). Note that this is a noop when called from inside Grid’5000.
- Parameters:
address – The ip address/fqdn of the targetted service
port – The port of the targetted service
- close()#
Close the tunnel.
Note that this won’t wait for any connection to finish first.
- start() Tuple[str, int, SSHTunnelForwarder | None] #
Start the tunnel.
- Returns:
A tuple composed of the local address , the local port and the tunnel object (if any)
- class enoslib.infra.enos_g5k.provider.G5kVlanNetwork(roles: List[str], id: str, site: str, vlan_id: str)#
Representation of a Vlan resource in G5k.
- Parameters:
roles – roles/tags to give to this network (set by the application)
id – the id to give to this network (set by the application)
site – the site of this network
vlan_id – the vlan id of the vlan
- property apinetwork: Vlan#
Gets the underlying RESTObject representing the network.
Only vlan and prod network have an equivalent entry in the API. None will be returned for subets.
- Returns:
The corresponding RESTObject, None otherwise.
- attach(fqdns: List[str], device: str)#
Attach the specific devices into this network.
- Parameters:
fqdns – list of hostnames (host uid in the API) to attach
device – the NIC names to put on this network.
- property cidr: str#
Get the network address in cid format.
- Returns:
The cidre of the network. None for subnets.
- property cidr6: str#
Get the ipv6 network address in cidr format.
- Returns:
The cidre of the network. None for subnets.
- property gateway: str#
Get the gateway for this network.
- Returns:
The gateway address as a string.
- property gateway6: str | None#
Get the gateway for this network (IPv6).
- Returns:
The gateway address as a string.
- to_enos() Tuple[List[str], List] #
Build the generic network type implementation.
A vlan in G5k has 2 flavours IPv4 and IPv6 so we generate both generic network type here corresponding to this vlan.
- translate(fqdns: Iterable[str], reverse: bool = False) List[Tuple[str, str]] #
Gets the DNS names of the passed fqdns in these networks and vice versa.
- Parameters:
fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.
- Returns:
List of translated names.
- translate6(fqdns: Iterable[str], reverse: bool = False) List[Tuple[str, str]] #
Gets the DNS names (resolved as ipv6) of the passed fqdns in these networks.
- Parameters:
fqdns – iterable of hostnames (host uid in the API) to translate.
reverse – Do the opposite operation.
- Returns:
List of translated names.
- property vlan_id: str#
Get a vlan id if any.
G5k Schema#
Grid5000 Configuration Schema#
type |
object |
|||
properties |
||||
|
Run dhcp client automatically on kavlan networks (default: True) |
|||
type |
boolean |
|||
|
Force systematic redeployment on nodes (deploy only) (default: False) |
|||
type |
boolean |
|||
|
The kadeploy3 environment to use (deploy only) |
|||
type |
string |
|||
|
Kadeploy3 env version to use (deploy only, optional) |
|||
type |
integer |
|||
minimum |
2016011914 |
|||
exclusiveMinimum |
True |
|||
|
Name of the job (default: EnOSlib) |
|||
type |
string |
|||
|
OAR job type (default: []). |
|||
format |
job_type |
|||
anyOf |
type |
string |
||
type |
array |
|||
items |
type |
string |
||
|
SSH public key to use (default: ~/.ssh/.id_rsa.pub) |
|||
type |
string |
|||
|
Activate on demand metrics (e.g |
|||
type |
string |
|||
|
Reload from existing job ids |
|||
type |
array |
|||
items |
||||
|
Project / team to use |
|||
type |
string |
|||
|
OAR queue to use (default: default) |
|||
type |
string |
|||
enum |
default, production, testing, besteffort |
|||
|
reservation date in YYYY-mm-dd HH:MM:SS format |
|||
type |
string |
|||
format |
reservation |
|||
|
Job duration (default: 02:00:00) |
|||
type |
string |
|||
format |
walltime |
|||
|
Grid’5000 Resources |
|||
type |
object |
|||
properties |
||||
|
Description of the servers to reserve |
|||
type |
array |
|||
items |
oneOf |
|||
|
Description of the networks to reserve |
|||
type |
array |
|||
items |
||||
uniqueItems |
True |
|||
additionalProperties |
False |
|||
additionalProperties |
False |
Grid5000 JobIds#
List of tuple (site, jobid) used to reload the jobs from |
||
type |
array |
|
items |
type |
string |
Grid5000 ComputeCluster#
Describe a group of machine based on a cluster name |
|||
type |
object |
||
properties |
|||
|
The concrete resources will be assigned this role |
||
type |
array |
||
items |
type |
string |
|
|
Which cluster to use |
||
type |
string |
||
|
Number of nodes (default: 1) |
||
type |
number |
||
|
Minimal number of nodes to get (default to nodes) |
||
type |
number |
||
|
Request access to reservable disks on nodes |
||
type |
boolean |
||
|
Network(id) to use on the primary NIC |
||
type |
string |
||
|
Additional networks(ids) to assign |
||
type |
array |
||
items |
type |
string |
|
uniqueItems |
True |
Grid5000 ComputeServers#
Description of a specific list of servers to get |
|||
type |
object |
||
properties |
|||
|
The concrete resources will be assigned this role |
||
type |
array |
||
items |
type |
string |
|
|
List of names (e.g [chetemi-1.lille.grid5000.fr]) |
||
type |
array |
||
items |
type |
string |
|
format |
hostname |
||
minItems |
1 |
||
|
Number of nodes to get (default: #servers) |
||
type |
number |
||
|
Minimal number of nodes to get (default to nodes) |
||
type |
number |
||
|
Request access to reservable disks on nodes |
||
type |
boolean |
||
|
Network to use on this NIC |
||
type |
string |
||
|
List of the network to use on the other NICs |
||
type |
array |
||
items |
type |
string |
|
uniqueItems |
True |
Grid5000 Network#
type |
object |
||
properties |
|||
|
Id used to identify network in machines |
||
type |
string |
||
|
Type of network to use supported by Grid’5000 |
||
enum |
prod, kavlan, kavlan-local, kavlan-global, slash_16, slash_22 |
||
|
The concrete resources will be assigned this role |
||
type |
array |
||
items |
type |
string |
|
|
On which site to reserve the network |
||
type |
string |
G5k API utils#
This module is composed of helpers functions to deal with the Grid’5000 REST API.
It wraps the python-grid5000 library to provide some usual routines to interact with the platform.
Classes:
|
Wrapper of the python-grid5000 client. |
|
Create new instance of OarNetwork(site, nature, descriptor) |
Functions:
|
Build the resources from the list of jobs. |
|
Check if #nodes can be started on a given cluster. |
|
Get all the corresponding sites of the passed clusters. |
|
|
|
Enable access to a group storage from ips. |
|
Enable access to home dir from ips. |
Get all the cluster of all the sites. |
|
Return the list of the sites. |
|
Gets the reference to the API client (singleton). |
|
Return username of client |
|
|
Get the network interfaces names corresponding to a criteria. |
|
Get the site of a given cluster. |
|
Returns for each cluster the available cluster interfaces |
|
Get the corresponding sites of given clusters. |
|
Get the status of the clusters (current and future reservations). |
|
Get the total number of cores in each machine for a given cluster |
|
|
|
|
|
Get the memory (in bytes) in each machine for a given cluster |
|
Get the network cards information |
|
|
|
Get all the nodes of a given cluster. |
|
Get a single site. |
|
|
|
Get the total number of threads in each machine for a given cluster |
|
|
|
|
|
Deploy and wait for the deployment to be finished. |
|
Destroy all the jobs with corresponding ids |
|
Destroy all the jobs with a given name. |
|
|
|
|
|
Reload all running or pending jobs of Grid'5000 from their ids |
|
Reload jobs of Grid'5000 from their ids |
|
Reload all running or pending jobs of Grid'5000 with a given name. |
|
|
|
Set the interface of the nodes in a specific vlan. |
|
Submit a job |
|
|
|
|
|
Waits for all the jobs to be runnning. |
- class enoslib.infra.enos_g5k.g5k_api_utils.Client(excluded_sites: List | None = None, **kwargs)#
Wrapper of the python-grid5000 client.
It accepts extra parameters to be set in the configuration file.
Constructor.
- Parameters:
excluded_sites (list) – sites to forget about when reloading the jobs. The primary use case was to exclude unreachable sites and allow the program to go on.
- class enoslib.infra.enos_g5k.g5k_api_utils.OarNetwork(site, nature, descriptor)#
Create new instance of OarNetwork(site, nature, descriptor)
Attributes:
Alias for field number 2
Alias for field number 1
Alias for field number 0
- descriptor#
Alias for field number 2
- nature#
Alias for field number 1
- site#
Alias for field number 0
- enoslib.infra.enos_g5k.g5k_api_utils.build_resources(jobs: Iterable[Job]) Tuple[List[str], List[OarNetwork]] #
Build the resources from the list of jobs.
- Parameters:
jobs (list) – The list of python-grid5000 jobs
- Returns:
- nodes, networks tuple where
nodes is a list of all the nodes of the various reservations
networks is a list of all the networks of the various reservation
- enoslib.infra.enos_g5k.g5k_api_utils.can_start_on_cluster(nodes_status: Mapping, number: int, exact_nodes: List[str], start: float, walltime: int) bool #
Check if #nodes can be started on a given cluster.
This is intended to give a good enough approximation. This can be used to prefiltered possible reservation dates before submitting them on oar.
- Parameters:
nodes_status – a dictionary with all the status of the nodes as returned by the api (cluster status endpoint)
number – number of node in the demand
exact_nodes – the list of the fqdn of the machines to get
start – start time of the job
walltime – walltime of the job
- Returns
True iff the job can start
- enoslib.infra.enos_g5k.g5k_api_utils.clusters_sites_obj(clusters: Iterable) Dict[str, Site] #
Get all the corresponding sites of the passed clusters.
- Parameters:
clusters (list) – list of the clusters (str)
- Returns:
dict corresponding to the mapping of cluster name to python-grid5000 site
- enoslib.infra.enos_g5k.g5k_api_utils.deploy(site: str, nodes: List[str], config: Dict) Tuple[List[str], List[str]] #
- enoslib.infra.enos_g5k.g5k_api_utils.enable_group_storage(storage_site: str, storage_server: str, storage_name: str, ips: List[str], termination_job: Job)#
Enable access to a group storage from ips.
- Parameters:
storage_site – a grid’5000 site. Site where the storage is located
storage_server – a server name Storage server where the storage is located (e.g. ~”storage1”~ or ~”home”~)
storage_name – name of the group storage The name is the one used to identify a Group Storage (this might be your username if you plan to allow your home dir)
ips – list of ips Ips to allow the access from (only IPv4 for now).
termination_job – a job This is used as a termination condition
- enoslib.infra.enos_g5k.g5k_api_utils.enable_home_for_job(job: Job, ips: List[str])#
Enable access to home dir from ips.
This allows to mount the home dir corresponding to the site of job from any of the ips provided.
Examples
roles, networks = provider.init() # get some ips to allow ips = [str(ip) for ip in networks["role"][0].network] # get the underlying job job = provider.jobs[0] enable_home_for_job(job, ips)
For enabling any group storage, please refer to
enable_group_storage()
- Parameters:
job – A (running) job. home site and access duration will be inferred from this job
ips – list of IPs Every machine connecting from one of this IPs will be granted an access to the home directory
- enoslib.infra.enos_g5k.g5k_api_utils.get_all_clusters_sites() Dict[str, str] #
Get all the cluster of all the sites.
- Returns:
dict corresponding to the mapping cluster uid to python-grid5000 site
- enoslib.infra.enos_g5k.g5k_api_utils.get_all_sites_obj() List[Site] #
Return the list of the sites.
- Returns:
list of python-grid5000 sites
- enoslib.infra.enos_g5k.g5k_api_utils.get_api_client() Client #
Gets the reference to the API client (singleton).
- enoslib.infra.enos_g5k.g5k_api_utils.get_api_username() str #
Return username of client
- Returns:
client’s username
- enoslib.infra.enos_g5k.g5k_api_utils.get_cluster_interfaces(cluster: str, extra_cond=<function <lambda>>) List[Tuple] #
Get the network interfaces names corresponding to a criteria.
Note that the cluster is passed (not the individual node names), thus it is assumed that all nodes in a cluster have the same interface names same configuration. In addition to
extra_cond
, only the mountable and Ehernet interfaces are returned.- Parameters:
cluster (str) – the cluster to consider
extra_cond (lambda) – boolean lambda that takes the nic(dict) as parameter
- enoslib.infra.enos_g5k.g5k_api_utils.get_cluster_site(cluster: str) str #
Get the site of a given cluster.
- Parameters:
cluster (str) – a Grid’5000 cluster
- Returns:
The corresponding site(str)
- enoslib.infra.enos_g5k.g5k_api_utils.get_clusters_interfaces(clusters: ~typing.Iterable, extra_cond=<function <lambda>>) Dict #
Returns for each cluster the available cluster interfaces
- Parameters:
clusters (str) – list of the clusters
extra_cond (lambda) – extra predicate to filter network card retrieved from the API. E.g. lambda nic: not nic[‘mounted’] will retrieve all the usable network cards that are not mounted by default.
- Returns:
dict of cluster with their associated nic names
Examples
# pseudo code actual = get_clusters_interfaces(["paravance"]) expected = {"paravance": ["eth0", "eth1"]} assertDictEquals(expected, actual)
- enoslib.infra.enos_g5k.g5k_api_utils.get_clusters_sites(clusters: Iterable[str]) Dict[str, str] #
Get the corresponding sites of given clusters.
- Parameters:
clusters (list) – list of the clusters (str)
- Returns:
dict of corresponding to the mapping cluster -> site
- enoslib.infra.enos_g5k.g5k_api_utils.get_clusters_status(clusters: Iterable[str]) Dict #
Get the status of the clusters (current and future reservations).
- enoslib.infra.enos_g5k.g5k_api_utils.get_cores(cluster: str) int #
Get the total number of cores in each machine for a given cluster
- Parameters:
cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)
- enoslib.infra.enos_g5k.g5k_api_utils.get_dns(site)#
- enoslib.infra.enos_g5k.g5k_api_utils.get_ipv6(site)#
- enoslib.infra.enos_g5k.g5k_api_utils.get_memory(cluster: str) int #
Get the memory (in bytes) in each machine for a given cluster
- Parameters:
cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)
- enoslib.infra.enos_g5k.g5k_api_utils.get_nics(cluster: str)#
Get the network cards information
- Parameters:
cluster (str) – Grid’5000 cluster name
- Returns:
dict of nic information
- enoslib.infra.enos_g5k.g5k_api_utils.get_node(site, cluster, uid) Node #
- enoslib.infra.enos_g5k.g5k_api_utils.get_nodes(cluster: str) List[Node] #
Get all the nodes of a given cluster.
- Parameters:
cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)
- enoslib.infra.enos_g5k.g5k_api_utils.get_site_obj(site) Site #
Get a single site.
- Returns:
the python-grid5000 site
- enoslib.infra.enos_g5k.g5k_api_utils.get_subnet_gateway(site)#
- enoslib.infra.enos_g5k.g5k_api_utils.get_threads(cluster: str) int #
Get the total number of threads in each machine for a given cluster
- Parameters:
cluster (str) – uid of the cluster (e.g ‘paravance’ for rennes)
- enoslib.infra.enos_g5k.g5k_api_utils.get_vlan(site, vlan_id) Vlan #
- enoslib.infra.enos_g5k.g5k_api_utils.get_vlans(site)#
- enoslib.infra.enos_g5k.g5k_api_utils.grid_deploy(site: str, nodes: List[str], config: Dict) Tuple[List[str], List[str]] #
Deploy and wait for the deployment to be finished.
- Parameters:
site (str) – the site
nodes (list) – list of nodes (str) to depoy
config (dict) – option of the deployment (refer to the Grid’5000 API Specifications)
- Returns:
tuple of deployed(list), undeployed(list) nodes.
- enoslib.infra.enos_g5k.g5k_api_utils.grid_destroy_from_ids(oargrid_jobids: Iterable[Tuple], wait: bool = False)#
Destroy all the jobs with corresponding ids
- Parameters:
oargrid_jobids (list) – the
(site, oar_job_id)
list of tuple identifying the jobs for each site.wait – True whether we should wait for a status change
- enoslib.infra.enos_g5k.g5k_api_utils.grid_destroy_from_name(job_name: str, wait: bool = False, restrict_to: Iterable[str] | None = None)#
Destroy all the jobs with a given name.
- Parameters:
job_name (str) – the job name
wait – True whether we should wait for a status change
restrict_to – restrict the action to these sites only. If None is passed, no restriction applies and the action will be taken on all possible sites
- enoslib.infra.enos_g5k.g5k_api_utils.grid_get_or_create_job(job_name, walltime, reservation_date, queue, job_type, monitor, project, machines, networks, wait=True, restrict_to: Iterable[str] | None = None) List[Job] #
- enoslib.infra.enos_g5k.g5k_api_utils.grid_make_reservation(job_name: str, walltime, reservation_date, queue, job_type: str | MutableSequence[str], monitor, project, machines: Iterable, networks: Iterable) List[Job] #
- enoslib.infra.enos_g5k.g5k_api_utils.grid_reload_from_ids(oargrid_jobids: Iterable[Tuple]) List[Job] #
Reload all running or pending jobs of Grid’5000 from their ids
- Parameters:
oargrid_jobids (list) – list of
(site, oar_jobid)
identifying the jobs on each site- Returns:
The list of python-grid5000 jobs retrieved
- enoslib.infra.enos_g5k.g5k_api_utils.grid_reload_jobs_from_ids(oargrid_jobids: Iterable[Tuple]) List[Job] #
Reload jobs of Grid’5000 from their ids
- Parameters:
oargrid_jobids (list) – list of
(site, oar_jobid)
identifying the jobs on each site- Returns:
The list of python-grid5000 jobs retrieved
- enoslib.infra.enos_g5k.g5k_api_utils.grid_reload_jobs_from_name(job_name: str, restrict_to: Iterable[str] | None = None) List[Job] #
Reload all running or pending jobs of Grid’5000 with a given name.
By default, all the sites will be searched for jobs with the name
job_name
. Using EnOSlib there can be only one job per site with namejob_name
.Note that it honors the
exluded_sites
attribute of the client so the scan can be reduced.- Parameters:
job_name (str) – the job name
restrict_to – restrict the action to these sites only. If None is passed, no restriction applies and the action will be taken on all possible sites
- Returns:
The list of the python-grid5000 jobs retrieved.
- Raises:
EnosG5kDuplicateJobsError – if there’s several jobs with the same name on a site.
- enoslib.infra.enos_g5k.g5k_api_utils.job_delete(job: Job, wait: bool = False)#
- enoslib.infra.enos_g5k.g5k_api_utils.set_nodes_vlan(nodes: List[str], interface: str, vlan_id: str)#
Set the interface of the nodes in a specific vlan.
All nodes need to belong to the same Grid’5000 site.
It is assumed that the same interface name is available on all nodes.
- Parameters:
nodes (list) – nodes to consider
interface (str) – the network interface to put in the vlan
vlan_id (str) – the id of the vlan
- Raises:
EnosG5kInvalidArgument – if not all nodes belong to the same site.
EnosG5kKavlanNodesError – if some nodes couldn’t be added to the VLAN.
- enoslib.infra.enos_g5k.g5k_api_utils.submit_jobs(job_specs: List[Tuple]) List[Job] #
Submit a job
- Parameters:
job_specs (dict) – The job specification (see Grid’5000 API reference)
- enoslib.infra.enos_g5k.g5k_api_utils.to_prod_nature() str #
- enoslib.infra.enos_g5k.g5k_api_utils.to_subnet_nature(cidr: str) str #
- enoslib.infra.enos_g5k.g5k_api_utils.to_vlan_nature(vlan_id: str) str #
- enoslib.infra.enos_g5k.g5k_api_utils.wait_for_jobs(jobs: Iterable)#
Waits for all the jobs to be runnning.
- Parameters:
jobs (list) – list of the python-grid5000 jobs to wait for
- Raises:
Exception – if one of the job gets in error state.
Virtual Machines on Grid5000 (VMonG5k)#
VMonG5k Provider Class#
Classes:
|
The provider to use when deploying virtual machines on Grid'5000. |
|
Internal data structure to manipulate virtual machines. |
Functions:
|
Generator function to get some macs out of G5k subnets |
|
Starts virtualmachines on G5K. |
- class enoslib.infra.enos_vmong5k.provider.VMonG5k(*args, **kwargs)#
The provider to use when deploying virtual machines on Grid’5000.
- async_init(start_time: int | None = None, force_deploy: bool = False, **kwargs)#
Partial init: secure the resources to the targeted infrastructure.
This is primarily used internally by
Providers
to get the resources from different platforms. As this method actually starts some real resources somewhere, errors may occur (e.g no more available resources, …). It’s up to the provider to indicate if the error is critical or not. For instance anInvalidReservationTime
can be raised to indicate the Providers to retry later.- Parameters:
kwargs – keyword arguments. Fit those from
init()
- Raises:
InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception
- destroy(wait: bool = False, **kwargs)#
Destroy the underlying job.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs) Tuple[Roles, Networks] #
Abstract. Provides resources and provisions the environment.
This calls the underlying provider and provision resources (machines and networks).
- Parameters:
force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.
- Returns:
roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see
Enos_vagrant
- Return type:
(roles, networks) tuple
- is_created() bool #
Is the provider already created.
- offset_walltime(difference: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
- set_reservation(timestamp: int)#
Change the internal reservation date.
Ignored on platform that aren’t based on a reservation system.
- Parameters:
timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.
- test_slot(start_time: int, end_time: int) bool #
Test if it is possible to reserve resources at start_time
- class enoslib.infra.enos_vmong5k.provider.VirtualMachine(name: str | None, eui: EUI, flavour_desc: Mapping, pm, extra: Dict | None = None, extra_devices='')#
Internal data structure to manipulate virtual machines.
- enoslib.infra.enos_vmong5k.provider.mac_range(g5k_subnets: List[G5kEnosSubnetNetwork], skip: int = 0, step: int = 1) Generator[str, None, None] #
Generator function to get some macs out of G5k subnets
- Parameters:
g5k_subnets – a list of g5k subnets (see :py:class::~enoslib.infra.enos_g5k.objects.G5kEnosSubnetNetwork)
skip – skip this amount of macs
step – step as in built-in range method
- Returns:
An iterator of mac addresses
- enoslib.infra.enos_vmong5k.provider.start_virtualmachines(provider_conf: Configuration, force_deploy: bool = False) Roles #
Starts virtualmachines on G5K.
This first distributes the virtual machine according to the undercloud attributes of the configuration, assign them IPs and start them. It is idempotent.
- Parameters:
provider_conf – This is the abstract description of your overcloud (VMs). Each configuration must have its undercloud attributes filled with the undercloud machines to use. Round Robin strategy to distribute the VMs to the PMs will be used for each configuration. Mac addresses will be generated according to the g5k_subnet parameter.
g5k_subnets – The subnets to use. Each element is a serialization of
enoslib.infra.enos_vmong5k.configuration.NetworkConfiguration
skip – number of addresses to skip when distributing them to the virtual machines. This can be useful when starting incrementally the virtual machines to avoid overlaping ip assignments between iterations.
force_deploy (boolean) – controls whether the virtual machines should be restarted from scratch.
- Returns:
roles
Examples
1import logging 2from itertools import islice 3from pathlib import Path 4 5import enoslib as en 6 7en.init_logging(level=logging.INFO) 8en.check() 9 10job_name = Path(__file__).name 11 12CLUSTER = "parasilo" 13SITE = "rennes" 14 15 16conf = ( 17 en.G5kConf.from_settings(job_type=[], job_name=job_name) 18 .add_network( 19 id="not_linked_to_any_machine", type="slash_22", roles=["my_subnet"], site=SITE 20 ) 21 .add_machine(roles=["role1"], cluster=CLUSTER, nodes=1) 22 .add_machine(roles=["role2"], cluster=CLUSTER, nodes=1) 23) 24 25provider = en.G5k(conf) 26roles, networks = provider.init() 27roles = en.sync_info(roles, networks) 28 29# Retrieving subnets 30subnet = networks["my_subnet"] 31logging.info(subnet) 32 33# We describe the VMs types and placement in the following 34# We build a VMonG5KConf with some extra fields: 35# - undercloud: where the VMs should be placed (round-robin) 36# - macs: list of macs to take: on G5k the dhcp is configured to assign specific 37# ip based on the configured mac 38 39n_vms = 16 40virt_conf = ( 41 en.VMonG5kConf.from_settings(image="/grid5000/virt-images/debian11-x64-base.qcow2") 42 # Starts some vms on a single role 43 # Here that means start the VMs on a single machine 44 .add_machine( 45 roles=["vms"], 46 number=n_vms, 47 undercloud=roles["role1"], 48 macs=list(islice(subnet[0].free_macs, n_vms)) 49 # alternative 50 # macs=list(islice(en.mac_range(subnet), n_vms)) 51 ) 52) 53 54# Start them 55vmroles = en.start_virtualmachines(virt_conf) 56print(vmroles) 57print(networks)
VMonG5k Schema#
VMonG5k schema. |
||||
type |
object |
|||
properties |
||||
|
Use TakTuk to distribute the VM image |
|||
type |
boolean |
|||
|
Remove and restart all virtual machines |
|||
type |
boolean |
|||
|
Name of the job (default: EnOslib-vmong5k) |
|||
type |
string |
|||
|
Grid’5000 queue to use (default: default) |
|||
type |
string |
|||
enum |
default, production, testing, besteffort |
|||
|
Job duration (default: 02:00:00) |
|||
type |
string |
|||
|
Path to the base image on the reserved nodes |
|||
type |
string |
|||
|
Skip this number of IPs |
|||
type |
number |
|||
|
Base image strategy (default: cow) |
|||
type |
string |
|||
enum |
copy, cow |
|||
|
Subnet type to use (default: slash_22) |
|||
type |
string |
|||
enum |
slash_16, slash_22 |
|||
|
Remote working directory (default: /tmp/enos_vmong5k) |
|||
type |
string |
|||
|
Domain type of the guest (default: kvm) |
|||
type |
string |
|||
|
Reservation date %Y-%m-%d %H:%M:%S (Paris Timezone) |
|||
type |
string |
|||
|
Project / team to use |
|||
type |
string |
|||
|
VMonG5k Resource |
|||
type |
object |
|||
properties |
||||
|
type |
array |
||
items |
||||
|
type |
array |
||
items |
type |
string |
||
additionalProperties |
False |
|||
additionalProperties |
False |
VMonG5k Compute#
VMonG5k Machine description |
|||
type |
object |
||
properties |
|||
|
EnOSlib’s roles |
||
type |
array |
||
items |
type |
string |
|
|
Number of VMs (default: 1) |
||
type |
number |
||
|
Predefined flavour (default: (‘tiny’, {‘core’: 1, ‘mem’: 512})) |
||
type |
string |
||
enum |
tiny, small, medium, big, large, extra-large |
||
|
Custom flavour description |
||
|
Type of vcore (default: thread) |
||
type |
string |
||
enum |
thread, core |
||
|
Grid’5000 cluster for the undercloud |
||
type |
string |
||
|
List of Host where the VM should be started. |
||
type |
array |
||
items |
type |
object |
|
|
List of MAC addresses to use for the vms |
||
type |
array |
||
items |
type |
string |
|
format |
mac |
||
|
Libvirt XML description for extra devices. |
||
type |
string |
||
additionalProperties |
False |
VMonG5k Flavour#
Custom flavour for a virtual machine. |
||
type |
object |
|
properties |
||
|
number of cores |
|
type |
number |
|
|
memory size in MB |
|
type |
number |
|
|
disk size in GB |
|
type |
number |
|
additionalProperties |
False |
Containers on Grid5000 (Distem)#
Distem Provider Class#
Classes:
|
Use Distem on G5k |
- class enoslib.infra.enos_distem.provider.Distem(provider_conf, name: str | None = None)#
Use Distem on G5k
- async_init(start_time: int | None = None, **kwargs)#
Partial init: secure the resources to the targeted infrastructure.
This is primarily used internally by
Providers
to get the resources from different platforms. As this method actually starts some real resources somewhere, errors may occur (e.g no more available resources, …). It’s up to the provider to indicate if the error is critical or not. For instance anInvalidReservationTime
can be raised to indicate the Providers to retry later.- Parameters:
kwargs – keyword arguments. Fit those from
init()
- Raises:
InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception
- destroy(wait=False)#
Abstract. Destroy the resources used for the deployment.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs)#
Abstract. Provides resources and provisions the environment.
This calls the underlying provider and provision resources (machines and networks).
- Parameters:
force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.
- Returns:
roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see
Enos_vagrant
- Return type:
(roles, networks) tuple
- is_created()#
Is the provider already created.
- offset_walltime(offset: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
- set_reservation(timestamp: int)#
Change the internal reservation date.
Ignored on platform that aren’t based on a reservation system.
- Parameters:
timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.
- test_slot(start_time: int, end_time: int) bool #
Test if it is possible to reserve the configuration corresponding to this provider at start_time
Distem Schema#
type |
object |
|||
properties |
||||
|
The job name (default: EnOslib-distem) |
|||
type |
string |
|||
|
Queue to use (default: default) |
|||
type |
string |
|||
enum |
default, production, testing, besteffort |
|||
|
Job duration (default: 02:00:00) |
|||
type |
string |
|||
|
Default base image to use |
|||
type |
string |
|||
|
reservation date in YYYY-mm-dd HH:MM:SS format |
|||
type |
string |
|||
format |
reservation |
|||
|
Clean the existing containers beforehand (default: False) |
|||
type |
boolean |
|||
|
Distem Resource |
|||
type |
object |
|||
properties |
||||
|
type |
array |
||
items |
||||
|
type |
array |
||
items |
type |
string |
||
additionalProperties |
False |
|||
additionalProperties |
False |
Distem Compute#
type |
object |
||
properties |
|||
|
The concrete resources will be assigned this role |
||
type |
array |
||
items |
type |
string |
|
|
Number of containers (default: 1) |
||
type |
number |
||
|
Predefined flavour (default: (‘tiny’, {‘core’: 1, ‘mem’: 512})) |
||
type |
string |
||
enum |
tiny, small, medium, big, large, extra-large |
||
|
|||
|
Type of vcore (default: thread) |
||
type |
string |
||
enum |
thread, core |
||
|
type |
string |
|
|
type |
array |
|
items |
type |
object |
|
additionalProperties |
False |
Distem Flavour#
Custom flavour/size for your container |
||
type |
object |
|
properties |
||
|
type |
number |
|
type |
number |
additionalProperties |
False |
FIT/IoT-LAB#
FIT/IoT-LAB Schema#
FIT/IoT-LAB configuration#
type |
object |
|||
properties |
||||
|
Name of the job (default: EnOSlib) |
|||
type |
string |
|||
|
Job duration (default: 00:01) |
|||
type |
string |
|||
format |
walltime |
|||
|
start time in YYYY-mm-dd HH:MM:SS format |
|||
type |
string |
|||
format |
start_time |
|||
|
Resource |
|||
type |
object |
|||
properties |
||||
|
oneOf |
type |
array |
|
items |
||||
minItems |
1 |
|||
type |
array |
|||
items |
||||
minItems |
1 |
|||
|
type |
array |
||
items |
||||
uniqueItems |
True |
|||
additionalProperties |
False |
|||
|
FIT/IoT-LAB Monitoring profiles |
|||
type |
object |
|||
properties |
||||
|
type |
array |
||
items |
||||
minItems |
1 |
|||
additionalProperties |
False |
|||
additionalProperties |
False |
FIT/IoT-LAB boards selected by architecture#
type |
object |
||
properties |
|||
|
The concrete resources will be assigned this role |
||
type |
array |
||
items |
type |
string |
|
|
Architecture to use |
||
type |
string |
||
|
Site to use |
||
type |
string |
||
|
Number of boards (defaut: 1) |
||
type |
number |
||
|
Firmware to use |
||
type |
string |
||
|
profile name to use |
||
type |
string |
FIT/IoT-LAB nodes selected by name#
type |
object |
||
properties |
|||
|
The concrete resources will be assigned this role |
||
type |
array |
||
items |
type |
string |
|
|
Exact name of physical nodes to use |
||
type |
array |
||
format |
hostname |
||
items |
type |
string |
|
minItems |
1 |
||
|
Firmware to use |
||
type |
string |
||
|
profile name to use |
||
type |
string |
FIT/IoT-LAB Network#
type |
object |
||
properties |
|||
|
Network to use |
||
|
type |
string |
|
|
enum |
prod |
|
|
type |
array |
|
items |
type |
string |
|
|
type |
string |
FIT/IoT-LAB Radio and Consumption profiles#
type |
object |
|||
properties |
||||
|
The profile name |
|||
type |
string |
|||
|
Target architecture |
|||
type |
string |
|||
enum |
m3, a8, custom |
|||
|
type |
object |
||
properties |
||||
|
type |
string |
||
enum |
rssi, sniffer |
|||
|
type |
integer |
||
maximum |
255 |
|||
minimum |
0 |
|||
|
type |
integer |
||
maximum |
65535 |
|||
minimum |
1 |
|||
|
type |
array |
||
items |
type |
integer |
||
maximum |
26 |
|||
minimum |
11 |
|||
additionalProperties |
False |
|||
|
type |
object |
||
properties |
||||
|
type |
boolean |
||
|
type |
boolean |
||
|
type |
boolean |
||
|
type |
integer |
||
enum |
140, 204, 332, 588, 1100, 2116, 4156, 8244 |
|||
|
type |
integer |
||
enum |
1, 4, 16, 64, 128, 256, 512, 1024 |
|||
additionalProperties |
False |
|||
additionalProperties |
False |
Sensor#
import socket
from typing import Dict, List, Optional
import iotlabcli.auth
import sshtunnel
from enoslib.api import play_on
from enoslib.infra.enos_iotlab.iotlab_api import IotlabAPI
from enoslib.infra.enos_iotlab.sensor import Sensor
from enoslib.log import getLogger
from enoslib.objects import DefaultNetwork, Host
logger = getLogger(__name__, ["IOTlab"])
def ssh_enabled(network_address: str) -> bool:
return network_address.startswith("a8") or network_address.startswith("rpi")
class IotlabHost(Host):
"""
A IoT-LAB host
IoT-LAB has several boards with different characteristics.
However, only A8 nodes are able to receive ssh connections
and run linux commands.
"""
def __init__(
self,
address: str,
roles: List[str],
site: str,
uid: str,
archi: str,
):
super().__init__(address, user="root")
# read only attributes
self.roles = roles
self.site = site
self.uid = uid
self.archi = archi
self._ssh_address: Optional[str] = None
if ssh_enabled(self.archi):
self._ssh_address = f"node-{self.address}"
@property
def ssh_address(self) -> Optional[str]:
"""Get an SSH reachable address for this Host.
Returns:
str: The address as a string.
"""
return self._ssh_address
def __repr__(self) -> str:
return (
"<IotlabHost("
f"roles={self.roles}, "
f"address={self.address}, "
f"ssh_address={self.ssh_address}, "
f"site={self.site}, "
f"uid={self.uid})>"
)
class IotlabSensor(Sensor):
"""A IoT-LAB sensor"""
def __init__(
self,
address: str,
roles: List[str],
site: str,
uid: str,
archi: str,
image: str,
iotlab_client: IotlabAPI,
):
alias = address.split(".")[0]
super().__init__(address, alias)
# read only attributes
self.roles: List[str] = roles
self.site: str = site
self.uid: str = uid
self.archi: str = archi
self.image: str = image
self.iotlab_client: IotlabAPI = iotlab_client
self.user, self.passwd = iotlabcli.auth.get_user_credentials()
self.exp_id = self.iotlab_client.get_job_id()
def __repr__(self) -> str:
return (
"<IotlabSensor("
f"roles={self.roles}, "
f"address={self.address}, "
f"site={self.site}, "
f"uid={self.uid})>"
f"image={self.image})>"
)
def stop(self):
"""
Stops this sensor
"""
self.iotlab_client.send_cmd_node(cmd="stop", nodes=[self.address])
def start(self):
"""
Starts this sensor
"""
self.iotlab_client.send_cmd_node(cmd="start", nodes=[self.address])
def reset(self):
"""
Resets this sensor
"""
self.iotlab_client.send_cmd_node(cmd="reset", nodes=[self.address])
def to_dict(self) -> Dict:
d = super().to_dict()
d.update(
roles=self.roles,
site=self.site,
uid=self.uid,
archi=self.archi,
image=self.image,
)
return d
class IotlabNetwork(DefaultNetwork):
"""Iotlab network class."""
def __init__(self, roles: List[str], *args, **kargs):
super().__init__(*args, **kargs)
self.roles = roles
class IotlabSerial:
def __init__(
self,
sensor: IotlabSensor,
serial_port: int = 20000,
interactive: bool = False,
timeout: int = 5,
):
"""
Create a serial connection to a sensor in IoT-LAB testbed
The serial has 2 possibilities:
1. Interactive: where you can interact with the serial,
reading and sending commands to it
2. Logging (default): only for logging purposes, collect the
output and save it to the file (.iot-lab/<exp_id>/log/<node>-serial.log).
No command write/read is allowed in this mode.
More details in open_serial_conn/enable_logging methods.
Args:
sensor: Sensor object
serial_port: serial port to connect to sensor
interactive: set to true to use write/read methods
timeout: Timeout for socket connection
"""
self.sensor = sensor
self.interactive = interactive
self.serial_port = serial_port
self.timeout = timeout
self._serial_tunnel: Optional[sshtunnel.SSHTunnelForwarder] = None
self._serial_socket: Optional[socket.socket] = None
self._filename = f"~/.iot-lab/{self.sensor.exp_id}/log/{self.sensor.alias}_serial.log" # noqa
def open_serial_conn(self):
"""
Opens serial connection to serial node.
This method creates a SSH tunnel to frontend and a socket
to the local port created by the SSH tunnel.
The sshtunnel library will create a thread to do process the forwarded packets.
Note:
Remember to call close_serial_conn, in order to stop
the thread and close the connection properly.
"""
self._serial_tunnel = sshtunnel.SSHTunnelForwarder(
self.sensor.site + ".iot-lab.info",
ssh_username=self.sensor.user,
ssh_password=self.sensor.passwd,
remote_bind_address=(self.sensor.address, self.serial_port),
)
self._serial_tunnel.start()
self._serial_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._serial_socket.connect(("127.0.0.1", self._serial_tunnel.local_bind_port))
self._serial_socket.settimeout(self.timeout)
msg = """
IotlabSensor(%s): Opening SSH tunnel for serial connection: \
server (%s) remote (%s:%s) local_port: %s"""
logger.info(
msg,
self.sensor.alias,
self.sensor.site,
self.sensor.address,
self.serial_port,
self._serial_tunnel.local_bind_port,
)
def close_serial_conn(self):
"""
Close serial connection.
Stops thread created by sshtunnel library.
"""
if self._serial_socket:
self._serial_socket.close()
self._serial_socket = None
if self._serial_tunnel:
self._serial_tunnel.stop()
self._serial_tunnel = None
def enable_logging_serial(self):
"""
Enables logging of serial output to a file
Runs serial_aggregator tool in frontend to write serial output
to a file.
Output file is saved in the experiment folder at:
.iot-lab/<exp_id>/log/<node>-serial.log.
Note:
Remember to call disable_logging_serial, in order to stop
serial_aggregator tool running on the frontend.
"""
# convert to seconds
wall_time = self.sensor.iotlab_client.get_walltime()
if wall_time is not None:
timeout = wall_time * 60
with play_on(
roles=[Host(f"{self.sensor.site}.iot-lab.info", user=self.sensor.user)]
) as p:
cmd = f"screen -dm bash -c 'serial_aggregator -l {self.sensor.site},{self.sensor.alias.replace('-', ',')} > {self._filename} 2>&1'" # noqa
p.shell(
cmd, task_name="Running serial_aggregator", asynch=timeout, poll=0
)
else:
logger.error("Wall time is None.")
def disable_logging_serial(self):
"""
Disables logging of serial output.
Stops serial_aggregator tool running on the frontend
"""
with play_on(
roles=[Host(f"{self.sensor.site}.iot-lab.info", user=self.sensor.user)],
on_error_continue=True,
) as p:
cmd = "pkill -f 'serial_aggregator -l {},{} > {} 2>&1'".format(
self.sensor.site,
self.sensor.alias.replace("-", ","),
self._filename,
)
p.command(cmd, task_name="Killing serial_aggregator")
def __enter__(self) -> "IotlabSerial":
if self.interactive:
self.open_serial_conn()
else:
self.enable_logging_serial()
return self
def __exit__(self, exc_type, exc_value, exc_traceback):
if self.interactive:
self.close_serial_conn()
else:
self.disable_logging_serial()
def write(self, content: str):
"""
Sends string on serial interface
Args:
content: String to be sent
"""
if not self.interactive:
logger.error("Not in interactive mode, impossible to write on serial")
return
logger.info("IotlabSerial(%s): Writing: %s", self.sensor.alias, content)
if self._serial_socket is not None:
self._serial_socket.sendall(content.encode())
else:
logger.error("Serial Socket is closed.")
def read(self, size: int = 1024) -> str:
"""
Reads string from serial interface
Args:
size: Maximum string size to be received
Returns:
str: String received
"""
data = b""
if not self.interactive:
logger.error("Not in interactive mode, impossible to read serial")
return data.decode()
if self._serial_socket is not None:
try:
data = self._serial_socket.recv(size)
except socket.timeout:
pass
return data.decode()
class IotlabSniffer:
def __init__(self, sensor: IotlabSensor, timeout: int = -1):
"""
Create a sniffer to a sensor in IoT-LAB testbed
Runs the sniffer_aggregator tool in the frontend node
to collect radio packets.
Pcap is saved in the experiment folder at:
.iot-lab/<exp_id>/sniffer/<node>.pcap.
Args:
sensor: Sensor object
timeout: Timeout for sniffer_aggregator command (-1 will run
until the experiment was finished (walltime cfg))
"""
self.sensor = sensor
self.timeout = timeout
wall_time = self.sensor.iotlab_client.get_walltime()
if timeout == -1 and wall_time is not None:
# convert to seconds
self.timeout = wall_time * 60
self._filename = (
f"~/.iot-lab/{self.sensor.exp_id}/sniffer/{self.sensor.alias}.pcap"
)
def start_sniffer(self):
"""
Starts radio sniffing
Run sniffer_aggregator tool in frontend to capture packets in
this sniffer node. The sniffer will run for "timeout" seconds,
or until the stop_sniffer method is called.
Pcap is saved in the experiment folder at:
.iot-lab/<exp_id>/sniffer/<node>.pcap.
Note:
Remember to call stop_sniffer, in order to stop
sniffer_aggregator tool running on the frontend.
"""
with play_on(
roles=[Host(self.sensor.site + ".iot-lab.info", user=self.sensor.user)]
) as p:
cmd = "sniffer_aggregator -l {},{} -o {}".format(
self.sensor.site,
self.sensor.alias.replace("-", ","),
self._filename,
)
p.shell(
cmd,
task_name="Running sniffer_aggregator",
asynch=self.timeout,
poll=0,
)
def stop_sniffer(self):
"""
Stops radio sniffing
Kills sniffer_aggregator tool running in the frontend.
"""
with play_on(
roles=[Host(f"{self.sensor.site}.iot-lab.info", user=self.sensor.user)],
on_error_continue=True,
) as p:
cmd = f'pkill -f "{self._filename}"'
p.command(cmd, task_name="Killing sniffer_aggregator")
def __enter__(self) -> "IotlabSniffer":
self.start_sniffer()
return self
def __exit__(self, exc_type, exc_value, exc_traceback):
self.stop_sniffer()
Openstack#
Openstack Provider Class#
Get resources from an Openstack [1] private cloud.
Openstack is a widely adopted Open Source Cloud Software. It powers infrastructure provider such as:
OVH: http://ovh.com
Chameleon Cloud: https://www.chameleoncloud.org/
Gulliver: https://gulliver.saclay.inria.fr/
and many others.
The supported deployment model lets you get a bunch of compute servers, a single private network. One of your server can act as a gateway and will be assigned a floating IP (EnOSlib hosts will be configured accordingly).
Classes:
|
- class enoslib.infra.enos_openstack.provider.Openstack(provider_conf, name: str | None = None)#
- destroy(wait: bool = False, **kwargs)#
Abstract. Destroy the resources used for the deployment.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs) Tuple[Roles, Networks] #
Abstract. Provides resources and provisions the environment.
This calls the underlying provider and provision resources (machines and networks).
- Parameters:
force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.
- Returns:
roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see
Enos_vagrant
- Return type:
(roles, networks) tuple
- offset_walltime(difference: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
Openstack Schema#
SCHEMA = {
"type": "object",
"properties": {
"resources": {"$ref": "#/resources"},
# Mandatory keys
"key_name": {"type": "string"},
"image": {"type": "string"},
"user": {"type": "string"},
"rc_file": {"type": "string"},
# optional keys
"allocation_pool": {"$ref": "#/os_allocation_pool"},
"configure_network": {"type": "boolean"},
"dns_nameservers": {"type": "array", "items": {"type": "string"}},
"gateway": {"type": "boolean"},
"gateway_user": {"type": "string"},
"network": {"$ref": "#/os_network"},
"subnet": {"$ref": "#/os_subnet"},
"prefix": {"type": "string"},
},
"additionalProperties": False,
"required": ["resources", "key_name", "image", "user", "rc_file"],
"os_allocation_pool": {
"title": "OSallocationPool",
"type": "object",
"properties": {"start": {"type": "string"}, "end": {"type": "string"}},
"required": ["start", "end"],
"additionalProperties": False,
},
"os_network": {
"title": "OSNetwork",
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
"additionalProperties": False,
},
"os_subnet": {
"title": "OSSubnet",
"type": "object",
"properties": {"name": {"type": "string"}, "cidr": {"type": "string"}},
"required": ["name", "cidr"],
"additionalProperties": False,
},
"resources": {
"title": "Resource",
"type": "object",
"properties": {
"machines": {"type": "array", "items": {"$ref": "#/machine"}},
"networks": {"type": "array", "items": {"type": "string"}},
},
"additionalProperties": False,
"required": ["machines", "networks"],
},
"machine": {
"title": "Compute",
"type": "object",
"properties": {
"roles": {"type": "array", "items": {"type": "string"}},
"flavour": {"type": "string"},
"number": {"type": "number"},
},
"required": ["roles", "number", "flavour"],
},
}
Chameleon#
Chameleon(kvm) Provider Class#
Classes:
|
- class enoslib.infra.enos_chameleonkvm.provider.Chameleonkvm(provider_conf, name: str | None = None)#
- destroy(wait: bool = False, **kwargs)#
Abstract. Destroy the resources used for the deployment.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs) Tuple[Roles, Networks] #
Abstract. Provides resources and provisions the environment.
This calls the underlying provider and provision resources (machines and networks).
- Parameters:
force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.
- Returns:
roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see
Enos_vagrant
- Return type:
(roles, networks) tuple
- offset_walltime(offset: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
Chameleon(kvm) Schema#
SCHEMA = {
"type": "object",
"properties": {
"resources": {"$ref": "#/resources"},
"key_name": {"type": "string"},
# everything is optional
"image": {"type": "string"},
"user": {"type": "string"},
"allocation_pool": {"$ref": "#/os_allocation_pool"},
"configure_network": {"type": "boolean"},
"dns_nameservers": {"type": "array", "items": {"type": "string"}},
"gateway": {"type": "boolean"},
"gateway_user": {"type": "string"},
"network": {"$ref": "#/os_network"},
"subnet": {"$ref": "#/os_subnet"},
"prefix": {"type": "string"},
},
"additionalProperties": True,
"required": ["resources", "key_name"],
"os_allocation_pool": {
"title": "OSallocationPool",
"type": "object",
"properties": {"start": {"type": "string"}, "end": {"type": "string"}},
"required": ["start", "end"],
"additionalProperties": False,
},
"os_network": {
"title": "OSNetwork",
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
"additionalProperties": False,
},
"os_subnet": {
"title": "OSSubnet",
"type": "object",
"properties": {"name": {"type": "string"}, "cidr": {"type": "string"}},
"required": ["name"],
"additionalProperties": False,
},
"resources": {
"title": "Resource",
"type": "object",
"properties": {
"machines": {"type": "array", "items": {"$ref": "#/machine"}},
"networks": {"type": "array", "items": {"type": "string"}},
},
"additionalProperties": False,
"required": ["machines", "networks"],
},
"machine": {
"title": "Compute",
"type": "object",
"properties": {
"roles": {"type": "array", "items": {"type": "string"}},
"flavour": {"type": "string"},
"number": {"type": "number"},
},
"required": ["roles", "number", "flavour"],
},
}
Chameleon(bare metal) Provider Class#
Classes:
|
- class enoslib.infra.enos_chameleonbaremetal.provider.Chameleonbaremetal(provider_conf, name: str | None = None)#
- destroy(wait: bool = False, **kwargs)#
Abstract. Destroy the resources used for the deployment.
- init(force_deploy: bool = False, start_time: int | None = None, **kwargs)#
Abstract. Provides resources and provisions the environment.
This calls the underlying provider and provision resources (machines and networks).
- Parameters:
force_deploy (bool) – Indicates that the resources must be redeployed.
start_time (timestamp (int)) – Date (UTC) when you want to have your resources ready.
- Returns:
roles is a dict whose key is a role and the value is the machines associated with this role. networks is the list of networks configured by the provider. see
Enos_vagrant
- Return type:
(roles, networks) tuple
- offset_walltime(offset: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
- set_reservation(timestamp: int)#
Change the internal reservation date.
Ignored on platform that aren’t based on a reservation system.
- Parameters:
timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.
Chameleon(bare metal) schema#
SCHEMA = {
"type": "object",
"properties": {
"resources": {"$ref": "#/resources"},
"key_name": {"type": "string"},
# everything is optional
"image": {"type": "string"},
"user": {"type": "string"},
"allocation_pool": {"$ref": "#/os_allocation_pool"},
"configure_network": {"type": "boolean"},
"dns_nameservers": {"type": "array", "items": {"type": "string"}},
"gateway": {"type": "boolean"},
"gateway_user": {"type": "string"},
"network": {"$ref": "#/os_network"},
"subnet": {"$ref": "#/os_subnet"},
"prefix": {"type": "string"},
},
"additionalProperties": True,
"required": ["resources", "key_name"],
"os_allocation_pool": {
"title": "OSallocationPool",
"type": "object",
"properties": {"start": {"type": "string"}, "end": {"type": "string"}},
"required": ["start", "end"],
"additionalProperties": False,
},
"os_network": {
"title": "OSNetwork",
"type": "object",
"properties": {"name": {"type": "string"}},
"required": ["name"],
"additionalProperties": False,
},
"os_subnet": {
"title": "OSSubnet",
"type": "object",
"properties": {"name": {"type": "string"}, "cidr": {"type": "string"}},
"required": ["name"],
"additionalProperties": False,
},
"resources": {
"title": "Resource",
"type": "object",
"properties": {
"machines": {"type": "array", "items": {"$ref": "#/machine"}},
"networks": {"type": "array", "items": {"type": "string"}},
},
"additionalProperties": False,
"required": ["machines", "networks"],
},
"machine": {
"title": "Compute",
"type": "object",
"properties": {
"roles": {"type": "array", "items": {"type": "string"}},
"flavour": {"type": "string"},
"number": {"type": "number"},
},
"required": ["roles", "number", "flavour"],
},
}
Providers Class#
Classes:
|
A provider that syncs resources of different infrastructures. |
Functions:
|
Search for a time slot at which all of the provider in "providers" are able to reserve their configurations time_window is how long in the future are we willing to be looking for start_time is when we start trying to look for a slot, by default a minute after the function is called |
|
Try to find a common time slot for all the Provider in providers to start and then start them |
|
Adjust provider walltime and reservation_date to fit into a slot |
- class enoslib.infra.providers.Providers(providers: List[Provider])#
Bases:
Provider
A provider that syncs resources of different infrastructures.
- Parameters:
providers – List of Provider instances that you wish to use
Methods:
async_init
([force_deploy, start_time, ...])Partial init: secure the resources to the targeted infrastructure.
destroy
([wait])Abstract.
init
([force_deploy, start_time, time_window])The provider to use when you want to sync multiple providers.
Is the provider already created.
offset_walltime
(offset)Offset the walltime.
set_reservation
(timestamp)Change the internal reservation date.
test_slot
(start_time, end_time)Test a slot that starts at a given point in time.
- async_init(force_deploy: bool = False, start_time: int | None = None, time_window: int | None = None, **kwargs)#
Partial init: secure the resources to the targeted infrastructure.
This is primarily used internally by
Providers
to get the resources from different platforms. As this method actually starts some real resources somewhere, errors may occur (e.g no more available resources, …). It’s up to the provider to indicate if the error is critical or not. For instance anInvalidReservationTime
can be raised to indicate the Providers to retry later.- Parameters:
kwargs – keyword arguments. Fit those from
init()
- Raises:
InvalidReservationTime – Resources can’t be reserved at the specific time.
InvalidReservationTooOld – The reservation time is in the past
_ – provider specific exception
- destroy(wait: bool = True, **kwargs)#
Abstract. Destroy the resources used for the deployment.
- init(force_deploy: bool = False, start_time: int | None = None, time_window: int | None = None, **kwargs) Tuple[Roles, Networks] #
The provider to use when you want to sync multiple providers.
This will call init on each provider after finding a common possible reservation date for each one of them. It uses
find_slot()
andstart_provider_within_bounds()
internally.Idempotency: ideally calling this function twice should reload existing reservations on each platform. However, the current behaviour might differ from this specification but we’ll be happy to get your feedback on this.
- Parameters:
time_window – duration in seconds How long in the future are you willing to look for a possible start time
start_time – timestamp in seconds The first start_time you will test, incremented after each try (5 minutes increment)
- Returns:
Providers’ roles and networks similar to
init()
return value.- Raises:
NoSlotError – If no common slot can be found
- is_created() bool #
Is the provider already created.
- offset_walltime(offset: int)#
Offset the walltime.
Increase or reduce the wanted walltime. This does not change the walltime of an already created provider but only affects the walltime configured before the provider calls ~init~.
- Raises:
NegativeWalltime – the walltime is negative
- set_reservation(timestamp: int)#
Change the internal reservation date.
Ignored on platform that aren’t based on a reservation system.
- Parameters:
timestamp (timestamp (int)) – The reservation date (UTC) as timestamp in seconds.
- test_slot(start_time: int, end_time: int) bool #
Test a slot that starts at a given point in time.
A slot is given by a start_time, a duration and an amount of resources. The two latter are found in the internal configuration.
- Parameters:
start_time (timestamp (int)) – Test for a possible slot that starts at start_time (UTC)
end_time – (timestamp (int)): How much time in the future we should look for possible reservation. This is used on some platform to know how much time in the future we look in the planning data. Timestamp is based on UTC.
- Returns:
True iff the slot is available
- enoslib.infra.providers.find_slot(providers: Sequence[Provider], time_window: int, start_time: int) int #
Search for a time slot at which all of the provider in “providers” are able to reserve their configurations time_window is how long in the future are we willing to be looking for start_time is when we start trying to look for a slot, by default a minute after the function is called
- Parameters:
providers – A list of providers
time_window – How long in the future are you willing to look for for a start time Must be positive.
start_time – The first start_time you will test, incremented after each try (5 minutes increment). Must be positive.
- Raises:
NoSlotError – If no compatible slot can be found for all provided providers
- enoslib.infra.providers.find_slot_and_start(providers: Sequence[Provider], start_time: int, time_window: int, **kwargs)#
Try to find a common time slot for all the Provider in providers to start and then start them
This search for a reservation date that will fit all provider in providers and the call do_init with them on that found reservation date If this fail it will try to raise an error indicating a next possible slot if possible
- Raises:
InvalidReservationTime – Happens if one of the provider cannot be initialized
with reservation_timestamp as reservation date. Will provide an indication –
for a potential next possible time slot –
- enoslib.infra.providers.start_provider_within_bounds(provider: Provider, start_time: int, **kwargs)#
Adjust provider walltime and reservation_date to fit into a slot
Mutate the reservation/walltime attributes until finding a slot within [start_time, start_time + provider.walltime] where the provider can be started.
The slot found is guaranteed to:
not exceed the right bound (a negative walltime would raise an error) in the current implementation, we chose to work with a fixed end time.
and start in the future: start_time might be in the past, and reserving resources in the past is weird (and not allowed in some testbeds). We do our best to fiddle with the start_time to make sure it’s in the future. If not we retry with an even further start_time.
Finding the slot depends on a loop on error strategy which is stopped when the retry limit is hit or the walltime become to small. In both cases NoSlotError is raised. Otherwise the same errors as
init()
can be raised (except InvalidReservationTooOld which can be caught internally)- Raises:
InvalidReservationTime – If a provider object cannot be initialized anymore, due to an update to it’s related platform status since we first fetched it
NoSlotError – If a Providers object cannot be initialized at its given start_time or if a provider fails to be initialized after too many retries.