Provider::VMonG5K#

This tutorial leverages the VmonG5k provider: a provider that provisions virtual machines for you on Grid’5000.

Hint

For a complete schema reference see VMonG5k Schema

Installation#

On Grid’5000, you can go with a virtualenv :

$ virtualenv -p python3 venv
$ source venv/bin/activate
$ pip install -U pip

$ pip install enoslib

Configuration#

Since python-grid5000 is used behind the scene, the configuration is read from a configuration file located in the home directory. It can be created with the following:

echo '
username: MYLOGIN
password: MYPASSWORD
' > ~/.python-grid5000.yaml

chmod 600 ~/.python-grid5000.yaml

With the above you can access the Grid’5000 API from you local machine aswell.

External access#

If you want to control you experiment from the outside of Grid’5000 (e.g from your local machine) you can refer to the following. You can jump this section if you work from inside Grid’5000.

SSH external access#

  • Solution 1: use the Grid’5000 VPN

  • Solution 2: configure you ~/.ssh/config properly:

Host !access.grid5000.fr *.grid5000.fr
   User <login>
   ProxyJump <login>@access.grid5000.fr
   StrictHostKeyChecking no
   UserKnownHostsFile /dev/null
   ForwardAgent yes

Accessing HTTP services inside Grid’5000#

If you want to control you experiment from the outside of Grid’5000 (e.g from your local machine). For instance the Distem provider is starting a web server to handle the client requests. In order to access it propertly externally you drom your local machine can either

  • Solution 1 (general): use the Grid’5000 VPN

  • Solution 2 (HTTP traffic only): create a socks tunnel from your local machine to Grid’5000

    # on one shell
    ssh -ND 2100 access.grid5000.fr
    
    # on another shell
    export https_proxy="socks5h://localhost:2100"
    export http_proxy="socks5h://localhost:2100"
    
    # Note that browsers can work with proxy socks
    chromium-browser --proxy-server="socks5://127.0.0.1:2100" &
    
  • Solution 3 (ad’hoc): create a forwarding port tunnel

    # on one shell
    ssh -NL 3000:paravance-42.rennes.grid5000.fr:3000 access.grid5000.fr
    
    # Now all traffic that goes on localhost:3000 is forwarded to paravance-42.rennes.grid5000.fr:3000
    
  • Solution 3’ : The same but programmatically with enoslib.infra.enos_g5k.provider.G5kTunnel (See also Create a tunnel to a service)

To accesss your virtual machines from your local machine, see below.

Basic example#

We’ll imagine a system that requires 5 compute machines and 1 controller machine. We express this using the ~VmonG5K~ provider:

 1from pathlib import Path
 2
 3import enoslib as en
 4
 5
 6_ = en.init_logging()
 7
 8job_name = Path(__file__).name
 9
10# claim the resources
11conf = (
12    en.VMonG5kConf
13    .from_settings(job_name=job_name)
14    .add_machine(
15        roles=["docker", "compute"],
16        cluster="paravance",
17        number=5,
18        flavour_desc={
19            "core": 1,
20            "mem": 1024
21        }
22    )
23    .add_machine(
24        roles=["docker", "control"],
25        cluster="paravance",
26        number=1,
27        flavour="large"
28    )
29
30    .finalize()
31)
32
33provider = en.VMonG5k(conf)
34
35roles, networks = provider.init()
36print(roles)
37print(networks)
38
39en.wait_for(roles)
40
41# install docker on the nodes
42# bind /var/lib/docker to /tmp/docker to gain some places
43docker = en.Docker(agent=roles["docker"], bind_var_docker="/tmp/docker")
44docker.deploy()
45
46# start containers.
47# Here on all nodes
48with en.actions(roles=roles) as a:
49    a.docker_container(
50        name="mycontainer",
51        image="nginx",
52        ports=["80:80"],
53        state="started",
54    )
  • You can launch the script using :

    $ python tuto_vmg5k.py
    
  • The raw data structures of EnOSlib will be displayed and you should be able to connect to any machine using SSH and the root account.

Notes#

  • The VmonG5K provider internally uses the G5k provider. In particular it sets the job_type to allow_classic_ssh and claim an extra slash_22 subnet.

  • SSH access will be granted to the VMs using the ~/.ssh/id_rsa | ~/.ssh/id_rsa.pub keypair. So these files must be present in your home directory.

  • The working_dir setting controls where the temporary files and virtual images disks will be stored. The default is to store everything in the temp folder of the physical nodes.

  • You might be interested in adding wait_ssh(roles) (from enoslib.api) just after init() to make sure SSH is up and running on all VMs. Otherwise you might get an unreachable error from SSH.

Warning

The working_dir and all its content is deleted by the provider when calling destroy.

Changing resource size of virtual machines#

As for the CPU and memory resources, you can simply change the name of the flavour (available flavours are listed here), or create your own flavour with flavour_desc.

[...]
.add_machine(
    [...],
    flavour_desc={"core": 1, "mem": "512"}
)

Notes on the disks of Virtual Machines#

  • Adding a new disk: Using the disk attribute of flavour_desc will create a new disk and make it available to the VM. For instance to get an extra disk of 10GB you can use this python configuration parameter:

    [...]
    .add_machine(
        [...],
        flavour_desc={"core": 1, "mem": 512, "disk": 10}
    )
    

Note that with the above configuration an extra disk of 10GB will be provisioned and available to the Virtual Machine. In the current implementation, the disk is neither formatted nor mounted in the Virtual Machine OS.

  • Make an external (not managed by EnOSlib) disk available to the Virtual Machine A typical use case is to use an hardware disk from the host machine. In this situation, use the extra_devices parameter of the configuration. It corresponds to the XML string of Libvirt.

    [...]
    .add_machine(
        [...],
        extra_devices = """
        <disk type='block' device='disk'>
        <driver name='qemu' type='raw'/>
        <source dev='/dev/disk/by-path/pci-0000:82:00.0-sas-phy1-lun-0'/>
        <target dev='vde' bus='virtio'/>
        </disk>
        """
    
  • Resize the root filesystem To do so, you will need to get the qcow2 file, put it in your public folder, and resize it. Location of the file is shown here.

    cp /grid5000/virt-images/debian10-x64-nfs.qcow2 $HOME/public/original.qcow2
    cd $HOME/public
    qemu-img info original.qcow2  # check the size (should be 10GB)
    qemu-img resize original.qcow2 +30G
    # at this stage, image is resized at 40GB but not the partition
    virt-resize --expand /dev/sda1 original.qcow2 my-image.qcow2
    rm original.qcow2
    # now you can check the size of each partition (/dev/sda1 should be almost 40GB)
    virt-filesystems –long -h –all -a my-image.qcow2
    

    Finally, you need to tell EnosLib to use this file with:

    Configuration.from_settings(...
                                image="/home/<username>/public/my-image.qcow2",
                                ...
                                )
    

EnOSlib primer using VMonG5k#

 1import logging
 2from pathlib import Path
 3
 4import enoslib as en
 5
 6logging.basicConfig(level=logging.DEBUG)
 7
 8job_name = Path(__file__).name
 9
10conf = (
11    en.VMonG5kConf()
12    .from_settings(job_name=job_name, gateway=True)
13    .add_machine(roles=["server"], cluster="paravance", number=1)
14    .add_machine(roles=["client"], cluster="paravance", number=1)
15    .finalize()
16)
17
18provider = en.VMonG5k(conf)
19
20roles, networks = provider.init()
21en.wait_for(roles)
22
23with en.actions(roles=roles) as p:
24    # flent requires python3, so we default python to python3
25    p.shell("update-alternatives --install /usr/bin/python python /usr/bin/python3 1")
26    p.apt_repository(
27        repo="deb http://deb.debian.org/debian stretch main contrib non-free",
28        state="present",
29    )
30    p.apt(
31        name=["flent", "netperf", "python3-setuptools", "python3-matplotlib"],
32        state="present",
33    )
34
35with en.actions(pattern_hosts="server", roles=roles) as p:
36    p.shell("nohup netperf &")
37
38with en.actions(pattern_hosts="client", roles=roles) as p:
39    server_address = roles["server"][0].address
40    p.shell(
41        "flent rrul -p all_scaled "
42        + "-l 60 "
43        + f"-H { server_address } "
44        + "-t 'bufferbloat test' "
45        + "-o result.png"
46    )
47    p.fetch(src="result.png", dest="result")

SSH external access to the virtual machines#

This is mandatory if you deployed from your local machine.

  • Solution 1: use the Grid’5000 VPN

  • Solution 2: Add the following in your configuration force Ansible to jump through a gateway (access.grid5000.fr):

    :: code-block:: python

    Configuration.from_settings(…

    gateway=True …

    )

Controlling the virtual machines placement#

 1from itertools import islice
 2import logging
 3from pathlib import Path
 4
 5import enoslib as en
 6
 7en.init_logging(logging.INFO)
 8
 9job_name = Path(__file__).name
10
11CLUSTER = "parasilo"
12SITE = "rennes"
13
14
15prod_network = en.G5kNetworkConf(
16    id="n1",
17    type="prod",
18    roles=["my_network"],
19    site=SITE)
20conf = (
21    en.G5kConf
22    .from_settings(
23        job_type="allow_classic_ssh",
24        job_name=job_name
25    )
26    .add_network_conf(prod_network)
27    .add_network(
28        id="not_linked_to_any_machine",
29        type="slash_22",
30        roles=["my_subnet"],
31        site=SITE
32    )
33    .add_machine(
34        roles=["role1"],
35        cluster=CLUSTER,
36        nodes=1,
37        primary_network=prod_network
38    )
39    .add_machine(
40        roles=["role2"],
41        cluster=CLUSTER,
42        nodes=1,
43        primary_network=prod_network
44    )
45    .finalize()
46 )
47
48provider = en.G5k(conf)
49roles, networks = provider.init()
50roles = en.sync_info(roles, networks)
51
52# Retrieving subnets
53subnet = networks["my_subnet"]
54logging.info(subnet)
55
56# We describe the VMs types and placement in the following
57# We build a VMonG5KConf with some extra fields:
58# - undercloud: where the VMs should be placed (round robin)
59# - macs: list of macs to take: on G5k the dhcp is configured to assign specific
60#   ip based on the configured mac
61
62n_vms = 16
63virt_conf = (
64    en.VMonG5kConf
65    .from_settings(image="/grid5000/virt-images/debian11-x64-base.qcow2")
66    # Starts some vms on a single role
67    # Here that means start the VMs on a single machine
68    .add_machine(
69        roles=["vms"],
70        number=n_vms,
71        undercloud=roles["role1"],
72        macs=list(islice(subnet[0].free_macs, n_vms))
73        # alternative
74        # macs=list(islice(en.mac_range(subnet), n_vms))
75    )
76    .finalize()
77)
78
79# Start them
80vmroles = en.start_virtualmachines(virt_conf)
81print(vmroles)
82print(networks)

Multisite Support#

You can specify clusters from different sites in the configuration. The provider will take care of reserving nodes and subnet on the different sites and configure the VMs’ network card accordingly.

Mounting your home directory (or a group storage)#

Mounting your home directory within the VMs is a two steps process. It first relies on a white list of IPS allowed to mount the NFS exported home: so you need to add your VM’s IPS to this list. This is done using an REST API call. Second, you need to mount the home inside your VMs.

 1from pathlib import Path
 2
 3import enoslib as en
 4
 5
 6_ = en.init_logging()
 7
 8job_name = Path(__file__).name
 9
10# claim the resources
11conf = (
12    en.VMonG5kConf.from_settings(job_name=job_name)
13    .add_machine(
14        roles=["vms"],
15        cluster="paravance",
16        number=5,
17        flavour_desc={"core": 1, "mem": 1024},
18    )
19    .finalize()
20)
21
22provider = en.VMonG5k(conf)
23
24roles, networks = provider.init()
25print(roles)
26print(networks)
27
28en.wait_for(roles)
29
30# get the job
31job = provider.g5k_provider.jobs[0]
32
33# get the ips to white list
34ips = [vm.address for vm in roles["vms"]]
35
36# add ips to the white list for the job duration
37en.g5k_api_utils.enable_home_for_job(job, ips)
38
39# mount the home dir
40username = en.g5k_api_utils.get_api_username()
41with en.actions(roles=roles) as a:
42    a.mount(
43        src=f"nfs:/export/home/{username}",
44        path=f"/home/{username}",
45        fstype="nfs",
46        state="mounted",
47    )

Note that you can allow any group storage using enable_group_storage()