Chameleon/ChameleonEdge tutorials#
This tutorial will let you get started using EnOSlib and Chameleon/ChameleonEdge. Chameleon provider is a specialization of the OpenStack provider (so this should be theoretically possible to work with a custom OpenStack platform, but we didn’t push too much effort on this lately :))
Here we present you the bare minimum to install the library with the required dependencies.
Hint
For a complete schema reference see Openstack Schema
Installation#
# This will install the openstack clients
$ pip install enoslib[chameleon]
Note
It’s a good practice to use a virtualenv or a python version manager like pyenv.
Chameleon Bare-metal Example#
The following reserve 2 nodes on the chameleon bare-metal infrastructure. The preferred way of authenticating to the Chameleon provider using EnOSlib is through application credentials (EnOSlib will transparently source your application credentials when needed)
1import enoslib as en
2
3en.init_logging()
4en.check()
5
6provider_conf = {
7 "key_name": "enos_matt",
8 "lease_name": "mylease",
9 "rc_file": "./my-app-cred-edge-openrc.sh", # FIXME use your OPENRC file
10 "resources": {
11 "machines": [
12 {
13 "roles": ["server"],
14 "flavour": "compute_skylake",
15 "number": 1,
16 },
17 {
18 "roles": ["client"],
19 "flavour": "compute_skylake",
20 "number": 1,
21 },
22 ],
23 "networks": ["network_interface"],
24 },
25}
26
27tc = {
28 "enable": True,
29 "default_delay": "20ms",
30 "default_rate": "1gbit",
31}
32conf = en.CBMConf.from_dictionary(provider_conf)
33provider = en.CBM(conf)
34
35roles, networks = provider.init()
36
37roles = en.sync_info(roles, networks)
38
39# Experimentation logic starts here
40with en.actions(roles=roles) as p:
41 # flent requires python3, so we default python to python3
42 p.apt_repository(
43 repo="deb http://deb.debian.org/debian stretch main contrib non-free",
44 state="absent",
45 )
46 p.apt(
47 name=["flent", "netperf", "python3-setuptools", "python3-matplotlib"],
48 allow_unauthenticated=True,
49 state="present",
50 )
51
52with en.actions(pattern_hosts="server", roles=roles) as p:
53 p.shell("nohup netperf &")
54
55with en.actions(pattern_hosts="client", roles=roles) as p:
56 p.shell(
57 "flent rrul -p all_scaled "
58 + "-l 60 "
59 + "-H {{ hostvars[groups['server'][0]].ansible_default_ipv4.address }} "
60 + "-t 'bufferbloat test' "
61 + "-o result.png"
62 )
63 p.fetch(src="result.png", dest="result")
Note
Similarly to other provider the configuration can be generated programmatically instead of using a dict.
The result of the above code is the following:
Chameleon / ChameleonEdge Example#
To mimic an FOG deployment you can claim resources on the Chameleon Infrastructure (acting as the Cloud) and the ChameleonEdge Infrastructure (acting as the Edge). The following example lets you deploy a simple Machine Learning pipeline: images are collected at the Edge and sent to the cloud for inference. We show the main script, other resources can be found in EnOSlib source code.
1# Chameleon User Guide: Edge to Cloud
2# This example is based on:
3# https://www.chameleoncloud.org/experiment/share/37991779-fd7b-4ab0-8d6f-e726a9204946
4
5import logging
6import os
7import time
8
9import enoslib as en
10
11en.init_logging(level=logging.INFO)
12logging.basicConfig(level=logging.INFO)
13en.check()
14
15prefix = os.getlogin()
16_walltime = "02:00:00"
17providers = []
18# Leasing resources on Chameleon Cloud
19provider_conf = {
20 "walltime": _walltime,
21 "lease_name": f"{prefix}-enoslib-chicloud-lease",
22 "rc_file": "./my-app-cred-cloud-openrc.sh", # FIXME use your OPENRC file
23 "key_name": "drosendo", # FIXME use your key_name
24 "image": "CC-Ubuntu20.04",
25 "resources": {
26 "machines": [
27 {
28 "roles": ["server"],
29 "flavour": "gpu_rtx_6000",
30 "number": 1,
31 }
32 ],
33 "networks": ["network_interface"],
34 },
35}
36conf = en.CBMConf().from_dictionary(provider_conf)
37provider = en.CBM(conf)
38roles, networks = provider.init()
39providers.append(provider)
40roles = en.sync_info(roles, networks)
41floating_ip = roles["server"][0].extra["gateway"] # 192.5.87.127
42
43# Leasing resources on Chameleon Edge
44provider_conf = {
45 "walltime": _walltime,
46 "lease_name": f"{prefix}-enoslib-chiedge-lease",
47 "rc_file": "./my-app-cred-edge-openrc.sh", # FIXME use your OPENRC file
48 "resources": {
49 "machines": [
50 {
51 "roles": ["client"],
52 "device_name": "iot-rpi4-03",
53 "container": {
54 "name": "cli-container",
55 "image": "arm64v8/ubuntu",
56 },
57 }
58 ],
59 },
60}
61conf = en.ChameleonEdgeConf.from_dictionary(provider_conf)
62provider = en.ChameleonEdge(conf)
63roles_edge, networks_edge = provider.init()
64providers.append(provider)
65
66# Merging Chameleon Cloud and Edge resources
67for role, hosts in roles_edge.items():
68 roles[role] = hosts
69logging.info("*" * 40 + f" roles{type(roles)} = {roles}")
70logging.info("*" * 40 + f" networks{type(networks)} = {networks}")
71
72# Experimentation logic starts here
73# Chameleon Cloud
74dest_dir = "/tmp"
75with en.play_on(roles=roles["server"]) as p:
76 p.copy(src="./artifacts_cloud/", dest=dest_dir)
77 p.shell(f"cd {dest_dir} && bash {dest_dir}/cloud_worker.sh > {dest_dir}/tests")
78# Chameleon Edge
79for device in roles["client"]:
80 cmd_upload = device.upload("./artifacts_edge/", "/")
81 cmd_execute = device.execute(f"bash /edge_worker.sh edge_data 100 {floating_ip}")
82
83logging.info("Running experiment for 600 secs...")
84time.sleep(600)
85# How to check execution? ssh to the Cloud server: ssh cc@<floating_ip>
86# cc@<floating_ip>:~# tail -f /tmp/predict.log you may also check mosquitto
87# topic (mosquitto_sub_img.py downloads images received in running dir):
88
89# $) python mosquitto_sub_img.py --topic edge_data --mqtt_broker 192.5.87.127
90
91# Releasing resources from Chameleon Cloud and Edge
92logging.info("Releasing resources.")
93for p in providers:
94 p.destroy()