{ "cells": [ { "cell_type": "markdown", "id": "functioning-experience", "metadata": { "tags": [] }, "source": [ "# Remote actions and variables\n", "\n", "Changing the state of remote resources\n", "\n", "\n", "---\n", "\n", "- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html\n", "- Instant chat: https://framateam.org/enoslib\n", "- Source code: https://gitlab.inria.fr/discovery/enoslib\n", "\n", "---\n", "\n", "## Prerequisites\n", "\n", "
\n", " Make sure you've run the one time setup for your environment\n", "
\n" ] }, { "cell_type": "code", "execution_count": null, "id": "funded-azerbaijan", "metadata": {}, "outputs": [], "source": [ "import enoslib as en\n", "\n", "# Enable rich logging\n", "_ = en.init_logging()" ] }, { "cell_type": "markdown", "id": "familiar-intention", "metadata": {}, "source": [ "## Setup on Grid'5000\n", "\n", "EnOSlib uses `Providers` to ... provide resources. They transform an abstract resource configuration into a concrete one.\n", "To do so, they interact with an infrastructure where they get the resources from. There are different providers in EnOSlib: \n", "\n", "- Vbox/KVM to work with locally hosted virtual machines\n", "- Openstack/Chameleon to work with bare-metal resources hosted in the Chameleon platform\n", "- FiT/IOT lab to work with sensors or low profile machines\n", "- VmonG5k to work with virtual machines on Grid'5000\n", "- Distem to work with lxc containers on Grid'5000\n", "- **Grid'5000**, with options to configure several networks easily\n", "\n", "A providers eases the use of the platform by internalizing some of the configuration tasks (e.g automatically managing the reservation on G5k, network configuration ...)\n", "\n", "### Describing the resources\n", "\n", "For the purpose of the tutorial we'll reserve 2 nodes in the production environment.\n", "\n", "First we build a configuration object describing the wanted resources: `machines` and `networks`." ] }, { "cell_type": "code", "execution_count": null, "id": "bulgarian-honolulu", "metadata": {}, "outputs": [], "source": [ "network = en.G5kNetworkConf(type=\"prod\", roles=[\"my_network\"], site=\"rennes\")\n", "\n", "conf = (\n", " en.G5kConf.from_settings(job_type=[], job_name=\"rsd-01\")\n", " .add_network_conf(network)\n", " .add_machine(\n", " roles=[\"control\"], cluster=\"parasilo\", nodes=1, primary_network=network\n", " )\n", " .add_machine(\n", " roles=[\"compute\"],\n", " cluster=\"parasilo\",\n", " nodes=1,\n", " primary_network=network,\n", " )\n", " .finalize()\n", ")\n", "conf" ] }, { "cell_type": "markdown", "id": "green-producer", "metadata": {}, "source": [ "### Reserving the resources\n", "\n", "We can pass the `Configuration` object to the `G5k` provider. " ] }, { "cell_type": "code", "execution_count": null, "id": "alike-immigration", "metadata": {}, "outputs": [], "source": [ "provider = en.G5k(conf)\n", "roles, networks = provider.init()" ] }, { "cell_type": "markdown", "id": "asian-stretch", "metadata": {}, "source": [ "Inspecting the ressources we own for the experiment's lifetime:\n", "\n", "- roles: this is somehow a dictionnary whose keys are the role names and the associated values are the corresponding list of hosts\n", "- networks: similar to roles but for networks" ] }, { "cell_type": "code", "execution_count": null, "id": "sunset-voluntary", "metadata": {}, "outputs": [], "source": [ "roles" ] }, { "cell_type": "code", "execution_count": null, "id": "regional-procedure", "metadata": {}, "outputs": [], "source": [ "# list of host on a given role\n", "roles[\"control\"]" ] }, { "cell_type": "code", "execution_count": null, "id": "nonprofit-excess", "metadata": {}, "outputs": [], "source": [ "# a single host\n", "roles[\"control\"][0]" ] }, { "cell_type": "code", "execution_count": null, "id": "latest-affiliate", "metadata": {}, "outputs": [], "source": [ "networks" ] }, { "cell_type": "markdown", "id": "danish-circle", "metadata": {}, "source": [ "`provider.init` is idempotent. In the Grid'5000 case, you can call it several time in a row. The same reservation will reloaded and the roles and networks will be the same." ] }, { "cell_type": "code", "execution_count": null, "id": "furnished-angle", "metadata": {}, "outputs": [], "source": [ "roles, networks = provider.init()\n", "roles" ] }, { "cell_type": "code", "execution_count": null, "id": "blocked-marker", "metadata": {}, "outputs": [], "source": [ "# sync some more information in the host data structure (for illustration purpose here)\n", "roles = en.sync_info(roles, networks)" ] }, { "cell_type": "code", "execution_count": null, "id": "fbbfe094-766b-4ddf-8e2c-78225d88761f", "metadata": {}, "outputs": [], "source": [ "# the hosts have been populated with some new information\n", "roles" ] }, { "cell_type": "markdown", "id": "abandoned-british", "metadata": {}, "source": [ "## Acting on remote nodes\n", "\n", "### run a command, filter results" ] }, { "cell_type": "code", "execution_count": null, "id": "suited-nickname", "metadata": {}, "outputs": [], "source": [ "results = en.run_command(\"whoami\", roles=roles)\n", "results" ] }, { "cell_type": "code", "execution_count": null, "id": "favorite-updating", "metadata": {}, "outputs": [], "source": [ "one_result = results.filter(host=roles[\"control\"][0].alias)[0]\n", "one_result" ] }, { "cell_type": "code", "execution_count": null, "id": "prepared-liverpool", "metadata": {}, "outputs": [], "source": [ "one_result.payload[\"stdout\"]" ] }, { "cell_type": "markdown", "id": "hollow-bulgaria", "metadata": {}, "source": [ "There are some specific shortcuts when the remote actions is a remote (shell) command: `.stdout`, `.stderr`, `.rc`" ] }, { "cell_type": "code", "execution_count": null, "id": "reported-intro", "metadata": {}, "outputs": [], "source": [ "print(f\"stdout = {one_result.stdout}\\n\", f\"stderr={one_result.stderr}\\n\", f\"return code = {one_result.rc}\")" ] }, { "cell_type": "markdown", "id": "operating-testimony", "metadata": {}, "source": [ "By default the user is `root` (this is common to all EnOSlib's provider).\n", "If you want to run command as your regular Grid'5000 user you can tell the command to `sudo` back to your regular user using `run_as` (the SSH login is still `root` though)" ] }, { "cell_type": "code", "execution_count": null, "id": "surprising-junior", "metadata": {}, "outputs": [], "source": [ "my_g5k_login = en.g5k_api_utils.get_api_username()\n", "results = en.run_command(\"whoami\", roles=roles, run_as=my_g5k_login)\n", "results" ] }, { "cell_type": "markdown", "id": "inner-monster", "metadata": {}, "source": [ "### Filtering hosts on which the command is run\n", "\n", "`run_command` acts on remote hosts. Those hosts can be given as a `Roles` type (output of `provider.init`) or as a list of `Host` or a single `Host`. \n" ] }, { "cell_type": "code", "execution_count": null, "id": "spectacular-wellington", "metadata": {}, "outputs": [], "source": [ "# some roles\n", "en.run_command(\"date\", roles = roles)" ] }, { "cell_type": "code", "execution_count": null, "id": "accompanied-coverage", "metadata": {}, "outputs": [], "source": [ "# a list of hosts\n", "en.run_command(\"date\", roles = roles[\"control\"])" ] }, { "cell_type": "code", "execution_count": null, "id": "mineral-transparency", "metadata": {}, "outputs": [], "source": [ "# a single host\n", "en.run_command(\"date\", roles=roles[\"control\"][0])" ] }, { "cell_type": "markdown", "id": "invisible-gabriel", "metadata": {}, "source": [ "A `pattern_hosts` can also be supplied. The pattern can be a regexp but [other patterns are possible](\n", "https://docs.ansible.com/ansible/latest/user_guide/intro_patterns.html#common-patterns)" ] }, { "cell_type": "code", "execution_count": null, "id": "detected-wholesale", "metadata": {}, "outputs": [], "source": [ "# co* matches all hosts\n", "en.run_command(\"date\", roles=roles, pattern_hosts=\"co*\")\n", "\n", "# com* only matches host with `compute` tags\n", "en.run_command(\"date\", roles=roles, pattern_hosts=\"com*\")" ] }, { "cell_type": "code", "execution_count": null, "id": "every-albany", "metadata": {}, "outputs": [], "source": [ "# you can forge some host yourself\n", "# Here we run the command on the frontend: this should work if your SSH parameters are correct\n", "en.run_command(\"date\", roles=en.Host(\"rennes.grid5000.fr\", user=en.g5k_api_utils.get_api_username()))" ] }, { "cell_type": "markdown", "id": "33bca3c7-0f2a-4350-b7bb-d40a31bb805f", "metadata": {}, "source": [ "### Dealing with failures\n", "\n", "By default, failures (command failure, host unreachable) raises on exception: this breaks your execution flow.\n", "Sometime you just want to allow some failures to happen. For this purpose you can add `on_error_continue=True`" ] }, { "cell_type": "code", "execution_count": null, "id": "be5eb4e5-6ce6-4233-b545-d4cd99e6b151", "metadata": {}, "outputs": [], "source": [ "en.run_command(\"non existing command\", roles=roles, on_error_continue=True)\n", "print(\"This is printed, so the execution can continue\")" ] }, { "cell_type": "markdown", "id": "extreme-carol", "metadata": { "tags": [], "toc-hr-collapsed": true }, "source": [ "### Remote actions\n", "\n", "Tools like Ansible, Puppet, Chef, Terraform ... are shipped with a set of predefined remote actions to ease the administrator life.\n", "\n", "Actions like copying file, adding some users, managing packages, making sure a line is absent from a configuration file, managing docker containers ... are first-class citizens actions and brings some nice garantees of correctness and idempotency.\n", "\n", "There are 1000+ modules available:\n", "https://docs.ansible.com/ansible/2.9/modules/list_of_all_modules.html\n", "\n", "---\n", "\n", "EnOSlib wraps Ansible module and let you use them from Python (without writting any YAML file). You can call any module by using the `actions` context manager:\n", "\n", "In the following we install docker (using g5k provided script) and a docker container. We also need to install the python docker binding on the remote machine so that Ansible can interact with the docker daemons on the remote machines. This block of actions is idempotent.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "organic-vanilla", "metadata": {}, "outputs": [], "source": [ "with en.actions(roles=roles) as a:\n", " # installing the docker daemon\n", " # prepending with a guard to make the command idempotent\n", " a.shell(\"which docker || /grid5000/code/bin/g5k-setup-docker\")\n", " # install the python docker binding on the remote host\n", " # mandatory by the docker_container module\n", " a.pip(name=\"docker\", state=\"present\")\n", " # fire up a container (forward port 80 at the host level)\n", " a.docker_container(name=\"myserver\", image=\"nginx\", state=\"started\", ports=[\"80:80\"])\n", " # wait for the connection on the port 80 to be ready\n", " a.wait_for(port=80, state=\"started\")\n", " # keep track of the result of each modules\n", " # not mandatory but nice :)\n", " results = a.results" ] }, { "cell_type": "code", "execution_count": null, "id": "enhanced-highlight", "metadata": {}, "outputs": [], "source": [ "results.filter(task=\"docker_container\")[0]" ] }, { "cell_type": "markdown", "id": "liquid-quest", "metadata": {}, "source": [ "### Background actions\n", "\n", "Sometime you need to fire a process on some remote machines that needs to survive the remote connection that started it. EnOSlib provides a `keyword` argument for this purpose and can be used when calling modules (when supported)." ] }, { "cell_type": "code", "execution_count": null, "id": "thorough-heritage", "metadata": {}, "outputs": [], "source": [ "# synchronous execution, will wait until the end of the shell command\n", "results = en.run_command(\"for i in $(seq 1 10); do sleep 1; echo toto; done\", roles=roles)\n", "results" ] }, { "cell_type": "code", "execution_count": null, "id": "protected-complexity", "metadata": {}, "outputs": [], "source": [ "# The remote command will be daemonize on the remote machines\n", "results = en.run_command(\"for i in $(seq 1 10); do sleep 1; echo toto; done\", roles=roles, background=True)\n", "results" ] }, { "cell_type": "code", "execution_count": null, "id": "returning-trout", "metadata": {}, "outputs": [], "source": [ "# you can get back the status of the daemonized process by reading the remote results_file\n", "# but we need to wait the completion, so forcing a sleep here (one could poll the status)\n", "import time\n", "time.sleep(15)\n", "h = roles[\"control\"][0]\n", "result_file = results.filter(host=h.alias)[0].results_file\n", "cat_result = en.run_command(f\"cat {result_file}\",roles=h)\n", "cat_result" ] }, { "cell_type": "code", "execution_count": null, "id": "executed-philosophy", "metadata": {}, "outputs": [], "source": [ "# the result_file content is json encoded so decoding it\n", "import json\n", "print(json.loads(cat_result[0].stdout)[\"stdout\"])" ] }, { "cell_type": "markdown", "id": "tracked-dialogue", "metadata": {}, "source": [ "## Using variables" ] }, { "cell_type": "markdown", "id": "alternate-assistant", "metadata": {}, "source": [ "### Same variable value for everyone\n", "\n", "Nothing surprising here, you can use regular python interpolation (e.g a `f-string`).\n", "String are interpolated by the interpreter before being manipulated." ] }, { "cell_type": "code", "execution_count": null, "id": "proved-finger", "metadata": {}, "outputs": [], "source": [ "host_to_ping = roles[\"control\"][0].alias\n", "host_to_ping\n", "\n", "results = en.run_command(f\"ping -c 5 {host_to_ping}\", roles=roles)\n", "results" ] }, { "cell_type": "code", "execution_count": null, "id": "young-sierra", "metadata": {}, "outputs": [], "source": [ "[(r.host, r.stdout) for r in results]" ] }, { "cell_type": "markdown", "id": "thrown-arkansas", "metadata": {}, "source": [ "### Using templates / Ansible variables\n", "\n", "There's an alternative way to pass a variable to a task: using `extra_vars`.\n", "The difference with the previous case (python interpreted variables) is the fact that the variable is interpolated right before execution happens on the remote node.\n", "One could imagine the the value is broadcasted to all nodes and replaced right before the execution.\n", "\n", "To indicate that we want to use this kind of variables, we need to pass its value using the `extra_vars` dictionnary and use a template (`{{ ... }}`) in the task description." ] }, { "cell_type": "code", "execution_count": null, "id": "intensive-parcel", "metadata": {}, "outputs": [], "source": [ "host_to_ping = roles[\"control\"][0].alias\n", "host_to_ping\n", "\n", "results = en.run_command(\"ping -c 5 {{ my_template_variable }}\", roles=roles, extra_vars=dict(my_template_variable=host_to_ping))\n", "results" ] }, { "cell_type": "markdown", "id": "mysterious-separate", "metadata": {}, "source": [ "### Host specific variables\n", "\n", "In the above, we've seen how a common value can be broadcasted to all remote nodes. What if we want host specific value ?\n", "\n", "For instance in our case we'd like `host 1` to ping `host 2` and `host 2` to ping `host 1`. That make the `host_to_ping` variable host-specific.\n", "\n", "For this purpose you can use the `extra` attribute of the `Host` objects and use a template as before." ] }, { "cell_type": "code", "execution_count": null, "id": "southeast-panel", "metadata": {}, "outputs": [], "source": [ "control_host = roles[\"control\"][0]\n", "compute_host = roles[\"compute\"][0]\n", "control_host.set_extra(host_to_ping=compute_host.address)\n", "compute_host.set_extra(host_to_ping=control_host.address)\n", "control_host" ] }, { "cell_type": "code", "execution_count": null, "id": "17295b0c-cb42-4752-b063-0c88f5580cb3", "metadata": {}, "outputs": [], "source": [ "compute_host" ] }, { "cell_type": "markdown", "id": "material-charm", "metadata": {}, "source": [ "> Note that the `extra` variable can be reset to their initial states with `Host.reset_extra()`" ] }, { "cell_type": "code", "execution_count": null, "id": "emotional-avenue", "metadata": {}, "outputs": [], "source": [ "results = en.run_command(\"ping -c 5 {{ host_to_ping }}\", roles=roles)\n", "results" ] }, { "cell_type": "code", "execution_count": null, "id": "english-answer", "metadata": {}, "outputs": [], "source": [ "[(r.host, r.stdout) for r in results]" ] }, { "cell_type": "markdown", "id": "infectious-packing", "metadata": { "tags": [] }, "source": [ "## Cleaning" ] }, { "cell_type": "code", "execution_count": null, "id": "spoken-modeling", "metadata": {}, "outputs": [], "source": [ "provider.destroy()" ] } ], "metadata": { "kernelspec": { "display_name": "my_venv", "language": "python", "name": "my_venv" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.2" }, "toc-autonumbering": false, "toc-showcode": false, "toc-showmarkdowntxt": false }, "nbformat": 4, "nbformat_minor": 5 }