{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "direct-johnston",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Setup and basic objects\n",
    "\n",
    "Get started with EnOSlib on Grid'5000.\n",
    "\n",
    "---\n",
    "\n",
    "- Website: https://discovery.gitlabpages.inria.fr/enoslib/index.html\n",
    "- Instant chat: https://framateam.org/enoslib\n",
    "- Source code: https://gitlab.inria.fr/discovery/enoslib\n",
    "\n",
    "---\n",
    "\n",
    "This is the first notebook in a serie that will let you discover the main features of EnOSlib on Grid'5000.\n",
    "\n",
    "If you want to actually execute them you'll need to setup your environment properly.\n",
    "We sum up here the different steps to achieve this process.\n",
    "\n",
    "1. Get a Grid'5000 account\n",
    "    - Register using this [page](https://www.grid5000.fr/w/Grid5000:Get_an_account).\n",
    "      Pay attention to the fact that uploading a SSH key (public part) is mandatory to perform any EnOSlib action from your local machine.\n",
    "    - Make sure the SSH connection is ready. You can follow this [tutorial](https://www.grid5000.fr/w/Getting_Started).\n",
    "2. Make sure EnOSlib is available in your notebook environment\n",
    "    - Follow the steps [here](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html#installation).\n",
    "      Using a virtualenv is the way to go, make sure to use one.\n",
    "      Also adding the optional `jupyter` will improve your experience. (`pip install enoslib[jupyter]`)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ceramic-burst",
   "metadata": {},
   "source": [
    "## Testing the import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "thorough-maker",
   "metadata": {},
   "outputs": [],
   "source": [
    "import enoslib as en"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "civilian-determination",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Resources abstractions\n",
    "\n",
    "In this notebook, we won't execute anything remotely. Instead we'll just cover some basic abstractions provided by the library.\n",
    "We start with the abstractions of resources (machines and networks that are usually given by an infrastructure)\n",
    "\n",
    "\n",
    "### Host\n",
    "\n",
    "An host is anything we can connect to and act on. Most of the time it corresponds to a machine reachable through SSH.\n",
    "The datastructure reflects this.\n",
    "\n",
    "Usually you don't instantiate hosts manually, instead they are brought to you by EnOSlib (because most likely they depend on a scheduler decision like OAR on Grid'5000).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cleared-presence",
   "metadata": {},
   "outputs": [],
   "source": [
    "bare_host = en.Host(\"192.168.0.1\")\n",
    "host_with_alias = en.Host(\"192.168.0.2\", alias=\"one_alias\")\n",
    "host_with_alias_and_username = en.Host(\"192.168.0.3\", alias=\"one_alias\", user=\"foo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "peripheral-campus",
   "metadata": {},
   "outputs": [],
   "source": [
    "bare_host"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "unlikely-credits",
   "metadata": {},
   "outputs": [],
   "source": [
    "host_with_alias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "empty-privilege",
   "metadata": {},
   "outputs": [],
   "source": [
    "host_with_alias_and_username"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attractive-stationery",
   "metadata": {},
   "source": [
    "The local machine can be represented by an instance of the `LocalHost` object. This is a specialization of an `Host`, the connection to this host will be made using sub-processes (instead of SSH). We can see it in the `extra` attribute of the `LocalHost` object. This `extra` attribute is actually interpreted when a \"remote\" action is triggered on our hosts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dental-population",
   "metadata": {},
   "outputs": [],
   "source": [
    "localhost = en.LocalHost()\n",
    "localhost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "temporal-adaptation",
   "metadata": {},
   "source": [
    "Other types of Hosts are possible. The library has a `DockerHost` which represents a docker container we want to reach using the docker TCP protocol. One needs to specify where this container is running by passing an host instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "educational-conviction",
   "metadata": {},
   "outputs": [],
   "source": [
    "docker_host = en.DockerHost(\"alias\", \"container_name\", host_with_alias_and_username)\n",
    "docker_host"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "executive-liberia",
   "metadata": {},
   "source": [
    "The above `extra` field suggest that the connection to this docker container will be made through an ssh jump to the remote host hosting the container.\n",
    "This will be done transparently by the library anyway.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "exceptional-techno",
   "metadata": {},
   "source": [
    "### Roles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tight-request",
   "metadata": {},
   "source": [
    "A common pratice when experimenting, especially with distributed applications, is to form logical group of machines.\n",
    "Indeed, during an experiment your hosts will serve different purposes: some will host the system you are studying while other will install third party tools to inject some load, observe ...\n",
    "\n",
    "A natural way of configuring differently several sets of hosts is to tag them and group them according to their tags.\n",
    "\n",
    "The `Roles` datastructure serves this purpose: it lets you group your hosts based on tags. It follow a `dict-like` interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "spatial-probe",
   "metadata": {},
   "outputs": [],
   "source": [
    "h1 = en.Host(\"10.0.0.1\")\n",
    "h2 = en.Host(\"10.0.0.2\")\n",
    "h3 = en.Host(\"10.0.0.3\")\n",
    "roles = en.Roles()\n",
    "roles[\"tag1\"] = [h1, h2]\n",
    "roles[\"tag2\"] = [h3]\n",
    "roles[\"tag3\"] = [h2, h3]\n",
    "\n",
    "roles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adapted-apparatus",
   "metadata": {},
   "source": [
    "### Network and Networks\n",
    "\n",
    "`Network` and `Networks`  are the same as `Host` and `Roles` but for networks:\n",
    "\n",
    "- `Network` represent a single Network\n",
    "- `Networks` represent a \"Roles\" of Network: networks indexed by their tags\n",
    ".\n",
    "\n",
    "Networks are usually given by an infrastructure and thus you won't really instantiate `Network` nor `Networks` by yourself.\n",
    "More precisely there exists a specific subclass of `Network` per infrastructure which will be returned automatically by EnOSlib when needed.\n",
    "\n",
    "Moreover `Network` datastructure isn't exposed in EnOSlib at the top level, let's see however how a `DefaultNetwork` can look like. A `DefaultNetwork` is a very common abstraction of a network that allows to represent a basic network with optionnally a pool of free ips/macs address. For instance a subnet or a vlan on Grid5000 are represented by a specific `DefaultNetwork`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "distributed-ceramic",
   "metadata": {},
   "outputs": [],
   "source": [
    "from enoslib.objects import DefaultNetwork"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "empirical-somewhere",
   "metadata": {},
   "outputs": [],
   "source": [
    "one_network = DefaultNetwork(\"192.168.1.0/24\")\n",
    "one_network_with_a_pool_of_ips = DefaultNetwork(\"192.168.1.0/24\", ip_start=\"192.168.1.10\", ip_end=\"192.168.1.100\") "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "stone-boxing",
   "metadata": {},
   "outputs": [],
   "source": [
    "one_network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aerial-coral",
   "metadata": {},
   "outputs": [],
   "source": [
    "one_network_with_a_pool_of_ips"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "appreciated-preservation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get one free ip\n",
    "ip_gen = one_network_with_a_pool_of_ips.free_ips\n",
    "next(ip_gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "determined-township",
   "metadata": {},
   "outputs": [],
   "source": [
    "# get another one\n",
    "next(ip_gen)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab4929f-1c39-46ca-b6ac-ae0b907cc3ed",
   "metadata": {},
   "source": [
    "## Providers (and their configurations)\n",
    "\n",
    "EnOSlib uses `Provider`s to ... provide resources. \n",
    "`Provider`s  let the user get ownership of some resources (for the time of the experiment) in good shape (e.g access granted, network configured ...). \n",
    "They transform an abstract `Configuration` to `Roles, Networks` : \n",
    "\n",
    "\n",
    "$Configuration \\xrightarrow{provider} Roles, Networks$\n",
    "\n",
    "\n",
    "There are different providers in EnOSlib: \n",
    "\n",
    "\n",
    "- **Vbox/KVM** to work with locally hosted virtual machines\n",
    "- **Openstack/Chameleon** to work with bare-metal resources hosted in the Chameleon platform\n",
    "- **FiT/IOT lab** to work with sensors or low profile machines\n",
    "- **Grid'5000** to get bare-metal resources from G5k.<br/>\n",
    "  There are also some composite providers that sit on top of the Grid'5000 provider\n",
    "  \n",
    "    - **VmonG5k** to work with virtual machines on Grid'5000**\n",
    "    - **Distem** to work with lxc containers on Grid'5000**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67a8c47b-5b0e-4e51-b17c-2ba3fd3d0d12",
   "metadata": {},
   "source": [
    "### Configurations\n",
    "\n",
    "A `Provider` must be fed with a `Configuration`.  `Configuration` objects are specific to each provider.\n",
    "\n",
    "You can build them from a dictionnary (e.g from a yaml/json file) or programmatically. For instance the schema for Grid'5000 is [here](https://discovery.gitlabpages.inria.fr/enoslib/apidoc/infra.html#g5k-schema).\n",
    "\n",
    "In this section, we'll only build some configurations (No resource will be reserved on Grid'5000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fef3970b-5d7a-4e18-bc6f-59d944396e83",
   "metadata": {},
   "outputs": [],
   "source": [
    "import enoslib as en\n",
    "\n",
    "# An empty configuration isn't really useful but let you see\n",
    "# some of the default parameters\n",
    "# Note that by default (empty job_type) the nodes are provisioned\n",
    "# with the standard Grid'5000 software environment.\n",
    "conf = en.G5kConf()\n",
    "conf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecdde599-41c3-420b-9ac9-88a1842cdfe4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# changing the top level options is done by calling the classmethod `from_settings`\n",
    "en.G5kConf.from_settings(walltime=\"10:00:00\", job_name=\"my awesome job\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98ecf1f3-027f-4754-baed-7a7188990868",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the canonical way of getting some machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e95e4ef-442f-4364-bfdf-f0c7f8271027",
   "metadata": {},
   "outputs": [],
   "source": [
    "prod_network = en.G5kNetworkConf(roles=[\"mynetwork\"], site=\"rennes\", type=\"prod\")\n",
    "conf = (\n",
    "    en.G5kConf()\n",
    "        .add_machine(cluster=\"paravance\", nodes=3, roles=[\"role1\", \"role2\"], primary_network=prod_network)\n",
    "        .add_machine(cluster=\"parasilo\", nodes=3, roles=[\"role2\", \"role3\"], primary_network=prod_network)\n",
    "        .add_network_conf(prod_network)\n",
    "        # optional, but do some sanity checks on the configuration \n",
    "        .finalize()\n",
    "    )\n",
    "conf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08d69a20-a7f0-453c-a066-5b1304913d9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Changing to a deploy job.\n",
    "# The operating system to deploy needs to be selected (debian, ubuntu, centos...)\n",
    "# See https://www.grid5000.fr/w/Advanced_Kadeploy#Search_an_environment\n",
    "prod_network = en.G5kNetworkConf(roles=[\"mynetwork\"], site=\"rennes\", type=\"prod\")\n",
    "conf = (\n",
    "    en.G5kConf.from_settings(job_type=[\"deploy\"], env_name=\"ubuntu2204-min\")\n",
    "        .add_machine(cluster=\"paravance\", nodes=3, roles=[\"role1\", \"role2\"], primary_network=prod_network)\n",
    "        .add_machine(cluster=\"parasilo\", nodes=3, roles=[\"role2\", \"role3\"], primary_network=prod_network)\n",
    "        .add_network_conf(prod_network)\n",
    "        # optional, but do some sanity checks on the configuration \n",
    "        .finalize()\n",
    "    )\n",
    "conf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6eb4609-a873-4bd6-8cd4-06b56ff670a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using a secondary networks\n",
    "prod_network = en.G5kNetworkConf(roles=[\"mynetwork\"], site=\"rennes\", type=\"prod\")\n",
    "kavlan_network = en.G5kNetworkConf(roles=[\"myprivate\"], site=\"rennes\", type=\"kavlan\")\n",
    "conf = (\n",
    "    en.G5kConf(job_type=[\"deploy\"], env_name=\"debian11-nfs\")\n",
    "        .add_machine(cluster=\"paravance\", nodes=3, roles=[\"role1\", \"role2\"], primary_network=prod_network, secondary_networks=[kavlan_network])\n",
    "        .add_machine(cluster=\"parasilo\", nodes=3, roles=[\"role2\", \"role3\"], primary_network=prod_network, secondary_networks=[kavlan_network])\n",
    "        .add_network_conf(prod_network)\n",
    "        .add_network_conf(kavlan_network)\n",
    "        # optional, but do some sanity checks on the configuration \n",
    "        .finalize()\n",
    "    )\n",
    "conf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4ab8b3-cdb2-46b8-be32-e709bc11fee0",
   "metadata": {},
   "source": [
    "### Discussion and references\n",
    "\n",
    "- Many configurations options are possible. [The documentation](https://discovery.gitlabpages.inria.fr/enoslib/tutorials/grid5000.html) will show you some more.\n",
    "- In EnOSlib `Roles` and `Networks` don't really depend on the provider that produced them. In other words you can substitute one provider's configuration to another one easily without changing the artifact code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce78182e-562a-46a1-b6f9-17ffade1d851",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}