{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Getting Started\n", "\n", "This notebook walks you through executing a `hello_world` application written with RADICAL-Pilot (RP) and locally executed on a GNU/Linux operating system. The application consists of a Bag of Tasks with heterogeneous requirements: different number of CPU cores/GPUs and different execution time for each task. In this simple application, tasks have no data requirements but see the [Data Staging](tutorials/staging_data.ipynb) tutorial for how to manage data in RP.\n", "\n", "
\n", " \n", "__Warning:__ We assume you understand what a `pilot` is and how it enables to concurrently and sequentially execute compute tasks on its resources. See our [Brief Introduction to RP](https://radical-cybertools.github.io/presentations/rp_internals.mp4) video to familiarize yourself with the architectural concepts and execution model of a pilot. \n", "\n", "
\n", "\n", "## Installation\n", "\n", "
\n", " \n", "__Warning:__ RP **must** be installed in a Python environment. RP will **not** work properly when installed as a system-wide package. You **must** create and activate a virtual environment before installing RP. \n", "\n", "
\n", "\n", "You can create a Python environment suitable to RP using [Virtualenv](https://virtualenv.pypa.io/en/latest/), [Venv](https://docs.python.org/3/library/venv.html) or [Conda](https://docs.conda.io/projects/conda/en/latest/index.html). Once created and activated a virtual environment, RP is a Python module installed via [pip](https://pip.pypa.io/en/latest/), Conda, or [Spack](https://spack.readthedocs.io/en/latest/).\n", "\n", "
\n", " \n", "__Note:__ Please see [using environment variables](envs.rst) with RP for more options and detailed information. That will be especially useful when executing RP on [supported high performance computing (HPC) platforms](supported.rst).\n", "\n", "
\n", "\n", "### Virtualenv\n", "\n", "```shell\n", "virtualenv ~/ve_rp\n", ". ~/ve_rp/bin/activate\n", "pip install radical.pilot\n", "```\n", "\n", "### Venv\n", "\n", "```shell\n", "python -m venv ~/ve_rp\n", ". ~/ve_rp/bin/activate\n", "pip install radical.pilot\n", "```\n", "\n", "### Conda\n", "\n", "If there is no `conda` pre-installed, here is a distilled set of commands to install Miniconda on a GNU/Linux x86_64 OS. Find more (and possibly updated) information on the official [Conda documentation](https://docs.conda.io/projects/conda/en/latest/user-guide/index.html))\n", "\n", "```shell\n", "wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ./miniconda.sh\n", "chmod +x ./miniconda.sh\n", "./miniconda.sh -b -p ./conda\n", "source ./conda/bin/activate\n", "```\n", "\n", "Once Conda is available:\n", "\n", "```shell\n", "conda create -y -n ve_rp python=3.9\n", "conda activate ve_rp\n", "conda install -y -c conda-forge radical.pilot\n", "```\n", "\n", "### Spack\n", "\n", "If there is no `spack` pre-installed, here is a distilled set of commands to install Spack on a GNU/Linux x86_64 OS. Find more (and possibly updated) information on the official [Spack documentation](https://spack.readthedocs.io/en/latest/).\n", "\n", "```bash\n", "git clone https://github.com/spack/spack.git\n", ". spack/share/spack/setup-env.sh\n", "```\n", "\n", "Once Spack is available:\n", "\n", "```shell\n", "spack env create ve_rp\n", "spack env activate ve_rp\n", "spack install py-radical-pilot\n", "```\n", "\n", "
\n", " \n", "__Note:__ It is recommended to use [PYTHONNOUSERSITE](https://docs.python.org/3/using/cmdline.html#envvar-PYTHONNOUSERSITE) environment variable before activating your virtual environment to prevent user site packages from interfering: `export PYTHONNOUSERSITE=True`.\n", "\n", "
\n", "\n", "### Check the installed version\n", "\n", "Often, we need to know what version of RP we installed. For example, you will need to know that when opening a [support ticket](https://github.com/radical-cybertools/radical.pilot/issues) with the RADICAL development team. \n", "\n", "We install a command with RP that prints information for all the installed [RADICAL Cybertools](https://radical-cybertools.github.io/):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!radical-stack" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Write your first application\n", "\n", "RP executes in batch mode:\n", "\n", "* Write an application using RP API.\n", "* Launch that application.\n", "* Wait a variable amount of time doing something else.\n", "* When the application exits, come back, collect and check the results.\n", "\n", "Each RP application has a distinctive pattern:\n", "\n", "1. Create a session\n", "1. Create a pilot manager\n", "1. Describe the pilot on which you want to run your application tasks: \n", " - Define the platform on which you want to execute the application\n", " - Define the amount/type of resources you want to use to run your application tasks.\n", "1. Assign the pilot description to the pilot manager\n", "1. Create a task manager\n", "1. Describe the computational tasks that you want to execute:\n", " - Executable launched by the task\n", " - Arguments to pass to the executable command if any\n", " - Amount of each type of resource used by the executable, e.g., CPU cores and GPUs\n", " - When the executable is MPI/OpenMP, number of ranks for the whole executable, number of ranks per core or GPU\n", " - Many other parameters. See the [API specification](apidoc.rst) for full details\n", "1. Assign the task descriptions to the task manager\n", "1. Submit tasks for execution\n", "1. Wait for tasks to complete execution\n", "\n", "Some of RP behavior can be configured via environment variables. RP's progression bar does not work properly with Jupyter notebooks. Thus, you may want to set it to FALSE." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%env RADICAL_REPORT_ANIME=FALSE" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As with every Python application, first you import all the required modules." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import radical.pilot as rp" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Enable user feedback\n", "\n", "As RP implements a batch programming model, by default, it returns a minimal amount of information. After submitting the tasks for execution, RP will remain silent until all the tasks have completed. In practice, when developing and debugging your application, you will want more feedback. We wrote a reporter module that you can use with RP and all the other RADICAL-Cybertools.\n", "\n", "To use the reporter:\n", "\n", "* Configure RP by exporting a shell environment variable.\n", "\n", " ```shell\n", " export RADICAL_PILOT_REPORT=TRUE\n", " ```\n", "\n", "* Import `radical.utils`, create a reporter and start to use it to print meaningful messages about the state of the application execution.\n", "\n", "
\n", " \n", "__Note:__ See our tutorial about [Profiling a RADICAL-Pilot Application](tutorials/profiling.ipynb) for a guide on how to trace and profile your application.\n", "\n", "
\n", "\n", "
\n", "\n", "__Note:__ See our tutorial about [Debugging a RADICAL-Pilot Application](tutorials/debugging.ipynb) for a guide on how to debug your application.\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import radical.utils as ru\n", "\n", "report = ru.Reporter(name='radical.pilot')\n", "report.title('Getting Started (RP version %s)' % rp.version)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating a session\n", "\n", "[rp.Session](apidoc.html#radical.pilot.Session) is the root object of all the other objects of RP." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "session = rp.Session()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating a pilot manager\n", "\n", "You need to manage the resources you will acquire with a pilot either locally or, more commonly and usefully, on a supported HPC platform. An instance of [rp.PilotManager](apidoc.html#radical.pilot.PilotManager) attached to your `session` will do that for you.\n", "\n", "
\n", " \n", "__Note:__ One [rp.PilotManager](apidoc.html#radical.pilot.PilotManager) can manage multiple pilots. See our tutorial about [Using Multiple Pilots with RADICAL-Pilot](tutorials/multiple_pilots.ipynb) to see why and how.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pmgr = rp.PilotManager(session=session)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Configuring pilot resources\n", "\n", "You can use a dictionary to specify location, amount and other properties of the resources you want to acquire with a pilot; and use that dictionary to initialize a [rp.PilotDescription](apidoc.html#radical.pilot.PilotDescription) object. See the [Configuration](tutorials/configuration.ipynb) tutorial for more details.\n", "\n", "In this example, we want to run our `hello_world` application on our local GNU/Linux, for not more than 30 minutes and use 2 cores. \n", "\n", "
\n", " \n", "__Warning:__ We choose a 30 minutes runtime, but the application could take less or more time to complete. 30 minutes are the upper bound but RP will exit as soon as all tasks have reached their final state (`DONE`, `CANCELED`, `FAILED`). Conversely, RP will always exit once the runtime expires, even if some tasks still need to be executed.\n", "\n", "
\n", "\n", "
\n", " \n", "__Note:__ We could choose to use as many CPU cores as we have available on our local machine. RP will allocate all of them, but it will use only the cores required by the application tasks. If all the tasks together require fewer cores than those available, the remaining cores will go unused. Conversely, if there are more tasks that cores, RP will schedule each task as soon as the required amount of cores becomes available. In this way, RP will maintain the available resources as busy as possible and the application tasks will run both concurrently and sequentially, depending on resource availability.\n", "\n", "
\n", "\n", "
\n", " \n", "__Note:__ `'exit_on_error': False` allows us to compile this notebook without errors. You should probably not use it with a standalone RP application.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pdesc = rp.PilotDescription({'resource' : 'local.localhost',\n", " 'project' : None,\n", " 'queue' : None,\n", " 'cores' : 4,\n", " 'gpus' : 0,\n", " 'runtime' : 30, # pilot runtime minutes\n", " 'exit_on_error': False})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Submitting the pilot\n", "\n", "We now have a pilot manager, we know how many resources we want and on what platform. We are ready to submit our request!\n", "\n", "
\n", " \n", "__Note:__ On a supported HPC platform, our request will queue a job into the platform's batch system. The actual resources will become available only when the batch system schedules the job. This is not under the control of RP and, barring reservation, the actual queue time will be unknown.\n", "\n", "
\n", "\n", "We use the [rp.PilotManager.submit_pilots()](apidoc.html#radical.pilot.PilotManager.submit_pilots) method of our pilot manager and pass to it the pilot description." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "report.header('submit pilot')\n", "pilot = pmgr.submit_pilots(pdesc)\n", "\n", "# preserve pilot sandbox\n", "pilot_sandbox = ru.Url(pilot.pilot_sandbox).path\n", "\n", "# wait for pilot to become ACTIVE\n", "pilot.wait(rp.PMGR_ACTIVE)\n", "\n", "# report pilot state\n", "report.info('<>{pilot.state}\\n')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Creating a task manager\n", "\n", "We have acquired the resources we asked for (or we are waiting in a queue to get them) so now we need to do something with those resources, i.e., executing our application tasks. First, we create a [rp.TaskManager](apidoc.html#radical.pilot.TaskManager) and associate it to our session. That manager will take care of taking our task descriptions and sending them to our pilot so that it can execute those tasks on the allocated resources." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tmgr = rp.TaskManager(session=session)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Registering the pilot with the task manager\n", "\n", "We tell the task manager what pilot it should use to execute its tasks." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tmgr.add_pilots(pilot)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Describing the application tasks\n", "\n", "In this example, we want to run simple tasks but that require different number of CPU cores and that run for a variable amount of time. Thus, we use the executable [radical-pilot-hello.sh](https://github.com/radical-cybertools/radical.pilot/blob/master/bin/radical-pilot-hello.sh) we crafted to occupy a configurable amount of resources for a configurable amount of time. \n", "\n", "Each task is an instance of [rp.TaskDescription](apidoc.html#radical.pilot.TaskDescription) with some defined properties:\n", "\n", "* `executable`: the name of the executable we want to launch with the task.\n", "* `arguments`: the arguments to pass to `executable`. In this case, the number of seconds it needs to run for.\n", "* `ranks`: this is the number of processes (i.e., MPI ranks) on which the task should run. See [Describing tasks in RADICAL-Pilot](tutorials/describing_tasks.ipynb#MPI-Tasks-and-Task-Resources) and the details about executing tasks that use the message passing interface (MPI).\n", "* `cores_per_rank`: the amount of cores that each rank of the task utilizes. In our case, each task will randomly use either 1 or 2 cores have requested.\n", "\n", "
\n", "\n", "__Warning:__ Executing MPI tasks (i.e., one with `rp.TaskDescription.ranks > 1`) requires for an MPI implementation to be available on the machine on which you will run the task. That is usually taken care of by the system administrator, but if you are managing your own cluster, you will have to install and make available one of the many MPI distributions available for GNU/Linux. \n", "\n", "
\n", "\n", "We run 10 tasks that should be enough to see both concurrent and sequential executions on the amount of resources we requested, but not enough to clog the example.\n", "\n", "
\n", " \n", "__Note:__ We use the reporter to produce a progress bar while we loop over the task descriptions.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "import random\n", "\n", "n = 10\n", "\n", "report.progress_tgt(n, label='create')\n", "tds = list()\n", "for i in range(n):\n", "\n", " td = rp.TaskDescription()\n", " td.executable = 'radical-pilot-hello.sh'\n", " td.arguments = [random.randint(1, 10)]\n", " td.ranks = 1\n", " td.cores_per_rank = random.randint(1, 2)\n", " td.named_env = 'rp'\n", "\n", " tds.append(td)\n", " report.progress()\n", "\n", "report.progress_done()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Submitting tasks for execution\n", "\n", "Now that we have all the elements of the application we can execute its tasks. We submit the list of application tasks to the task manager that, in turn, will submit them to the indicated pilot for execution. Upon receiving the list of task descriptions, the pilot will schedule those tasks on its available resources and then execute them.\n", "\n", "
\n", " \n", "__Note:__ For RP, tasks are black boxes, i.e., it knows nothing about the code executed by the task. RP just knows that a task has been launched on the requested amount of resources, and it will wait until the tasks exits. In that way, RP is **agnostic** towards task details like language used for its implementation, the type of scientific computation it performs, how it uses data, etc. This is why RP can serve a wide range of scientists, independent on their scientific domain.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "report.header('submit %d tasks' % n)\n", "tasks = tmgr.submit_tasks(tds)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Waiting for the tasks to complete\n", "\n", "Wait for all tasks to reach a final state (`DONE`, `CANCELED` or `FAILED`). This is a blocking call, i.e., the application will wait without exiting and, thus, the shell from which you launched the application should not exit either. Thus, no closing your laptop or no exiting from a remote connection without first leaving the shell running in background or using a terminal multiplexer like [tmux](https://github.com/tmux/tmux).\n", "\n", "
\n", "\n", "__Note:__ After the wait call returns, you can describe and/or submit more tasks/pilots as your RP session will still be open.\n", "\n", "
\n", "\n", "
\n", "\n", "__Note:__ You can wait for the execution of a **subset** of the tasks you defined. See [Describing tasks in RADICAL-Pilot](tutorials/describing_tasks.ipynb) for more information.\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tmgr.wait_tasks(uids=[t.uid for t in tasks])\n", "report.header(f'finished {len(tasks)} tasks')\n", "\n", "task_states = [t.state for t in tasks]\n", "state_counter = {state: task_states.count(state) for state in set(task_states)}\n", "for state, counter in sorted(state_counter.items()):\n", " report.info('<>({counter}) {state}\\n')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the wait is finished, let us know and exit!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "report.header('finalize')\n", "session.close()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Generated Output\n", "\n", "RP is a distributed system, even when all its components run on a single machine as with this example. RP has two main components (Client and Agent) and both store their output into a sandbox, stored at a specific filesystem location:\n", "\n", "- **Client sandbox**: A directory created within the working directory from where the RP application was launched. The sandbox is named after the session ID, e.g., `rp.session.nodename.username.018952.0000`.\n", "- **Agent sandbox**: A directory created at a different location, depending on the machine on which the application executes. The Agent sandbox, named `radical.pilot.sandbox`, contains the following nested directories: `//` which represent session-sandbox, pilot-sandbox(es) and task-sandbox(es) respectively.\n", "\n", "When running RP on localhost, the Agent sandbox is located at `$HOME/radical.pilot.sandbox`. When using a [supported HPC platform](supported.rst), the location of the Agent sandbox depends on the filesystem capabilities of the platform. You can see the pre-configured location for the Agent sandbox in the RP git repository, at [src/radical/pilot/configs/resource_*.json](https://github.com/radical-cybertools/radical.pilot/tree/master/src/radical/pilot/configs).\n", "\n", "
\n", "\n", "__Warning:__ When executing RP on a supported HPC platform, the output file(s) of each task are saved in the task-sandbox, named after `task_ID` of that task. Without specific staging instructions (see our tutorial [Staging Data with RADICAL-Pilot](tutorials/staging_data.ipynb)), you will have to manually retrieve those files. When executing on localhost, you can retrieve them from `$HOME/radical.pilot.sandbox///`.\n", "\n", "
\n", "\n", "
\n", "\n", "__Note:__ When enabling debugging (see our tutorial [Debugging a RADICAL-Pilot Application](tutorials/debugging.ipynb)) and/or tracing (see our tutorial [Tracing and Profiling a RADICAL-Pilot Application](tutorials/profiling.ipynb)), RP writes the debug and/or trace files in the Client and Agent sandboxes. On large/production runs, RP can produce hundreds of debug files. Please [contact](https://github.com/radical-cybertools/radical.pilot/issues) the RADICAL development team if you need further assistance.\n", "\n", "
\n", "\n", "Here are the output files of the `task.000000` of the application we just executed in this notebook:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!echo $pilot_sandbox\n", "!ls $pilot_sandbox/task.000000/" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here is the \"result\" produced by `task.000000`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!cat $pilot_sandbox/task.000000/task.000000.out" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 4 }