3.4. Use Multiple Pilots

We have seen in the previous examples how an RP pilot acts as a container for multiple task executions. There is in principle no limit on how many of those pilots are used to execute a specific workload, and specifically, pilot don’t need to run on the same resource!

03_multiple_pilots.py demonstrates that, and features the following diff to the previous examples:


Instead of creating one pilot description, we here create one for any resource specified as command line parameter, no matter if those parameters point to the same resource targets or not.

The tasks are distributed over the created set of pilots according to some scheduling mechanism – section Selecting a Task Scheduler will discuss how an application can choose between different scheduling policies. The default policy used here is Round Robin.

3.4.1. Running the Example

The workload of our example has now changed to report the respectively used pilot on stdout, and the output shows that. We here exemplarily start a pilot on local.localhost, and one on xsede.stampede:


3.4.2. What’s Next?

Using multiple pilots is very powerful – it becomes more powerful if you allow RP to load-balance tasks between them. Selecting a Task Scheduler will show how to do just that.