RADICAL-Pilot 0.50.22 Documentation

RADICAL-Pilot (RP) is a Pilot Job system written in Python. It allows a user to run large numbers of computational tasks (called ComputeUnits) concurrently on one or more remote ComputePilots that RADICAL-Pilot can start transparently on a multitude of different distributed resources, like HPC clusters and Clouds.

In this model, a part (slice) of a resource is acquired by a user’s application so that the application can directly schedule ComputeUnits into that resource slice, rather than going through the system’s job scheduler. In many cases, this can drastically shorten overall exeuction time as the individual ComputeUnits don’t have to wait in the system’s scheduler queue but can execute directly on the ComputePilots.

ComputeUnits can be sequential, multi-threaded (e.g. OpenMP), parallel process (e.g. MPI) executables, Hadoop or Spark applications.

RADICAL-Pilot is not a static system, but it rather provides the user with a programming library (“Pilot-API”) that provides abstractions for resource access and task management. With this library, the user can develop everything from simple “submission scripts” to arbitrarily complex applications, higher- level services and tools.

Links

• repository: https://github.com/radical-cybertools/radical.pilot
• user list: https://groups.google.com/d/forum/radical-pilot-users
• developer list: https://groups.google.com/d/forum/radical-pilot-devel

Contents:

• 1. Introduction
• 2. RADICAL-Pilot - Overview
  • 2.1. What Problems does RP solve?
  • 2.2. What is a Pilot?
  • 2.3. What is a Compute Unit (CU)?
  • 2.4. How about data?
  • 2.5. Why do I need a MongoDB to run RP?
  • 2.6. How do I know what goes on in the pilot? With my CUs?
  • 2.7. What about logging and profiling?
• 3. Installation
  • 3.1. Requirements
  • 3.2. Installation
  • 3.3. Preparing the Environment
  • 3.4. Troubleshooting
• 4. User Guide
  • 4.1. Getting Started
  • 4.2. Obtaining Unit Details
  • 4.3. Handle Failing Units
  • 4.4. Use Multiple Pilots
  • 4.5. Selecting a Unit Scheduler
  • 4.6. Staging Unit Input Data
  • 4.7. Staging Unit Output Data
  • 4.8. Sharing Unit Input Data
  • 4.9. Setup Unit Environment
  • 4.10. MPI Applications
  • 4.11. Using Pre- and Post- exec commands
• 5. Examples
  • 5.1. Simple Bag-of-Tasks
  • 5.2. Chained Tasks
  • 5.3. Coupled Tasks
  • 5.4. MPI tasks
• 6. API Reference
  • 6.1. Sessions and Security Contexts
  • 6.2. Pilots and PilotManagers
  • 6.3. ComputeUnits and UnitManagers
  • 6.4. Exceptions
• 7. Data Staging
  • 7.1. Compute Unit I/O
  • 7.2. Compute Pilot I/O
  • 7.3. Examples
• 8. Using Local and Remote HPC Resources
  • 8.1. Introduction
  • 8.2. Configuring SSH Access
  • 8.3. Configuring GSISSH Access for XSEDE resources
  • 8.4. Pre-Configured Resources
  • 8.5. Writing a Custom Resource Configuration File
  • 8.6. Customizing Resource Configurations Programatically
• 9. Unit Scheduler
  • 9.1. Introduction
  • 9.2. Round-Robin Scheduler (SCHEDULER_ROUND_ROBIN)
  • 9.3. Backfilling Scheduler (SCHEDULER_BACKFILLING)
• 10. Testing
  • 10.1. Introduction
  • 10.2. Remote Testing
  • 10.3. Adding New Tests
• 11. Benchmarks
• 12. Details on Profiling
• 13. Frequently Asked Questions
  • 13.1. How do I…
  • 13.2. Other Questions
• 14. Developer Documentation
  • 14.1. Installation from Source
  • 14.2. License
  • 14.3. Style Guide
  • 14.4. Debugging
  • 14.5. RADICAL-Pilot Architecture