RADICAL-Pilot 1.6.0 Documentation¶
RADICAL-Pilot (RP) is a Pilot system [1] [2] written in Python and specialized in executing applications composed of many computational tasks on high performance computing (HPC) platforms. As a Pilot system, RP separates resource acquisition from using those resources to execute application tasks. Resources are acquired by submitting a job to the batch system of an HPC machine. Once the job is scheduled on the requested resources, RP can directly schedule and launch application tasks on those resources. Thus, tasks are not scheduled via the batch system of the HPC platform, but directly on the acquired resources.
As every Pilot system, RP offers two main benefits: (1) high-throughput task execution; and (2) concurrent and sequential task executions on the same pilot. High-throughput is possible because the user exclusively owns the resources on which those tasks are executed for as long as the job submitted to the HPC platform remains available. Depending on resource availability, tasks can be scheduled concurrently and, if more tasks need to be executed, one after the other. In this way, tasks can execute both concurrently and sequentially on the same pilot.
RP offers four unique features when compared to other pilot systems or tools that enable the execution of multi-task applications on HPC platforms: (1) execution different types of tasks concurrently on the same pilot, e.g., single-core, OpenMP, MPI, single- and multi-GPU; (2) support of all the major HPC batch systems, e.g., slurm, torque, pbs, lsf, etc.; (3) support of more than 14 methods to launch tasks, e.g., ssh, mpirun, aprun, jsrun, prrte, etc.; and (4) a general purpose architecture.
RADICAL-Pilot is not a static system, but it rather provides the user with a programming library (“Pilot-API”) that provides abstractions for resource access and task management. With this library, the user can develop everything from simple “submission scripts” to arbitrarily complex applications, higher-level services and tools.
Chapter RADICAL-Pilot Overview offers more information about tasks, workloads, pilot, pilot systems, and RP implementation. The user is strongly invited to carefully read that section before starting to use RP.
Links
Contents:¶
- 1. RADICAL-Pilot (RP) - Overview
- 1.1. What problems does RP solve?
- 1.2. What is a Task?
- 1.3. What is a Workload?
- 1.4. What is a Task (Task)?
- 1.5. What is a Pilot?
- 1.6. How about data?
- 1.7. Why do I need a MongoDB to run RP?
- 1.8. Why do I need RADICAL-SAGA to run RP?
- 1.9. How do I monitor pilots and CUs?
- 1.10. What about logging?
- 1.11. What about tracing and profiling?
- 2. Installation
- 3. User Guide
- 3.1. Getting Started
- 3.2. Obtaining Task Details
- 3.3. Handle Failing Tasks
- 3.4. Use Multiple Pilots
- 3.5. Selecting a Task Scheduler
- 3.6. Staging Task Input Data
- 3.7. Staging Task Output Data
- 3.8. Sharing Task Input Data
- 3.9. Setup Task Environment
- 3.10. MPI Applications
- 3.11. Using Pre- and Post- exec commands
- 4. Examples
- 5. API Reference
- 6. Data Staging
- 7. Using Local and Remote HPC Resources
- 8. Task Scheduler
- 9. Testing
- 10. Benchmarks
- 11. Frequently Asked Questions
- 11.1. Error “OperationFailure: too many namespaces/collections”
- 11.2. Error “Permission denied (publickey,keyboard-interactive).” in AGENT.STDERR or STDERR
- 11.3. Error “Failed to execvp() ‘mybinary’: No such file or directory (2)”
- 11.4. Errors from setuptools when trying to use a virtualenv
- 11.5. Error “Received message too long 1903391841”
- 11.6. Pop-up “Do you want the application python to accept incoming network connections?” on macOS
- 11.7. Error “Could not detect shell prompt (timeout)”
- 11.8. Number of concurrent RADICAL-Pilot scripts that can can be executed
- 12. Developer Documentation
- 13. Release Notes
- 13.1. 1.6.0 Release 2021-02-13
- 13.2. 1.5.12 Release 2021-02-02
- 13.3. 1.5.11 Release 2021-01-19
- 13.4. 1.5.10 Release 2021-01-18
- 13.5. 1.5.8 Release 2020-12-09
- 13.6. 1.5.7 Release 2020-10-30
- 13.7. 1.5.4 Release 2020-10-01
- 13.8. 1.5.4 Release 2020-09-14
- 13.9. 1.5.2 Hotfix Release 2020-08-11
- 13.10. 1.5.1 Hotfix Release 2020-08-05
- 13.11. 1.4.1 Hotfix Release 2020-06-09
- 13.12. 1.4.0 Release 2020-05-12
- 13.13. 1.3.0 Release 2020-04-10
- 13.14. 1.2.1 Hotfix Release 2020-02-11
- 13.15. 1.2.0 Release 2020-02-11
- 13.16. 1.1.1 Hotfix Release 2020-02-11
- 13.17. 1.1 Release 2020-02-11
- 13.18. 1.0.0 Release 2019-12-24
- 13.19. 0.90.1 Release 2019-10-12
- 13.20. 0.90.0 Release 2019-10-07
- 13.21. 0.73.1 Release 2019-10-07
- 13.22. 0.72.2 Hotfix Release 2019-09-30
- 13.23. 0.72.0 Release 2019-09-11
- 13.24. 0.70.3 Hotfix Release 2019-08-02
- 13.25. 0.70.2 Hotfix Release 2019-07-31
- 13.26. 0.70.1 Hotfix Release 2019-07-30
- 13.27. 0.70.0 Release 2019-07-07
- 13.28. 0.63.0 Release 2019-06-25
- 13.29. 0.62.0 Release 2019-06-08
- 13.30. 0.61.0 Release 2019-05-07
- 13.31. 0.60.1 Hotfix 2019-04-12
- 13.32. 0.60.0 Release 2019-04-10
- 13.33. 0.50.23 Release 2019-03-20
- 13.34. 0.50.22 Release 2019-02-11
- 13.35. 0.50.21 Release 2018-12-19
- 13.36. 0.50.20 Release 2018-11-25
- 13.37. 0.50.19 Release 2018-11-15
- 13.38. 0.50.18 Release 2018-11-13
- 13.39. 0.50.17 Release 2018-10-31
- 13.40. 0.50.16 Release 2018-10-26
- 13.41. 0.50.15 Release 2018-10-24
- 13.42. 0.50.14 Release 2018-10-24
- 13.43. 0.50.13 Release 2018-10-16
- 13.44. 0.50.12 Release 2018-10-12
- 13.45. 0.50.11 Release 2018-10-09
- 13.46. 0.50.10 Release 2018-09-29
- 13.47. 0.50.9 Release 2018-09-28
- 13.48. 0.50.8 Release 2018-08-03
- 13.49. 0.50.7 Release 2018-08-01
- 13.50. 0.50.6 Release 2018-07-31
- 13.51. 0.50.5 Release 2018-07-30
- 13.52. 0.50.4 Release 2018-07-13
- 13.53. 0.50.3 Release 2018-07-11
- 13.54. 0.50.2 Release 2018-07-08
- 13.55. 0.50.1 Release 2018-07-05
- 13.56. 0.47.14 Release 2018-06-13
- 13.57. 0.47.13 Release 2018-06-02
- 13.58. 0.47.12 Release 2018-05-19
- 13.59. 0.47.11 Release 2018-05-08
- 13.60. 0.47.10 Release 2018-04-19
- 13.61. 0.47.9 Release 2018-04-18
- 13.62. 0.47.8 Release 2018-04-16
- 13.63. 0.47.7 Release 2018-04-15
- 13.64. 0.47.6 Release 2018-04-12
- 13.65. 0.47.5 Release 2018-04-12
- 13.66. 0.47.4 Release 2018-03-20
- 13.67. 0.47.3 Release 2018-03-20
- 13.68. 0.47.2 Release 2018-02-28
- 13.69. 0.47 Release 2017-11-19
- 13.70. 0.46.2 Release 2017-09-02
- 13.71. 0.46.1 Release 2017-08-23
- 13.72. Version 0.46 2017-08-11
- 13.73. 0.45.3 Release 2017-05-09
- 13.74. 0.45.1 Release 2017-03-05
- 13.75. 0.45 Release 2017-02-28
- 13.76. 0.44.1 Release 2016-11-01
- 13.77. 0.43.1 Release 2016-09-09
- 13.78. 0.43 Release 2016-09-08
- 13.79. 0.42 Release 2016-08-09
- 13.80. 0.41 Release 2016-07-15
- 13.81. 0.40.4 Release 2016-05-18
- 13.82. 0.40.3 Release 2016-05-17
- 13.83. 0.40.2 Release 2016-05-13
- 13.84. 0.40.1 Release 2016-02-11
- 13.85. 0.40 Release 2016-02-03
- 13.86. 0.38 Release 2015-12-22
- 13.87. 0.37.10 Release 2015-10-20
- 13.88. 0.37.9 Release 2015-10-20
- 13.89. 0.37.8 Release 2015-10-20
- 13.90. 0.37.7 Release 2015-10-20
- 13.91. 0.37.6 Release 2015-10-20
- 13.92. 0.37.5 Release 2015-10-19
- 13.93. 0.37.3 Release 2015-10-19
- 13.94. 0.37.2 Release 2015-10-18
- 13.95. 0.37.1 Release 2015-10-18
- 13.96. 0.37 Release 2015-10-15
- 13.97. 0.36 Release 2015-10-08
- 13.98. 0.35.1 Release 2015-09-29
- 13.99. 0.35 Release 2015-07-14
- 13.100. 0.34 Release 2015-07-14
- 13.101. 0.33 Release 2015-05-27
- 13.102. 0.32 Release 2015-05-18
- 13.103. 0.31 Release 2015-04-30
- 13.104. 0.30 Release 2015-04-29
- 13.105. 0.29 Release 2015-04-28
- 13.106. 0.28 Release 2015-04-16
- 13.107. 0.26 Release 2015-04-08
- 13.108. 0.25 Release 2015-04-01
- 13.109. 0.24 Release 2015-03-30
- 13.110. 0.23 Release 2014-12-13
- 13.111. 0.22 Release 2014-12-11
- 13.112. 0.21 Release 2014-10-29
- 13.113. 0.19 Release September 12. 2014
- 13.114. 0.18 Release July 22. 2014
- 13.115. 0.17 Release June 18. 2014
- 13.116. 0.16 Release June 17. 2014
- 13.117. 0.15 Release June 12. 2014
- 13.118. 0.14 Release June 11. 2014
- 13.119. 0.13 Release May 19. 2014
- 13.120. 0.12 Release May 09. 2014
- 13.121. 0.11 Release Apr. 29. 2014
- 13.122. 0.10 Release Apr. 29. 2014
- 13.123. 0.9 Release Apr. 16. 2014
- 13.124. 0.8 Release Mar. 24. 2014
- 13.125. 0.7 Release Feb. 25. 2014
- 13.126. 0.6 Release Feb. 24. 2014
- 13.127. 0.5 Release Feb. 06. 2014
- 13.128. 0.4 Release
- 13.129. 0.1.3 Release
- 13.130. 0.1.2 Release