Multiple objective scheduling of HPC workloads through dynamic prioritization
Loading...
Links to Files
Permanent Link
Author/Creator
Author/Creator ORCID
Date
2013-04-07
Type of Work
Department
Program
Citation of Original Publication
Simon, Tyler A., Phuong Nguyen, and Milton Halem. “Multiple Objective Scheduling of HPC Workloads through Dynamic Prioritization.” In Proceedings of the High Performance Computing Symposium, 1–8. HPC ’13. San Diego, CA, USA: Society for Computer Simulation International, 2013.
Rights
This item is likely protected under Title 17 of the U.S. Copyright Law. Unless on a Creative Commons license, for uses protected by Copyright Law, contact the copyright holder or the author.
Subjects
Abstract
We have developed an efficient single queue scheduling system that utilizes a greedy knapsack algorithm with dynamic job priorities. Our scheduler satisfies high level objectives while maintaining high utilization of the HPC system or collection of distributed resources such as a computational GRID. We provide simulation analysis of our approach in contrast with various scheduling strategies of shortest job first; longest waiting jobs first; and large jobs first. Further, we look at the effects of system size on the total workload response time and find that for real workloads, the relationship between response time and system size follows an inverse power law. Our approach does not require system administrators or users to identify a specific priority queue for each of their jobs. The proposed scheduler performs an exhaustive parameter search for a priority calculation per job to balance high level objectives and provide guaranteed performance jobs in a workload. The system administrator needs only tune the prioritization parameters (knobs) and the system scheduler will behave accordingly, such as reducing wait time for jobs that are above average size with small runtimes. We demonstrate that our approach works very well on workloads that have many independent tasks. We evaluate our scheduler on a realistic mixed scientific data processing workload and with a realistic HPC workload trace from the parallel workloads archive.