Scheduling Data-Parallel Computations on Heterogeneous and Time-Shared Environments

This paper addresses the problem of load balancing data-parallel computations on heterogeneous and time-shared parallel computing environments, where load imbalance may be introduced by the different capacities of processors populating a computer, or by the sharing of the same computational resources among several users. To solve this problem we propose a run-time support for parallel loops based upon a hybrid (static + dynamic) scheduling strategy. The main features of our technique are the absence of centralization and synchronization points, the prefetching of work toward slower processors, and the overlapping of communication latencies with useful computation.