k-Dispatch's Performance Modules for Advanced Workflow Submission

Complex scientific workflows describing challenging
real-world problems are composed of a lot of computational tasks
which require high performance computing or cloud facilities to
be computed in a sensible time. Most of such tasks are also
written as distributed parallel programs being able to run across
multiple compute nodes. The amount of requested resources
per task influences both the overall execution time (makespan)
and the computational cost. Optimal resource assignment to
particular task is thus crucial. Since the exact execution time
cannot be measured for every possible combination of task, input
data, and assigned resources, its prediction may be challenging. This poster introduces the idea of performance modules implemented within k-Dispatch that employ space-searching methods together with fitting and machine learning methods. The optimal amount of resources is found using those methods and evaluated using a cluster simulator that estimates the workflow makespan quickly. Using the performance modules, the execution plans for the input workflows are found and can be submitted to the real cluster. k-Dispatch then monitors those remote calculations within a provided service.