On an integrated mapping and scheduling solution to large-scale scientific workflows in resource sharing environments

Next-generation e-science applications feature large-scale data-intensive workflows comprised of many interrelated computing modules. The end-to-end performance of such scientific workflows depends on both the mapping scheme, which determines module assignment, and the scheduling policy, which determines resource allocation if multiple modules are mapped to the same node. These two aspects of workflow optimization are traditionally treated as two separated topics, and the interactions between them have not been fully explored by any existing efforts. As the scale of scientific workflows and the complexity of network environments rapidly increase, each individual aspect of performance optimization alone can only meet with limited success. We conduct an in-depth investigation into workflow execution dynamics of both mapping and scheduling, and propose an integrated solution, referred to as Mapping and Scheduling Interaction (MSI), to achieve a higher level of resource utilization and workflow performance. The efficacy of MSI is illustrated by extensive simulation-based workflow experiments.