Maximizing workflow throughput for streaming applications in distributed environments

Yi Gu, Qishi Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

17 Scopus citations

Abstract

Large-scale computation-intensive applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate intermodule dependencies. Mapping such workflows in heterogeneous network environments and maximizing their throughput are crucial to the success of large-scale scientific applications that process streaming datasets. We construct analytical cost models and formulate workflow mapping as an optimization problem for maximum frame rate. The difficulty of this problem essentially arises from the topological matching nature in the spatial domain, which is further compounded by the resource sharing complicacy in the temporal dimension if multiple modules are deployed on the same node. We conduct a rigorous workflow stability analysis and design a workflow mapping scheme based on a topological layer-oriented dynamic programming solution to identify and minimize the global bottleneck. The performance superiority of the proposed mapping scheme is illustrated by extensive simulation-based comparisons with existing algorithms.

Original languageEnglish (US)
Title of host publication2010 Proceedings of 19th International Conference on Computer Communications and Networks, ICCCN 2010
DOIs
StatePublished - Oct 29 2010
Externally publishedYes
Event2010 19th International Conference on Computer Communications and Networks, ICCCN 2010 - Zurich, Switzerland
Duration: Aug 2 2010Aug 5 2010

Publication series

NameProceedings - International Conference on Computer Communications and Networks, ICCCN
ISSN (Print)1095-2055

Other

Other2010 19th International Conference on Computer Communications and Networks, ICCCN 2010
Country/TerritorySwitzerland
CityZurich
Period8/2/108/5/10

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Hardware and Architecture
  • Software

Keywords

  • Distributed computing
  • Frame rate
  • Workflow

Fingerprint

Dive into the research topics of 'Maximizing workflow throughput for streaming applications in distributed environments'. Together they form a unique fingerprint.

Cite this