Optimizing the performance of big data workflows in multi-cloud environments under budget constraint

Chase Q. Wu, Huiyan Cao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

18 Scopus citations

Abstract

Workflow techniques have been widely used as a major computing solution in many science domains. With the rapid deployment of cloud infrastructures around the globe and the economic benefit of cloud-based computing and storage services, an increasing number of scientific workflows have been shifted or are in active transition to clouds. As the scale of scientific applications continues to grow, it is now common to deploy data-and network-intensive computing workflows in multi-cloud environments, where inter-cloud data transfer oftentimes plays a significant role in both workflow performance and financial cost. We construct rigorous mathematical models to analyze the intra-and inter-cloud execution process of scientific workflows and formulate a budget-constrained workflow mapping problem to optimize the network performance of scientific workflows in multi-cloud environments. We show the proposed problem to be NP-complete and design a heuristic solution that takes into consideration module execution, data transfer, and I/O operations. The performance superiority of the proposed solution over existing methods is illustrated through extensive simulations.

Original languageEnglish (US)
Title of host publicationProceedings - 2016 IEEE International Conference on Services Computing, SCC 2016
EditorsJia Zhang, John A. Miller, Xiaofei Xu
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages138-145
Number of pages8
ISBN (Electronic)9781509026289
DOIs
StatePublished - Aug 31 2016
Event2016 IEEE International Conference on Services Computing, SCC 2016 - San Francisco, United States
Duration: Jun 27 2016Jul 2 2016

Publication series

NameProceedings - 2016 IEEE International Conference on Services Computing, SCC 2016

Other

Other2016 IEEE International Conference on Services Computing, SCC 2016
CountryUnited States
CitySan Francisco
Period6/27/167/2/16

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Hardware and Architecture
  • Software

Keywords

  • Cloud computing
  • Performance optimization
  • Scientific workflows
  • Workflow mapping

Fingerprint Dive into the research topics of 'Optimizing the performance of big data workflows in multi-cloud environments under budget constraint'. Together they form a unique fingerprint.

Cite this