Energy-efficient mapping of large-scale workflows under deadline constraints in big data computing systems

Tong Shu, Chase Q. Wu

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

Large-scale workflows for big data analytics have become a main consumer of energy in data centers where moldable parallel computing models such as MapReduce are widely applied to meet high computational demands with time-varying computing resources. The granularity of task partitioning in each moldable job of such big data workflows has a significant impact on energy efficiency, which remains largely unexplored. In this paper, we analyze the properties of moldable jobs and formulate a workflow mapping problem to minimize the dynamic energy consumption of a given workflow request under a deadline constraint in big data systems. Since this problem is strongly NP-hard, we design a fully polynomial-time approximation scheme (FPTAS) for a special case with a pipeline-structured workflow on a homogeneous cluster and a heuristic for the generalized problem with an arbitrary workflow on a heterogeneous cluster. The performance superiority of the proposed solution in terms of dynamic energy saving and deadline missing rate is illustrated by extensive simulation results in comparison with existing algorithms, and further validated by real-life workflow implementation and experimental results in Hadoop/YARN systems.

Original languageEnglish (US)
Pages (from-to)515-530
Number of pages16
JournalFuture Generation Computer Systems
Volume110
DOIs
StatePublished - Sep 2020

All Science Journal Classification (ASJC) codes

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Keywords

  • Big data
  • Green computing
  • Workflow mapping

Fingerprint Dive into the research topics of 'Energy-efficient mapping of large-scale workflows under deadline constraints in big data computing systems'. Together they form a unique fingerprint.

Cite this