Performance Modeling and Prediction of Big Data Workflows: An Exploratory Analysis

Wuji Liu, Chase Q. Wu, Qianwen Ye, Aiqin Hou, Wei Shen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many next-generation scientific and business applications feature large-scale data-intensive workflows, which require massive computing resources for execution on high-performance clusters in cloud environments. Such computing resources (e.g., VCores and virtual memory) requested through parameter setting in big data systems, if not fully utilized by workloads, are simply wasted due to the nature of exclusive access made possible by containerization. This necessitates accurate modeling and prediction of workflow performance to make an effective recommendation of appropriate parameter settings to end users. However, it is challenging to determine optimal workflow and system configurations due to the large parameter space and the interaction between various technology layers of big data systems. Towards this goal, we propose a machine learning-based feature selection method to identify influential parameters based on historical performance measurements of Spark-based computing workloads executed in big data systems with YARN. We first identify a comprehensive set of parameters across multiple layers in the big data technology stack including workflow input structure, Spark computing engine, and YARN resource management. We then conduct an in-depth exploratory analysis of their individual and coupled impact on workflow performance, and develop a performance-influence model using random forest for prediction. Experimental results show that the proposed approach identifies important features for performance modeling and achieves high accuracy in performance prediction.

Original languageEnglish (US)
Title of host publicationICCCN 2020 - 29th International Conference on Computer Communications and Networks
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728166070
DOIs
StatePublished - Aug 2020
Event29th International Conference on Computer Communications and Networks, ICCCN 2020 - Honolulu, United States
Duration: Aug 3 2020Aug 6 2020

Publication series

NameProceedings - International Conference on Computer Communications and Networks, ICCCN
Volume2020-August
ISSN (Print)1095-2055

Conference

Conference29th International Conference on Computer Communications and Networks, ICCCN 2020
CountryUnited States
CityHonolulu
Period8/3/208/6/20

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Hardware and Architecture
  • Software

Keywords

  • Big data workflows
  • machine learning
  • performance modeling and prediction
  • representation learning
  • Spark

Fingerprint Dive into the research topics of 'Performance Modeling and Prediction of Big Data Workflows: An Exploratory Analysis'. Together they form a unique fingerprint.

Cite this