On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters

Yang Liu; Chase Q. Wu; Meng Wang; Aiqin Hou; Yongqiang Wang

doi:10.1109/ISNCC.2018.8530970

On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters

Yang Liu, Chase Q. Wu, Meng Wang, Aiqin Hou, Yongqiang Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

12 Scopus citations

Abstract

Hadoop is one of the most popular distributed systems for big data computing in both industry and science communities. The default data placement strategy of Hadoop Distributed File System (HDFS), which was initially designed for homogenous environments, may suffer from performance degradation when deployed in heterogeneous clusters comprised of data nodes with disparate computing power and disk capacity, hence undermining the performance of MapReduce applications. In this paper, we use a Grey Forecast model to predict data hotness dynamically and determine an appropriate number of data block replicas on the fly. Based on such information, we further propose a dynamic data placement strategy (DDPS) to decide the best location for new replicas according to their hotness. The proposed method is able to dynamically adjust data replicas stored on each node in a heterogeneous Hadoop cluster and reduce the response time of big data applications. Experimental results on a heterogeneous Hadoop cluster show that DDPS together with the prediction model significantly increases application execution efficiency and improve MapReduce performance over the default HDFS configuration.

Original language	English (US)
Title of host publication	2018 International Symposium on Networks, Computers and Communications, ISNCC 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781538637784
DOIs	https://doi.org/10.1109/ISNCC.2018.8530970
State	Published - Nov 9 2018
Event	2018 International Symposium on Networks, Computers and Communications, ISNCC 2018 - Rome, Italy Duration: Jun 19 2018 → Jun 21 2018

Publication series

Name	2018 International Symposium on Networks, Computers and Communications, ISNCC 2018

Other

Other	2018 International Symposium on Networks, Computers and Communications, ISNCC 2018
Country/Territory	Italy
City	Rome
Period	6/19/18 → 6/21/18

All Science Journal Classification (ASJC) codes

Computational Theory and Mathematics
Computer Networks and Communications
Hardware and Architecture
Energy Engineering and Power Technology
Safety, Risk, Reliability and Quality

Keywords

HDFS
Heterogeneous Hadoop
MapReduce
data placement
prediction model

Access to Document

10.1109/ISNCC.2018.8530970

Cite this

Liu, Y., Wu, C. Q., Wang, M., Hou, A., & Wang, Y. (2018). On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters. In 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018 Article 8530970 (2018 International Symposium on Networks, Computers and Communications, ISNCC 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISNCC.2018.8530970

@inproceedings{99af723f48d347fd86aaa5bf98c466f5,

title = "On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters",

abstract = "Hadoop is one of the most popular distributed systems for big data computing in both industry and science communities. The default data placement strategy of Hadoop Distributed File System (HDFS), which was initially designed for homogenous environments, may suffer from performance degradation when deployed in heterogeneous clusters comprised of data nodes with disparate computing power and disk capacity, hence undermining the performance of MapReduce applications. In this paper, we use a Grey Forecast model to predict data hotness dynamically and determine an appropriate number of data block replicas on the fly. Based on such information, we further propose a dynamic data placement strategy (DDPS) to decide the best location for new replicas according to their hotness. The proposed method is able to dynamically adjust data replicas stored on each node in a heterogeneous Hadoop cluster and reduce the response time of big data applications. Experimental results on a heterogeneous Hadoop cluster show that DDPS together with the prediction model significantly increases application execution efficiency and improve MapReduce performance over the default HDFS configuration.",

keywords = "HDFS, Heterogeneous Hadoop, MapReduce, data placement, prediction model",

author = "Yang Liu and Wu, {Chase Q.} and Meng Wang and Aiqin Hou and Yongqiang Wang",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018 ; Conference date: 19-06-2018 Through 21-06-2018",

year = "2018",

month = nov,

day = "9",

doi = "10.1109/ISNCC.2018.8530970",

language = "English (US)",

series = "2018 International Symposium on Networks, Computers and Communications, ISNCC 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2018 International Symposium on Networks, Computers and Communications, ISNCC 2018",

address = "United States",

}

Liu, Y, Wu, CQ, Wang, M, Hou, A & Wang, Y 2018, On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters. in 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018., 8530970, 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018, Institute of Electrical and Electronics Engineers Inc., 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018, Rome, Italy, 6/19/18. https://doi.org/10.1109/ISNCC.2018.8530970

On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters. / Liu, Yang; Wu, Chase Q.; Wang, Meng et al.
2018 International Symposium on Networks, Computers and Communications, ISNCC 2018. Institute of Electrical and Electronics Engineers Inc., 2018. 8530970 (2018 International Symposium on Networks, Computers and Communications, ISNCC 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters

AU - Liu, Yang

AU - Wu, Chase Q.

AU - Wang, Meng

AU - Hou, Aiqin

AU - Wang, Yongqiang

PY - 2018/11/9

Y1 - 2018/11/9

N2 - Hadoop is one of the most popular distributed systems for big data computing in both industry and science communities. The default data placement strategy of Hadoop Distributed File System (HDFS), which was initially designed for homogenous environments, may suffer from performance degradation when deployed in heterogeneous clusters comprised of data nodes with disparate computing power and disk capacity, hence undermining the performance of MapReduce applications. In this paper, we use a Grey Forecast model to predict data hotness dynamically and determine an appropriate number of data block replicas on the fly. Based on such information, we further propose a dynamic data placement strategy (DDPS) to decide the best location for new replicas according to their hotness. The proposed method is able to dynamically adjust data replicas stored on each node in a heterogeneous Hadoop cluster and reduce the response time of big data applications. Experimental results on a heterogeneous Hadoop cluster show that DDPS together with the prediction model significantly increases application execution efficiency and improve MapReduce performance over the default HDFS configuration.

AB - Hadoop is one of the most popular distributed systems for big data computing in both industry and science communities. The default data placement strategy of Hadoop Distributed File System (HDFS), which was initially designed for homogenous environments, may suffer from performance degradation when deployed in heterogeneous clusters comprised of data nodes with disparate computing power and disk capacity, hence undermining the performance of MapReduce applications. In this paper, we use a Grey Forecast model to predict data hotness dynamically and determine an appropriate number of data block replicas on the fly. Based on such information, we further propose a dynamic data placement strategy (DDPS) to decide the best location for new replicas according to their hotness. The proposed method is able to dynamically adjust data replicas stored on each node in a heterogeneous Hadoop cluster and reduce the response time of big data applications. Experimental results on a heterogeneous Hadoop cluster show that DDPS together with the prediction model significantly increases application execution efficiency and improve MapReduce performance over the default HDFS configuration.

KW - HDFS

KW - Heterogeneous Hadoop

KW - MapReduce

KW - data placement

KW - prediction model

UR - http://www.scopus.com/inward/record.url?scp=85058473721&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058473721&partnerID=8YFLogxK

U2 - 10.1109/ISNCC.2018.8530970

DO - 10.1109/ISNCC.2018.8530970

M3 - Conference contribution

AN - SCOPUS:85058473721

T3 - 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018

BT - 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018

Y2 - 19 June 2018 through 21 June 2018

ER -

Liu Y, Wu CQ, Wang M, Hou A, Wang Y. On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters. In 2018 International Symposium on Networks, Computers and Communications, ISNCC 2018. Institute of Electrical and Electronics Engineers Inc. 2018. 8530970. (2018 International Symposium on Networks, Computers and Communications, ISNCC 2018). doi: 10.1109/ISNCC.2018.8530970

On a Dynamic Data Placement Strategy for Heterogeneous Hadoop Clusters

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this