TY - GEN
T1 - Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning
AU - Yun, Daqing
AU - Liu, Wuji
AU - Wu, Chase Q.
AU - Rao, Nageswara S.V.
AU - Kettimuthu, Rajkumar
N1 - Publisher Copyright:
© 2020 IFIP.
PY - 2020/6
Y1 - 2020/6
N2 - Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.
AB - Big data transfer in next-generation scientific applications is now commonly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) through advance bandwidth reservation. To use HPN resources efficiently, provisioning agents need to carefully schedule data transfer requests and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized by the requesting user, could be simply wasted or cause extra overhead and complexity in management due to exclusive access. This calls for the capability of performance prediction to reserve bandwidth resources that match actual needs. Towards this goal, we employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements, which are collected over a span of several years from a large number of data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated HPN testbeds. We first identify a comprehensive list of attributes involved in a typical big data transfer process, including end host system configurations, network connection properties, and control parameters of data transfer methods. We then conduct an in-depth exploratory analysis of their impacts on application-level throughput, which provides insights into big data transfer performance and motivates the use of machine learning. We also investigate the applicability of machine learning algorithms and derive their general performance bounds for performance prediction of big data transfer in HPNs. Experimental results show that, with appropriate data preprocessing, the proposed machine learning-based approach achieves 95% or higher prediction accuracy in up to 90% of the cases with very noisy real-life performance measurements.
KW - Performance prediction
KW - big data transfer
KW - experimental analysis
KW - machine learning
UR - http://www.scopus.com/inward/record.url?scp=85090047916&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85090047916&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85090047916
T3 - IFIP Networking 2020 Conference and Workshops, Networking 2020
SP - 181
EP - 189
BT - IFIP Networking 2020 Conference and Workshops, Networking 2020
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2020 IFIP Networking Conference and Workshops, Networking 2020
Y2 - 22 June 2020 through 25 June 2020
ER -