TY - JOUR
T1 - Advising big data transfer over dedicated connections based on profiling optimization
AU - Yun, Daqing
AU - Wu, Chase Q.
AU - Rao, Nageswara S.V.
AU - Kettimuthu, Rajkumar
N1 - Funding Information:
Manuscript received September 5, 2018; revised April 29, 2019 and July 8, 2019; accepted September 17, 2019; approved by IEEE/ACM TRANS-ACTIONS ON NETWORKING Editor J. Liu. Date of publication October 14, 2019; date of current version December 17, 2019. This work was supported in part by the Department of Energy’s Office of Science under Grant DESC0015892 and Contract DE-AC05-00OR22725 at Oak Ridge National Laboratory managed by UT-Battelle, LLC, in part by the Harrisburg University’s Presidential Research Grant, and in part by the National Science Foundation with New Jersey Institute of Technology under Grant CNS-1828123. Some preliminary results of this article were presented at IEEE ICCCN 2016 [19] and IEEE LCN 2017 [20]. (Corresponding author: Chase Q. Wu.) D. Yun is with the Computer and Information Sciences Program, Harrisburg University, Harrisburg, PA 17101 USA (e-mail: dyun@harrisburgu.edu).
Publisher Copyright:
© 1993-2012 IEEE.
PY - 2019/12
Y1 - 2019/12
N2 - Big data transfer in next-generation scientific applications is now commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.
AB - Big data transfer in next-generation scientific applications is now commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.
KW - Profiling optimization
KW - big data transfer
KW - high-performance networks
KW - stochastic approximation
UR - http://www.scopus.com/inward/record.url?scp=85077313347&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85077313347&partnerID=8YFLogxK
U2 - 10.1109/TNET.2019.2943884
DO - 10.1109/TNET.2019.2943884
M3 - Article
AN - SCOPUS:85077313347
SN - 1063-6692
VL - 27
SP - 2280
EP - 2293
JO - IEEE/ACM Transactions on Networking
JF - IEEE/ACM Transactions on Networking
IS - 6
M1 - 8867859
ER -