Advising big data transfer over dedicated connections based on profiling optimization

Daqing Yun, Chase Q. Wu, Nageswara S.V. Rao, Rajkumar Kettimuthu

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

Big data transfer in next-generation scientific applications is now commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.

Original languageEnglish (US)
Article number8867859
Pages (from-to)2280-2293
Number of pages14
JournalIEEE/ACM Transactions on Networking
Volume27
Issue number6
DOIs
StatePublished - Dec 2019

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Electrical and Electronic Engineering

Keywords

  • Profiling optimization
  • big data transfer
  • high-performance networks
  • stochastic approximation

Fingerprint

Dive into the research topics of 'Advising big data transfer over dedicated connections based on profiling optimization'. Together they form a unique fingerprint.

Cite this