Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Daqing Yun, Wuji Liu, Chase Q. Wu, Nageswara S.V. Rao, Rajkumar Kettimuthu

Research output: Contribution to journalArticlepeer-review

Abstract

Big data transfer in large-scale scientific and business applications is increasingly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) via advance bandwidth reservation. Provisioning agents need to carefully schedule data transfer requests, compute network paths, and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized, could be simply wasted due to the exclusive access during the approved time window, and cause extra overhead and complexity for resource management. This calls for accurate performance prediction to reserve bandwidths that match actual needs and avoid over-provisioning. We employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements collected in the past several years from data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated testbeds. We first analyze the performance patterns in response to a comprehensive list of parameters in end-host systems, network connections, and data transfer applications, which motivate the use of machine learning and also help us identify the effects of latent factors. We then propose threshold- and clustering-based methods to eliminate negative effects of latent factors in data preprocessing and build a robust performance predictor based on customized domain-oriented loss functions. The performance of the proposed methods is verified by extensive experiments using SVR and RFR as well as theoretical analysis of the general performance bound.

Original languageEnglish (US)
Article number104285
JournalEngineering Applications of Artificial Intelligence
Volume102
DOIs
StatePublished - Jun 2021

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Keywords

  • Big data transfer
  • Latent effect
  • Machine learning
  • Performance prediction

Fingerprint

Dive into the research topics of 'Exploratory analysis and performance prediction of big data transfer in High-performance Networks'. Together they form a unique fingerprint.

Cite this