On Performance Prediction of Big Data Transfer in High-performance Networks

Wuji Liu, Daqing Yun, Chase Q. Wu, Nageswara S.V. Rao, Aiqin Hou, Wei Shen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations


Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.

Original languageEnglish (US)
Title of host publication2020 IEEE International Conference on Communications, ICC 2020 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781728150895
StatePublished - Jun 2020
Event2020 IEEE International Conference on Communications, ICC 2020 - Dublin, Ireland
Duration: Jun 7 2020Jun 11 2020

Publication series

NameIEEE International Conference on Communications
ISSN (Print)1550-3607


Conference2020 IEEE International Conference on Communications, ICC 2020

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Electrical and Electronic Engineering


  • Performance prediction
  • big data transfer
  • high-performance networks
  • latent variables
  • machine learning


Dive into the research topics of 'On Performance Prediction of Big Data Transfer in High-performance Networks'. Together they form a unique fingerprint.

Cite this