GLEE: GPR-based Latent Effect Elimination for Performance Prediction of Big Data Transfer

A wide spectrum of large-scale applications in science, industry, and business domains generate colossal amounts of data that need to be moved across geographical locations for various purposes. Such big data transfer is increasingly carried out over connections with guaranteed bandwidth provisioned in high-performance networks, as exemplified by ESnet via advance bandwidth reservation agents such as OSCARS. Accurate performance prediction of big data transfer is vital for resource management to reserve appropriate bandwidth and optimize resource utilization. Data-driven methods using machine learning offer a promising solution to such prediction capabilities. However, most of these methods highly rely on the quality of training data and might suffer from low prediction accuracy when performance measurements are collected in environments with high dynamics or under latent effects. We propose a performance prediction method to help end users obtain a good estimate of achievable throughput, which could also be used by network managers to infer expected bandwidth usage and hence optimize bandwidth reservation and utilization. Based on Gaussian Process Regression (GPR), our approach is able to automatically detect intrinsic noise and identify latent effects to enhance prediction accuracy. We implement and evaluate the proposed method using performance measurements collected in real-life networks and the results demonstrate its effectiveness.