BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer

Qiming Lu, Liang Zhang, Sajith Sasidharan, Wenji Wu, Phil Demar, Chin Guok, John MacAuley, Inder Monga, Se Young Yu, Jim Hao Chen, Joe Mambretti, Jin Kim, Seo Young Noh, Xi Yang, Tom Lehman, Gary Liu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

6 Scopus citations

Abstract

Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.

Original languageEnglish (US)
Title of host publicationProceedings of INDIS 2018
Subtitle of host publicationInnovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages75-84
Number of pages10
ISBN (Electronic)9781728101941
DOIs
StatePublished - Feb 21 2019
Event2018 IEEE/ACM Innovating the Network for Data-Intensive Science, INDIS 2018 - Dallas, United States
Duration: Nov 11 2018 → …

Publication series

NameProceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference2018 IEEE/ACM Innovating the Network for Data-Intensive Science, INDIS 2018
Country/TerritoryUnited States
CityDallas
Period11/11/18 → …

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Computer Networks and Communications

Keywords

  • Big-data
  • Co-scheduling
  • DTN
  • High-performance-data-transfer
  • High-speed-networking
  • Performance
  • SDN

Fingerprint

Dive into the research topics of 'BigData Express: Toward Schedulable, Predictable, and High-Performance Data Transfer'. Together they form a unique fingerprint.

Cite this