TY - GEN
T1 - BigData Express
T2 - 2018 IEEE/ACM Innovating the Network for Data-Intensive Science, INDIS 2018
AU - Lu, Qiming
AU - Zhang, Liang
AU - Sasidharan, Sajith
AU - Wu, Wenji
AU - Demar, Phil
AU - Guok, Chin
AU - MacAuley, John
AU - Monga, Inder
AU - Yu, Se Young
AU - Chen, Jim Hao
AU - Mambretti, Joe
AU - Kim, Jin
AU - Noh, Seo Young
AU - Yang, Xi
AU - Lehman, Tom
AU - Liu, Gary
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2018/7/2
Y1 - 2018/7/2
N2 - Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.
AB - Big Data has emerged as a driving force for scientific discoveries. Large scientific instruments (e.g., colliders, and telescopes) generate exponentially increasing volumes of data. To enable scientific discovery, science data must be collected, indexed, archived, shared, and analyzed, typically in a widely distributed, highly collaborative manner. Data transfer is now an essential function for science discoveries, particularly within big data environments. Although significant improvements have been made in the area of bulk data transfer, the currently available data transfer tools and services can not successfully address the high-performance and time-constraint challenges of data transfer required by extreme-scale science applications for the following reasons: disjoint end-to-end data transfer loops, cross-interference between data transfers, and existing data transfer tools and services are oblivious to user requirements (deadline and QoS requirements). Fermilab has been working on the BigData Express project to address these problems. BigData Express seeks to provide a schedulable, predictable, and high-performance data transfer service for big data science. The BigData Express software is being deployed and evaluated at multiple research institutions, which include UMD, StarLight, FNAL, KISTI, KSTAR, SURFnet, Ciena, and other sites. Meanwhile, the BigData Express research team is collaborating with the StarLight International/National Communications Exchange Facility to deploy BigData Express at various research platforms, including Pacific Research Platform, National Research Platform, and Global Research Platform. It is envisioned that we are working toward building a high-performance data transfer federation for big data science.
KW - Big-data
KW - Co-scheduling
KW - DTN
KW - High-performance-data-transfer
KW - High-speed-networking
KW - Performance
KW - SDN
UR - http://www.scopus.com/inward/record.url?scp=85063320038&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85063320038&partnerID=8YFLogxK
U2 - 10.1109/INDIS.2018.00011
DO - 10.1109/INDIS.2018.00011
M3 - Conference contribution
AN - SCOPUS:85063320038
T3 - Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis
SP - 75
EP - 84
BT - Proceedings of INDIS 2018
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 11 November 2018
ER -