Co-sites: The autonomous distributed dataflows in collaborative scientific discovery

Yanwei Zhang, Matthew Wolf, Karsten Schwan, Qing Liu, Greg Eisenhauer, Scott Klasky

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Online "big data" processing applications have seen increas- ing importance in the high performance computing domain, including online analytics of large volumes of data output by various scientific applications. This work contributes to answering the question of how to promote efficient collaborative science in face of unpre- dictable analytics workloads and dynamics in available re- sources? It proposes the Co-Sites solution employing online resource management at the sites participating online collab- oration, including geographically distributed sites that may spread across large distances. Co-Sites operates by each site observing its local progress and making its own decisions to better utilize local resources and to maintain acceptable rates of global progress. Co-Sites further enriches such dis- tributed data ows to permit just-in-time data sharing to better leverage collaborators' diverse domain expertise. Experiments with a combustion workow demonstrate the Co-Sites solution with (i) improved end-to-end completion times, (ii) good scalability, and (iii) with good data sharing latencies.

Original languageEnglish (US)
Title of host publicationProceedings of WORKS 2015
Subtitle of host publication10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450339896
DOIs
StatePublished - Nov 15 2015
Externally publishedYes
Event10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015 - Austin, United States
Duration: Nov 15 2015 → …

Publication series

NameProceedings of WORKS 2015: 10th Workshop on Workflows in Support of Large-Scale Science - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

Other

Other10th Workshop on Workflows in Support of Large-Scale Science, WORKS 2015
CountryUnited States
CityAustin
Period11/15/15 → …

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'Co-sites: The autonomous distributed dataflows in collaborative scientific discovery'. Together they form a unique fingerprint.

Cite this