Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, Manish Parashar, Nagiza Samatova, Karsten Schwan, Arie Shoshani, Matthew Wolf, Kesheng Wu, Weikuan Yu

Research output: Contribution to journalArticlepeer-review

130 Scopus citations

Abstract

Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

Original languageEnglish (US)
Pages (from-to)1453-1473
Number of pages21
JournalConcurrency Computation Practice and Experience
Volume26
Issue number7
DOIs
StatePublished - May 2014
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Computer Science Applications
  • Computer Networks and Communications
  • Computational Theory and Mathematics

Keywords

  • I/O middleware
  • high performance I/O
  • high performance computing

Fingerprint Dive into the research topics of 'Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks'. Together they form a unique fingerprint.

Cite this