Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Qing Liu; Jeremy Logan; Yuan Tian; Hasan Abbasi; Norbert Podhorszki; Jong Youl Choi; Scott Klasky; Roselyne Tchoua; Jay Lofstead; Ron Oldfield; Manish Parashar; Nagiza Samatova; Karsten Schwan; Arie Shoshani; Matthew Wolf; Kesheng Wu; Weikuan Yu

doi:10.1002/cpe.3125

Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, Manish Parashar, Nagiza Samatova, Karsten Schwan, Arie Shoshani, Matthew Wolf, Kesheng Wu, Weikuan Yu

Research output: Contribution to journal › Article › peer-review

181 Scopus citations

Abstract

Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

Original language	English (US)
Pages (from-to)	1453-1473
Number of pages	21
Journal	Concurrency Computation Practice and Experience
Volume	26
Issue number	7
DOIs	https://doi.org/10.1002/cpe.3125
State	Published - May 2014
Externally published	Yes

All Science Journal Classification (ASJC) codes

Software
Theoretical Computer Science
Computer Networks and Communications
Computer Science Applications
Computational Theory and Mathematics

Keywords

I/O middleware
high performance I/O
high performance computing

Access to Document

10.1002/cpe.3125

Cite this

Liu, Q., Logan, J., Tian, Y., Abbasi, H., Podhorszki, N., Choi, J. Y., Klasky, S., Tchoua, R., Lofstead, J., Oldfield, R., Parashar, M., Samatova, N., Schwan, K., Shoshani, A., Wolf, M., Wu, K., & Yu, W. (2014). Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks. Concurrency Computation Practice and Experience, 26(7), 1453-1473. https://doi.org/10.1002/cpe.3125

@article{dd1dab53bc914935940f4d871d353462,

title = "Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks",

abstract = "Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.",

keywords = "I/O middleware, high performance I/O, high performance computing",

author = "Qing Liu and Jeremy Logan and Yuan Tian and Hasan Abbasi and Norbert Podhorszki and Choi, {Jong Youl} and Scott Klasky and Roselyne Tchoua and Jay Lofstead and Ron Oldfield and Manish Parashar and Nagiza Samatova and Karsten Schwan and Arie Shoshani and Matthew Wolf and Kesheng Wu and Weikuan Yu",

year = "2014",

month = may,

doi = "10.1002/cpe.3125",

language = "English (US)",

volume = "26",

pages = "1453--1473",

journal = "Concurrency Computation Practice and Experience",

issn = "1532-0626",

publisher = "John Wiley and Sons Ltd",

number = "7",

}

Liu, Q, Logan, J, Tian, Y, Abbasi, H, Podhorszki, N, Choi, JY, Klasky, S, Tchoua, R, Lofstead, J, Oldfield, R, Parashar, M, Samatova, N, Schwan, K, Shoshani, A, Wolf, M, Wu, K & Yu, W 2014, 'Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks', Concurrency Computation Practice and Experience, vol. 26, no. 7, pp. 1453-1473. https://doi.org/10.1002/cpe.3125

TY - JOUR

T1 - Hello ADIOS

T2 - The challenges and lessons of developing leadership class I/O frameworks

AU - Liu, Qing

AU - Logan, Jeremy

AU - Tian, Yuan

AU - Abbasi, Hasan

AU - Podhorszki, Norbert

AU - Choi, Jong Youl

AU - Klasky, Scott

AU - Tchoua, Roselyne

AU - Lofstead, Jay

AU - Oldfield, Ron

AU - Parashar, Manish

AU - Samatova, Nagiza

AU - Schwan, Karsten

AU - Shoshani, Arie

AU - Wolf, Matthew

AU - Wu, Kesheng

AU - Yu, Weikuan

PY - 2014/5

Y1 - 2014/5

N2 - Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

AB - Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

KW - I/O middleware

KW - high performance I/O

KW - high performance computing

UR - http://www.scopus.com/inward/record.url?scp=84899424893&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84899424893&partnerID=8YFLogxK

U2 - 10.1002/cpe.3125

DO - 10.1002/cpe.3125

M3 - Article

AN - SCOPUS:84899424893

SN - 1532-0626

VL - 26

SP - 1453

EP - 1473

JO - Concurrency Computation Practice and Experience

JF - Concurrency Computation Practice and Experience

IS - 7

ER -

Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Abstract

All Science Journal Classification (ASJC) codes

Keywords

Access to Document

Other files and links

Fingerprint

Cite this