Transparent in situ data transformations in ADIOS

David A. Boyuka, Sriram Lakshminarasimham, Xiaocheng Zou, Zhenhuan Gong, John Jenkins, Eric R. Schendel, Norbert Podhorszki, Qing Liu, Scott Klasky, Nagiza F. Samatova

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Scopus citations

Abstract

Though an abundance of novel "data transformation" technologies have been developed (such as compression, level-of-detail, layout optimization, and indexing), there remains a notable gap in the adoption of such services by scientific applications. In response, we develop an in situ data transformation framework in the ADIOS I/O middleware with a 'plug in' interface, thus greatly simplifying both the deployment and use of data transform services in scientific applications. Our approach ensures user-transparency, runtime-configurability, compatibility with existing I/O optimizations, and the potential for exploiting read-optimizing transforms (such as level-of-detail) to achieve I/O reduction. We demonstrate use of our framework with the QLG simulation at up to 8,192 cores on the leadership-class Titan supercomputer, showing negligible overhead. We also explore the read performance implications of data transforms with respect to parameters such as chunk size, access pattern, and the 'opacity' of different transform methods including compression and level-of-detail.

Original languageEnglish (US)
Title of host publicationProceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2014
PublisherIEEE Computer Society
Pages256-266
Number of pages11
ISBN (Print)9781479927838
DOIs
StatePublished - Jan 1 2014
Externally publishedYes
Event14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014 - Chicago, IL, United States
Duration: May 26 2014May 29 2014

Publication series

NameProceedings - 14th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2014

Other

Other14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGrid 2014
CountryUnited States
CityChicago, IL
Period5/26/145/29/14

All Science Journal Classification (ASJC) codes

  • Software

Keywords

  • ADIOS
  • I/O middleware
  • compression
  • data transforms
  • indexing
  • level-of-detail
  • storage layout optimization

Fingerprint Dive into the research topics of 'Transparent in situ data transformations in ADIOS'. Together they form a unique fingerprint.

Cite this