SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data

Zhenbo Qiao, Tao Lu, Huizhang Luo, Qing Liu, Scott Klasky, Norbert Podhorszki, Jinzhen Wang

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Scientific simulations on high performance computing (HPC) platforms generate large quantities of data. To bridge the widening gap between compute and I/O, and enable data to be more efficiently stored and analyzed, simulation outputs need to be refactored, reduced, and appropriately mapped to storage tiers. However, a systematic solution to support these steps has been lacking in the current HPC software ecosystem. To that end, this paper develops SIRIUS, a progressive JPEG-like data management scheme for storing and analyzing big scientific data. It co-designs data decimation, compression, and data storage, taking the hardware characteristics of each storage tier into considerations. With reasonably low overhead, our approach refactors simulation data, using either topological or uniform decimation, into a much smaller, reduced-Accuracy base dataset, and a series of deltas that is used to augment the accuracy if needed. The base dataset and deltas are compressed and written to multiple storage tiers. Data saved on different tiers can then be selectively retrieved to restore the level of accuracy that satisfies data analytics. Thus, SIRIUS provides a paradigm shift towards elastic data analytics and enables end users to make trade-offs between analysis speed and accuracy on-The-fly. This paper further develops algorithms to preserve statistics for data decimation, a common requirement for reducing data. We assess the impact of SIRIUS on unstructured triangular meshes, a pervasive data model used in scientific simulations. In particular, we evaluate two realistic use cases: The blob detection in fusion and high-pressure area extraction in computational fluid dynamics.

Original languageEnglish (US)
Article number8576666
Pages (from-to)900-913
Number of pages14
JournalIEEE Transactions on Multi-Scale Computing Systems
Issue number4
StatePublished - Oct 1 2018

All Science Journal Classification (ASJC) codes

  • Control and Systems Engineering
  • Information Systems
  • Hardware and Architecture


  • High-performance computing
  • compression
  • data analytics
  • data reduction
  • progressive refactoring
  • storage


Dive into the research topics of 'SIRIUS: Enabling Progressive Data Exploration for Extreme-Scale Scientific Data'. Together they form a unique fingerprint.

Cite this