Compression Ratio Modeling and Estimation across Error Bounds for Lossy Compression

Jinzhen Wang, Tong Liu, Qing Liu, Xubin He, Huizhang Luo, Weiming He

Research output: Contribution to journalArticlepeer-review

Abstract

Scientific simulations on high-performance computing (HPC) systems generate vast amounts of floating-point data that need to be reduced in order to lower the storage and I/O cost. Lossy compressors trade data accuracy for reduction performance and have been demonstrated to be effective in reducing data volume. However, a key hurdle to wide adoption of lossy compressors is that the trade-off between data accuracy and compression performance, particularly the compression ratio, is not well understood. Consequently, domain scientists often need to exhaust many possible error bounds before they can figure out an appropriate setup. The current practice of using lossy compressors to reduce data volume is, therefore, through trial and error, which is not efficient for large datasets which take a tremendous amount of computational resources to compress. This paper aims to analyze and estimate the compression performance of lossy compressors on HPC datasets. In particular, we predict the compression ratios of two modern lossy compressors that achieve superior performance, SZ and ZFP, on HPC scientific datasets at various error bounds, based upon the compressors' intrinsic metrics collected under a given base error bound. We evaluate the estimation scheme using twenty real HPC datasets and the results confirm the effectiveness of our approach.

Original languageEnglish (US)
Article number8821342
Pages (from-to)1621-1635
Number of pages15
JournalIEEE Transactions on Parallel and Distributed Systems
Volume31
Issue number7
DOIs
StatePublished - Jul 1 2020

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Keywords

  • High-performance computing
  • data reduction
  • lossy compression
  • performance modeling

Fingerprint Dive into the research topics of 'Compression Ratio Modeling and Estimation across Error Bounds for Lossy Compression'. Together they form a unique fingerprint.

Cite this