Total Variation Reduction for Lossless Compression of HPC Applications

Junqi Wang, Yida Li, Qing Liu, Huizhang Luo, Kenli Li

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the growing size of high-performance computing (HPC) applications, a major challenge that domain scientists are facing is how to efficiently store and analyze the vast volume of output data. Compression can reduce the amount of data that needs to be transferred and stored. However, most of the large datasets are of floating-point format, which exhibit high entropy. As a result, existing lossless compressors usually achieve a modest reduction ratio of less than 2X. To address this problem, we propose a total variation reduction method to improve the compression ratio of lossless compressors. In particular, we first try to exploit space-filling curve (SFC), a well-known technique to preserve data locality for a multi-dimensional HPC dataset. We show and explain why a raw SFC, such as Hilbert curve and Z-order curve, cannot improve the compression ratio. Then, we explore the opportunity and theoretical feasibility of the proposed total variation reduction based algorithm. The experiment results show the effectiveness of the proposed method. The compression ratios are improved by 20.6% for FPZIP, and 18.4% for FPC, on average.

Original languageEnglish (US)
Title of host publicationProceedings - 34th IEEE International System-on-Chip Conference, SOCC 2021
EditorsGang Qu, Jinjun Xiong, Danella Zhao, Venki Muthukumar, Md Farhadur Reza, Ramalingam Sridhar
PublisherIEEE Computer Society
Pages129-134
Number of pages6
ISBN (Electronic)9781665429313
DOIs
StatePublished - 2021
Externally publishedYes
Event34th IEEE International System-on-Chip Conference, SOCC 2021 - Virtual, Online, United States
Duration: Sep 14 2021Sep 17 2021

Publication series

NameInternational System on Chip Conference
Volume2021-September
ISSN (Print)2164-1676
ISSN (Electronic)2164-1706

Conference

Conference34th IEEE International System-on-Chip Conference, SOCC 2021
Country/TerritoryUnited States
CityVirtual, Online
Period9/14/219/17/21

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Keywords

  • High-performance computing
  • floating-point data
  • lossless compression

Fingerprint

Dive into the research topics of 'Total Variation Reduction for Lossless Compression of HPC Applications'. Together they form a unique fingerprint.

Cite this