Parallel in situ indexing for data-intensive computing

Jinoh Kim, Hasan Abbasi, Luis Chacón, Ciprian Docan, Scott Klasky, Qing Liu, Norbert Podhorszki, Arie Shoshani, Kesheng Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

35 Scopus citations

Abstract

As computing power increases exponentially, vast amount of data is created by many scientific research activities. However, the bandwidth for storing the data to disks and reading the data from disks has been improving at a much slower pace. These two trends produce an ever-widening data access gap. Our work brings together two distinct technologies to address this data access issue: indexing and in situ processing. From decades of database research literature, we know that indexing is an effective way to address the data access issue, particularly for accessing relatively small fraction of data records. As data sets increase in sizes, more and more analysts need to use selective data access, which makes indexing an even more important for improving data access. The challenge is that most implementations of indexing technology are embedded in large database management systems (DBMS), but most scientific datasets are not managed by any DBMS. In this work, we choose to include indexes with the scientific data instead of requiring the data to be loaded into a DBMS.We use compressed bitmap indexes from the FastBit software which are known to be highly effective for query-intensive workloads common to scientific data analysis. To use the indexes, we need to build them first. The index building procedure needs to access the whole data set and may also require a significant amount of compute time. In this work, we adapt the in situ processing technology to generate the indexes, thus removing the need of reading data from disks and to build indexes in parallel. The in situ data processing system used is ADIOS, a middleware for high-performance I/O. Our experimental results show that the indexes can improve the data access time up to 200 times depending on the fraction of data selected, and using in situ data processing system can effectively reduce the time needed to create the indexes, up to 10 times with our in situ technique when using identical parallel settings.

Original languageEnglish (US)
Title of host publication1st IEEE Symposium on Large-Scale Data Analysis and Visualization 2011, LDAV 2011 - Proceedings
Pages65-72
Number of pages8
DOIs
StatePublished - 2011
Externally publishedYes
Event1st IEEE Symposium on Large-Scale Data Analysis and Visualization 2011, LDAV 2011 - Providence, RI, United States
Duration: Oct 23 2011Oct 24 2011

Publication series

Name1st IEEE Symposium on Large-Scale Data Analysis and Visualization 2011, LDAV 2011 - Proceedings

Other

Other1st IEEE Symposium on Large-Scale Data Analysis and Visualization 2011, LDAV 2011
Country/TerritoryUnited States
CityProvidence, RI
Period10/23/1110/24/11

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Keywords

  • D.4.2 [Storage Management]: Access methods
  • H.3.3 [Information Search and Retrieval]
  • [D.4.2]: Storage Management

Fingerprint

Dive into the research topics of 'Parallel in situ indexing for data-intensive computing'. Together they form a unique fingerprint.

Cite this