Size oblivious programming with infinimem

Sai Charan Koduru, Rajiv Gupta, Iulian Neamtiu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Scopus citations

Abstract

Many recently proposed BigData processing frameworks make programming easier, but typically expect the datasets to fit in the memory of either a single multicore machine or a cluster of multicore machines. When this assumption does not hold, these frameworks fail. We introduce the InfiniMem framework that enables size oblivious processing of large collections of objects that do not fit in memory by making them disk-resident. InfiniMem is easy to program with: the user just indicates the large collections of objects that are to be made diskresident, while InfiniMem transparently handles their I/O management. The InfiniMem library can manage a very large number of objects in a uniform manner, even though the objects have different characteristics and relationships which, when processed, give rise to a wide range of access patterns requiring different organizations of data on the disk. We demonstrate the ease of programming and versatility of InfiniMem with 3 different probabilistic analytics algorithms, 3 different graph processing size oblivious frameworks; they require minimal effort, 6–9 additional lines of code. We show that InfiniMem can successfully generate a mesh with 7.5 million nodes and 300 million edges (4.5GB on disk) in 40min and it performs the PageRank computation on a 14GB graph with 134 million vertices and 805 million edges at 14 min per iteration on an 8-core machine with 8GB RAM. Many graph generators and processing frameworks cannot handle such large graphs. We also exploit InfiniMem on a cluster to scale-up an object-based DSM.

Original languageEnglish (US)
Title of host publicationLanguages and Compilers for Parallel Computing - 28th International Workshop, LCPC 2015, Revised Selected Papers
EditorsXipeng Shen, Frank Mueller, James Tuck
PublisherSpringer Verlag
Pages3-19
Number of pages17
ISBN (Print)9783319297774
DOIs
StatePublished - 2016
Externally publishedYes
Event28th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2015 - Raleigh, United States
Duration: Sep 9 2015Sep 11 2015

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9519
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other28th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2015
Country/TerritoryUnited States
CityRaleigh
Period9/9/159/11/15

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Size oblivious programming with infinimem'. Together they form a unique fingerprint.

Cite this