Experiences in using OS-level virtualization for block I/O

Dan Huang, Jun Wang, Qing Liu, Jiangling Yin, Xuhong Zhang, Xunchao Chen

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Today, HPC clusters commonly use Resource Management Systems such as PBS and TORQUE to share physical resources. These systems enable resources to be shared by assigning nodes to users exclusively in non-overlapping time slots. With virtualization technology, users can run their applications on the same node with low mutual interference. However, the overhead introduced by the virtual machine monitor or hypervisor is too high to be accepted, because efficiency is key to many HPC applications. OS-level virtualization (such as Linux Containers) offers a lightweight virtualization layer, which promises a near-native performance and is adopted by some BigData resource sharing platforms such as Mesos. Nevertheless, OS-level virtualization's overhead and isolation on block devices have not been completely evaluated, especially when applied to a shared distributed/parallel file system (D/PFS) such as HDFS or Lustre. In this paper, we thoroughly evaluate the overhead and isolation involved in sharing block I/O via OS-level virtualization on the local disk and D/PFSs. Meanwhile, to assign D/PFS storage resources to users, a middleware system is proposed and implemented to bridge the configuration gap between virtual clusters and remote D/PFSs.

Original languageEnglish (US)
Title of host publicationProceedings of PDSW 2015
Subtitle of host publication10th Parallel Data Storage Workshop - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherAssociation for Computing Machinery, Inc
Pages13-18
Number of pages6
ISBN (Electronic)9781450340083
DOIs
StatePublished - Nov 15 2015
Externally publishedYes
Event10th Parallel Data Storage Workshop, PDSW 2015 - Austin, United States
Duration: Nov 16 2015 → …

Publication series

NameProceedings of PDSW 2015: 10th Parallel Data Storage Workshop - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

Other

Other10th Parallel Data Storage Workshop, PDSW 2015
CountryUnited States
CityAustin
Period11/16/15 → …

All Science Journal Classification (ASJC) codes

  • Information Systems

Fingerprint Dive into the research topics of 'Experiences in using OS-level virtualization for block I/O'. Together they form a unique fingerprint.

Cite this