DFS-container: Achieving containerized block I/O for distributed file systems

Dan Huang, Jun Wang, Qing Liu, Xuhong Zhang, Xunchao Chen, Jian Zhou

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Today BigData systems commonly use resource management systems such as TORQUE, Mesos, and Google Borg to share the physical resources among users or applications. Enabled by virtualization, users can run their applications on the same node with low mutual interference. Container-based virtualizations (e.g., Docker and Linux Containers) offer a lightweight virtualization layer, which promises a near-native performance and is adopted by some Big-Data resource sharing platforms such as Mesos. Nevertheless, using containers to consolidate the I/O resources of shared storage systems is still at an early stage, especially in a distributed file system (DFS) such as Hadoop File System (HDFS). To overcome this issue, we propose a distributed middleware system, DFS-Container, by further containerizing DFS.We also evaluate and analyze the unfairness of using containers to proportionally allocate the I/O resource of DFS. Based on these analyses and evaluations, we propose and implement a new mechanism, IOPS-Regulator, which improve the fairness of proportional allocation by 74.4% on average.

Original languageEnglish (US)
Title of host publicationSoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing
PublisherAssociation for Computing Machinery, Inc
Pages660
Number of pages1
ISBN (Electronic)9781450350280
DOIs
StatePublished - Sep 24 2017
Event2017 Symposium on Cloud Computing, SoCC 2017 - Santa Clara, United States
Duration: Sep 24 2017Sep 27 2017

Publication series

NameSoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing

Other

Other2017 Symposium on Cloud Computing, SoCC 2017
CountryUnited States
CitySanta Clara
Period9/24/179/27/17

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Theoretical Computer Science

Keywords

  • CGroups
  • HDFS
  • Proportional I/O Allocation

Fingerprint Dive into the research topics of 'DFS-container: Achieving containerized block I/O for distributed file systems'. Together they form a unique fingerprint.

Cite this