Enhancing Proportional IO Sharing on Containerized Big Data File Systems

Dan Huang, Jun Wang, Qing Liu, Nong Xiao, Huafeng Wu, Jiangling Yin

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Big Data platforms recently employ resource management systems, such as YARN, Mesos, and Google Borg, to provision computational resources. These systems adopt containerization to share the computing resources in a multi-tenant setting with low performance overhead and interference. However, it may be observed that tenants often interfere with each other on the underlying Big Data File Systems (BDFS), e.g., Hadoop File System, which have been widely deployed as a persistent layer in current data centers. A solution with systematic generality is to containerize BDFS itself to isolate and allocate its IO sources to multiple tenants. To this end, we conduct analysis on the ineffectiveness of proportionally sharing BDFS IO resource via containerization. This ineffectiveness is due to the scheduler of containerization in 'pseudo-starvation' status, in which most of IO requests are backlogged in BDFS rather than in containerization scheduler. Without enough backlogged IO requests, existing schedulers might have to maximize device utilization rather than enforce proportional sharing policy. To resolve this ineffectiveness issue, we develop a cross-layer system called BDFS-Container, which containerizes BDFS at the Linux block IO level. Central to BDFS-Container, we propose and design a proactive IOPS throttling-based mechanism named IOPS Regulator, which achieves a trade-off between maximizing IO utilization and accurately proportional IO sharing. The evaluation results show that our method can improve proportionally sharing BDFS IO resources by 74.4 percent on average.

Original languageEnglish (US)
Pages (from-to)2083-2097
Number of pages15
JournalIEEE Transactions on Computers
Issue number12
StatePublished - Dec 1 2021

All Science Journal Classification (ASJC) codes

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture
  • Computational Theory and Mathematics


  • Containerization
  • big data storage
  • hadoop file system
  • resource sharing


Dive into the research topics of 'Enhancing Proportional IO Sharing on Containerized Big Data File Systems'. Together they form a unique fingerprint.

Cite this