TY - GEN
T1 - DFS-container
T2 - 2017 Symposium on Cloud Computing, SoCC 2017
AU - Huang, Dan
AU - Wang, Jun
AU - Liu, Qing
AU - Zhang, Xuhong
AU - Chen, Xunchao
AU - Zhou, Jian
PY - 2017/9/24
Y1 - 2017/9/24
N2 - Today BigData systems commonly use resource management systems such as TORQUE, Mesos, and Google Borg to share the physical resources among users or applications. Enabled by virtualization, users can run their applications on the same node with low mutual interference. Container-based virtualizations (e.g., Docker and Linux Containers) offer a lightweight virtualization layer, which promises a near-native performance and is adopted by some Big-Data resource sharing platforms such as Mesos. Nevertheless, using containers to consolidate the I/O resources of shared storage systems is still at an early stage, especially in a distributed file system (DFS) such as Hadoop File System (HDFS). To overcome this issue, we propose a distributed middleware system, DFS-Container, by further containerizing DFS.We also evaluate and analyze the unfairness of using containers to proportionally allocate the I/O resource of DFS. Based on these analyses and evaluations, we propose and implement a new mechanism, IOPS-Regulator, which improve the fairness of proportional allocation by 74.4% on average.
AB - Today BigData systems commonly use resource management systems such as TORQUE, Mesos, and Google Borg to share the physical resources among users or applications. Enabled by virtualization, users can run their applications on the same node with low mutual interference. Container-based virtualizations (e.g., Docker and Linux Containers) offer a lightweight virtualization layer, which promises a near-native performance and is adopted by some Big-Data resource sharing platforms such as Mesos. Nevertheless, using containers to consolidate the I/O resources of shared storage systems is still at an early stage, especially in a distributed file system (DFS) such as Hadoop File System (HDFS). To overcome this issue, we propose a distributed middleware system, DFS-Container, by further containerizing DFS.We also evaluate and analyze the unfairness of using containers to proportionally allocate the I/O resource of DFS. Based on these analyses and evaluations, we propose and implement a new mechanism, IOPS-Regulator, which improve the fairness of proportional allocation by 74.4% on average.
KW - CGroups
KW - HDFS
KW - Proportional I/O Allocation
UR - http://www.scopus.com/inward/record.url?scp=85032439257&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85032439257&partnerID=8YFLogxK
U2 - 10.1145/3127479.3132568
DO - 10.1145/3127479.3132568
M3 - Conference contribution
AN - SCOPUS:85032439257
T3 - SoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing
SP - 660
BT - SoCC 2017 - Proceedings of the 2017 Symposium on Cloud Computing
PB - Association for Computing Machinery, Inc
Y2 - 24 September 2017 through 27 September 2017
ER -