Autonomous learning of load and traffic patterns to improve cluster utilization

Hukeun Kwak, Andrew Sohn, Kyusik Chung

Research output: Contribution to journalArticlepeer-review

4 Scopus citations

Abstract

Adaptive clustering aims at improving cluster utilization for varying load and traffic patterns. Locality-based least-connection with replication (LBLCR) scheduling that comes with Linux is designed to help improve cluster utilization through adaptive clustering. A key issue with LBLCR, however, is that cluster performance depends much on a single threshold value that is used to determine adaptation. Once set, the threshold remains fixed, regardless of the load and traffic patterns. If a cluster of PCs is to adapt to different traffic patterns for high utilization, a good threshold has to be selected and used dynamically. We present in this paper an adaptive clustering framework that autonomously learns and adapts to different load and traffic patterns at runtime with no administrator intervention. The cluster is configured once and for all. As the patterns change, the cluster automatically expands/contracts to meet the changing demands. At the same time, the patterns are proactively learned so that when similar patterns emerge in the future, the cluster knows what to do to improve utilization. We have implemented this autonomous learning method and compared it with LBLCR using published Web traces. Experimental results indicate that our autonomous learning method produces high performance scalability and adaptability for different patterns. On the other hand LBLCR-based clustering suffers from performance scalability and adaptability for different traffic patterns since it is not designed to obtain good threshold values and use them at runtime.

Original languageEnglish (US)
Pages (from-to)397-417
Number of pages21
JournalCluster Computing
Volume14
Issue number4
DOIs
StatePublished - Dec 1 2011

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Networks and Communications

Keywords

  • Adaptive clustering
  • Autonomous traffic learning
  • Cluster utilization
  • Load and traffic patterns

Fingerprint Dive into the research topics of 'Autonomous learning of load and traffic patterns to improve cluster utilization'. Together they form a unique fingerprint.

Cite this