SmokeOut: An approach for testing clustering implementations

Vincenzo Musco, Xin Yin, Iulian Neamtiu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Clustering is a key Machine Learning technique, used in many high-stakes domains from medicine to self-driving cars. Many clustering algorithms have been proposed, and these algorithms have been implemented in many toolkits. Clustering users assume that clustering implementations are correct, reliable, and for a given algorithm, interchangeable. We challenge these assumptions. We introduce SmokeOut, an approach and tool that pits clustering implementations against each other (and against themselves) while controlling for algorithm and dataset, to find datasets where clustering outcomes differ when they shouldn't, and measure this difference. We ran SmokeOut on 7 clustering algorithms (3 deterministic and 4 nondeterministic) implemented in 7 widely-used toolkits, and run in a variety of scenarios on the Penn Machine Learning Benchmark (162 datasets). SmokeOut has revealed that clustering implementations are fragile: on a given input dataset and using a given clustering algorithm, clustering outcomes and accuracy vary widely between (1) successive runs of the same toolkit; (2) different input parameters for that tool; (3) different toolkits.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages473-480
Number of pages8
ISBN (Electronic)9781728117355
DOIs
StatePublished - Apr 2019
Event12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019 - Xi'an, China
Duration: Apr 22 2019Apr 27 2019

Publication series

NameProceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019

Conference

Conference12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
CountryChina
CityXi'an
Period4/22/194/27/19

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Safety, Risk, Reliability and Quality

Keywords

  • Clustering
  • Differential Testing
  • Machine Learning
  • Software Reliability

Fingerprint Dive into the research topics of 'SmokeOut: An approach for testing clustering implementations'. Together they form a unique fingerprint.

Cite this