Quantifying Nondeterminism and Inconsistency in Self-organizing Map Implementations

Sydur Rahaman, Raina Samuel, Iulian Neamtiu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Self-organizing maps (SOMs) are a popular approach for neural network-based unsupervised learning. However the reliability of self-organizing map implementations has not been investigated. Using internal and external metrics, we define and check two basic SOM properties. First, determinism: A given SOM implementation should produce the same SOM when run repeatedly on the same training dataset. Second, consistency: Two SOM implementations should produce similar SOMs when presented with the same training dataset. We check these properties in four popular SOM implementations. We ran our approach on 381 popular datasets used in health, medicine, and other critical domains. We found that implementations violate these basic properties. For example, 375 out of 381 datasets have nondeterministic outcomes; for 51-92% of datasets, toolkits yield significantly different SOM clusterings; and clustering accuracy might be so inconsistent as to vary by a factor of four between toolkits. This undermines SOM reliability, and the reliability of results obtained via SOMs. Our study shines a light on what to expect, in practice, when running actual SOM implementations. Our findings suggest that for critical applications, SOM users should not take reliability for granted; rather, multiple runs and different toolkits should be considered and compared.

Original languageEnglish (US)
Title of host publicationProceedings - 3rd IEEE International Conference on Artificial Intelligence Testing, AITest 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages85-92
Number of pages8
ISBN (Electronic)9781665434812
DOIs
StatePublished - Aug 2021
Event3rd IEEE International Conference on Artificial Intelligence Testing, AITest 2021 - Virtual, Online, United Kingdom
Duration: Aug 23 2021Aug 26 2021

Publication series

NameProceedings - 3rd IEEE International Conference on Artificial Intelligence Testing, AITest 2021

Conference

Conference3rd IEEE International Conference on Artificial Intelligence Testing, AITest 2021
Country/TerritoryUnited Kingdom
CityVirtual, Online
Period8/23/218/26/21

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Safety, Risk, Reliability and Quality
  • Modeling and Simulation

Keywords

  • AI reliability
  • AI testing
  • neural networks
  • nondeterminism
  • Self-organizing maps
  • validation

Fingerprint

Dive into the research topics of 'Quantifying Nondeterminism and Inconsistency in Self-organizing Map Implementations'. Together they form a unique fingerprint.

Cite this