Subjective confidence and source reliability in soft data fusion

Donald J. Bucci, Sayandeep Acharya, Timothy J. Pleskac, Moshe Kam

Research output: Contribution to conferencePaperpeer-review

3 Scopus citations

Abstract

There is ongoing interest in constructing data fusion systems which are capable of using human (i.e., soft) decisions and confidence assessments as inputs. Most relevant studies involved experimentation with humans which is often expensive, subject to strict institutional regulations, and hard to validate and replicate. Here we make use of a mathematical model of human decision-making and human confidence assessment developed by Pleskac and Busemeyer (2010) in order to compare four types of fusion operators: (1) operators that use human-subject decisions (such as the k-out-of-N majority rule); (2) operators that use subject decisions and error rates (the Chair and Varshney fusion rule); (3) operators that use subject decisions and confidence assessments (Yager's rule and the Proportional Conflict Redistribution rule #5); and (4) operators that use subject decisions, confidence assessments, and the average strength of each subject's confidence assessment, namely the average Brier scores (Dempster's rule of combination and Bayes' rule of probability combination). The ability of each fusion system to discriminate between alternatives was determined by computing the normalized area under the receiver operating characteristic curves (AUC). When the number of sources used by the fusion algorithm exceeded five, fusion operators which made use of decisions and confidence assessments alone (i.e., type (3)) produced the lowest (namely, worst) normalized AUC values. Operators which made use of subject reliabilities (i.e., types (2) and (4)) produced larger (namely, better) normalized AUC values which, in addition, were similar to those of fusion algorithms that relied on decisions alone (i.e., type (1)). For the city size discrimination task studied by Pleskac and Busmeyer, these results suggest that as the number of sources increases, the importance of decision self-assessment diminishes.

Original languageEnglish (US)
DOIs
StatePublished - Jan 1 2014
Externally publishedYes
Event2014 48th Annual Conference on Information Sciences and Systems, CISS 2014 - Princeton, NJ, United States
Duration: Mar 19 2014Mar 21 2014

Other

Other2014 48th Annual Conference on Information Sciences and Systems, CISS 2014
CountryUnited States
CityPrinceton, NJ
Period3/19/143/21/14

All Science Journal Classification (ASJC) codes

  • Information Systems

Keywords

  • Data fusion
  • Dempster-Shafer Theory
  • human simulation
  • subjective confidences

Fingerprint Dive into the research topics of 'Subjective confidence and source reliability in soft data fusion'. Together they form a unique fingerprint.

Cite this