Subjective confidence and source reliability in soft data fusion

There is ongoing interest in constructing data fusion systems which are capable of using human (i.e., soft) decisions and confidence assessments as inputs. Most relevant studies involved experimentation with humans which is often expensive, subject to strict institutional regulations, and hard to validate and replicate. Here we make use of a mathematical model of human decision-making and human confidence assessment developed by Pleskac and Busemeyer (2010) in order to compare four types of fusion operators: (1) operators that use human-subject decisions (such as the k-out-of-N majority rule); (2) operators that use subject decisions and error rates (the Chair and Varshney fusion rule); (3) operators that use subject decisions and confidence assessments (Yager's rule and the Proportional Conflict Redistribution rule #5); and (4) operators that use subject decisions, confidence assessments, and the average strength of each subject's confidence assessment, namely the average Brier scores (Dempster's rule of combination and Bayes' rule of probability combination). The ability of each fusion system to discriminate between alternatives was determined by computing the normalized area under the receiver operating characteristic curves (AUC). When the number of sources used by the fusion algorithm exceeded five, fusion operators which made use of decisions and confidence assessments alone (i.e., type (3)) produced the lowest (namely, worst) normalized AUC values. Operators which made use of subject reliabilities (i.e., types (2) and (4)) produced larger (namely, better) normalized AUC values which, in addition, were similar to those of fusion algorithms that relied on decisions alone (i.e., type (1)). For the city size discrimination task studied by Pleskac and Busmeyer, these results suggest that as the number of sources increases, the importance of decision self-assessment diminishes.