Performance of probability transformations using simulated human opinions

Probability transformations provide a method of relating Dempster-Shafer sources of evidence to subjective probability assignments. These transforms are constructed to facilitate decision making over a set of mutually exclusive hypotheses. The probability information content (PIC) metric has been recently proposed for characterizing the performance of different probability transforms. To investigate the applicability of the PIC metric, we compare five probability transformations (i.e., BetP, PrPl, PrNPl, PrHyb, and DSmP) using a simulator of human responses from cognitive psychology known as two-stage dynamic signal detection. Responses were simulated over two tasks: a line length discrimination task and a city population size discrimination task. Human decision-makers were modeled for these two tasks by Pleskac and Busemeyer (2010). Subject decisions and confidence assessments were simulated and combined for both tasks using Yager's rule and mapped into subjective probabilities using the five probability transforms. Receiver operating characteristic (ROC) curves, normalized areas under the ROC curves (AUCs), along with average PIC values were obtained for each probability transform. Our results indicate that higher PIC values do not necessarily equate to higher discriminability (i.e., higher normalized AUCs) between probability transforms. In fact, all five probability transforms exhibited nearly the same normalized AUC values. At lower, fixed false alarm rates, the BetP, PrPl, PrNPl, and PrHyb transforms yielded higher detection rates over the DSmP transform. For higher, fixed false alarm rates, the DSmP transform yielded higher detection rates over the other four transforms. These trends were observed over both tasks, which suggests that the PIC may not be sufficient for evaluating the performance of probability transforms.