Evaluating rank accuracy based on incomplete pairwise preferences

Brian Ackerman, Yi Chen

Research output: Contribution to journalConference articlepeer-review

4 Scopus citations


Existing methods to measure the rank accuracy of a recommender system assume the ground truth is either a set of user ratings or a total ordered list of items given by the user with possible ties. However, in many applications we are only able to obtain implicit user feedback, which does not provide such comprehensive information, but only gives a set of pairwise preferences among items. Generally such pairwise preferences are not complete, and thus may not deduce a total order of items. In this paper, we propose a novel method to evaluate rank accuracy, expected discounted rank correlation, which addresses the unique challenges of handling incomplete pairwise preferences in ground truth and also puts an emphasis on properly ranking items that users most prefer.

Original languageEnglish (US)
Pages (from-to)74-77
Number of pages4
JournalCEUR Workshop Proceedings
StatePublished - 2011
Externally publishedYes
EventJoint Workshop on Human Decision Making in Recommender Systems, Decisions@RecSys 2011 and User-Centric Evaluation of Recommender Systems and Their Interfaces-2, UCERSTI 2 - Affiliated with the 5th ACM Conference on Recommender Systems, RecSys 2011 - Chicago, IL, United States
Duration: Oct 23 2011Oct 26 2011

All Science Journal Classification (ASJC) codes

  • General Computer Science


  • Evaluation
  • Pairwise preference
  • Rank accuracy
  • Recommender systems


Dive into the research topics of 'Evaluating rank accuracy based on incomplete pairwise preferences'. Together they form a unique fingerprint.

Cite this