TY - JOUR
T1 - Evaluation of a UMLS Auditing Process of Semantic Type Assignments.
AU - Gu, Huanying Helen
AU - Hripcsak, George
AU - Chen, Yan
AU - Morrey, C. Paul
AU - Elhanan, Gai
AU - Cimino, James
AU - Geller, James
AU - Perl, Yehoshua
PY - 2007
Y1 - 2007
N2 - The UMLS is a terminological system that integrates many source terminologies. Each concept in the UMLS is assigned one or more semantic types from the Semantic Network, an upper level ontology for biomedicine. Due to the complexity of the UMLS, errors exist in the semantic type assignments. Finding assignment errors may unearth modeling errors. Even with sophisticated tools, discovering assignment errors requires manual review. In this paper we describe the evaluation of an auditing project of UMLS semantic type assignments. We studied the performance of the auditors who reviewed potential errors. We found that four auditors, interacting according to a multi-step protocol, identified a high rate of errors (one or more errors in 81% of concepts studied) and that results were sufficiently reliable (0.67 to 0.70) for the two most common types of errors. However, reliability was low for each individual auditor, suggesting that review of potential errors is resource-intensive.
AB - The UMLS is a terminological system that integrates many source terminologies. Each concept in the UMLS is assigned one or more semantic types from the Semantic Network, an upper level ontology for biomedicine. Due to the complexity of the UMLS, errors exist in the semantic type assignments. Finding assignment errors may unearth modeling errors. Even with sophisticated tools, discovering assignment errors requires manual review. In this paper we describe the evaluation of an auditing project of UMLS semantic type assignments. We studied the performance of the auditors who reviewed potential errors. We found that four auditors, interacting according to a multi-step protocol, identified a high rate of errors (one or more errors in 81% of concepts studied) and that results were sufficiently reliable (0.67 to 0.70) for the two most common types of errors. However, reliability was low for each individual auditor, suggesting that review of potential errors is resource-intensive.
UR - http://www.scopus.com/inward/record.url?scp=56149090083&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=56149090083&partnerID=8YFLogxK
M3 - Article
C2 - 18693845
AN - SCOPUS:56149090083
SN - 1559-4076
SP - 294
EP - 298
JO - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium
JF - AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium
ER -