A Formal Approach to Evaluating Medical Ontology Systems using Naturalness

Ontologies, terminologies and vocabularies are popular repositories for collecting the terms used in a domain. It may be expected that in the future more such ontologies will be created for domain experts. However, there is increasing interest in making the language of experts understandable to casual users. For example, cancer patients often research their cases on the Web. The authors consider the problem of objectively evaluating the quality of ontologies (QoO). This article formalizes the notion of naturalness as a component of QoO and quantitatively measures naturalness for well-known ontologies (UMLS, WordNet, OpenCyc) based on their concepts, IS-A relationships and semantic relationships. To compute numeric values characterizing the naturalness of an ontology, this article defines appropriate metrics. As absolute numbers in such a pursuit are often meaningless, we concentrate on using relative naturalness metrics. That allows us to say that a certain ontology is relatively more natural than another one.