Crowdsourcing Ontology Verification.
Jonathan Mortensen, Paul R. Alexander, Mark A. Musen, Natalya Fridman Noy · 2013
Biomedical ontologies are becoming increasingly large and complex. A single user cannot easily develop or maintain them. Researchers have developed various automated techniques to assist with ontology development and engineering at scale. However, these solutions are not always complete. Microtask crowdsourcing, wherein workers are paid small amounts to complete simple, short tasks, may be one technique to alleviate some of the development difficulties. Previously, we developed a method to verify an ontology hierarchy using microtask crowdsourcing. In this work, we investigated the finer details of the design and configuration of a hierarchyverification task. For example, when we provided definitions and required qualifications, workers performed with 82 % accuracy on the hierarchy-verification task, compared to 50 % without. We showed that to achieve reasonable performance on such a task, workers require context via definitions, tasks require qualifications that select a worker with proper domain knowledge, and a question must be phrased with the least cognitive load (i.e., in the simplest way).