Multiple Choice Online Algorithms for Technology-Assisted Reviews
Otto Benjamin Piramuthu · 2023
Exhaustive manual review of documents to determine their relevancy for a given purpose is error-prone and resource intensive. This has led to the consideration of computer-aided processes where only a small subset of the entire set of documents is manually reviewed with comparable performance as exhaustive manual review, resulting in the reduction of human-introduced error and allocated resources. As more evidence for the superiority of Technology-Assisted Reviews (TAR) becomes available, researchers and practitioners have resorted to the exploration of various methods to improve process efficiency without significant degradation in output quality. Of particular interest in this process is the decision on when to stop reviewing additional documents. We consider and evaluate online algorithms, specifically a solution to the multiple choice secretary problem, which is a natural fit for this purpose. Unlike in extant TAR methods, only the relative ranking among the documents that have already been retrieved is needed for this algorithm. Our results indicate that online algorithms are a competitive choice for TAR applications.