Exploring Approaches to Optimize the Performance of Predictive Coding on Multilanguage Data Sets
Christian J. Mahoney, Nathaniel Huber-Fliflet, Peter Gronvall, Chris Clark, Jianping Zhang, Fusheng Wei, Qiang Mao · 2023
Predictive modeling - known in the legal industry as ‘predictive coding’ or ‘Technology Assisted Review (TAR)’ - is a popular tool used by legal professionals to augment a historically manual document review and classification process in responding to data requests for various legal proceedings. It has become more commonplace over the past decade due to its proven ability to minimize manual document classification, thus reducing the time and cost associated with this aspect of legal proceedings. There is significant research supporting the effectiveness of this technology that includes topics, such as identifying the most performant machine learning algorithms (e.g., logistic regression) or establishing the best methodologies for selecting representative training and testing data. Primarily, this research has been performed without a focus on multilanguage data sets because many legal proceedings typically involve one primary, dominant language. As acceptance of predictive modeling technology grows, legal practitioners have developed independent preferences for handling multiple languages in data sets. These preferences have been primarily based on anecdotal experience rather than empirical assessments and have resulted in two modeling approaches. The first approach uses a single model, which is less complex, and more cost effective than the next approach. The second develops multiple language-specific models, which can create workflow complexity that may impact the overall cost savings that predictive coding seeks to achieve. Proponents of the second approach believe that creating models per language results in better performance over a single model approach. In this study, we empirically explore the performance differences between the two approaches - single model and multiple language-specific models. We hypothesize that a single model approach performs similarly to a language-specific modeling approach when classifying documents for relevance. We evaluate both approaches by comparing their resulting precision-recall curves across real-world test data from legal document review projects. Excitingly, our results demonstrate that in most scenarios, a single model (trained with multiple languages) approach will perform as well, or sometimes better than the approach that uses a group of language-specific models. The results of our research provide a roadmap for legal practitioners and other parties to thoughtfully engage in dialogue around the most effective way to deploy predictive coding within a multilanguage data set. This research will change how predictive models are deployed for multilanguage data sets across the legal industry and enable significant cost savings for clients and legal practitioners by utilizing the most efficient modeling method.