Using unsupervised topic modeling to uncover document hierarchy and latent topics in prostate cancer clinical texts
Tuulia Denti, Wei Sun, Shaoxiong Ji, Hans Jonas Fossum Moen, Oleg Kerro, Antti Sakari Rannikko, Pekka Marttinen, Miika Koskinen · medRxiv · 2024
Abstract Objective Medical records reflect patients’ health status and journey through healthcare services and medical specialties. As a result, document collections contain various text types, even about one patient, which makes medical texts unique. Our aim was to take advantage of the implicit document hierarchy of medical records to evaluate contextualized neural topic modeling as a tool to gain insight into a mixed set of notes related to prostate cancer treatment, i.e., to uncover document types, their contents, and relations. Materials and Methods We collected clinical text documents from 21,872 prostate cancer patients and organized the documents into a hierarchy using the document metadata. We trained neural topic models without the metadata to index the document collections, performed rigorous numerical evaluations of topic and clustering quality to optimize the topic count, visualized the latent representation of the models, and evaluated the topic clusters with respect to document metadata. Results Topic clusters reflected the structure of the document hierarchy and provided information about the contents of different text types. The determination of the optimal number of topics required complementary information by topic and clustering quality metrics. Discussion The topic modeling was found useful in visualizing and indexing large document collections, in providing an understanding of document contents, and in revealing document organization comparable to metadata-based hierarchy. Conclusion Hospital databases contain masses of text documents, and topic modeling can provide means for analysts and researchers to group documents into discernable and explainable classes.