Automated Detection of Core Comments in Online UAV Discussion Forums
Doheon Han, Nuno Moniz, Jane Cleland‐Huang, Nitesh V. Chawla · Research Square · 2024
Abstract Unmanned aerial vehicles (UAVs) have become popular, and accordingly, diagnosing failure is essential but not easy due to its difficulty and complexity. Online UAV forums can help users diagnose the failures since they contain abundant information from diverse users. However, matching responses to user needs is not easy because forum comments are often unclear or contradictory, and it can be even harder when forums contain difficult content. Large Language Models have shown good achievement for this task, but they have limitations such as their inexplicableness, or inconsistent results. Therefore, we try a traditional machine-learning approach, text classification in a supervised manner, with human-annotated labels. This approach shows competitive performance with analyzable and consistent results, however, some issues such as imbalance and user subjectivity are identified. The subjectivity issue is more serious in difficult forums like UAV forums, meaning each user defines correctness or helpfulness with their standards. In this paper, we tackle determining core comments, i.e., answers or helpful comments, in online UAV discussion forums while dealing with the imbalance and subjectivity issues. With a thorough experimental evaluation, we confirm the effect of our strategy. Empirically, we discover how individual subjectivity may harm the training and evaluation process. In addition, we analyze the results in quantitative and qualitative ways to define helpfulness in the context of online forums.