LLMs Cannot (Yet) Match the Specificity and Simplicity of Online Communities in Long Form Question Answering
Kris-Fillip Kahl, Tolga Buz, Russa Biswas, Gerard de Melo · 2024
Retail investing is on the rise, and a growing number of users are relying on online finance communities to educate themselves.However, as Large Language Models (LLMs) are increasingly viewed as powerful question-answering (QA) tools, users have shifted away from interacting in communities towards discourse with AI-driven conversational interfaces.Such AI tools are currently constrained by the availability of labelled data providing domain-specific financial knowledge.Therefore, in this work, we curate a QA preference dataset called SO-CIALFINANCEQA for fine-tuning and aligning LLMs, extracted from more than 7.4 million submissions and 82 million comments from 2008 to 2022 in Reddit's 15 largest finance communities.Additionally, we propose the novel framework SOCIALQA-EVAL as a generally applicable method to evaluate generated QA responses.We evaluate various LLMs fine-tuned on this dataset, using traditional metrics, LLMbased evaluation, and human annotation.Our results demonstrate the value of high-quality Reddit data, with even state-of-the-art LLMs improving on producing simpler and more specific responses.