DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents
Varun Nair, Elliot Schumacher, Kwok Fai Tso, Anitha Kannan · 2024
Large language models (LLMs) have emerged as valuable tools for many natural language understanding tasks.In safety-critical applications such as healthcare, the utility of these models is governed by their ability to generate factually accurate and complete outputs.In this work, we present dialog-enabled resolving agents (DERA).DERA is a paradigm made possible by the increased conversational abilities of LLMs.It provides a simple, interpretable forum for models to communicate feedback and iteratively improve output.We frame our dialog as a discussion between two agent types -a Researcher, who processes information and identifies crucial problem components, and a Decider, who has the autonomy to integrate the Researcher's information and makes judgments on the final output.We test DERA against three clinically-focused tasks, with GPT-4 serving as our LLM.DERA shows significant improvement over the base GPT-4 performance in both human expert preference evaluations and quantitative metrics for medical conversation summarization and care plan generation.In a new finding, we also show that GPT-4's performance (70%) on an open-ended version of the MedQA questionanswering (QA) dataset ( Jin et al. (2021), USMLE) is well above the passing level (60%), with DERA showing similar performance.We will release the open-ended MedQA dataset.