Evaluating Robustness of Open Dialogue Summarization Models in the Presence of Naturally Occurring Variations
Ankita Gupta, Chulaka Gunasekara, Hui Wan, Jatin Ganhotra, Sachindra Joshi, Marina Danilevsky · 2024
Dialogue summarization involves summarizing long conversations while preserving the most salient information.Real-life dialogues often involve naturally occurring variations (e.g., repetitions, hesitations), and in this study, we systematically investigate the impact of such variations on state-of-the-art open dialogue summarization models whose details are publicly known (e.g., architectures, weights, and training corpora).To simulate real-life variations, we introduce two types of perturbations: utterance-level perturbations that modify individual utterances with errors and language variations, and dialogue-level perturbations that add non-informative exchanges (e.g., repetitions, greetings).We perform our analysis along three dimensions of robustness: consistency, saliency, and faithfulness, which aim to capture different aspects of performance of a summarization model.We find that both finetuned and instruction-tuned models are affected by input variations, with the latter being more susceptible, particularly to dialogue-level perturbations.We also validate our findings via human evaluation.Finally, we investigate whether the robustness of fine-tuned models can be improved by training them with a fraction of perturbed data and find that this approach does not yield consistent performance gains, warranting further research.Overall, our work highlights robustness challenges in current open models and provides insights for future research. * Work done during an internship at IBM ResearchAny news on what happened to the 9.13am train?It was delayed, now it appears to have disappeared.Hi there.There were animals on the line so the train had to bypass the station at a reduce speed.[....] How do I formally complain?Sorry for the inconvenience.We always try to impact as few customers as possible.Make a complaint by emailing __ email__ sorry, couldn't hear you, can you repeat?Sure, we try to impact as few customers as possible and you can make a complaint by emailing __ email__ Request to repeatCustomer is complaining about the delay in the train.Agent states that there were animals on the line and train had to bypass at reduced speed.Customer is complaining about the delay in a train.Agent states that they always try to impact as few customers as possible and requests to make a complaint by emailing Summary of the perturbed dialogue Summary of the original dialogue Summarize Summarize