PAIR: Prompt-Aware margIn Ranking for Counselor Reflection Scoring in Motivational Interviewing
Do June Min, Verónica Pérez‐Rosas, Ken Resnicow, Rada F. Mihalcea · 2022
Reflections are a core verbal skill used by mental health counselors to express understanding and acknowledgement of the client's experience and concerns.In this paper, we propose a system for the automatic evaluation of counselor reflections.Specifically, our system takes as input one dialog turn containing a client prompt likely leading to a reflection and a counselor response to it, and outputs a numeric score indicating the quality of the reflection made by the counselor.We compile a dataset consisting of reflections portraying different levels of reflective listening skills, and propose Prompt-Aware margIn Ranking (PAIR), a novel framework for reflection scoring that contrasts positive and negative prompt and response pairs using adhoc multi-gap and prompt-aware margin ranking losses.Through empirical evaluations and deployment of our system in a real-life educational environment, we show that our scoring model outperforms several baselines on different metrics, and can be used to provide useful feedback to counseling trainees.