Mean Average Distance to Resolver: An Evaluation Metric for Ticket Routing in Expert Network
Jianglei Han, Aixin Sun · 2017
In the technical support division of a large enterprise software provider, customers' technical incidents, problems, and change requests are processed as tickets. Each ticket is assigned to a support engineer for processing. Due to the limited expertise of individuals, resolving a ticket may involve routing the ticket among multiple groups of engineers. Each routing step costs time and resources. It is desirable for experts to route a ticket to its most likely resolver with minimum steps. Automated or semi-automated systems are proposed to improve routing efficiency. To evaluate the performance of any system, including human routing, two metrics are commonly used, namely Mean Steps to Resolver (MSTR) and Resolution Rate (RR). The two measures are designed independently, with different objectives and at different scales, making it difficult to compare systems. Moreover, the current measures only consider the resolver group as the ground truth, even during path-level evaluation. They disregard the contribution of intermediate groups during the ticket resolution. In this paper, we propose a distance-based unified evaluation measure named Mean Average Distance to Resolver (MADR). This new framework addresses the aforementioned limitations, and it can be easily modified to adapt to different business requirements in different organizations. In addition, existing evaluation paradigm does not consider human routing steps except the resolver. We argue that the predicted paths may not be followed exactly by expert groups in real operation. An assistive routing evaluation framework is therefore designed to take into account expert's choice when recommendation fails, for each routing. Experiments using proprietary data from a large enterprise demonstrate that MADR can be used to benchmark and compare routing systems.