Did You Mean...? Confidence-based Trade-offs in Semantic Parsing
Elias Stengel-Eskin, Benjamin Van Durme · 2023
We illustrate how a calibrated model can help balance common trade-offs in task-oriented parsing.In a simulated annotator-in-the-loop experiment, we show that well-calibrated confidence scores allow us to balance cost with annotator load, improving accuracy with a small number of interactions.We then examine how confidence scores can help optimize the tradeoff between usability and safety.We show that confidence-based thresholding can substantially reduce the number of incorrect lowconfidence programs executed; however, this comes at a cost to usability.We propose the DidYouMean system (cf.Fig. 1) which better balances usability and safety by rephrasing low-confidence inputs.