ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments
Gili Lior, Eliya Habba, Shahar Levy, Avi Caciularu, Gabriel Stanovsky · 2025
LLMs are highly sensitive to prompt phrasing, yet standard benchmarks typically report performance using a single prompt, raising concerns about the reliability of such evaluations.In this work, we argue for a stochastic method of moments evaluation over the space of meaningpreserving prompt perturbations.We introduce a formal definition of reliable evaluation that accounts for prompt sensitivity, and suggest RELIABLEEVAL -a method for estimating the number of prompt resamplings needed to obtain meaningful results.Using our framework, we stochastically evaluate five frontier LLMs and find that even top-performing models like GPT-4o and Claude-3.7-Sonnetexhibit substantial prompt sensitivity.Our approach is model-, task-, and metric-agnostic, offering a recipe for meaningful and robust LLM evaluation. 1