When Aggregate Accuracy Is Not Enough: Decision Impact, Validation Governance, and a Causal Evaluation Blueprint for Financial Complaint Triage
Charles Ancheta · Zenodo (CERN European Organization for Nuclear Research) · 2026
This validation-only study examines financial complaint triage as both a machine-learning and governance problem. It compares TF-IDF logistic regression with a compact MiniLM classifier under temporal separation and duplicate isolation, then evaluates calibration and confidence-based abstention. Although MiniLM improves class-balanced performance, no tested policy satisfies every required route safeguard, resulting in a manual-review-only recommendation. The paper also presents a prospective randomized evaluation blueprint for measuring whether AI assistance improves reviewer accuracy and handling time without harming important complaint categories. No deployment benefit or causal effect is claimed.