Caveats and Solutions for Characterising General-Purpose AI
José Hernández‐Orallo · Frontiers in artificial intelligence and applications · 2024
The concept of General-Purpose AI (GPAI) has recently been permeating research papers, policy reports and legal regulations, as a way of referring to current and future models with high levels of capability and generality. Yet precisely characterising GPAI models remains elusive. Current definitions often describe GPAI models as those that ‘competently perform a wide range of distinct tasks’. To properly characterise GPAI we need well-grounded definitions of capability and generality. In this paper, I will briefly introduce –or revisit– the concept of capability, going well beyond aggregate performance on benchmarks, and discuss practical procedures to evaluate the capability profile of AI systems, and derive generality metrics from them.