Performance Evaluation of Intelligent Systems at the National Institute of Standards and Technology (NIST)
Craig I. Schlenoff, Harry A. Scott, Stephen B. Balakirsky · 2011
The ability to comprehensively evaluate the quantitative and qualitative performance of anintelligent system is critical to accurately predicting how it will perform in various situations.The design of such evaluations is often as much of a research challenge as is the design of theintelligent systems themselves. Over the past decade, the Intelligent Systems Division, a part ofthe National Institute of Standards and Technology, has been at the forefront of assessing theperformance of the various intelligent systems. This article gives a broad overview of some of theevaluation efforts that have been pursued by the Intelligent Systems Division over the past fewyears, including performance evaluation of emergency response robots, sensor systems onunmanned ground vehicles, speech-to-speech translation systems, and the development ofperformance metrics for mixed-palletizing through the use of a simulation environment.Key words: Design of evaluations; emergency response robots; intelligent systems; mixedpalletizing, testing, simulation; robotic vehicle perception systems; search and rescuerobots; speech translation systems.