The BICA Cognitive Decathlon: A Test Suite for Biologically-Inspired Cognitive Agents
Shane T. Mueller, Matt Jones, Brandon S. Minnery, Julia M. H. Hiland · 2007
ABSTRACT: BICA (Biologically-Inspired Cognitive Architectures) is a DARPA Phase-I program whose goal is to create the next generation of cognitive architecture models based on principles of psychology and neuroscience. This project is motivated by the belief that traditional artificial intelligence research has hit a wall in its quest to develop truly intelligent agents: although agents can be engineered to perform exceedingly well at specific tasks, they are typically quite brittle, unable to deal with unforeseen situations and unable to learn from others. This paper describes the BICA Cognitive Decathlon, Challenge Scenarios, and Biovalidity Assessment, a set of tests we designed to evaluate the performance of such agents in a variety of situations that cover a core set of cognitive, perceptual, and motor skills typical for a two-year-old human child. These include behavioral tasks related to search, navigation, manipulation, memory, language, and three pathways to procedural knowledge: instruction, demonstration, and reinforcement/exploration. The test suite has three distinct components: a set of four integrative challenge scenarios that support the goals of building coherent, systematic, integrated cognitive agents; a set of focused tasks that can better determine the extent to which the core cognitive competencies match the capabilities of humans; and a set of biovalidity assessments to determine the extent to which the agents architecture resembles the human brain. Ultimately, this three-level set of tests was designed to evaluate whether systems are flexible, comprehensive, and taskable in complex situations, while still performing tasks in ways similar to human performers. The test