Universal psychometrics: Measuring cognitive abilities
in the machine kingdom
Cognitive Systems Research, 27 (0):
50--74 (2014)DOI: http://dx.doi.org/10.1016/j.cogsys.2013.06.001
Abstract
Abstract We present and develop the notion of
‘universal psychometrics’ as a subject of study,
and eventually a discipline, that focusses on the
measurement of cognitive abilities for the machine
kingdom, which comprises any (cognitive) system,
individual or collective, either artificial, biological
or hybrid. Universal psychometrics can be built, of
course, upon the experience, techniques and
methodologies from (human) psychometrics, comparative
cognition and related areas. Conversely, the
perspective and techniques which are being developed in
the area of machine intelligence measurement using
(algorithmic) information theory can be of much broader
applicability and implication outside artificial
intelligence. This general approach to universal
psychometrics spurs the re-understanding of most (if
not all) of the big issues about the measurement of
cognitive abilities, and creates a new foundation for
(re)defining and mathematically formalising the concept
of cognitive task, evaluable subject, interface, task
choice, difficulty, agent response curves, etc. We
introduce the notion of a universal cognitive test and
discuss whether (and when) it may be necessary for
exploring the machine kingdom. On the issue of
intelligence and very general abilities, we also get
some results and connections with the related notions
of no-free-lunch theorems and universal priors.
