Seminar: Dr Bryan Maddox 'The rise of the machines? What place for fairness and validity in AI based educational assessments?'

Dr Bryan Maddox, University of East Anglia

12-1.30pm, 10th February, Paterson's Land rm 1.18

Link to book

Abstract

Educational assessments have made a technological leap from paper to digital tests delivered on computers, tablets and wearable devices.  Any initial concern with mode equivalence (i.e. ‘computers can do assessment as well as humans’) has been replaced by the ambitious agenda of next generation assessments with new types of data, new assessment domains, and the application of artificial intelligence and machine learning (i.e. ‘computers have changed what we think of as assessment’).  

The use of AI in assessment involves three forms of innovation: Adaptive design (the ability of computers to deliver dynamic, differentiated assessment content tailored to the abilities of each respondent); Automation (real-time marking and selection of assessment content, including test item generation); and Expansion (the use of multi-modal process data as performance data). The expansion of digital data creates the most profound change to what we understand as educational assessment. Test validity was previously concerned with the interpretation and use of test scores (Kane, 2016).  The use of process data involves information captured by sensors, including keystrokes, response times, physiological and behavioural data such as heart rate, and affective and communicative data on speech, emotion and gaze. These are not ‘test scores’ in the conventional sense, but they are increasingly treated as performance data.

In this paper I present case studies on the use of automated recognition of emotion, gesture, speech and gaze in educational assessments. Following the work of Kane (2016), I propose that the use of process data must be accompanied by theoretical and empirical understanding and arguments about their appropriate use, interpretation and consequences.  There are other challenges. The use of process data in AI based assessments should recognise the diversity that is present in all human populations to avoid unintended negative consequences. It requires the kind of accountability to test takers and end users, that we associate with published mark schemes. That may lead to performative responses by participants who want to game the AI system, and ontological changes in how we recognise ourselves and others. 

 

Biography

Bryan Maddox is Associate Professor in Educational Assessment at the University of East Anglia and Executive Director of Assessment Micro-Analytics Ltd (a university spin-out company). He specialises in small-scale observational studies of interaction in assessment including work on gesture, talk and gaze in testing situations.  He has conducted assessment research in Nepal, Mongolia, Senegal, Slovenia, France, the United States and the UK. His edited book on ‘International Large-Scale Assessments in Education’ is published by Bloomsbury (2018).

Date of Event
Event Leader
Ben Williamson
Location
Paterson's Land rm 1.18
Research Area
Data Society