AI Survey Network WebinarWe are pleased to invite you to the next AI Survey Network webinar, taking place on Thursday 22 October 2026, 13:00–14:00, via Microsoft Teams. The webinar is organised as part of our UKRI-funded research on digital skills development.
At this meeting we will also launch the new ESRA AI Special Interest Group, with a brief introduction to its aims and information on how colleagues can get involved. This group is organised by Liam Wright (University College London), David Bann (University College London), Paulo Serodio (University of Essex), and Gabi Durrant (University of Southampton).
Dr Laone Maphane, Lecturer in Research Methods and AI Skills at the University of Southampton, will present joint work with Professor Gabriele Durrant from the University of Southampton and the National Centre for Research Methods (NCRM). The presentation is titled:
Evaluating Large Language Models (LLMs) for Coding Open-Ended Survey Responses: A Total Survey Error Framework
The abstract is below.
There will also be time for discussion on the wider developments in the use of AI in survey research.
Please feel free to share this invitation with colleagues who may be interested. We hope you will be able to join us.
Best wishes,
Gabi, Liam and David
Abstract
Open-ended questions are commonly used in survey research to supplement closed-ended questions by capturing respondents’ views, reasons and priorities in their own words. However, coding these responses for analysis remains time-consuming and difficult to standardise. Recent advances in automated coding, especially through large language models (LLMs), have renewed interest in using open-ended responses at scale, but their evaluation is often framed mainly in terms of classification accuracy and agreement with human coding. This presentation develops a Total Survey Error framework for evaluating automated coding of open-ended survey responses as part of survey measurement and data production. Model outputs are assessed using existing human-coded data as a high-quality operational benchmark, with the evaluation examining six complementary dimensions, including processing agreement, measurement validity, prevalence distortion, coding density, response-form sensitivity and operational reliability. The framework is applied using open-ended survey responses from the National Centre for Social Research (NatCen). The presentation aims to inform how automated coding systems, including LLMs, can be assessed in practice, what issues survey researchers should consider when evaluating their suitability for open-ended survey coding, and how improved coding may enable wider use of open-ended questions in large-scale quantitative surveys.