Which analytical choices are better? – Comparing Traditional Statistical Methods with Machine Learning and AI-supported methods
Date:
05/03/2026
Organised by:
NCRM, University of Southampton
Presenter:
Dr Christian Bokhove and Dr Hayward Godwin
Level:
Entry (no or almost no prior knowledge)
Contact:
Jacqui Thorp
Training and Capacity Building Coordinator, National Centre for Research Methods, University of Southampton
Email: jmh6@soton.ac.uk
Location:
View in Google Maps (SO17 1BJ)
Venue:
Building 54, Room 4001, University of Southampton, Highfield, Hants
Description:
This course is one of a series of four. You may register for any number of sessions individually. If you choose to register for all four, a discount will be applied. Further information about the series can be found at the end of this listing.
Session Four - Which analytical choices are better? – Comparing Traditional Statistical Methods with Machine Learning and AI-supported methods
Using different analysis methods even on the same dataset can sometimes lead to different conclusions — not by coincidence, but as a common reality in secondary data analysis across the social sciences.
This session looks at how researchers’ degrees of freedom create specification uncertainty — the idea that different, yet equally reasonable, model choices can yield divergent conclusions. Such uncertainty challenges the replicability and robustness of social science research and directly affects the credibility and interpretability of policy evidence.
Through the frameworks of Multiverse Analysis and Specification Curves, this session will explore how traditional selective modelling can evolve into a systematic and transparent exploration of analytical alternatives. The session will give an introduction to Multiverse Analysis and Specification Curves.
The session also demonstrates how AI-supported approaches can enhance the efficiency and scalability of model building and data analysis. Ultimately, we will ask: Can research findings become less dependent on what we choose — and instead reveal stable, choice-insensitive truths?
We highlight this challenge using secondary data, including ILSA (International Large-Scale Assessment) data, whose complexity — including multi-stage sampling, weight structures, plausible values, hierarchical designs, and multi-country contexts — grants researchers considerable analytical flexibility. Every decision, from handling missing data and standardisation to choosing control variables and model specifications, however, can substantially alter the results.
When faced with this, how would you, as a researcher, explain such discrepancies? And if you were a policy analyst, which conclusion would you trust?
By the end of this session, participants will be able to:
- Explain the concept of specification uncertainty and its implications for reproducibility and robustness.
- Apply the logic of multiverse analysis and specification curves to identify stable and choice-insensitive results.
- Compare traditional statistical approaches with AI-driven model exploration techniques.
- Critically reflect on how model selection shapes interpretation and the trustworthiness of research evidence.
This course is aimed at researchers, graduate students, data analysts, and education professionals interested in quantitative reasoning, data-driven inquiry, and evidence-based research. No prior experience with statistical software or large-scale assessment data is required.
Delivery
This course is being delivered in a hybrid format on Thursday 5th March from 09:00-11:00:
In person - Room 54/4001 (limited capacity, offered on first-come first-served basis) or Online.
Series details:
Session One - £25 - https://www.ncrm.ac.uk/training/show.php?article=14610
Session Two – £50 - https://www.ncrm.ac.uk/training/show.php?article=14611
Session Three – £50 - https://www.ncrm.ac.uk/training/show.php?article=14612
Special offer: Register for all four sessions for £120
Cost:
The fee for this session is:
• £25 per person for all participants.
In the event of cancellation by the delegate a full refund of the course fee is available up to two weeks prior to the course. NO refunds are available after this date.
If it is no longer possible to run a course due to circumstances beyond its control, NCRM reserves the right to cancel the course at its sole discretion at any time prior to the event. In this event every effort will be made to reschedule the course. If this is not possible or the new date is inconvenient a full refund of the course fee will be given. NCRM shall not be liable for any costs, losses or expenses that may be incurred as a result of its cancellation of a course, including but not limited to any travel or accommodation costs.
The University of Southampton’s Online Store T&Cs also continue to apply.
Website and registration:
Region:
South East
Keywords:
Quantitative Data Handling and Data Analysis, Mixed Methods Data Handling and Data Analysis, AI and machine learning, International large-scale assessment data. Specification Uncertainty, Researcher Degrees of Freedom, Multiverse Analysis, Specification Curve, Artificial Intelligence (AI), Robustness and Reproducibility, Model Transparency, Evidence-Based Research, Cross-Sectional Research, Secondary Analysis
Related publications and presentations from our eprints archive:
Quantitative Data Handling and Data Analysis
Mixed Methods Data Handling and Data Analysis
AI and machine learning