Professor George Leckie and UCAS: NCRM Impact Prize entry
This NCRM Impact Prize application describes ongoing two-way knowledge exchange between the Universities and Colleges Admissions Service (UCAS) and Professor George Leckie of the University of Bristol.
As a result of this collaboration, UCAS has improved its knowledge of student behaviour, benefiting the 1.5 million students who use UCAS services, such as applying for places at university.
Summary of NCRM participation
The knowledge exchange between the two parties arose after data scientists from UCAS attended two NCRM training courses, Introduction to Multilevel Modelling Using MLwiN, R or Stata, which were run by Professor Leckie.
The three-day course includes lectures, software demos, and practical sessions. The first course, in January 2023, was attended by three data scientists, while the second, in July 2023, was attended by two data scientists.
Background
The Universities and Colleges Admissions Service (UCAS) is an independent charity, and the UK's shared admissions service for higher education. The UCAS Modelling and Research team (five data scientists) conduct a continuous programme of sophisticated applied quantitative research using UCAS’s admissions data on student choices and outcomes. The team has several projects ongoing at any one time. The outcomes of these projects feed into UCAS decision and policy recommendations and HE admissions reform.
A fundamental feature of the UCAS admissions data is its complex multilevel data structure. Each student potentially applies to many degree courses at different universities through the UCAS main scheme. Students makes these choices within the context of their schools and local authorities. Courses are nested within universities. The data also contain student, school, course, and university characteristics. There are also multiple cohorts of data. Thus, there are many levels at which the data can be analysed and student application choices and outcomes vary across schools and local authorities. The team therefore routinely apply multilevel modelling to analyse these complex data.
Attending NCRM training
In January 2023, the UCAS team decided to attend the NCRM three-day multilevel modelling short course run by Leckie, to enhance the knowledge of newer team members and generally identify any additional approaches for their ‘modelling toolkit’. The training covered number of more advanced areas of multilevel modelling including options available for fixed-effects vs. random-effects models modelling within, between, and contextual effects in multilevel analysis.
UCAS team members found the training informative and thorough. They especially appreciated the broad range of examples presented– including examples from education - and the presenters’ enthusiasm for the subject matter throughout, and readiness to share their own experiences. They also commented very positively on the organisation of the course by Lucy Haslam from the Centre for Multilevel Modelling. Additionally, they communicated their intention to send any further new team members on the course as part of their initial training.
Ongoing knowledge exchange
Immediately following the course, UCAS contacted Leckie to see whether he might provide feedback on their use of more advanced multilevel modelling to a number of specific complex challenges of their data in their various projects. Leckie was very happy to do so, especially as part of his current ESRC Secondary Data Analysis Initiative Grant he is analysing UCAS data.
Thus, what has arisen from the initial NCRM training is an ongoing two-way knowledge exchange whereby Leckie is used as an independent reviewer of UCAS’ advanced multilevel modelling and other modelling outputs, providing valuable feedback that has informed subsequent development, while UCAS supports Leckie’s understanding of UCAS data and HE admissions and UCAS advises Leckie on substantive and policy insights and implications for his research. A fourth member of the UCAS team and another UCAS data scientist have now attended the July 2023 running of the same short course.
In terms of knowledge exchange from Leckie to UCAS: Leckie has given written comments on multilevel methodology in three different UCAS projects. UCAS and Leckie have had six MS Teams meetings to discuss this work.
In terms of knowledge exchange from UCAS to Leckie: UCAS have given Leckie written feedback on his submitted journal article. UCAS also invited Leckie to present this work to senior UCAS staff more generally including several heads of section and area leads. Valuable feedback and suggestions to improve the work were provided which has led Leckie to revise and improve his now submitted journal article.
UCAS also recorded this presentation for those UCAS staff who could not attend as well as circulating the draft journal article. UCAS found several of the visualisations and changes to effects with incorporation of additional explanatory variables that Leckie presented very accessible and informative ways to present data and models and UCAS have started to use these visualisations in their research reports.
UCAS and Leckie have agreed to continue their regular two-way knowledge exchange meetings over the autumn. Crucially, this impactful relationship for both UCAS and Leckie only arose as a result of the initial NCRM training events.
Impact achieved
The resulting upskilling of both parties has built our knowledge and understanding of student behaviour, which ultimately benefits the nearly 1.5m students that use UCAS’ services.