Matching and weighting for quasi-experimental policy evaluation: a primer (online)
Date:
28/06/2023
Organised by:
NatCen Social Research
Presenter:
Dr Andi Fugard
Level:
Intermediate (some prior knowledge)
Contact:
Emily Adams
emily.adams@natcen.ac.uk
Venue: Online
Description:
Policymakers need to know whether social programmes and policies make a difference, and often when randomised experiments are unethical or unfeasible. Causal matching and weighting, types of quasi-experiments, can satisfy this need. Using a range of examples, we will begin by exploring intuitive understandings of how matching and weighting methods work and progress towards a gentle introduction to key concepts that underpin them. A central message will be that a robust theory from the substantive topic area is an essential ingredient for quasi-experiments to deliver the conclusions that policymakers demand. By the end of the session, participants will be empowered to tackle the technical literature and learn more.
This half-day (09:00 - 12:30) online course covers:
- The case for programme and policy evaluation – why evaluation matters and what it can offer policymakers.
- A running example: estimating the average causal effect of an intervention when there is one comparison group and data from two time points: baseline and endpoint.
- What a quasi-experiment is trying to estimate, couched in terms of the potential outcomes framework, and what you lose when you can’t run an RCT.
- Exact matching, why it usually fails, and how to fix it through coarsening.
- The idea of a multivariate distance and how distance metrics (Euclidean and Mahalanobis) can be used for matching.
- The idea of a propensity score and how it can be used for matching and weighting.
- Common themes when assessing matching/weighting analyses, regardless of the approach used.
By the end of the course participants will be able to:
- Understand where quasi-experiments fit into the broader landscape of evaluations.
- Make a compelling case for running quasi-experiments and how they differ from RCTs.
- Explain the potential outcomes framework and how it applies to quasi-experiments.
- Understand what a causal estimand is and be able to choose an appropriate estimand for a given evaluation question.
- Understand the steps of estimating propensity scores and common ways to use them for matching and weighting.
- Understand a selection of approaches of matching directly on covariates (coarsened exact matching and distance-based matching).
- Appraise the quality of key elements of a quasi-experimental analysis that each of the methods introduced have in common.
- Engage with literature in the field with more confidence.
It is assumed that participants have a firm grasp of the foundation of quantitative methods used in social science, particularly linear and logistic regression and confidence intervals. It will also facilitate the course if participants have a specific idea for a quasi-experiment they would like to run.
Dr Andi Fugard (they/them) is Deputy Director of NatCen’s Centre for Evaluation. They have experience designing, project managing, and analysing data from quasi-experimental and randomised controlled trials across a range of policy areas, including mental health and education. Before joining NatCen, Andi was a Senior Lecturer at Birkbeck, University of London and Lecturer at University College London, in both roles teaching statistics for social science. They are an Associate Fellow of the Higher Education Academy and a member of the UK government Evaluation and Trial Advice Panel, which advises civil servants and What Works Centres.
Cost:
The fee per teaching day is £30 per day for students / £60 per day for staff working for academic institutions, Research Councils and other recognised research institutions, registered charity organisations and the public sector / £100 per day for all other participants. In the event of cancellation by the delegate a full refund of the course fee is available up to two weeks prior to the course. NO refunds are available after this date. If it is no longer possible to run a course due to circumstances beyond its control, NCRM reserves the right to cancel the course at its sole discretion at any time prior to the event. In this event every effort will be made to reschedule the course. If this is not possible or the new date is inconvenient a full refund of the course fee will be given. NCRM shall not be liable for any costs, losses or expenses that may be incurred as a result of its cancellation of a course, including but not limited to any travel or accommodation costs. The University of Southampton’s Online Store T&Cs also continue to apply.
Website and registration:
Region:
Greater London
Keywords:
Explanatory Research and Causal analysis, Experimental Research , Quasi-Experimental Research, Evaluation Research, Behavioural Research, Hypothesis testing research, Regression Methods, Evidence-Based Policy and Practice
Related publications and presentations:
Explanatory Research and Causal analysis
Experimental Research
Quasi-Experimental Research
Evaluation Research
Behavioural Research
Hypothesis testing research
Regression Methods
Evidence-Based Policy and Practice