Over the last couple of years, NCRM has worked with TNS BMRB to derive a general model of the impact of fieldwork effort on those total population estimates that are drawn from face-to-face random sample surveys. On the whole, the team has found only modest effects, suggesting that much of the expense entailed from repeatedly visiting initially unproductive addresses is unnecessary1.
However, modest effects on total population estimates may hide larger effects on subpopulation estimates. Subpopulations will often be more homogeneous than the total population with respect to a target variable but that does not mean that the impact of fieldwork effort must also be smaller. In particular, (i) the correlation between response propensity and measured characteristics may be greater for a subpopulation than for the total population, (ii) the variance of response propensity within a subpopulation may be greater than for the total population, and (iii) some subpopulations have lower than average response propensities. Any of these factors may mean that fieldwork effort has a larger effect on some subpopulation estimates than it has on the total population estimate. It is usually very difficult to estimate the impact of fieldwork effort on subpopulation estimates because any systematic effects are confounded with substantial random sampling error. The Crime Survey of England & Wales (CSEW) is an exception: 35,000 interviews per year is large enough to separate out the effect of fieldwork effort for plenty of subpopulations, albeit recognising that the generalisability of any findings is limited due to topic specificity.
After a fall in the CSEW response rate from c74% to c70% in 2014-15, the Office for National Statistics (ONS) asked TNS BMRB to explore the impact of a lower response rate on the headline statistics that are published. To do this, we used data from 2012-14 and stripped out the interviews obtained after re-issuing initially unproductive addresses. This transformed the response rate from 74% to 66% and allowed us to obtain survey estimates that reflected a lower than usual level of fieldwork effort. Of course, putting in less fieldwork effort is not the same as what happened in 2014-15 when TNS BMRB put in the same amount of effort but obtained a lower response rate! However, it is reasonable to assume that less effort or equal effort but lower success would result in a similar responding sample.
Before describing the results, it is worth noting that reissuing initially unproductive addresses is a disproportionately costly element of fieldwork. Per-interview pay rates are very high, reflecting the difficulty of getting these interviews. While interviews at the original issue stage are obtained after an average of 3-4 visits to the address, interviews obtained at the reissue stage are obtained after an average of 15-20 visits in total. It is reasonable to question the proportionality of this activity even without addressing the impact on the survey estimates themselves. So why are initially unproductive addresses reissued? The most accurate answer is that it allows fieldwork agencies to meet contractual response targets and avoid financial penalties. Naturally, the cost of this work is passed on to their clients which means – for government research - the taxpayer foots the bill. Consequently, the value of all this extra work needs to be obvious.
After discussion with ONS, we identified three variables defining subpopulations. These were selected because of the apparent variability in response rates between each sub-population. The variables were (i) age group, (ii) ACORN category (a five-category postcode segmentation based on multiple sources), and (iii) housing tenure. ONS wanted us to look at all the key published estimates: a mixture of (i) prevalence/incidence of crimes, (ii) behaviours, and (iii) reported attitudes. In total, there were 77 variables (across 37 questions) and, while not a random selection, there are variables from most of the ‘ask all’ modules within the adult questionnaire.
To obtain subpopulation estimates before and after the reissue stage, we post-stratified the sample each time as would be ONS standard practice. We also standardised the differences between the estimates before and after reissuing, allowing us to summarise across groups with different sample sizes and across variables with different measurement properties. Prior work demonstrated that these standardised differences – t scores - should broadly follow the theoretical t-distribution if the reissue stage made no systematic difference to the estimates. In summary, the limited impact of fieldwork effort suggested by the general model appears to also be true for the subpopulations assessed for this study and for this survey. It is unclear how transportable these findings are to other surveys and other subpopulations but they are unlikely to be unique.
Assuming these findings are generalisable to some degree, it is hard to argue on statistical grounds for committing funds to re-issuing initially unproductive cases. However, research commissioners like high response rates for non-statistical reasons too: they provide public credibility, an intangible that is worth a lot to them. Nevertheless, even while accepting this is important, it seems to us that targeting specific response rates ends up larding surveys with cost and puts them entirely out of reach of many research buyers. Paradoxically, a less macho approach to response maximisation might ultimately protect the random sample method and perhaps should be more vigorously promoted by both statisticians and the industry at large.
Notes:
1 See http://eprints.ncrm.ac.uk/3771/ for a working paper describing this project in detail.