Article by Patrick Sturgis, NCRM, University of Southampton (MethodsNews Spring 2015)
As interviewing costs continue to rise while response rates decline throughout the world, pressure is mounting for large-scale population surveys to be implemented in more cost-effective ways.
Chief among the list of mooted possibilities for achieving significant cost efficiencies is administration of surveys on the web. Web surveys offer huge potential for collecting information about populations at low cost, with rapid turnaround speeds, and with the ability to harness all the ICT functionality and flexibility of the internet. Thus, in addition to reducing the cost-burden of high quality population inference, the web offers the potential not only to reduce costs but to transform the ways in which social surveys are undertaken and, as a consequence, the sorts of questions that it is possible for researchers and policy-makers to address.
There has been huge growth in recent years in the number of ‘opt-in’ web panels that are widely used for market research and opinion polling. Opt-in panels do not use probability sampling methods but recruit panel members using banners, pop-ups, advertising, recruitment websites, and so on. Commonly, quota sampling and weighting are used to match the recruited sample to known population totals for key demographic categories.
Although there is no agreed-upon statistical theory to underpin claims that these opt-in samples are representative of the general population, they have been shown to accurately predict election outcomes and TV popularity contests. Model-based calibration adjustments and sample-matching methods appear to show promise for enabling opt-in panels to closely mirror population totals on a range of social and economic outcomes. However, serious doubts remain about the suitability of opt-in panels for use in academic research and official statistics.
Surveys using probability sampling methods have, to date, made limited use of web data collection. Web data collection is particularly problematic for cross-sectional surveys because a cost-effective way of selecting, contacting and persuading people to go online to complete questionnaires has yet to be adequately developed in the UK. The dependency on the Postcode Address File (PAF) for selecting a probability sample of the general population in the UK means that it is necessary to use either postal or in-person visits in order to make initial contact. Postal contact at sampled PAF addresses produces low response rates and, therefore, increases the risk of self-selection bias.
There is more opportunity for using web administration in longitudinal surveys because names and email addresses have been collected at an earlier wave of data collection. Most key longitudinal studies in the UK are currently testing and using the web in mixed mode designs, for example Understanding Society, the UK Birth Cohorts and the Longitudinal Study of Young People in England. However, the scope for exploiting the full technological capabilities of the web on existing longitudinal studies is constrained by the requirement to replicate face-to-face modes, in order to maintain data comparability over time.
The potential cost-saving and speed of turnaround advantages of online panels have motivated a number of countries (the Netherlands, France, Norway, Sweden, the USA and Germany) to set up web panels based on probability designs. Some of these studies have been able to achieve good population coverage and response rates.
In these designs, conventional sampling frames and methods are used to select a probability sample of the general population. Substantial effort and resource is then invested in recruiting panel members using traditional modes of contact. Offline households are included in the panel either by providing them with free internet access, or by allowing participation using a different mode, typically self-completion by paper questionnaire. The high costs of the initial recruitment effort are then offset by multiple data collections using web rather than more expensive traditional modes.
Such a web-based probability panel has the potential to be a hugely valuable resource for social scientists in the UK. It would provide a vehicle for developing and testing new methods and procedures that could be applied to surveys more generally, both in the UK and internationally, as well as serving as a key resource for undertaking high quality substantive research.
The ESRC has commissioned a team from the Centre for Longitudinal Studies (CLS) at the UCL Institute of Education (UCL IoE) and TNS-BMRB to undertake a scoping study of potential designs and costs for a probability-based web panel in the UK. The findings of the study will inform the ESRC’s decision on whether to make the large scale investment needed to establish and maintain such a panel.
The team has recently been hosting a consultative workshop of stakeholders and interested parties to gauge the potential interest in a web-based probability panel from the user community in the UK, to elaborate possible research opportunities such a panel may provide or stimulate, and to gain concrete input into the development of the potential design options.
For more information on the wider consultation and how you might contribute, go to www.cls.ioe.ac.uk/ukwebpanel