In the months and weeks leading up to the 2015 general election, the opinion polls told a consistent story; the Conservatives and Labour were tied in a dead heat in the popular vote. This led media commentators, party strategists, and the public to focus attention on the likely composition of a coalition, rather than on a single-party government led by the Conservatives who, of course, ultimately won the election with a 6.6% lead over Labour and an absolute majority in the House of Commons. The expectation of a hung parliament in the final days and weeks of the campaign was so strong and widely held that the sense of shock and disbelief was palpable when the result of the exit poll was announced at 10pm on May 7th.
In response to these polling errors, the British Polling Council and the Market Research Society asked me to chair an independent inquiry into what went wrong in 2015. The inquiry has now published its report1, in which it concludes that the primary cause of the polling miss was unrepresentative samples. The methods the pollsters used to collect samples of voters systematically over-represented Labour and under-represented Conservative supporters. The statistical adjustment procedures applied to the raw data did not mitigate this basic problem to any notable degree.
This conclusion was arrived at partly by elimination of other possible causes of the errors. The inquiry was able to exclude the possibility that postal voters, overseas voters, and un-registered voters made any detectable contribution to the polling miss. The ways that pollsters asked respondents about their voting intentions was also eliminated as a possible cause of what went wrong.
There was weak evidence of a very modest late swing to the Conservatives, although this can have contributed – at most – around one percentage point to the error on the Conservative lead. The widely held view that the polling miss was due to deliberate misreporting - ‘shy Tories’ telling pollsters they intended to vote for other parties - is difficult to reconcile with the results of re-contact surveys carried out by the pollsters and with the two random surveys undertaken after the election. Differential turnout by party was also pointed to after the election as a likely cause of the errors; so-called ‘lazy Labour’ supporters telling pollsters they would vote Labour but ultimately not turning out. However, data from a number of sources showed no support for this making anything but a very small contribution to the polling errors.
In addition to ruling out other possibilities, the inquiry found direct evidence that the poll samples were unrepresentative. A key assumption of quota sampling – the procedure used by all pollsters – is that, within each weighting cell, the vote intention should be the same in the sample as it is in the population. The inquiry found that this assumption was violated. The inquiry also compared the polls to ‘gold standard’ probability surveys: the British Election Study and the British Social Attitudes survey. These produced estimates of the Conservative lead that were close to the actual election result. The inquiry was also able to show that the poll samples were unrepresentative in other ways, for example, they under-represented voters aged over 74 and over-represented the more politically engaged.
A diagnosis of unrepresentative samples naturally leads to a desire to identify the cause of the unrepresentativeness. However, the inquiry refrained from advancing a causal ‘story’ to explain exactly how the samples ended up being biased toward Labour and away from the Conservatives. This was partly because the data considered did not support any such conclusion but it was also because the inquiry concluded that bias in the vote intention estimate is unlikely to be accounted for in any simple way.
Take, for example, political engagement which has been pointed to by several commentators as the cause of the problems in the poll samples. The argument here is that the polls substantially over-represent the politically engaged and that this drives the bias in vote intention. The polls certainly over-represent the politically engaged, but how this relates to bias in vote intention is not so straightforward because the relationship between political interest and support for the main parties is not linear. Rather, it is the most and the least politically engaged who tend to support Labour, while people with a moderate level of political interest are most likely to vote Conservative. Thus, over-representing the politically engaged will tend to produce an over-estimate of Labour support but this will be at least partially compensated by under-representation of the least politically interested, who are also most likely to support Labour. And, indeed, the pollsters’ early experiments with weighting their samples by political interest appear, thus far, to make little or no difference to estimates of vote intention.
Despite its limitations, polling remains the most accurate means of predicting election outcomes and is likely to remain so for the foreseeable future. The report makes a number of recommendations for improving the ways that polls are conducted and reported. While these should reduce the risk of the 2015 polling miss recurring in the future, there will be no silver bullet – opinion polls will always be subject to random and systematic errors that are difficult to control. Stakeholders on all sides must be more realistic about the level of accuracy opinion polls are capable of delivering.
References
1 Sturgis, P. et al. (2016) Report of the Inquiry into the 2015 British general election opinion polls.
http://eprints.ncrm.ac.uk/3789/