In February this year the European Union Agency for Fundamental Rights (FRA) hosted a meeting with survey research experts from different countries and backgrounds. The purpose was to discuss the possibility of developing a new EU-wide survey on fundamental rights and discrimination using a mixed mode approach.
Mixed mode surveys combine different ways of interviewing people, such as face-to-face, telephone or web. These can be combined concurrently, thus giving the respondent the possibility of choosing their preferred mode, or sequentially. The latter approach is the most popular in practice. The typical design starts with a cheaper mode, such as web or mail, followed by interview attempts with non-respondents using a more expensive mode, such as face-to-face. By using mixed modes in this way, researchers typically hope to achieve cost savings while maintaining the quality of the survey. An indication of the increased interest in mixed mode designs both in academia and among survey practitioners is the large number of presentations and sessions on this topic at the most recent European Survey Research Association conference, in Reykjavik, and the experiments carried out by the European Social Survey and Understanding Society.
Although mixed mode surveys have the potential to save costs, numerous questions remain. Is the data quality really the same as in a single mode approach? How much money does it save? In what conditions can it lead to biased estimates? These issues become even more complicated in multi-country surveys, where differences in the implementation of the mixed mode design can introduce bias to country comparisons. As such, survey practitioners have to consider a number of different aspects when implementing mixed mode designs across countries.
Firstly, the modes used should be as similar as possible. For example, previous research has shown that people tend to answer questions more honestly in self-administered modes, such as web or mail surveys, than in interviewer-administered modes, such as face to face or telephone. This means that implementing a sequential web and face-to-face design may be problematic as answers might be different between people due to the mode in which they answer. This issue becomes even more complex in a cross-cultural context. For example, if the percentage of people answering by web is very different in the UK and Romania, then comparison between them on the questions of interest can be biased.
These differences also stem from a second challenge, which relates to implementing a survey design in a similar way across multiple countries. The way in which the design is applied can be heavily influenced by the data collection agency and by the common practices in each country. An example is the availability of sampling frames. Here, countries vary considerably in the type of frames available, from individual level registries to postal address frames or to the absence of such frames entirely, for example in Greece or Portugal. This will influence the possibility of using certain modes. For example, if no household or individual registry is available then people cannot be recruited to the survey using invitation letters without a separate enumeration stage which adds to the costs of the survey. Additionally, the quality of sampling frames can differ significantly leading to country differences in coverage, even when the same design is implemented.
At the meeting in Vienna, the use of web surveys received considerable attention due to some of their apparent strengths, such as lower costs and lower social desirability effects. Also, internet penetration has steadily increased in the EU Member States, and is expected to continue improving in the foreseeable future. Nevertheless, a number of limitations are also obvious, such as the lack of a sampling frame that could be used to directly recruit respondents to an online survey as well as the exclusion of people who do not have access to the internet. Here, once again, using a mixed mode approach could tackle some of the issues. For example, mail invitations can be used for the web survey. Additionally, non-respondents can be interviewed using an alternative mode, such as face-to-face. Mode effects related to combining web data collection with face-to-face interviews could be partially addressed by including a self-completion component as a part of the in-person interview, which is more similar to the web survey questionnaire.
So one might ask, what was the conclusion of the meeting in Vienna? To mix or not to mix? At this point there is no clear answer, with different countries having variously applied mixed modes. The implications of using mixed modes must be looked at with respect to both their advantages and possible disadvantages. Here, it has to be acknowledged that many EU-wide surveys – on which a proportion of Eurostat data is based – currently employ different modes and sampling frames for their data collection across countries, which is not always critically reviewed. In this case FRA has the advantage of considerable flexibility in developing a new survey and the possibility of centrally managing it. Their experience with implementing a mixed mode survey across Europe will bring important insights in this field.
Notes
The views expressed in this article are solely those of the authors and its content does not necessarily represent the views or position of the European Union Agency for Fundamental Rights.