While the vast majority of survey questions use fixed response options, there is a long tradition in survey research of obtaining less structured data from respondents. These types of ‘verbatim’ responses require respondents to express their thoughts about a given topic ‘in their own words’. Open-ended questions potentially provide richer data than closed-format questions and allow people to frame an issue in their own terms, rather than in those selected in advance by the researcher.
However, in face-to-face surveys, interviewers typically type open-ended responses as the respondent speaks, something which can be difficult to do fully and accurately. The quality of verbatim responses can therefore be quite variable, depending to a large extent on interviewer motivation and typing skills. Recent technological developments mean it is now possible for interviewers to digitally record respondents rather than typing as they speak, offering potentially substantial gains in data quality. In a recent NCRM study, we weighed the costs and benefits of implementing audio-recording of verbatim responses within a CAPI interview1.
We used the 2012 Wellcome Trust Monitor2, which is a random face-to-face survey on public attitudes to biomedical science. Respondents were randomly assigned to one of two experimental groups; in one group, verbatim answers to questions about scientific terms were audio-recorded, while in the other group they were typed by the interviewer in the conventional manner. Because it was necessary for ethical reasons to ask respondents for their consent to be audio-recorded, the design produced three groups: (1) those who gave consent and were allocated to the ‘audio’ group, so had their responses audio-recorded; (2) those who gave consent but were allocated to the ‘typed’ condition, so had their responses typed; and (3) those who did not give consent and regardless of whether they were allocated to the ‘audio’ or ‘typed’ condition, had their responses typed.
A key limitation of audio-recording is that a substantial minority, a third of respondents in this case, do not give consent to be recorded. Moreover, we found that the probability of giving consent was related to respondent characteristics. Those with higher educational qualifications, higher scientific knowledge, and those with a greater willingness to participate in research were the most likely to give consent. Thus, analysis of the raw response data would over-represent these groups relative to the general population.
Three indicators were used to compare the data quality of the responses between groups: the total number of words recorded, the number of ‘thematic’ codes applied by coders, and the average word length. The audio-recorded responses were significantly and substantially longer than the two typed groups. However, there were no significant differences with regard to the number of codes applied or the average word length.
We also examined whether audio-recording responses reduced between interviewer variability on these indicators, as audio-recording should, in principle, reduce the interviewer’s influence on these outcomes. We found significantly lower between interviewer variance in both the number of words and the number of codes applied for the audio-recorded group compared to the typed groups, though not for the average word length.
In conclusion, audio-recording of verbatim responses appears to offer significant gains in data quality compared to requiring interviewers to type responses as they are articulated by respondents. However, this must be weighed against the substantial and non-random rate of refusal to be audio-recorded. Future research could usefully focus on methods for increasing the rate of consent to be recorded.
References
1 Sturgis, Patrick and Luff, Rebekah (2015) Audio-recording of Open-ended Survey Questions: A Solution to the Problem of Interviewer Transcription? In: Engel, U (Ed) Survey Measurements: Techniques, Data Quality and Sources of Error. Frankfurt: Campus, pp.42-57.
2 Wellcome Trust Monitor: http://www.wellcome.ac.uk/About-us/Publications/Reports/Public-engagement/WTX058859.htm