Web surveys have become a widely adopted approach for data collection in many fields due to their low costs and efficiency. Nevertheless, due to the high competition for respondents’ time and attention, web survey companies are struggling to recruit web respondents, keep them engaged in online surveys, and ensure adequate data quality. One possible avenue to ameliorate these threats is through the use of innovative survey designs. Interest in this area is apparent in the attention that concepts like gamification have received.
One important visual aspect of web surveys that could be changed to improve the experience of respondents is the response scales. Traditionally, HTML code allowed the use of radio buttons for responses with mutually exclusive categories. An early alternative to this approach has been the drop-down menu, while more recently slider questions and visual analog scales have been proposed. Nevertheless, radio buttons have remained the standard approach to collecting answers to close-ended questions in online surveys.
In a recent paper1, we have experimentally compared four alternative graphic designs to the traditional radio button response. Thus, we replace the traditional round button with stars, smileys, hearts and thumbs. By replacing the radio buttons with these familiar symbols we expected to make the survey more appealing. At the same time we are interested in any unintentional impact on response patterns. In order to investigate the influence of these symbols on responses, we ran a series of web survey experiments that experimentally manipulated other possible moderating factors: the number of categories (5 vs. 7), type of response scale (unipolar vs. bipolar) and the use of verbal labelling (with and without labels). Additionally, we investigated how these are influenced by the device used (PC vs. mobile). We crossed all of these experimental groups and applied them in SurveyMonkey nonprobability panel in the US.
Overall, there are no differences in data quality or respondent satisfaction between the radio buttons and the new response scales. Two differences stand out. Firstly, the smiley face scale had higher item missing and lower satisfaction, both compared to radio buttons and the other symbols. At this point it is unclear why the smiley face scale significantly underperformed compared with the other symbols. One explanation can come through the implied reasoning behind the use of the scales. When the scales are not labelled we expect respondents to infer that more agreement/satisfaction leads to more points on the scale. This is in line to how we already use some of these symbols: 5 stars is better than 3 stars. This inference may work less well in the case of the smiley response scale (as seen in Figure 2). An alternative design could make this a bi-polar scale by including a frowning face at one extreme of the scale and a smiley face at the end, with a neutral face in the middle.
Figure 1 Alternative scales proposed in the paper
Figure 2. How choosing different points on the scale worked
The second finding of the paper is around response time. The new response scales were faster to complete. The alternative buttons take on average around 41 second less to complete. This difference is larger on PCs than on mobile, 56 seconds versus 24. Given that the data quality of the new scales was similar to that from the radio buttons it might indicate the cognitive burden is lower (which might be an advantage).
In conclusion, should we get rid of radio buttons? It’s still early days. Further testing is probably needed, especially in longer and more complex surveys. That being said there is some indication that we can replace the radio buttons in surveys without loss to data quality but also with no increase in the satisfaction of the respondents. Combining the new response scales with other approaches such as gamification might be worth considering. If you want to read more about this you can have a look at our recently published paper in the International Journal of Market Research. Find out more about Alex’s work at: www.alexcernat.com
References
1 Cernat, A., & Liu, M. (2018). Radio buttons in web surveys: Searching for alternatives. International Journal of Market Research. https://doi.org/10.1177/147078531881352