You are here:

Cognitive Interviewing and what it can be used for

Presenter(s): Olivia Sexton, Sophie Pilley, Jo d’Ardenne and Richard Bull


decorative image to accompany text

This resource introduces the cognitive interviewing method, a qualitative method which explores an individual’s thought processes when presented with a task or information. Cognitive interviews traditionally involve administering a survey questionnaire whilst collecting additional verbal and non-verbal information about how individual’s respond to the question. The types of verbal and non-verbal information which are typically collected during a cognitive interview are participant observations, any naturalistic verbalisations from participants, and responses to semi-structured interviews. This additional verbal information is used to evaluate whether the question is measuring what the researcher intends it to. Cognitive interviewing is traditionally used in social research to pre-test survey questions. 

We will explain:

  • how to conduct a cognitive interview and the other uses of cognitive interviewing in addition to pre-testing survey questions.
  • where best to use the method as well as its limitations.

 

What cognitive interviewing is and why we might use this as a research method

Cognitive interviewing is a qualitative research method used to explore how individuals perceive stimuli and information. Typically, when we design surveys and questionnaires, we design them to be used in quantitative research projects. When designing surveys and questionnaires, we want the questions that we use to be able to generate good quality data, in that our data is valid, reliable, sensitive, unbiased and complete. 



   Download transcript    |   Download slides [ 709 Views ]

To be able to check that our questions will likely generate good quality data, we can use the cognitive interviewing method. Traditionally this technique has been used when pre-testing questions which are used in quantitative data collection. 

As a research method, the cognitive interviewing technique is used within qualitative interviews to explore how individuals comprehend and judge questions, how they retrieve the information to answer a question and ultimately how they respond to the question.


How to conduct cognitive interviews

Cognitive interviews typically occur in a one-to-one setting between an interviewer and a participant. During these interviews, the participant is asked survey questions, but the focus is on the mental processes used to come up with an answer. 

There are four key elements of a cognitive interview.

The first key element is administering the survey question. When interviewers administer the survey question, it should be asked in a format which is as close as possible to how participants would complete this in the real survey context. For example, if the survey is interviewer-administered, the interviewer should read the questions aloud to the participant.

The second key element is participant observation. During a cognitive interview, interviewers should observe any non-verbal signs from participants when they are thinking about or answering the questions. For example, if you administer a question and a participant appears to hesitate, this should be noted.

The third key element is the think-aloud technique. This technique developed by Ericsson and Simon (1980) encourages participants to verbalise their thought processes as they answer the questions. As this is often not the easiest thing for participants to do, we recommend training participants to think aloud, by first demonstrating the technique yourself.



   Download transcript    |   Download slides [ 410 Views ]

The final key element of the cognitive interview is interviewer probing, which is where interviewers, following an interview protocol, ask semi-structured questions. During our cognitive interviews, this is where we get the majority of our information about how the questions are working. There are different ways in which the probing questions can be asked. For example, they can be asked at the same time as the question being administered or can be asked at the end of the survey.

The most important part of conducting a cognitive interview as an interviewer is to not bias the participant. This means that you should avoid trying to add clarifications to questions unless there are specific interview instructions. Remember, it is important to know which questions are working well, and which questions may need amending to help with understanding.

> Download example probe sheet here.


What are the other uses of cognitive interviewing?

Whilst cognitive interviewing is traditionally used to pre-test survey questions, it does have other uses as a research method. As the cognitive interview ultimately explores how individual’s respond to stimuli, the stimulus does not always have to be survey questions.



   Download transcript    |   Download slides [ 111 Views ]

Cognitive interviewing can be used in a variety of other ways. It can be used to test advance leaflets and letters (for example informational leaflets and invitational letters), in combination with user testing of digital devices and forms and to test permission forms for data linkage. 

> Download case studies document.


Strengths and limitations

Cognitive interviewing adds something to the survey development process: it tells us something about what goes on in respondents’ minds when asked the survey question or to complete a questionnaire. It is particularly good at identifying problems with comprehension that relate to the ‘reasonableness’ of the task, the way in which the request is structured (syntax) and the words used. As such it is a useful tool to allow us to examine whether the question achieves construct and face validity. It can also identify other problems, related to recall, judgement, response, visual layout and clarity of objectives.

However, as with any technique, there are boundaries to using this technique.

StrengthsLimitations

Adds to questionnaire development process – 

Cognitive interviewing adds something to the questionnaire development process: it tells us something about what goes on in respondents’ minds when asked the survey question or to complete a questionnaire

Unable to indicate size or extent of problems – 

Cognitive testing samples are generally small. The purpose is not statistical estimation but rather probing around potential issues. Cognitive testing sample sizes, therefore, are not large enough to supply precision in statistical estimates around the size or extent of issues observed.

Able to clarify question objectives and concepts – 

Cognitive testing is a useful tool which allows researchers to examine whether the question or materials being tested achieve construct and face validity. In other words, that it does what it is supposed to do.

Unable to guarantee all problems are mapped – 

Because cognitive testing can’t always reach data saturation (due to budget and time constraints), or the point at which no new information or themes are observed in the data, it is unable to determine for certain that all problems have been mapped.

Able to reveal ‘hidden’ problems – 

Cognitive testing allows unforeseen issues to be identified by encouraging participants to discuss how they found using the questions or materials.  

Unable to guarantee problems are ‘real’ – 

Cognitive testing is unable to say definitively that any problems encountered are ‘real’. It may be that participant behaviour during cognitive testing does not fully replicate a real world setting, and as such they are more likely to mention ‘issues’ which under normal conditions would not be identified as such.

Able to identify comprehension difficulties – 

Comprehension problems result when questionnaire designer and respondent’s interpretation don’t match. Cognitive testing is particularly good at identifying problems with comprehension that relate to the ‘reasonableness’ of the task, the way in which the request is structured (syntax), and the words used.

Unable to fully test every question on the full sample if questions are filtered (routed) – 

Cognitive testing relies on the sample having been designed well: that is that it is large enough and including a range of different types of respondent so that all questions are sufficiently tested. When test questionnaires are heavily filtered (routed), there is the risk that some questions may only be asked to a small number of participants.

Able to identify recall difficulties – 

Cognitive testing is able to identify if participants experience any difficulty in recalling events from a specific period by probing around how they found doing this and how confident they are that their response is accurate.

Unable to replicate real survey conditions – 

Cognitive testing is not the same as a real world survey. It is assumed that the problems identified during cognitive testing would occur in the survey itself, rather than being an artefact of the cognitive interview process.  

Able to reveal where potentially inaccurate data may be collected (e.g., shortcuts, estimates and editing) – 

Cognitive testing is useful in uncovering areas where potentially unreliable data may be collected. This could include when participants provide estimates or ‘short cuts’ to a question when a higher degree of accuracy is intended. Participants may give answers in an incorrect format, refuse to answer certain questions, or may be unable to answer the way they wish using the answer options provided.

Unable to definitively say whether the new version is better than the old – 

Whilst it is hoped that recommendations by way of changes to questions will lead to improvements in data quality, cognitive testing does not usually offer the opportunity to test these recommendations. As such, it is difficult to determine how successful these recommendations are or what other problems they may cause.

Able to identify missing response options – 

Cognitive testing is able to identify if participants experience any difficulty in selecting a response option, and if so, if there are other response options that should potentially be included in the question.

 

 




Supporting materials

About the author

Dr. Olivia Sexton is Senior Researcher at the National Centre for Social Research (NatCen). NatCen is a registered charity and the largest independent and not-for-profit social research organisation in the UK. The centre is driving innovation in survey methods and methodology, providing a variety of research solutions to help clients understand their stakeholders better.

Primary author profile page


⌃
BACK TO TOP