Cognitive Interviews

Our mission is to foster the improvements in the analysis of health related phenomena among older minority; encourage the development of methods and measures that better capture the health and determinants of health of diverse elders; promote collaboration between RCMAR sites on analysis issues; and disseminate new knowledge in this area.

Cognitive Interviews

Using Cognitive Interviews to Develop Structured Surveys

Prepared by the Center for Aging in Diverse Communities (CADC) University of California, San Francisco

Cognitive interviews are used widely during the pre-testing phase of questionnaires to detect items that are not understood by respondents as intended by the survey developers. Generally, cognitive interview methods reflect a theoretical model of the survey response process that involves four stages: comprehension or interpretation, information retrieval, judgment formation, and response editing. In other words, the respondent must first understand the question, then recall information, then decide upon its relevance, and finally formulate an answer in the format provided by the interviewer. Cognitive interviews thus identify the types of errors made by respondents and help us understand how they interpret and answer questions. In studies of health disparities, cognitive interview testing can be critical to assessing the conceptual adequacy of new or adapted self-report measures across groups varying in socioeconomic status or race/ethnicity. Cognitive interview results can be used to revise or develop new items so that they are appropriate to respondents’ cultural context and lifestyle. Cognitive interviewing methods rely primarily on verbal probes about the interpretation of questions and recall strategies. Such probes may be scripted or spontaneously created by the interviewer. Behavioral coding is another approach to identifying problems with items administered regardless of whether probes are used. Below we identify several key articles, books and book chapters that illustrate the use of these methods for pre-testing surveys.

  • Beatty P. The dynamics of cognitive interviewing. In: Presser S, Rothgeb JM, Couper MP, et al., eds. Methods for testing and evaluating survey questionnaires. Hoboken, N.J.: John Wiley & Sons, Inc.;2004:45-66.
As a way to formulate standard methods for cognitive interviewing, the author examines specific interviewer behaviors and their impact on the results of cognitive interviews. In a series of small laboratory based cognitive tests the author determined the effects that different types of probing had on the participants’ ability to choose an answer (precision), and how interviewer behavior influenced or biased participants’ responses. Recommendations for interviewing practices are made.

This article describes the use of cognitive interviews to assess comprehension of nutrition survey items and nutrition intervention messages in low-income families in North Carolina. Twenty-three technical community college students responded to a posted school notice about the study. The sample included 8 males and 15 females, all were over 18 years old, and half were African American. The cognitive interviews identified problems with unclear instructions, confusing terms and response options, variable interpretations of terms, and misinterpretation of dietary recommendations. The results were utilized to clarify ambiguous instructions and terms, expand response options, and more clearly specify dietary recommendations to make the survey and intervention messages more suitable for use in low-income groups.

This article reviews methods used to assure that respondents are able to understand the questions being asked, that questions are understood in the same way by all respondents, and that respondents are willing and able to answer such questions. The paper describes some of the theories and tools used to test these assumptions in order to develop better survey instruments and proposal software questionnaires. It argues that cognitive testing should be a standard part of the development process of any survey instrument.

  • DeMaio TJ, Rothgeb JM. Cognitive Interviewing Techniques: In the Lab and in the Field. In: Schwarz N, Sudman S, eds. Answering Questions. Methodology for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: Jossey-Bass, Inc.; 1996: 177-196.
This article provides a general review of cognitive interviewing techniques used in laboratory and field settings. Using three real life examples, the authors demonstrate the use of cognitive testing in both interviewer administered and self report surveys, review the procedures used in each case, and provide a brief analysis of the results. They also describe varying methods used during respondent debriefing, a technique commonly used to conduct cognitive testing in the field.

This article discusses how cognitive interviewing can be used to identify problematic questions prior to using the questionnaire in the field. Literature is reviewed. It is recommended that cognitive interviewing methods are most valuable in pretesting questions that are complex, sensitive and intrusive, and for specific groups for whom questionnaire completion may pose particular difficulties.

The authors performed 150 cognitive interviews to facilitate the development of the CAHPS core survey questionnaire, a standardized survey used to obtain information on the experiences of health care plan enrollees. The sample included 14 low literacy adults, 22 parents or guardians of children, 29 adults with chronic conditions, 31 Medicare beneficiaries and 54 Medicaid recipients. The authors describe the process of conducting cognitive interviews and their role in survey development. They report their findings pertaining to three methodological areas: 1) using ratings versus reports when asking respondents to assess their experience; 2) the length of the reference time specific to the questions; and 3) ensuring respondents utilize the ‘not applicable’ response when appropriate in self-report surveys.

The authors review several National Center for Health Statistics (NCHS) reports and demonstrate how cognitive testing techniques can be used to improve patient understanding of questions, recall of past events, and understanding of medical conditions in national health surveys such as the NHIS.

The authors performed behavioral coding of 345 face-to-face interviews with African Americans (n=86), Mexican Americans (n=101), Puerto Ricans (n=74), and non-Latino Whites (n=84) to identify question characteristics that might account for cultural variability in responses. Behavior coding involves the systematic coding of respondent and interviewer behaviors using a pre-determined coding scheme to identify problems with survey questions. The interview included standard questions from the National Health Interview Survey (NHIS), the Behavioral Risk Factor and Surveillance Survey (BRFSS), and the former National Household Survey of Drug Abuse. All interviews were conducted in English. Race/ethnicity was found to be independently associated with question comprehension problems, with minority group members more likely to report comprehension problems.

This article illustrates the importance of cognitive testing in assessing the conceptual adequacy of new or adapted self-report measures across ethnic groups in studies of health disparities. The authors demonstrate the use of two complementary cognitive testing techniques, interaction analysis and behavior coding, to evaluate a survey administered to an ethnically diverse population. Forty-eight cognitive interviews were completed, including 14 with African Americans, 20 with Latinos and 14 with Whites. Six Latinos were interviewed in English and 14 in Spanish. The mean age was 46 years and 58% were women.Results of cognitive testing permitted the identification of both general and ethnic group specific problems with wording and concepts, revealed the nature of the problems, and enabled decisions to modify or drop items based on respondents’ comments.

  • Oksenberg L, Cannell C, Kalton G. New Strategies for pretesting survey questions. Journal of Official Statistics. 1991;3:349-365.
This article demonstrates the need for more rigorous pre-testing methods of surveys and explores cognitive interviewing techniques involving behavior coding and probe questions. The authors detail the process of conducting both behavior coding and cognitive probes on a sample of 164 Michigan residents through a general telephone health survey. Participants were selected through a probability sampling of telephone numbers of residents of southeastern Michigan. Once the household was contacted the participant was selected to obtain an equal number of males and females in the final sample. The authors explore the benefits and limitations of using either behavior coding or directed probes independently, and suggest that using the two techniques together provides the researcher with the most systematic and objective method for evaluating the survey questions.

  • Presser S, Couper MP, Lessler JT, Martin E, Martin J, Rothgeb JM, Singer E. Methods for testing and evaluating survey questions. Public Opinion Quarterly. 2004;68(1):109-130.
The authors provide a detailed review of cognitive pre-testing including its history and development, supplemental techniques, experiments to test the usefulness of revisions made to surveys based on cognitive interview results, statistical modeling and different modes of data collection. The authors argue that the methods used to conduct cognitive testing influence the results, and thus developing optimal cognitive interviewing techniques is imperative. Directions for future research are explored in depth.

This article describes the use of cognitive interviewing to adapt measures for use in populations for whom they were not originally developed. Cognitive interviewing was used to adapt four instruments (the Audit of Diabetes Knowledge, the Insulin Management Diabetes Self-Efficacy Scale, and the Center for Epidemiological Studies-Depression Scale) for use with low-literate, Caribbean Spanish-speaking older adults with diabetes. The 5 participants in the study were Puerto Rican (3) and Dominican (2), aged 57-80 years, and 3 were female. All of the participants had 5 years or less of schooling. Adaptations included development of interviewer administered versions of these instruments since they were originally designed for self-administration, translation of measures into Spanish where necessary, modification of negatively worded items, deletion of items that were not culturally relevant, and clarification of response options to make them more concrete. Probes addressed the suitability of specific words used in items, the clarity of response options, and suggestions for alternate wording. The results of content analysis of the interviews were used to modify instructions and specific survey items for improved comprehension.

  • Schwarz N, Sudman S (Eds). Answering Questions: Methodology for Determining the Cognitive and Communicative Processes in Survey Research. San Francisco, CA: Jossey-Bass Publishers; 1995.
The authors summarize research on techniques and methods used to uncover cognitive and communicative processes used in answering survey questions. Chapters include using behavioral coding to identify cognitive problems with survey questions, questionnaire pretesting, interactive aspects of cognitive processing in standardized survey interviews, cognitive interviewing techniques. Other more in-depth chapters are also included such as expanding the use of verbal protocols in survey research and the validity of verbal reports about attitudes.

  • Sudman S, Bradburn NM, Schwarz N. Thinking About Answers: The Application of Cognitive Processes to Survey Methodology. San Francisco, CA: Jossey-Bass Publishers; 1996.
This book explores the methods that people use to answer questions, the problems that they have, and the implications that it has for survey based research. The authors explore the four steps in cognitive processing and identify possible sources of difficulty or measurement error at each stage including autobiographical event recall, estimation, frequency counts, ranking, effects of attitude or context on answers, and the influence of question ordering on subsequent questions. They provide in-depth examples and evaluations of several methods used to measure response error such as think aloud, cognitive experts, and behavioral coding. The implications of the cognitive processes involved in survey response for questionnaire design are discussed.

The article reviews the theory of survey response and how differences in race/ethnicity lead to systematic measurement error in survey responses to standardized questionnaires. The authors performed cognitive testing on questions from several well established national health surveys (BRFSS, etc) in an ethnically diverse sample. The purposive sample of 423 adults included equal numbers of African Americans, Puerto Ricans, Mexican Americans, and non-Hispanic whites. They specifically evaluated how race and ethnicity influenced respondents’ performance of the four cognitive tasks involved in responding to survey questions: question interpretation, information retrieval, judgment formation, and editing. Differences were found by race/ethnicity for several of the stages of question response, suggesting that more work is needed to develop surveys that are valid across racial and ethnic groups.

  • Willis GB. Cognitive Interviewing. Thousand Oaks, CA: Sage Publications; 2005.
This book is a complete guide to using cognitive interviews to develop and refine questionnaires. The author explores the theory behind cognitive interviewing, specific techniques to conduct interviews including developing probes, training interviewers, recruitment of participants, and documenting and analyzing results. Other helpful sections include cognitive testing of sensitive questions and across age groups.

On-line manual available

Updated June 2010