Practical Guidance for Studies Using Freelisting Interviews

Shimrit Keddem, PhD, MPH^1,2,3; Frances K. Barg, PhD, MEd²; Rosemary Frasso, PhD, MSc, CPH³ (View author affiliations)

Suggested citation for this article: Keddem S, Barg FK, Frasso R. Practical Guidance for Studies Using Freelisting Interviews. Prev Chronic Dis 2021;18:200355. DOI: http://dx.doi.org/10.5888/pcd18.200355external icon.

PEER REVIEWED

Abstract

Freelisting is a qualitative interviewing technique that has recently grown in popularity. It is an excellent tool for rapidly exploring how groups of people think about and define a particular health-related domain and is well suited for engaging communities and identifying shared priorities. In this article, we outline 7 practical considerations for conducting freelisting studies summarized from 16 articles conducted by the authors at the University of Pennsylvania and Thomas Jefferson University in partnership with community-based organizations and students. Our recommendations can inform study design, data collection, and data analysis for investigators who are interested in using freelisting interviews in their research.

Top

Introduction

Freelisting, a qualitative method developed by anthropologists in the early 1980s (1,2), can be used as an elicitation technique during one-on-one interviews or in group settings to define a particular domain. Researchers start by asking participants to name all the items that come to mind in response to a prompt (2) (eg, “What comes to mind when you think about staying healthy while in school?”). Items are sorted and ranked by the researcher on the basis of frequency or salience, a measure indicating the importance of an item to the respondents. The data allow researchers to understand how a population defines various health-related topics, from perceptions of illness to health behavior to health care needs. The strength of this method is that it elicits unimagined, spontaneous responses that can be rapidly collected, analyzed, and quantified, and results from these analyses can be incorporated into mixed-methods studies in different populations and settings. The information gained from freelisting interviews can be invaluable to understanding community needs and priorities and informing culturally appropriate health communication, health promotion, and research materials. The terms “freelisting,” “free listing,” and “free-listing” are often used interchangeably. A Web of Science search including these terms returned only 2 articles during 1999; by 2019, the cumulative number of articles using freelisting had increased to 361.

The increasing use of qualitative and mixed-methods approaches has led to the growing application of freelisting in health research, especially among vulnerable populations such as people living in low-income urban settings, people who inject drugs, and people with serious mental illness. Despite this growth, few published resources detail the intricacies of designing and implementing freelisting studies. The lack of resources providing guidance is a barrier to researchers conducting thoughtful and rigorous studies that incorporate freelisting either on its own or in combination with other methods.

We analyzed a convenience sample of 16 studies (3–18) conducted by the authors at 2 Philadelphia institutions (the University of Pennsylvania and Thomas Jefferson University) in partnership with community-based organizations and students (Table). Our goal was to summarize the methodologic information in the studies to provide practical guidance for conducting freelisting interviews. The 16 studies include varying health topics in differing populations, with both freelisting used on its own or as part of mixed-methods studies or community-based participatory work. We outline the potential challenges and considerations in conducting freelisting interviews and suggest practical solutions, including tactics for designing a freelisting study; collecting, managing, and analyzing freelisting data; and developing the interview guide. As with all other qualitative methods, freelisting studies should incorporate rigorous systematic and reflective processes that ensure both internal and external validity and limit bias (19). Although complete objectivity is unachievable in research, researchers should strive for “strong” objectivity, whereby the researcher and their team are reflective and transparent throughout the research process (20).

Top

Guidelines for Conducting Freelisting Interviews

Design the freelisting study to examine specific populations’ perspectives and compare across domains or groups

Freelisting rapidly explores the “emic,” or insider, perspective of a group or culture by asking members of that group to list all the elements of a particular domain. Central to this approach is the notion that shared cultural beliefs tend to produce shared concepts among members of that group. As a rule of thumb, a sample of 20 freelisting interviews is adequate to reach saturation in homogeneous groups, meaning that beyond this, additional interviews will likely not yield novel data (2). Freelisting allows the researcher to conduct 2 types of comparisons: 1) between different constructs (eg, different psoriasis treatments) or 2) between different groups or cultures (Figure). For comparisons across groups, we recommend aiming for at least 20 interviews per group. Of the 16 articles we analyzed, 11 contained some type of comparison across groups or domains.

Figure.
Phases of the freelisting process. [A text description of this figure is available.]

These types of comparisons can be particularly informative when researching groups with unique perspectives. For example, Barg et al (4) interviewed adolescents to understand teens’ and predrivers’ definitions of the phrases “good driver” versus “safe driver.” Each class of students was asked by a facilitator to produce 2 written lists in response to 2 prompts: “What words do you think of when you think of a good driver?” and “What words do you think of when you think of a safe driver?” Analyses aimed to compare the most salient words used for “good driver” versus “safe driver” to understand how these domains differ for young adults. In addition, the authors examined differences in responses by sex and ethnicity.

Comparisons can be used to understand linguistic and cultural differences between groups. Karlawish et al (11) compared the perceptions of 4 groups — Latinx versus non-Latinx and caregivers versus noncaregivers —of Alzheimer disease and its causes and treatments. Interviews were conducted in Spanish and English and comparisons were made across the 4 groups.

Use freelisting in mixed-methods or qualitative studies

Freelisting can be used on its own or be incorporated into a sequential or concurrent mixed-methods design. At the beginning of any study, freelisting interviews can provide the definition and boundaries of the topic being studied (2) and therefore help develop recruitment materials, surveys, or interview guides. Freelisting findings can also aid in the development of educational materials or help determine priorities for policy or practice.

A freelisting exercise can be the first step in a study to develop an interview guide. To better understand issues surrounding supervised injection facilities, Harris et al (9) conducted freelisting interviews with people who inject drugs and health care providers recruited from a syringe exchange program. Findings were combined with policy/makers’ priorities to inform the development of a semi-structured interview guide to explore whether people who inject drugs believe that safe injection facilities could improve safety and mitigate risk of infection.

Keddem et al (12) conducted a freelisting study as part of a sequential mixed-methods design. Freelisting questions were included at the beginning of an in-depth interview guide to understand neighborhood and environmental barriers to asthma control. Freelists were collected from a stratified sample of adults with asthma living in an urban neighborhood. Freelist responses, sorted by salience, were used to generate a composite map of geographic areas of poor asthma control, as perceived by residents of the neighborhood.

Freelisting is an excellent method to use for working with communities. Minimal training is required to conduct the freelisting interviews, and the process of cleaning and analysis is enhanced by community member participation. In their freelisting study of fruit, vegetable, and fast food consumption, Lucan and colleagues (13) asked selected community members to review and help revise study documents, provide space to conduct interviews, and advertise the study to the community. Community stakeholder perspectives can also be incorporated into the data collection, management, and analysis phases. For example, Ahmad et al (3) included a patient representative with heart failure in the data cleaning and review process of a study that examined heart failure management and hospital readmissions.

Freelist interviews can take as few as 5 minutes to complete, depending on the number of questions (6). Interviews can be scheduled in advance or conducted on the spot in community settings, waiting rooms, or other places where people can be approached and can provide consent (21). Freelisting is an efficient way to gather data from large samples of people in a short time (6). Freelisting can be used when issues are time sensitive or if a group happens to be gathered together (eg, at a convention or community event) to quickly obtain and understand community goals and needs. This feature could prove useful in the context of disaster response.

Create a freelisting interview guide

Designing the interview guide is an integral component of the freelisting study but asking the “right” questions is not necessarily easy, so piloting the guide is important. As with all qualitative research endeavors, the interview guide is strongest when reviewed by an interdisciplinary team and, if possible, with input from the community. For example, in the study by Dress et al (7) of people with early psychosis, the guide was reviewed by 4 faculty members and piloted during a focus group consisting of research faculty and staff who were familiar with the study population.

Questions should aim to elicit as extensive a list of items as possible. When respondents’ lists are too short, this can be an indication that questions are not well conceived (22). To capture the respondents’ own words, responses should be recorded verbatim by the interviewer. Because people are not accustomed to providing answers in the form of a list, starting the interview with a practice question (eg, “If I asked you to list all the names of animals that you can think of, what would you say?”) can be beneficial. The interviewer should encourage 1-word answers, but more descriptive phrases are also acceptable. It is important and appropriate for the interviewer to clarify responses that are unclear after the participant completes their list.

Data collection can be conducted in various formats and settings. Lists may be collected by the interviewer in person or on the telephone (3) or provided by the respondents, written by hand or through an electronic survey (10). Allowing interviewees an unlimited number of responses is ideal.

Eliciting listed responses to vignettes or visual cues is also possible. In the study by Lucan et al, interviewees were asked to provide a list in response to images of different foods to understand their perceptions of the food (13). Karlawish et al (11) incorporated vignettes about Alzheimer disease into their interview guide to understand the dynamics in relationships where one spouse is a caregiver for a partner who has Alzheimer disease. All caregivers were also asked to provide a list in response to a 3-paragraph description of a clinical trial for persons with Alzheimer disease.

Develop a systematic approach for cleaning the freelisting data

Freelisting data must be carefully cleaned before analysis by reviewing raw lists to combine root words, synonyms, and similar concepts, a process that can be done in multiple iterations. On a first round, researchers, and when possible community members, should group grammatical forms of the same word (eg, “smell” and “smells”). On a second round, synonyms can be combined (eg, “scent” and “smells”), and on a third round, words representing similar concepts can be grouped (eg, “perfume,” “deodorant,” “scent,” and “smell”). Categories should be created inductively without any a priori set of rules. At this phase of the process, reviewers should be blinded to any demographics to reduce bias. To ensure the integrity of the process, it can be constructive for team members to check each other. For example, to reduce bias, the Dress et al team had 2 researchers combine synonyms, and a third researcher review the agreed-upon categories to produce the final list. If lists are collected in different languages (as with Karlawish et al), creating 2 independent translations of the lists and reviewing both to generate a final English translation for analysis is ideal.

Use a scree plot and design a meaningful review process to interpret salience indices

The goal of the freelisting analysis is threefold: 1) to identify the most salient items across multiple respondents’ lists; 2) to present only the terms on the list that the researcher believes are truly shared by the group or groups; and 3) to develop a manageable list that is not too long (23). Anthropac software (Analytic Technologies) is a free program (www.analytictech.com/anthropac/anthropac.htm) created for the analysis of freelisting data. Freelists can be stored in many spreadsheet formats but need to be converted to a text file before they are entered into Anthropac. If multiple groups or questions are included in the study, a separate file should be generated for each question and each group within each question before importing into the software.

Smith’s S, the salience index, is the main statistic used for analyzing freelists (24). This statistic considers both the frequency of an item across all respondents’ lists as well as its rank order on these lists, reflecting the assumption that items mentioned early and often across respondents are the most salient. The salience index is a gross mean percentile rank for each item across all lists, as represented in the following equation (24):

Freelist salience of an item (Smith’s S) equals the sum of the item’s percentile ranks divided by the total number of lists, where L is the length of each list, R_j is the rank of item j in the list, and N is the number of lists in the sample.

Anthropac combines all respondents’ lists into a composite list and provides a salience index for every item on the composite list. Selecting items to display in a final report or article may be determined by many factors, including the purpose of the study, the number and frequency of the items, the number of questions asked, and the study design and format. Examining salience indices using a scree plot (in Excel [Microsoft Corporation] or other program), placing the salience index on the y axis and the items on the x axis, can help in the selection process (4,14). This setup will display the results of each question as a downward curve on a graph. There is almost always an “elbow” in the curve, which is the point where data level off. This point can be used as a cutoff, and the researcher can choose to display only items that appear before the cutoff as the most salient terms. In some cases, researchers may choose their own cutoff, for example, including only the top 5 items on each list.

Researchers can display differences between groups in several ways. Visuals and diagrams can be particularly helpful for highlighting differences. Of the 16 articles we included, 6 used Venn diagrams and 9 used tables to display differences between groups. Jonas et al (10) chose to transform the salience index into a relative salience score for comparing across groups by dividing the raw salience indices for each term by the raw index for the most salient term in the list, then converting to a percentage. In their article, they displayed these differences in tables with color coding to show differences in relative salience.

Build on your study by adding cultural consensus analysis

Beyond the salience index, the researcher can also use consensus analysis to determine the degree of similarity among respondents. Consensus analysis is based on the theory that members of the same cultural group will have more knowledge about a given cultural domain (19). The cultural consensus analysis model relies on 3 assumptions: 1) that there is a “common truth,” or one unifying cultural reality; 2) that respondents are independent of each other, and that if they don’t know the “culturally appropriate answer,” they make one up independently; and 3) that all questions are on the same topic (25). Cultural consensus analysis generates eigenvalues, used to measure agreement among persons in a group and indicating the strength of the consensus. If there is 1 culture, there must be 1 large eigenvalue. Therefore, results are analyzed in terms of the ratio of the first eigenvalue to the second eigenvalue. If the ratio is less than 3 to 1, the researcher can assume that there is more than 1 “correct” cultural definition within the group (25). Of the 16 articles we included, 3 presented results from cultural consensus analysis.

Fiks and colleagues (8) conducted a freelisting study to understand perception of attention deficit hyperactivity disorder (ADHD) among pediatricians and parents of affected children. They included cultural consensus analysis to determine the degree to which subgroups of parents and clinicians shared an understanding of ADHD. For each group and for each question, they used Anthropac to measure how each person’s response was weighted in relation to the most frequent responses of the group. They looked for an eigenvalue greater than 3 to indicate group consensus and found that none existed among clinicians for any of the freelist questions. However, parental consensus was found for 2 questions: 1) words participants think about when they hear “ADHD” and 2) getting help for ADHD. They also assessed consensus by race and found no strong consensus among White participants to any of the questions. Among Black participants, consensus was found for items describing ADHD.

Understand the limitations of freelisting

Freelisting interviews have several limitations. Freelisting responses consist only of individual words or phrases and do not provide the type of depth that can be gained from a semi-structured or extensive qualitative interview or focus group. However, freelisting elicits a set of spontaneous responses in the respondents’ own words. To overcome this barrier, researchers can and should incorporate freelisting in mixed-methods studies as a step in the research process to provide triangulation and inform subsequent steps of the process. In the study by Bennett et al (5) of the role of low literacy in maternal care utilization, the researchers conducted confirmatory focus groups following freelisting to confirm and explore the items identified by respondents. Another limitation is the potential bias introduced during data cleaning. To reduce this bias, it is important that the cleaned categories be inductive and use the respondents’ words. In addition, developing a rigorous and systematic cleaning process is also necessary, ideally one involving multiple perspectives from team and community members.

Top

Conclusion

This article outlines guidelines and considerations for conducting freelisting studies summarized from 16 articles conducted by researchers at the University of Pennsylvania and Thomas Jefferson University. Our findings can be useful to investigators interested in conducting freelisting studies by providing practical and methodologic considerations that may arise at the design, data collection, and analysis stages of the research.

Freelisting is an ideal technique for understanding the community perspective and eliciting spontaneous responses that can allow researchers to gain insight into the insider perceptions of a variety of health topics. Results from freelisting interviews can be gained rapidly, are quantifiable, and can be immediately applied in research and practice. Researchers who are interested in using freelisting should consider their population of interest, identify the purpose of their study, and understand the theoretical and methodological underpinnings of the technique.

Top

Acknowledgments

No copyrighted tools or materials were used in this research.

Top

Author Information

Corresponding Author: Rosemary Frasso, PhD, MSc, CPH, College of Population Health, Thomas Jefferson University, 901 Walnut St,10th Floor, Philadelphia, PA 19107. Telephone: 215-503-8901. Email: rosie.frasso@jefferson.edu.

Author Affiliations: ¹Michael J. Crescenz VA Medical Center, Philadelphia, Pennsylvania. ²University of Pennsylvania, Perelman School of Medicine, Philadelphia, Pennsylvania. ³Jefferson College of Population Health, Thomas Jefferson University, Philadelphia, Pennsylvania.

Top

References

1. Weller SC. Consistency and consensus among informants: disease concepts in a rural Mexican village. Am Anthropol 1984;86(4):966–75. CrossRefexternal icon
2. Weller SC, Romney AK. Systematic data collection, volume 10. Thousand Oaks (CA): Sage Publications; 1988.
3. Ahmad FS, Barg FK, Bowles KH, Alexander M, Goldberg LR, French B, et al. Comparing perspectives of patients, caregivers, and clinicians on heart failure management. J Card Fail 2016;22(3):210–7. CrossRefexternal icon
4. Barg FK, Keddem S, Ginsburg KR, Winston FK. Teen perceptions of good drivers and safe drivers: implications for reaching adolescents. Inj Prev 2009;15(1):24–9. CrossRefexternal icon
5. Bennett I, Switzer J, Aguirre A, Evans K, Barg F. “Breaking it down”: patient–clinician communication and prenatal care among African American women of low and higher literacy. Ann Fam Med 2006;4(4):334–40. CrossRefexternal icon
6. Cunningham A, Riley AH, Puskarz K, Frasso R. Experiential learning: moving from data collection to health promotion in one term. Pedagogy Health Promot 2019;6(3):162–7.
7. Dress EM, Frasso R, Calkins ME, Curry AE, Kohler CG, Schmidt LR, et al. Comparing patient, clinician, and caregiver perceptions of care for early psychosis: a free listing study. Narrat Inq Bioeth 2018;8(2):157–78. CrossRefexternal icon
8. Fiks AG, Gafen A, Hughes CC, Hunter KF, Barg FK. Using freelisting to understand shared decision making in ADHD: parents’ and pediatricians’ perspectives. Patient Educ Couns 2011;84(2):236–44. CrossRefexternal icon
9. Harris RE, Richardson J, Frasso R, Anderson ED. Perceptions about supervised injection facilities among people who inject drugs in Philadelphia. Int J Drug Policy 2018;52:56–61. CrossRefexternal icon
10. Jonas JA, Davies EL, Keddem S, Barg FK, Fieldston ES. Freelisting on costs and value in health care by pediatric attending physicians. Acad Pediatr 2015;15(4):461–6. CrossRefexternal icon
11. Karlawish J, Barg FK, Augsburger D, Beaver J, Ferguson A, Nunez J. What Latino Puerto Ricans and non-Latinos say when they talk about Alzheimer’s disease. Alzheimers Dement 2011;7(2):161–70. CrossRefexternal icon
12. Keddem S, Barg FK, Glanz K, Jackson T, Green S, George M. Mapping the urban asthma experience: using qualitative GIS to understand contextual factors affecting asthma control. Soc Sci Med 2015;140:9–17. CrossRefexternal icon
13. Lucan SC, Barg FK, Long JA. Promoters and barriers to fruit, vegetable, and fast-food consumption among urban, low-income African Americans — a qualitative approach. Am J Public Health 2010;100(4):631–5. CrossRefexternal icon
14. Meghani SH, Wool J, Davis J, Yeager KA, Mao JJ, Barg FK. When patients take charge of opioids: self-management concerns and practices among cancer outpatients in the context of opioid crisis. J Pain Symptom Manage 2020;59(3):618–25. CrossRefexternal icon
15. Mollen CJ, Fernando M, Hayes KL, Barg FK. Pregnancy, contraception, and emergency contraception: the language of urban adolescent young women. J Pediatric Adolesc Gynecol 2012;25(4):238–40.
16. Takeshita J, Eriksen WT, Raziano VT, et al. Racial differences in perceptions of psoriasis therapies: implications for racial disparities in psoriasis treatment. J Invest Dermatol 2019;139(8):1672–9.
17. Hwang J, Lee C, Mastrolonardo E, Frasso R. Where there’s smoke, there’s fire: what current and future providers do and do not know about electronic cigarettes. BMC Public Health 2020;20(1):1145. CrossRefexternal icon
18. Auriemma CL, Lyon SM, Strelec LE, Kent S, Barg FK, Halpern SD. Defining the medical intensive care unit in the words of patients and their family members: a freelisting analysis. Am J Crit Care  2015;24(4):47–55.
19. Malterud K. Qualitative research: standards, challenges, and guidelines. Lancet 2001;358(9280):483–8.
20. Harding S. “Strong objectivity”: a response to the new objectivity question. Synthese 1995;104(3):331–49. CrossRefexternal icon
21. Frasso R, Keddem S, Golinkoff JM. Qualitative methods: tools for understanding and engaging communities. In: Handbook of community movements and local organizations in the 21st century. New York (NY): Springer Publishing; 2018. p. 527-49.
22. Brewer DD. Supplementary interviewing techniques to maximize output in free listing tasks. Field Methods 2002;14(1):108–18. CrossRefexternal icon
23. Borgatti S. Using ANTHROPAC to investigate a cultural domain. Cultural Anthropology Methods Newsletter 1989;1(2):11.
24. Smith JJ. Using ANTHROPAC 3.5 and a spreadsheet to compute a free-list salience index. Cultural Anthropology Methods 1993;5(3):1–3. CrossRefexternal icon
25. Borgatti SP, Halgin DS. Consensus analysis. A companion to cognitive anthropology. Thousand Oaks (CA): Sage Publications; 2011.

Top

Table

Abbreviations: ADHD, attention deficit hyperactivity disorder; BMI, body mass index; NA, not applicable.

Top