A Quality Approach to Qualitative Content Analysis: Similarities and Differences Compared to Other Qualitative Methods

Abstract: Qualitative content analysis is a method that shares many of the unique attributes associated with all qualitative research methods. These shared attributes extend to a key consideration in all qualitative research designs, i.e., the integration of quality concepts at each step of the research process. In this article I discuss one such approach, the total quality framework. With this framework at hand, researchers can think about quality in qualitative research regardless of the qualitative method, including qualitative content analysis. For example, quality constructs associated with sampling, researcher effects, and data verification in qualitative content analysis are not unlike those in the in-depth interview, focus group, or observation method. There are, however, a few quality considerations distinctive to the qualitative content analysis method, such as the definition of "data" and the necessary thick description associated with the two-phase, eight-step qualitative content analysis process. In this article I discuss the similarities and differences between the qualitative content analysis method and other qualitative methods from a quality perspective; specifically, the total quality framework.

Key words: qualitative content analysis; qualitative research; qualitative methods; quality criteria; total quality framework

Scholarly discourse about what it means to collect and analyze qualitative data is a dynamic discussion in the qualitative community. At the center of this discourse is the shared understanding that qualitative research involves the examination of nuanced connections, along with the social and contextual dimensions, that give meaning to qualitative data. Qualitative researchers strive to discover these nuanced connections and contextual dimensions with all methods, and most assuredly with qualitative content analysis (QCA) (ELO & KYNGÄS, 2008; GRANEHEIM & LUNDMAN, 2004; HSIEH & SHANNON, 2005; LATTER, YERRELL, RYCROFT-MALONE & SHAW, 2000; SCHREIER, 2012; TOWNSEND, AMARSI, BACKMAN, COX & LI, 2011). Yet, in every instance, qualitative researchers are presented with the challenge of conceptualizing and implementing research designs that result in rich contextual data, while also incorporating principles of quality research to maximize the discovery of valid interpretations that lead to the ultimate usefulness (i.e., the "so what?") of their research. [1]

In this article I discuss what makes QCA similar to and different from other qualitative research methods from the standpoint of a quality approach. In order to establish the basis from which quality concerns can be discussed, I begin with defining the QCA method (Section 2) and, in so doing, identifying the fundamental similarities and differences between QCA and other methods (Section 3) from the perspective of the ten unique attributes of qualitative research (ROLLER & LAVRAKAS, 2015). With this as a foundation, I continue with a brief contextual discussion of a quality approach to qualitative research and the QCA method (Section 4), followed by an introduction to one such approach, i.e., the total quality framework (TQF) (ibid.), in which I give researchers a way to think about quality design throughout each phase of the qualitative research process (Section 5). With these preparatory sections—defining and contrasting the QCA method with other qualitative methods, discussing quality approaches, and a brief description of the TQF approach—I lay the necessary groundwork for a meaningful discussion of the similarities and differences when adapting the TQF to the QCA method, which is my focus with this article (Section 6). [2]

In order to compare and contrast QCA with other qualitative methods, it is imperative to begin with a definition—What is QCA? There are many definitions of QCA from various researchers, ranging from those that adhere to a narrow view that QCA is "a systematic quantitative description of the manifest content" (KVALE & BRINKMANN, 2009, p.203) to those who define QCA in terms of the subjective understanding of "patterns, themes, and categories important to a social reality" (ZHANG & WILDEMUTH, 2009, p.322). For the purposes of the current discussion, the QCA method is defined as "the systematic reduction of content, analyzed with special attention to the context in which it was created, to identify themes and extract meaningful interpretations of the data" (ROLLER & LAVRAKAS, 2015, p.232). The reference to "content" in this definition is not unlike MAYRING's inclusion of "all sort of recorded communication" (2000, §4) in that the "content" in QCA embraces all appropriate data sources, moving beyond text to include images, video, audio, graphics, and symbols (KUCKARTZ, 2014; MACNAMARA, 2005; MAYRING, 2000; SCHREIER, 2012). The reference to "context" in this definition pertains to the idea that "useful claims in content analysis require contextual understanding" (BOCK, 2009, p.40), emphasizing the idea that "textual units are rarely ever entirely independent of each other" and that "words may have many meanings" (KRIPPENDORFF & BOCK, 2009, p.44). For the purpose of this article, I interpret context as "the juxtaposition of words, substance, and 'broader environment' of the content" (ROLLER & LAVRAKAS, 2015, p.4). [3]

An important and more granular definition of the QCA method encompasses the exact nature of the method; specifically, the two-phase, eight-step process associated with conducting a QCA, as shown in Figure 1.

Figure 1: Phases and steps in qualitative content analysis (p.235). Please click here for an enlarged version of Figure 1. [4]

In Phase 1, the QCA researcher is in essence creating the data that will be analyzed in Phase 2. Unlike in-depth interview (IDI) or focus group data (text, images, video, audio) that are gathered directly from the research participants, the data gathered in the QCA method are the codes developed from IDI, focus group, observational, media, or other content. It is this data that is created in Phase 1 of the QCA method that is analyzed in the second phase (Phase 2). Underlying both phases (i.e., the data generation and data analysis processes), however, is the uniformly critical task of clearly defining the research objective and identifying the constructs to measure. This is true of all qualitative research methods, no less so in the QCA method where the researcher's intent is on matching the constructs of interest to the study objectives, while remaining open to objective-related data that may fall outside the realm of the presumed constructs. It should be noted that these phases of the QCA method remain the same regardless of whether the researcher is conducting QCA as a "primary method" or a "secondary method" (p.241). As a primary method, QCA research is one where the researcher is analyzing naturally occurring data sources such as media accounts of the news, films, and historical documents. As a secondary method, QCA is a study where the researcher's analysis is directed at data derived from other qualitative methods, such as IDIs and focus group discussions. [5]

As stated in the aforementioned definition, inherent to the QCA method is the importance of context and meaning. These are central characteristics of all qualitative methods and, indeed, context and meaning are two of the ten unique attributes of qualitative research identified by ROLLER and LAVRAKAS (2015) (see Table 1).

Table 1: The ten unique attributes of qualitative research (see pp.4-9 for a discussion of the ten unique attributes of qualitative research) [6]

In addition to context and meaning, QCA has many other unique attributes similar to other qualitative methods, such as the "absence of 'truth'" (i.e., the influence of many factors, including subjectivity, in qualitative data, Table 1), the unique skill set required of the researcher, the distinctive types of research questions that can be addressed and the online and mobile modes that can be used to address them, the flexibility built into the research designs, and the multi-faceted often intricate ("messy") analysis process. [7]

There are, however, two important distinctions between QCA and other qualitative methods. One is the attribute associated with the participant-researcher relationship. Unlike other methods where the relationship between the participant and researcher plays a fundamental role in data gathering (not only in the act of obtaining data but also in the integrity of the data collected), the QCA researcher is distanced from the research participants and is working with content (i.e., the text transcripts or other documents, images, video, graphics, or symbols) obtained at an earlier point in time. Therefore, from a quality perspective, potential bias from the participant-researcher relationship is embedded in the content to be analyzed by the QCA method. As such, the ways in which the participant-researcher relationship may have jeopardized the integrity of the content to be coded is not of concern in QCA. [8]

Another similar yet important distinction of the QCA method is the researcher-as-instrument attribute. The distinction lies in the fact that the QCA researcher is not the "instrument" collecting data from participant interviews, group discussions, or observations but rather the "instrument" creating the codes from the content that act as the data for analysis. In this regard, it is the coders in a QCA study that are the research instruments (FORMAN & DAMSCHRODER, 2008). It should be noted, however, that regardless of how the "instrument" role is defined, the fact remains that a potential for researcher-as-instrument bias is a shared concern in all qualitative methods (GUBA & LINCOLN, 1982; MERRIAM, 2009; MORROW, 2005). The relevance of this discussion on the similarities and differences of the QCA method compared to other qualitative methods with respect to the unique attributes will become clear in subsequent discussions of quality and specifically the total quality framework, which is the focus of this article. [9]

A quality approach to research design in qualitative methods is not new. Verification strategies, the role of reliability and validity, and attention to "qualitative rigor" are well-documented in the literature (ATKINSON & DELAMONT, 2006; BERG & LUNE, 2012; BRINKMANN & KVALE, 2015; CRESWELL, 2013; FLICK, 2019; LINCOLN & GUBA, 1985; MARSHALL & ROSSMAN, 2011; MAXWELL, 2013; MERRIAM, 2009; MORSE, BARRETT, MAYAN, OLSON & SPIERS, 2002; POLKINGHORNE, 2007; RIESSMAN, 2008; SEALE & SILVERMAN, 1997). More recently, LEVITT, MOTULSKY, WERTZ, MORROW and PONTEROTTO proposed a quality model based on the concept of "methodological integrity," which they associate with "the methodological foundation of trustworthiness" (2017, p.9). Importantly, LEVITT et al. do not advocate for a quality approach that is confined by set procedures, but rather a context-driven way of thinking about the methodological integrity (i.e., whether the research goals and the researcher's world view are supported and whether the design and processes are tailored to the research topic and investigators) related to the data collection and data analysis components of qualitative research design. [10]

A quality approach specific to the QCA method—as opposed to a quality orientation within the quantitative paradigm (KRIPPENDORFF, 2013)—has been put forth by a number of researchers. For instance, GRANEHEIM and LUNDMAN (2004) discuss the trustworthiness of QCA research, leaning on the familiar concepts of credibility, dependability, and transferability made popular by LINCOLN and GUBA (1985). Similarly, ZHANG and WILDEMUTH (2009) discuss the trustworthiness of the QCA method as defined by LINCOLN and GUBA (1985) and include the fourth criterion of confirmability. And, as a final example of how researchers have employed quality standards to the QCA method, FORMAN and DAMSCHRODER (2008) focus on issues of credibility, validity, and reliability throughout a QCA study, e.g., how memos add credibility to the research, how team coding establishes content validity as well as coding reliability, and how the examination and reporting of "negative cases" instills credibility in the findings. [11]

With a few exceptions, a discussion of a quality approach to the QCA method as a way to think about and incorporate quality principles at each phase of the research process has been lacking in the literature. ELO et al. (2014), for example, offer a checklist to improve the trustworthiness of a QCA study at each of three phases, i.e., the preparation, organization, and reporting phases. Also, in his discussion of the internal quality standards associated with qualitative text analysis, KUCKARTZ (2014) outlines essential questions covering a broad scope of the research process, including the selection of method, coding, category development, consideration of outliers (i.e., "any unusual or abnormal cases," p.154), and justification of the conclusions. And, as a third example of researchers who have looked at quality in QCA throughout the process, SCHREIER discusses the importance of reliability and validity in QCA at length, instructing researchers to

"make sure that you have gone about your research in a systematic way, that you make your procedure and your reasoning transparent to your readers, that your design and your method are appropriate to your research question, [and] that you have taken negative cases and alternative interpretations into account" (2012, p.27). [12]

By considering quality standards at each step in the research design, the researcher acknowledges that a quality qualitative research design is only "as strong as its weakest link"; meaning, for example, that a deliberate quality approach to data collection and analysis yet a failure to write a quality transparent final document, effectively masks the integrity of the research and undermines its ultimate value. A holistic quality-centric approach to qualitative research design and, specifically to the QCA method, is my focus in this article. This approach—the total quality framework (ROLLER & LAVRAKAS, 2015)—is introduced and discussed in the remaining sections, with particular attention paid to the similarities and differences between QCA and other qualitative methods when applying this framework. [13]

The TQF is a holistic approach, in which the researcher considers the entire research process and which can be applied to each research phase regardless of the qualitative research method. There are several very basic concepts that are fundamental to the purpose of the TQF. At its core, the TQF is built on the belief that if "qualitative research can, in fact, be used for worthwhile ('good') purposes, then logically this would only be to the degree that it is used ('executed') well" (p.20). This may seem like a simplistic and basic idea on the surface; however, with the exceptions discussed earlier, it is an idea that is often overlooked in the discussions of qualitative research generally and QCA design specifically. [14]

A second concept that is fundamental to the purpose of the TQF is the idea that a quality approach to qualitative research design does not need to, and should not, stifle or otherwise compromise the essence of qualitative research. It is this belief that defines a key goal of the researcher who is using the TQF, which is to bring greater rigor to qualitative research without diminishing the role of context and meaning, which are equally important in all qualitative methods (GUBA & LINCOLN, 1982; MORROW, 2005; THOMAS & MAGILVY, 2011). The TQF is thus a tool with which qualitative researchers can develop critical thinking skills by way of giving particular attention to quality issues related to the conceptualization, implementation, analysis, and reporting of their qualitative studies for any specific method, including the QCA method. In doing so, with the framework researchers can: 1. critically examine the possible sources of variability and bias in their qualitative methods, 2. incorporate features into their designs that attempt to mitigate these effects, 3. acknowledge and take the implications of these effects into consideration during analysis, and 4. identify the strengths and limitations of qualitative research studies (their own and those of other researchers). [15]

The third fundamental concept underlying the TQF is what defines it. The framework is rooted in the belief that qualitative research, regardless of the particular method, must be credible, analyzable, transparent, and useful. This basic understanding of qualitative research defines the four interconnected components of the framework—credibility, analyzability, transparency, and usefulness (Figure 2).

The credibility component of the TQF is relevant for the data collection phase of the research design. There are two aspects associated with data collection (credibility): Scope and data gathering. [17]

Scope is relevant for the coverage and representation of the sampled elements to the overall population, and is an important consideration in all qualitative research designs. In IDIs, focus groups, and ethnography, scope pertains to sampling, coverage, sample size (number of participants, observable events), and cooperation, addressing such issues as where the researcher will find participants, how participants will be sampled for possible inclusion in the research, how many participants are needed, and how the researcher will recruit participants and gain cooperation. With respect to sampling, for example, the researcher applying a TQF approach to the research design would consider the completeness and accuracy of sample lists (if applicable), as well as the representativeness of the sample as a result of the sampling process. [18]

QCA is similar to other methods in that the researcher must carefully define the target population of study, the method by which elements of the population will be sampled, and the techniques the researcher will use to maximize the inclusion of sample members in order to mitigate potential bias (BROD, TESLER & CHRISTENSEN, 2009; MAYS & POPE, 1995; SHENTON, 2004). These considerations are no different when conducting QCA as a primary method (i.e., the analysis of naturally occurring data sources such as diaries and historical documents) or as a secondary method (i.e., the analysis of data derived from other qualitative methods, such as IDIs and focus group discussions). [19]

As in many, if not most, qualitative studies, the ability to sample from the entire population of interest (in text, images, video, or audio format) is often not possible when utilizing the primary QCA method. This may be for any number of reasons, including difficulty accessing the content as well as limited resources (i.e., staff, time, and financial considerations). BEEMAN (2007), for example, used the primary QCA method to examine interracial relationships as depicted in films made in the United States between 1980 and 2001. She clearly did not have access or the resources to analyze all films in this time period and, therefore, relied on a sample design. To that end, BEEMAN used a sampling approach involving the selection of 40 films from a "comprehensive" (p.696) and well-respected movie guide. BEEMAN's careful choice of a sample source is not unlike the systematic selection of newscasts by ALTHEIDE (1987) to study the Iranian hostage crisis, both of which are examples of a quality approach consistent with the facet of scope in the framework's credibility component. [20]

The construct of sample representation is also relevant when conducting QCA as a secondary method, i.e., when using QCA to analyze interview, focus group, or ethnographic data. For instance, one of the first steps in the analysis process is to identify the unit of analysis (Step 2, Figure 1). From a TQF standpoint, the decision of how to define the unit of analysis is an important one because it serves as the basis for the development of codes. Too narrow of a unit of analysis (e.g., sentences) is unlikely to represent the full context and meaning in the content, resulting in a limited understanding of the data; and a unit of analysis that is too broad (e.g., the entire set of diaries or interviews in an in-depth interview study) will underrepresent individual differences in the data (GRANEHEIM & LUNDMAN, 2004). The appropriate unit of analysis for each QCA study may vary; however, the recommendation when using TQF is to use a unit of analysis that is sufficiently broad to give the researcher contextual understanding of the data (e.g., an entire interview). [21]

Similar to other qualitative methods, weak representation or coverage of the content under investigation in a QCA study leads to non-coverage bias, which seriously jeopardizes the integrity of the data. If, for example, the researcher had access to only a portion of the diaries in a multi-mode in-depth interview study, the final interpretations and conclusions reached by the researcher would be seriously in doubt. This bias and the dubious findings arising from this research are grounded in the fact that the researcher has no way of knowing if the inaccessible content is different from the content that is accessible in aspects that are important to the research objective. [22]

In addition to scope, the other broad facet of the TQF credibility component is data gathering. Data gathering concerns the steps that are taken to actually integrate best practices in the collection of qualitative data. For the IDI, focus group, and ethnography methods, this includes the care that is given to defining the relevant constructs and associated attributes, the choice of mode(s), the instrument tool (e.g., guide) development, the researcher's consideration of the sources of interviewer or moderator or observer bias (i.e., researcher effects), and the researcher's consideration of participant effects (i.e., having to do with the participant's ability and willingness to engage with the researcher's questions or observations). [23]

For the QCA method, data gathering pertains to Phase 1 (Figure 1) of the process, when the researcher creates the data to be analyzed by coding the content (e.g., existing documents or videos [as in the primary QCA method] or interview or focus group discussion transcripts [as in the secondary QCA method]). As such, quality concerns pertaining to the QCA method are distinctively focused on how the researcher finds meaning in and codes the content, not with how the content was created (e.g., the researcher is not reflecting on how the interviewer may have biased a set of content from an IDI study). A quality approach to this data generation phase of the QCA method is not unlike other qualitative research methods in that great care is taken to mitigate researcher effects. Similar to the in-depth interview, focus group discussion, or observational methods, the integrity of the data gathered in the QCA method—which, in the QCA method, is the coding of content—depends on the validity of the coding process. [24]

In using a TQF approach to data gathering in the QCA method, the researcher strives to maximize the accuracy of the final coding—i.e., Does the coding clearly reflect the manifest and latent meanings within the content?—while also minimizing researcher effects that weaken or corrupt this data. For example, researcher inconsistency (i.e., introducing variability in the data beyond the natural variability of the phenomena being studied) and bias are potential threats to data quality regardless of the qualitative research method (MILES & HUBERMAN, 1984; ONWUEGBUZIE & LEECH, 2007; SCHREIER, 2012; WHITLEY & CRAWFORD, 2005). In the QCA method, researcher effects are reduced (thereby enhancing data quality) by considering the consistency of coding among coders during the preliminary coding phase (Step 4, Figure 1) as well as the final coding of content (Step 5). The consistency, or inter-coder reliability, associated with coding can be assessed in several ways. There are, for instance, various statistical calculations that can be used to measure inter-coder reliability, such as COHEN's (1960) kappa coefficient and KRIPPENDORFF's (2013) alpha, with the acceptable minimum consistency typically deemed to be in the range of 75-80%. From the perspective of researchers who are using the TQF, however, these statistical measures are inappropriate for qualitative research and the QCA method because these calculations are grounded in underlying assumptions suitable for quantitative methods and statistical formulas based on discrete items. This is in striking contrast to qualitative research and the QCA method, where the purpose is to investigate the interrelatedness and nuanced meanings of the content. For this reason, a TQF approach to achieving consistency in QCA is not to rely on statistical coefficients, but rather procedures that are centered on consensus building among the research team (FONTEYN, VETTESE, LANCASTER & BAUER-WU, 2008; FORMAN & DAMSCHRODER, 2008). [25]

The development of a codebook that is complete and accurate, as well as flexible (i.e., can be modified as necessary throughout the coding process to account for new discoveries), is essential to maintaining consistent coding (and high inter-coder reliability) and mitigating bias in the data gathering process (FONTEYN et al., 2008). A researcher using the TQF approach to content coding, however, goes beyond a codebook to also include a coding form (Table 2). Coders can use the coding form to help them stay mindful of the key constructs related to the research objectives, while also enabling coders to record their thoughts and reflect on their interpretations of content and its relevance to the constructs. In this way, the TQF coding form is a type of memoing or reflexive journal which is not only valuable when monitoring the coding process but also an important asset during analysis and in the reporting document, allowing the user of the research "to assess the link between raw data and analyses" (AKKERMAN, ADMIRAAL, BREKELMANS & OOST, 2006, p.267).

Table 2: Example of a codebook and coding form. Click here to download the PDF file. [26]

Another important element for researcher consistency and reducing bias is training (NEUENDORF, 2016). The importance of training is not unique to the QCA method and, indeed, training is a basic necessity in all qualitative methods. For the QCA method, it is recommended that the researcher using the TQF conduct a thorough five-step coder training curriculum, including 1. introduction to the study and content; 2. detailed explanation of the codebook and coding form; 3. practice working with the codebook and coding form; 4. use of a reflexive journal; and 5. other project logistics. Allocating adequate resources (i.e., staff, time, and money) to coder training is essential to maximizing the integrity of the data gathering process in the QCA method. [27]

In addition to, and in conjunction with, coder training, it is critically important for the lead researcher to continually monitor the coding of each coder to review the coding accuracy. This is because there may be a high degree of coding consistency for a particular coder or across coders (i.e., high intra-coder or inter-coder reliability) yet errors in the coding decisions being made. From a quality perspective, these inaccuracies weaken the validity of the coding and therefore the data being generated in the data gathering process (SCHREIER, 2012). The regular monitoring of coders in the QCA method is not unlike the monitoring and supervision of interviewers, moderators, and observers that should occur in the context of other qualitative research methods. [28]

The second essential component of the TQF is analyzability, in which the researcher considers how well the analysis is conducted and, specifically, the procedures that were used to conduct a thorough and transparent interpretation of the data. There are two facets of the analyzability component—processing and verification. [29]

For the in-depth interview and focus group methods, processing includes the creation of transcripts which, when using the TQF, involves the careful selection of a transcriptionist, including the particular questions the researcher should consider before assigning someone the job of transcribing the data. For the QCA method, processing involves the data analysis phase (i.e., Phase 2). As shown in Figure 1, Phase 2 of the QCA method is comprised of three steps: identifying categories across codes (Step 6 in the overall QCA process), identifying themes/patterns across categories (Step 7), and drawing interpretations and implications (Step 8). Steps 6 and 7 pertain to the processing facet of the TQF analyzability component. Not unlike processing in other qualitative research methods, researchers conducting QCA identify categories by looking for groups of codes that, in conjunction with their corresponding manifest and latent content, share an underlying construct. A TQF approach to category development is the same regardless of whether the researcher is working on a primary QCA (i.e., analyzing codes derived directly from the primary source, such as historical documents) or a secondary QCA (i.e., analyzing codes derived from the content of another qualitative method, such as in-depth interview or focus group transcripts). In either case, category development consists of four stages:

The next step in the processing of QCA data (Step 7, Figure 1) is to identify themes or patterns across the categories derived in Step 6. This is what GRANEHEIM, LINDGREN and LUNDMAN described as finding "a unifying 'red thread' running through several categories that brings meaning to a recurrent topic or experiences and its various manifestations" (2017, p.32). This step in the analysis process is facilitated by the researcher's use of data visualization, by which they can see all the categories to discover patterns. There are various options for creating a data visualization of categories. One option is to create a table or worksheet (e.g., using a program such as Word or Excel) and place the categories as column headers with each case in the QCA study (e.g., each public service announcement, each diary, each in-depth interview participant) representing a row. Another option is to create a "concept map" as SCHILLING (2006) did to visualize categories associated with "positive leadership" using PowerPoint. A third option for the data visualization of categories in QCA is the use of computer-assisted qualitative data analysis software (CAQDAS). The many available CAQDAS programs are complete with convenient methods for visualizing categories and identifying connections that translate into themes, such as the ability to label, organize, search, sort, and choose a particular graphical display. One such example is from FRIESE (2013), who used ATLAS.ti for a study on happiness. [31]

At the completion of deriving themes (i.e., Step 7), a recommendation within TQF for the processing of QCA data is for the researcher to take a break from the analysis and return at a later time (i.e., in a day or two) to look again at the data. By taking a "time out" in the analysis process, the researcher is conducting an exercise in reflexivity that enables the researcher to reflect anew on the data (SCHREIER, 2012). In doing so, the researcher should re-analyze the derived categories and look for themes or patterns without reference to those discovered earlier. [32]

In addition to processing, the analyzability component of the TQF includes verification. Verification is about drawing interpretations and implications from the data and pertains to Step 8 of the overall QCA process, which is the third and final step in the data analysis phase (Figure 1). Verification may not encompass a distinctly separate dimension from processing and, indeed, may be conducted in conjunction with the processing of the data. In verification of qualitative research data, the researcher must look beyond and look differently at the key categories and themes that are emerging from the processing steps of the analysis in order to consider alternative explanations of the data. In doing so, the researcher attempts to discover new evidence in the data that supports and/or refutes preliminary interpretations. Verification is an essential analytical step that acts as "a self-correcting mechanism to ensure the quality of the project" (MORSE et al., 2002, p.14) regardless of the research method, i.e., in QCA as well as in-depth interviews, focus groups discussions, and ethnography. [33]

There are various techniques or approaches that the researcher can take in the verification process. Four broad forms of verification are peer debriefings, triangulation, deviant cases, and the reflexive journal. The first three—debriefings, triangulation, and deviant cases—are particularly relevant to QCA. For example, peer debriefings are especially important in the QCA method where the integrity of the data generation and analysis rests on the understanding of both manifest and latent content. It is only by way of continual monitoring and ongoing discussions with the research team that a range of interpretations can be identified and a consensus derived around the final understanding of the research findings (GRANEHEIM et al., 2017). Ideally, a peer debriefing for a QCA study would entail an independent review in which colleagues who were not involved in the earlier steps of the QCA process are given the data and reflective components (e.g., the coding forms), as well as the derived categories and themes, and asked to provide their own interpretations. At the completion of the reviews, the researcher should meet with reviewers (individually or as a group) to discuss their understanding of the data, along with their reasons for supporting or contradicting the researcher's (or research team's) initial interpretations. [34]

Similar to all qualitative research methods, triangulation is another critical verification approach in the QCA method. Triangulation

"uses multiple sources to contrast and compare study data to establish supporting and/or contradictory information to ultimately give the researcher and users of the research a more balanced and deeper understanding of the outcomes than relying on the study data alone" (ROLLER & LAVRAKAS, 2015, p.364). [35]

Investigator triangulation, i.e., comparing the independent results from multiple researchers, is not unlike peer debriefings and reviews discussed earlier. Method triangulation, e.g., comparing primary QCA study data with data obtained from a different method such as in-depth interviews or observations, is another useful form of triangulation in the QCA method. Researchers using a TQF approach, however, do not utilize a method for comparison that is quantitative in nature such as that espoused by KOHLBACHER (2006). In data triangulation, QCA data and initial interpretations are compared to data obtained from a different source, such as documents related to the topic but outside the scope of the current QCA study. Regardless of the particular approach or whether these comparisons support or contradict the preliminary findings, researchers can use triangulation to solidify their final interpretations and implications, as well as the credibility of the study overall. [36]

Deviant cases, or negative case analysis (CRESWELL, 2013; LINCOLN & GUBA, 1985), is a verification approach that involves the deliberate identification and examination of QCA data that does not conform to the researcher's preliminary interpretations. By looking at the outliers, i.e., the data situated outside of or in contradiction to the prevailing themes, the researcher may find evidence that casts doubt on or helps to support the initial findings. Verification by way of deviant cases is often overlooked by researchers who focus too closely on data that is consistent with their main interpretations, ignoring the negative cases that can be used to shed important light on the analysis. [37]

The Transparency component of the TQF focuses on the third broad phase of the qualitative research process, i.e., reporting. This component of the framework is concerned with how well and how thoroughly the researcher has discussed the details of the study, and is built around the idea that "[i]n order for the outcomes of qualitative research to be valued, the strategies used must be transparent and open to scrutiny" (MARTIN, SADLO & STEW, 2006, p.198). It is within the Transparency component that the researcher considers the concept of "thick description," i.e., "[a] complete account in the final research document of the phenomena under investigation as well as rich details of the data collection and analysis processes and interpretations of the findings" (ROLLER & LAVRAKAS, 2015, p.363). [38]

These details might include the design considerations, the sampling and coverage of the target content, the data collection procedures and in-the-field realities of design implementation, and the analytical process. In the TQF, the researcher's goal with the final document should be to provide the user of the document enough information about the research to provide something of value, such as the ability to arrive at similar conclusions or transfer the design and/or findings to a separate but comparable context. Although the inclusion of an exhaustive list of study details may be impractical, researchers using a TQF approach to qualitative design are encouraged to consider all of these details in their reporting phase, and incorporate as many of them as feasible, regardless of whether the final document is to be used internally or for publication. [39]

There are many types of details to be included in the thick description of a QCA study that are not unlike those that would be reported for other qualitative methods, e.g., the research objectives and primary research questions, justification for the chosen method, sampling and coverage of the target population. There are, however, study details unique to the QCA method that should be included (or seriously considered for inclusion) in the final reporting document:

Each of the three TQF components—credibility, analyzability, and transparency—feed into the fourth and final component, i.e., usefulness. Usefulness is the "so what?" component of the framework and, not unlike all research, it pertains to the ultimate objective of allowing users of the research to do something of value with the findings. When a TQF approach to a qualitative study is used to integrate best practices at each phase of the research process, the user is able to address questions of usefulness such as "How confident am I that these findings allow me to form certain hypotheses, or offer me viable next steps, or can be transferred to another context (i.e., transferability)?" It is these considerations that define the usefulness component. [41]

It is because usefulness relies on the other three TQF components that it is only achieved to the extent that the researcher adopted quality principles associated with these first three phases of the research. In a QCA study, the ultimate usefulness of the research rests on the researcher's attention to, not only best practices associated with all qualitative methods but also, the unique facets of the QCA method (FORMAN & DAMSCHRODER, 2008; GRANEHEIM & LUNDMAN, 2004; ZHANG & WILDEMUTH, 2009). This includes the careful consideration given to: 1. Credibility issues such as the content sampling (as necessary), the unit of analysis selected, code development, and the coding process; 2. Analyzability processes such as the completion and review of ancillary materials, e.g., such as coding forms and reflexive journals, the identification of categories and themes, and the verification strategies; and 3. Transparency in the final document and the inclusion of important details associated with the conceptualization and implementation of the QCA method, as well as the interpretation of QCA data. [42]

QCA is a method with many similar attributes associated with all qualitative research methods. Similar to other methods, the QCA method is grounded in the importance of context and meaning, as well as the absence of "truth" and other unique attributes of a qualitative approach. I proposed here that these shared attributes should appropriately extend to a key consideration in all qualitative research designs, i.e., the integration of quality concepts at each step of the research process, while maintaining the contextualized and often creative approaches that are fundamental to qualitative research. In this article I discussed the total quality framework, which is a quality approach to qualitative research in which the researcher strives to enhance the integrity of the research effort without stifling the essence of what it means to conduct qualitative research. When applied to the QCA method, researchers using the framework have a tool for thinking about quality-related constructs not unlike those associated with other qualitative methods. This includes scope and sampling, i.e., the manner in which the researcher targets and samples the population of interest; researcher effects due to inconsistency and bias; reflexivity and verification strategies; and a thick description of the research in the final document.

Table 3: Total quality framework design considerations—similar and unique QCA considerations compared to other methods. Click here to download the PDF file. [43]

In Table 3, I have summarized the many similarities between how the researcher might use the TQF to think about a quality approach to a QCA study, as well as the considerations distinctive to QCA. One of these considerations has to do with what is considered "data" in data gathering. Unlike other qualitative methods that create data from interviews, discussions, and observations, the QCA method is uniquely disassociated from how the content was created, and "data" is instead defined as the coded content generated by the analyst. As such, researchers using the TQF approach to QCA focus on the quality strategies that were used to find meaning in and the coding of content, not how the content was created. For this reason, the researcher effects that the QCA researcher hopes to mitigate are associated with the accuracy of the coding. [44]

Other unique considerations associated with the QCA method are a byproduct of the two-phase, eight-step process, such as the four distinct stages in category development conducted as part of the TQF analyzability-processing component. In this process the TQF transparency component is also uniquely defined by the QCA-specific details the researcher must track and report on concerning the decisions, procedures, techniques, and reflections that emerged in the course of the QCA process. [45]

QCA is an increasingly useful and important method, particularly as qualitative approaches continue to widen and include a range of content formats, e.g., text, audio, video, photographs, graphic images. Whether it is being conducted as the primary method or as the secondary method in support of the IDI, focus group, or ethnography method, QCA is at the core of qualitative research. The central role of the QCA method mandates a way to think about quality principles associated with QCA design and implementation, in order to maximize the integrity and ultimate usefulness of the outcomes. [46]

I want to thank my co-author, Paul LAVRAKAS, for his collaboration on the total quality framework generally and its relevance to the QCA method specifically. I also want to thank the anonymous reviewers of FQS, as well as the guest editors of this special issue, for their constructive feedback on early versions of this article and for their important contribution to the development and finalization of this article.

Akkerman, Sanne; Admiraal, Wilfried; Brekelmans, Mieke & Oost, Heinze. (2006). Auditing quality of research in social sciences. Quality & Quantity, 42(2), 257-274.

Altheide, David L. (1987). Ethnographic content analysis. Qualitative Sociology, 10(1), 65-77.

Atkinson, Paul & Delamont, Sara (2006). Rescuing narrative from qualitative research. Narrative Inquiry, 16(1), 164-172.

Beeman, Angie K. (2007). Emotional segregation: A content analysis of institutional racism in US films, 1980-2001. Ethnic and Racial Studies, 30(5), 687-712.

Berg, Bruce L. & Lune, Howard (2012). Qualitative research methods for the social sciences (8th ed.). Boston, MA: Pearson.

Bock, Mary Angela (2009). Impressionistic content analysis: Word counting in popular media. In Klaus Krippendorff & Mary Angela Bock (Eds.), The content analysis reader (pp.38-41). Thousand Oaks, CA: Sage.

Brinkmann, Svend & Steinar, Kvale (2015). Interviews: Learning the craft of qualitative research interviewing (3rd ed.). Thousand Oaks, CA: Sage.

Brod, Meryl; Tesler, Laura E. & Christensen, Torsten L. (2009). Qualitative research and content validity: Developing best practices based on science and experience. Quality of Life Research, 18(9), 1263-1278.

Christensen, Pia; Mikkelsen, Miguel R.; Nielsen, Thomas A.S. & Harder, Henrik (2011). Children, mobility, and space: Using GPS and mobile phone technologies in ethnographic research. Journal of Mixed Methods Research, 5(3), 227-246.

Cohen, Jacob (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.

Creswell, John W. (2013). Qualitative inquiry and research design (3rd ed.). Thousand Oaks, CA: Sage.

Elo, Satu & Kyngäs, Helvi (2008). The qualitative content analysis process. Journal of Advanced Nursing, 62(1), 107-115.

Elo, Satu; Kääriäinen, Maria; Kanste, Outi; Pölkki, Tarja; Utriainen, Kati & Kyngäs, Helvi (2014). Qualitative content analysis: A focus on trustworthiness. SAGE Open, 4(1), 1-10, https://doi.org/10.1177%2F2158244014522633 [Date of Access: May 7, 2019].

Flick, Uwe (2019). An introduction to qualitative research (6th ed.). Thousand Oaks, CA: Sage.

Fonteyn, Marsha E.; Vettese, Margaret; Lancaster, Diane R. & Bauer-Wu, Susan (2008). Developing a codebook to guide content analysis of expressive writing transcripts. Applied Nursing Research, 21(3),165-168.

Forman, Jane & Damschroder, Laura (2008). Qualitative content analysis. In Liva Jacoby & Laura A. Siminoff (Eds.), Empirical methods for bioethics: A primer (pp.39-62). Oxford: Elsevier.

Friese, Susanne (2013). Proposing a new method for computer-assisted qualitative data analysis. IIQM Master Class Webinar, ATLAS.ti, March 15, https://www.youtube.com/watch?v=2N3bwa5Lt94 [Date of Access: June 5, 2013].

Graneheim, Ulla H. & Lundman, Berit (2004). Qualitative content analysis in nursing research: Concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24(2), 105-112.

Graneheim, Ulla H.; Lindgren, Britt Marie & Lundman, Berit (2017). Methodological challenges in qualitative content analysis: A discussion paper. Nurse Education Today, 56, 29-34.

Guba, Egon G. & Lincoln, Yvonna S. (1982). Epistemological and methodological bases of naturalistic inquiry. Educational Communication and Technology, 30(4), 233-252.

Hsieh, Hsiu-Fang & Shannon, Sarah E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277-1288.

Kohlbacher, Florian (2006). The use of qualitative content analysis in case study research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 7(1), Art. 21, http://dx.doi.org/10.17169/fqs-7.1.75 [Date of Access: April 15, 2013].

Krippendorff, Klaus (2013). Content analysis (3rd ed.). Thousand Oaks, CA: Sage.

Krippendorff, Klaus & Bock, Mary Angela (2009). Unitizing and sampling. In Klaus Krippendorff & Mary Angela Bock (Eds.), The content analysis reader (pp.43-44). Thousand Oaks, CA: Sage.

Kuckartz, Udo (2014). Qualitative text analysis: A guide to methods, practice and using software. London: Sage.

Kvale, Steinar & Brinkmann, Svend (2009). Interviews: Learning the craft of qualitative research interviewing (2nd ed.). Thousand Oaks, CA: Sage.

Latter, Sue; Yerrell, Paul; Rycroft-Malone, Joanne & Shaw, David (2000). Nursing, medication education and the new policy agenda: The evidence base. International Journal of Nursing Studies, 37(6), 469-479.

Levitt, Heidi M.; Motulsky, Sue L.; Wertz, Fredrick J.; Morrow, Susan L. & Ponterotto, Joseph G. (2017). Recommendations for designing and reviewing qualitative research in psychology: Promoting methodological integrity. Qualitative Psychology, 4(1), 2-22.

Lincoln, Yvonna S. & Guba, Egon G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

Marshall, Catherine & Rossman, Gretchen B. (2011). Designing qualitative research. Thousand Oaks, CA: Sage.

Martin, Marion; Sadlo, Gaynor & Stew, Graham (2006). The phenomenon of boredom. Qualitative Research in Psychology, 3(3), 193-211.

Maxwell, Joseph A. (2013). Qualitative research design: An interactive approach (3rd ed.). Thousand Oaks, CA: Sage.

Merriam, Sharan B. (2009). Qualitative research: A guide to design and implementation. San Fransicso, CA: Jossey-Bass.

Miles, Matthew B. & Huberman, A. Michael (1984). Drawing valid meaning from qualitative data: Toward a shared craft. Educational Researcher, 13(5), 20-30.

Morrow, Susan L. (2005). Quality and trustworthiness in qualitative research in counseling psychology. Journal of Counseling Psychology, 52(2), 250-260.

Morse, Janice M.; Barrett, Michael; Mayan, Maria; Olson, Karin & Spiers, Jude (2002). Verification strategies for establishing reliability and validity in qualitative research. International Journal of Qualitative Methods, 1(2), 13-22, https://doi.org/10.1177/160940690200100202 [Date of Access: October 20, 2012].

Neuendorf, Kimberly A. (2016). The content analysis guidebook. Thousand Oaks, CA: Sage.

Onwuegbuzie, Anthony J. & Leech, Nancy L. (2007). Validity and qualitative research: An oxymoron?. Quality & Quantity, 41(2), 233-249.

Polkinghorne, Donald E. (2007). Validity issues in narrative research. Qualitative Inquiry, 13(4), 471-486.

Riessman, Catherine K. (2008). Narrative methods for the human sciences. Thousand Oaks, CA: Sage.

Roller, Margaret R. & Lavrakas, Paul J. (2015). Applied qualitative research design: A total quality framework approach. New York, NY: Guilford Press.

Schilling, Jan (2006). On the pragmatics of qualitative assessment. European Journal of Psychological Assessment, 22(1), 28-37.

Schreier, Margrit (2012). Qualitative content analysis in practice. London: Sage.

Seale, Clive & Silverman, David (1997). Ensuring rigour in qualitative research. European Journal of Public Health, 7(4), 379-384, https://doi.org/10.1093/eurpub/7.4.379 [Date of Access: October 21, 2012].

Shenton, Andrew K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22(2), 63-75.

Thomas, Eileen & Magilvy, Joan K. (2011). Qualitative rigor or research validity in qualitative research. Journal for Specialists in Pediatric Nursing, 16(2), 151-155.

Townsend, Anne; Amarsi, Zubin; Backman, Catherine L.; Cox, Susan M. & Li, Linda C. (2011). Communications between volunteers and health researchers during recruitment and informed consent: Qualitative content analysis of email interactions. Journal of Medical Internet Research, 13(4), https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3222195/?tool=pmcentrez&report=abstract [Date of Access: April 16, 2013].

Whitley, Rob & Crawford, Mike (2005). Qualitative research in psychiatry. Canadian Journal of Psychiatry, 50(2), 108-114, https://doi.org/10.1177/070674370505000206 [Date of Access: February 5, 2014].

Zhang, Yan & Wildemuth, Barbara M. (2009). Qualitative analysis of content. In Barbara M. Wildemuth (Ed.), Applications of social research methods to questions in information and library science (pp.318-329). Westport, CT: Libraries Unlimited.

Margaret R. ROLLER, M.A. is an independent consultant with more than 35 years in the research profession. Her expertise is in research design with a particular concentration in quality approaches to qualitative research designs. Prior to establishing her consulting firm, Ms. ROLLER held research managerial positions at AT&T, the Los Angeles Times, and Prentice Hall publishing. She has designed and implemented hundreds of qualitative studies, and conducts qualitative workshops and training. She is the lead author of "Applied Qualitative Research Design: A Total Quality Framework Approach" (ROLLER & LAVRAKAS, 2015), and her blog—Research Design Review—has been a resource for the research and academic community since 2009. Ms. ROLLER is an associate editor for Survey Practice and she is on the communications task force for the Society for Qualitative Inquiry in Psychology within the American Psychological Association. Her discipline of study is psychology. Her psychology degrees include a B.A. from California State University and graduate degree from the New School for Social Research, Graduate Faculty of Political and Social Science.

Roller Research (Independent Consultant)
Post Office Drawer 2436
Gloucester, Virginia 23061
USA

Roller, Margaret R. (2019). A Quality Approach to Qualitative Content Analysis: Similarities and Differences Compared to Other Qualitative Methods [46 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 20(3), Art. 31, http://dx.doi.org/10.17169/fqs-20.3.3385.

Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (FQS)

ISSN 1438-5627

Creative Commons Attribution 4.0 International License