Abstract: In this article, I explore the nature of secondary analysis and provide a brief history of the method. Qualitative secondary analysis is a relatively under-used method in education and the social sciences, often due to the lack of easily accessible, relevant, trustworthy, and complete data. I address some of the potentials and limitations that influence its use and explore criteria for assessing the quality and sufficiency of preexisting qualitative research data. Qualitative secondary analysis has important implications for qualitative researchers, students and practitioners interested in generating new knowledge via unobtrusive, reliable, valid, and time/cost effective research through the wider use of existing qualitative data.

Key words: qualitative research; qualitative secondary analysis; data evaluation

4. Evaluating Quality and Sufficiency of Qualitative Data for Secondary Analysis

In recent decades, secondary analysis has become popular among educational and social science researchers due to the increased quantity and accessibility of quantitative and qualitative data (BURSTEIN, 1978; DARGENTAS, 2006; FIELDING, 2000; HAKIM, 1982; HEATON, 2004; MURPHY & SCHLAERTH, 2010; VARTANIAN, 2011). These publicly available data provide researchers with resources to examine and analyze new hypotheses, inform research questions apart from the original objective of data collection, and derive new and/or additional interpretations and conclusions that were absent in the original research findings. Whichever research purpose is favored in a secondary analysis study, an important question arises: "Are preexisting data suitable, sufficient, and of a high enough quality to obtain new theoretical, empirical, and/or methodological understandings?" The objective of this article is to suggest evaluation criteria for the suitability, sufficiency, and quality of preexisting data employed for purposes of secondary analysis research. In the first sections, I outline the history and purpose of secondary analysis with a focus on the advantages and limitations of re-analysis of qualitative data. In the last section, I describe an assessment rubric to evaluate preexisting data's capacity to fulfill objectives of secondary research. [1]

Secondary analysis has a long tradition in the social, behavioral and education sciences, and has been an integral part of the development of research inquiry. In this sense, nearly 60 years ago, Seymour LIPSET and Reinhard BENDIX (1959) discussed an opportunity to re-analyze "existing data which were originally collected for other purposes" (p.xxvii). It is opined that the history of secondary analysis as a research strategy started in the last century before the Second World War and employed survey data. With the first national population census in 1790, followed by the collection of attitudinal surveys, extensive opportunities for secondary analysis were created (GLASER, 1963; SMITH, 2008). The original study conducted by Samuel STOUFFER and his team (1949) outlined the lives, relationships, attitudes, and adaptations of service personnel. Its later re-analysis has led to the development of theoretical propositions on race, class position, and social adjustment, as well as methodological insights on latent structure analysis and attitude scaling (GLASER, 1963; SMITH, 2008). [2]

During the 1970s, Janet ELASHOFF and Richard SNOW (1971) made an effort to re-analyze Robert ROSENTHAL and Lenore JACOBSEN's (1968) study on the effect of favorable teacher expectations on student achievement during the first and second grade. The original study found that high teacher expectations toward student academic success and intellectual growth greatly positively correlated with student successful learning and performance. A re-analysis was largely prompted by the huge interest the findings generated, as well as questions concerning the accuracy of results based on the original data analysis, methodology, and statistical procedures (ROSENTHAL & RUBIN, 1971). Another example of the important re-examination of influential studies is Frederick MOSTELLER and Daniel MOYNIHAN's (1972) secondary analysis study, "On Equality of Educational Opportunity." They revisited findings generated by James COLEMAN and his team in 1966. Their secondary analysis identified inaccuracies, discrepancies, and issues associated with the sampling and measurement instruments, and thereby led to a questioning of the validity of the original findings (MILNER, 1972). [3]

With the growing popularity of re-examining preexisting survey data, the concept of qualitative secondary analysis appears to have first entered the literature in the 1960s, when Barney GLASER (1963) discussed the limiting nature of survey data in the light of secondary analysis: "The emphasis on survey data neglects other kinds of data, particularly field data, and hence limits the potential use of secondary analysis" (p.11). He posited that qualitative data could and should be used in secondary analysis, as a way to independently contribute to this growing body of knowledge (GLASER, 1963). [4]

Since the 1960s, numerous definitions of secondary analysis have appeared in the literature, many with subtle differences and emphases on its theoretical, practical, and/or methodological implications. One of the first definitions of secondary analysis is found in the scholarship of Gene GLASS (1976), who describes it as "the re-analysis of data for the purpose of answering the original research question with better statistical techniques, or answering new questions with old data" (p.3). This understanding of secondary analysis for answering research questions divergent from the original ones and generating new conclusions and interpretations has been actively emphasized (GLEIT & GRAHAM, 1988; IRWIN & WINTERTON, 2011; LONG-SUTEHALL, SQUE & ADDINGTON-HALL, 2010). In 1978 BURSTEIN wrote about the capacity of secondary analysis "to contribute to knowledge because it has the potential to consider important questions without some of the limitations, or with a different set of limitations than those encountered in the original investigation" (p.7). A few years later, Catherine HAKIM (1982) comprehensively defined secondary analysis as "any further analysis of an existing dataset which presents interpretations, conclusions, or knowledge additional to, or different from, presented in the first report on the inquiry as a whole and its main results" (p.2). HAKIM suggested the use of secondary analysis was appropriate if secondary research aimed to identify additional indicators of an examined phenomenon; to reveal additional detail on the same research matter; to review a research matter from the perspective of a new theoretical framework not applied in the original study; or to answer research questions more thoroughly than in the original study through the application of a more sophisticated analytical methodology. [5]

HAKIM's definition of secondary analysis greatly contributes to the understanding of secondary analysis as a method to "generate new knowledge, new hypotheses, or supporting existing theories" (HINDS, VOGEL & CLARKE-STEFFEN, 1997, p.419). As an effective methodology that allows for exploring dimensions not explored in the primary study (DU PLESSIS & HUMAN, 2009), secondary analysis aims to uncover new meanings of information familiar to a researcher along with revisiting original research findings to assure their reliability and novelty (HEATON, 2004; KELDER, 2005; SAVAGE, 2005). Thus, the analysis of data collected by someone else (JOHNSTON, 2014) is common in research "pursuing a research interest that is distinct from that of the original work" (BUSTAMANTE-GAVINO, RATTANI & KHAN, 2012, p.36). This can be demonstrated by secondary analysis studies conducted by THOMPSON (2000) and RUSH, WATTS and STANBURY (2011). In the first case, THOMPSON examined three different datasets, one of which provided information regarding the extent to which family and work conditions before 1918 created economic and social difficulties in workers' earlier experiences. The study employed archived documented materials collected to investigate British trade unionism from the 1880s. The original data were collected by Beatrice and Sidney WEBB (1920), and included personal diaries and a large number of interviews accompanied by hand-written notes. Although the information was originally collected to record personal activity, politics and research work, it was suitable and sufficient to inform the history of the London labor movement in the 19th century. [6]

Recent research has also demonstrated the capacity of secondary analysis to generate valuable practical insights in the field of social research. For instance, Kathy RUSH, Wilda WATTS, and Janice STANBURY sought to understand the accounts of community-living older adults on adaptations they made in regard to mobility, and factors that influenced those adaptations. The secondary study was based on the interview data and demographic questionnaires collected by Margarete SANDELOWSKI (2011) to examine older adults' perceptions associated with aging along with the experiences of weakness in daily life. The secondary analysis study found several adaptation techniques available to older adults such as selection, optimization, and compensation. Moreover, the findings also offered a basis to maximize mobility with aging as well as a framework to identify older adults' motivations in overcoming mobility challenges, which had not existed in the previous literature. [7]

Another example that illustrates how re-analysis of archived data allowed for the generation of new theoretical knowledge and practical recommendations is found in Pamela KIDD, Ted SCHARF and Mark VEAZIE's study (1996). Conducted by the same researchers, the original study sought to identify safety decision making patters of older farmers and determine what health risks they perceived and how they were prioritized in respect to risk avoidance behavior. Re-analysis of the original data revealed new perspectives on the relationship between occupational context and worker behavior. It also allowed the researchers to design injury prevention strategies, identify economic factors influencing farmers' safety decisions, and develop an educational module to prevent farming-related injuries. [8]

In addition to furthering theory and practice (TRZESNIEWSKI, DONNELLAN & LUCAS, 2011), "the re-use of qualitative data provides an opportunity to study the raw materials of recent or earlier research to gain ... methodological insights" (CORTI & BISHOP, 2005, p.7). Recent studies have demonstrated how secondary analysis has informed method's assessments, ethical considerations in data re-use, and the use of various sources and types of secondary data (BISHOP, 2007; BRIASSOULIS, 2010; CORTI & THOMPSON, 2004; GOODWIN & O'CONNOR, 2009; LONG-SUTEHALL et al., 2011). While working on two different projects that focused on the re-use of interview data collected by Margot JEFFERY, Joanna BORNAT (2003) discovered methodological challenges associated with preparing data for archiving, data deposit and access, and level of anonymization. However, "the possibility that a researcher might find new evidence, or be able to draw new conclusions from archived data" (p.314) presents many opportunities to deeply engage in somebody else's data, actively learn from the original study and its researchers, when possible, and conduct research independently of any funding body obligation. [9]

The substantive examples presented earlier highlight a number of reasons for conducting secondary analysis including: examining new research questions with preexisting data sources (ANDREWS, HIGGINS, ANDREWS & LALOR, 2012; MITCHELL, 2015); developing knowledge that would have not otherwise been generated without secondary analysis (SALES et al., 2006); and expanding practical and methodological implications of the method (NOTZ, 2005; SALES, LICHTENWALTER & FEVOLA, 2006; SZABO & STRANG, 1997). Despite the fact that secondary data analysis has been employed for a long time, it has been used primarily in quantitative research, and its use with qualitative data is relatively recent (BISHOP, 2014; BISHOP & KUULA-LUMMI, 2017;). The most common purpose of qualitative secondary analysis, according to FIELDING (2000), is to gain new insights by re-analyzing the data from new perspectives. Although there are many advantages of this method, practical and ethical drawbacks do exist (BISHOP, 2014; MASON, 2007). Taking them into consideration is crucial when evaluating the quality, suitability, and sufficiency of qualitative data for their re-use. Therefore, in the following sections, prior to suggesting criteria for the evaluation of qualitative data for the purpose of secondary analysis, some benefits and limitations of the method are described. [10]

The definitions of secondary analysis, and qualitative secondary analysis specifically, described in the previous section have been accompanied by an ongoing academic debate over the potentials and limitations of qualitative data re-analysis. This debate is focused on the possibilities and methodological, archiving, and ethical issues (HEATON, 2004; LICHTMAN, 2005; MAUTHNER, 2012; YARDLEY, WATTS, PEARSON & RICHARDSON, 2014). [11]

As Janet HEATON (2004) has pointed out, qualitative secondary analysis as a methodology "for conducting free-standing studies using preexisting data originally collected for other purposes" (p.9) expands on data collected via observations, interviews, and document reviews (KAUFMAN, GUERRA & PLATT, 2006). Qualitative data sources also include researcher's notes, diaries, autobiographies, and open-ended questionnaires. Qualitative data used in secondary analysis research are often collected and archived as a product of independent qualitative studies conducted by a research team or independent investigators, as well as longitudinal research funded by private and government agencies (ANDRANOVICH & RIPOSA, 2012; DWORKIN, 2012; JACOBSON, HAMILTON & GALLOWAY, 1993; KAUFMAN et al., 2006; MAZZOCCHI, 2008). [12]

Secondary analysis of qualitative data can benefit the researcher intending to answer exploratory research questions or (re-)examine perceptions and experiences of a target audience (WINDLE, 2010). A more precise interpretation or emergence of new conceptual frameworks becomes possible, especially when the primary investigator conducts the secondary analysis inquiry themselves or serves as an advisor on the secondary research, such as with faculty and student projects (CORTI & BISHOP, 2005; CORTI & BLACKHOUSE, 2005; EBBINGHAUS, 2005; O'CONNOR & GOODWIN, 2010; PARRY & MAUTHNER, 2005; WINDLE, 2010). This is due to their extensive familiarity with the context of the original research. Qualitative secondary research, therefore, can generally broaden and deepen knowledge by stimulating a comprehensive understanding of the nature of an issue, especially when such issue is examined by the authors of preexisting data. It can reveal additional context for educational and social encounters and collaborations, as well as inform situated narratives around a particular topic (BROOM, CHESHIRE & EMMINSON, 2009). For example, in my doctoral dissertation (SHERIF, 2016), I re-analyzed longitudinal data from a sample of high school youth to determine practices of leadership development. The original study I conducted aimed to explore youth's perceptions, motives, and attitudes toward leadership and its development. Results of secondary analysis deepened the understanding of leadership development contexts as recounted by youth. [13]

Qualitative secondary analysis can also be beneficial in terms of economics. Re-analysis of already collected qualitative data saves time and money. The method is efficient because data collection is often a time consuming and expensive part of the research process (REW, KONIAK-GRIFFIN, LEWIS, MILES & O'SULLIVAN, 2000). Collection of data from various research sites and multiple sample subjects may be financially and time-restrictive as well. Many researchers, especially graduate students, have limited resources and are unable to cover indirect costs associated with obtaining access to specific populations for collection of specific data. Qualitative secondary analysis allows the researcher to overcome these types of data collection problems, thus, creating an equal opportunity for novice and other researchers to obtain and develop research independence, knowledge, and skills (SMITH, 2008). [14]

In addition to economic advantages, qualitative secondary analysis has social benefits. Research findings generated through this method can contribute to the specific body of knowledge without any intrusion into vulnerable populations (JOHNSTON, 2014; SMITH, 2008; ROBERTS, 1996). It has the advantage of not collecting additional data from individuals who require special treatment with respect to safeguards for their well-being and privacy (such as children, pregnant women, individuals with special needs, prisoners, etc.) or are challenging to recruit or access. Qualitative secondary analysis respects an individual's right to be left free from research inquiries about themselves or their activities/experiences (BULMER, 1979; SMITH, 2008). In cases in which samples may be difficult to recruit or hard to reach, qualitative secondary analysis can provide insights into sensitive issues, while protecting identities and privacy (REW et al., 2000). For example, Lenore BORIS (2015) aimed to explore experiences of HIV positive women in Kenya. Reanalysis of existing qualitative data enabled the investigator to share stories of these women on their "priorities and concerns post HIV diagnosis and how this diagnosis has reshaped their lives" (p.50), which otherwise would be challenging due to the remote location and limited access to this authentic population sample. [15]

Despite its apparent advantages, qualitative secondary analysis has its limitations. One of the major limitations, by definition, is that the data reflect insights collected for purposes different than those in the secondary research and may not adequately fulfill the research objectives of a new investigator (REW et al., 2000). The investigator is then challenged to explore new research questions derived directly from the data or shape the data so they would match the aims of the secondary research (HEATON, 2004). Therefore, qualitative secondary analysis appears to be useful as an extension of primary research, rather than for following up issues that emerged from the original study. [16]

Another major limitation of qualitative secondary analysis that expands on the issue of data fit is the completeness of preexisting data (HINDS et al., 1997; THORNE, 1998). The extent of data completeness is evaluated based on the accessibility of the original context (MAUTHNER, ODETTE & BACKETT-MILBURN, 1998). The context of the original study is often referred to as background information, and includes knowledge about interactions between the researcher and study participants; their age, gender, social status, race, occupation, etc.; research settings, time, and place; sample selection decisions; data collection methods and procedures; and complete interview, observation, focus group, document analysis protocols, interview audio materials and their detailed transcripts (BORIS, 2015; IRWIN & WINTERTON, 2011). When the secondary analysis is conducted by one who did not collect the data (ANDREWS et al., 2012; CORTI & BISHOP, 2005; THORNE, 1998), it often becomes challenging to address data flaws and eliminate a misfit between the data and new research objectives (BISHOP, 2007; CORTI & THOMPSON, 2004; REW et al., 2000). In either case, "the lack of control over conceptualization of the study, data collection decisions, ... and/or biases in the original study that cannot be overcome in the secondary analysis" (BORIS, 2015, p.46) limits qualitative secondary analysis to only exploratory designs. [17]

Archived qualitative data are bound by time, which can make the process of accessing high quality data and interpreting them in the light of current issues challenging (REW et al., 2000). Archiving requires abundant human and time resources; researchers who usually work alone may lack the motivation, funding, time, and specific guidelines to assure high quality preservation of qualitative data (MAUTHNER et al., 1998). Moreover, systematic data organization and archiving is an ongoing process occurring simultaneously with data collection rather than an "'add-on-task' at the end of the project" (HADFIELD, 2010, p.61). A more proactive approach enables the researcher to anticipate data archiving challenges and prepare data for archiving at various stages of material collection to assure protection of participant rights and identity. [18]

Archived qualitative data are also susceptible to time. Helene MORIARTY and her team (1999) argue that archived data may be outdated and lack information crucial to understanding research problems in the present day. Natasha MAUTHNER, Odette PARRY and Kathryn BACKETT-MILBURN (1998) note the potential discrepancy of theoretical and methodological lenses between existing data and secondary research questions. In this case, the data are inextricably connected to the unique settings of their collection. Thus, while it may be methodologically interesting to revisit archived data, "for the purpose of generating either new substantive findings or theories the data [are] wholly inadequate" (p.740). [19]

Ethical considerations for conducting qualitative secondary analysis include the process by which the subjects provided informed consent. Although secondary research implies no face-to-face involvement with study subjects and/or any intervention, there are challenges associated with confidentiality and subject agreement (CORTI & THOMPSON, 2004; HINDS et al., 1997; MOORE, 2007; THORNE, 1998). Archived qualitative studies embody an array of data about participants' backgrounds, knowledge, and experiences. Although datasets are typically cleared of identifying information, a secondary researcher may still receive data violating anonymity guidelines. While conducting secondary research involving vulnerable populations, BORIS (2015) cleared secondary data from any identifying information to assure participants' confidentiality, as well as verified the accuracy of original informed consents accompanying the dataset. As she later wrote, the lack of informed consent and noncompliance of secondary data with research anonymity guidelines may become a risk to conducting secondary research. Similarly, MCLEOD and THOMSON (2009) in their book, "Revisiting Social Change," highlight confidentiality and participants' understanding of informed consent as cornerstones of a valuable, informative, and reusable qualitative data archive. Therefore, to address ethical challenges associated with the secondary analysis of qualitative data, researchers should be cognizant of the risks imposed by ethical considerations of the method and make an effort to verify the alignment of the primary research with research integrity guidelines. [20]

Still, although the secondary analysis of data is not without challenges, there are indeed fewer risks with the archiving and reuse of one's own data (HEATON, 2004). Conducting qualitative secondary analysis using one's own data generally avoids the limitations discussed above, and may present some additional benefits. For instance, Sarah IRWIN (2013) argues that revisiting data by the original data collectors can create a "form of critical distance." As a result, the primary researchers can critically evaluate the quality and efficiency of collected data from the perspective of new research questions and fill in the blanks in the original study background, data collection procedures, and missing information. The in-depth knowledge of the original research may also be beneficial in protecting participants' privacy and confidentiality, since primary researcher's immersion in the data enhances their ability to determine which information might be identifying (THORNE, 1998). Their familiarity and closeness with the data can, therefore, allow for eliminating epistemological and ethical concerns associated with the interpretation of data created by other researchers and participants, as well as more effectively assessing the fit between the original data and new research questions. [21]

4. Evaluating Quality and Sufficiency of Qualitative Data for Secondary Analysis

Current scholarship informs numerous examples of the re-use of qualitative data. However, there are only a few commentators on qualitative secondary analysis that seem to describe specific criteria for the evaluation of the quality and sufficiency of existing data. This section presents such criteria, which emerged as a result of careful examination of studies employing qualitative secondary analysis, as well as methodological, archiving, and ethical concerns. To identify the studies, a literature review of peer-reviewed publications was conducted. The search resulted in 40 sources that included doctoral dissertations, book chapters, and articles. The sources were identified through the following search terms: secondary data analysis, qualitative data analysis, secondary qualitative data, secondary data, and secondary qualitative data analysis. As the review of full text sources revealed, the process of evaluating the sufficiency and quality of existing qualitative data includes evaluation of the fit and relevance of the original dataset to the present research, the background of primary research and collected data, dataset breadth and depth, data trustworthiness, and the timeliness of primary data. [22]

The process of evaluating preexisting data quality and sufficiency begins with conceptualization of a secondary study and selection of a research question (BORIS, 2015; DU PLESSIS & HUMAN, 2009; HINDS et al., 1997; JOHNSTON, 2014; KELDER, 2005; MEDJEDOVIC, 2011; MURPHY & SCHLAERTH, 2010; ROBERTS, 1996; SANDELOWSKI, 2011; SZABO & STRANG, 1997). In some cases, research can be conceptualized during examination of preexisting data with the research questions emerging directly from the data (GLADSTONE, VOLPE & BOYDELL, 2007; SHERIF, 2016). For example, "[t]here [should be] a logical link between the original data set and the question/s asked in the analysis, as the qualitative secondary analysis questions [arise] from the original data set" (DU PLESSIS & HUMAN, 2009, p.76). For instance, Brenda GLADSTONE's team conducted a study to examine the subjective experience between motivation and schizophrenia. It aimed to explore psychosocial factors influencing motivation while living with schizophrenia, and resulted in sixty in-depth interviews with individuals diagnosed with schizophrenia and parents of ill family members. Descriptively rich data were later re-read and provided foundations for new research questions focusing on parents' help-seeking experiences. [23]

The quality of a primary dataset is determined by examining the breadth, depth, completeness, and accuracy of preexisting data (HINDS et al., 1997). The original data must allow the researcher conducting secondary analysis to understand examined processes, relationships, and subjective meanings (BORNAT, 2005). Only holistic and rich data can reveal the complexity of the original study and an examined issue (MILES & HUBERMAN, 1994). Such data, if re-examined, provide vivid and rich descriptions of a researched phenomenon while having a strong impact on the findings of the secondary study. The holistic and rich data also have the capacity to present every detail of the examined issue and illustrate how people perceive, react, and experience it. [24]

The breadth and depth of the primary data refer to the richness of data content, whereas completeness of the dataset includes the technical condition of the dataset and its accuracy. Specifically, having a complete dataset means there should be no or minimal missing or damaged documents (CORTI & BACKHOUSE, 2005; HARRIS, 2001; HINDS et al., 1997). The documents also should be accompanied by extensive description of methodologies employed to collect the data, field notes, study and sample design decisions, a detailed sample plan, and any additional information. [25]

Dataset quality is also addressed by examining data accuracy. Accurate transcription of interview and/or focus group data should have no or minimal typographical errors, incomplete sentences, and missing words (MacLEAN, MEYER & ESTABLE, 2004). Transcribed documents should be time stamped and accompanied by transcription protocols indicating instructions for transcribers and decisions that address inaudible text segments, overlapping speech, unfamiliar terminology, and language-specific nuances. An accurate dataset is also member-checked, which is completed during the data collection process and consists of the primary researcher restating, summarizing, and/or paraphrasing the information received from a study participant (GUBA & LINCOLN; 1981; KUZEL & LIKE, 1991). As part of this process, the respondent affirms that what was shared, heard or written down is recorded and understood correctly. Although dataset accuracy is critical to high-quality secondary analysis, assurance and compliance to accuracy standards require human and time resources (MAUTHNER et al., 1998). [26]

The trustworthiness of preexisting data is often questioned by opponents of the method, perhaps because the reliability and validity cannot be addressed in the same manner as quantitative data (SHENTON, 2004). Nevertheless, several writers have demonstrated how researchers could address this issue. They note that comprehensive background information about the original study increases dataset quality and sufficiency (ANDREWS et al., 2012; ELLIOTT, 2015; FIELDING, 2000). Research background supports understanding and interpretation "because no data can be seen outside of a viewing context" (SANDELOWSKI, 2011, p.347). "The detailed contextual knowledge about the circumstances of the data collection possessed by the primary researcher" can foster both description and explanation (MITCHELL, 2015, p.3). In particular, knowing the research background allows for a complex understanding of nuances of the original study, as well as its theoretical and/or empirical contribution to the overall knowledge base. [27]

How detailed should the description of the original study background be to make a judgment regarding dataset sufficiency and quality? According to Emmerentia DU PLESSIS and Susara HUMAN (2009), the original study background should include a description of research questions, the study population, sample selection choices, and the employed research methodology. An important question is how well the study settings and sample "of the original research project match those of the present project" (NOTZ, 2005, p.2). The settings of the original study and its sample must meet the expectations and needs of the secondary research (IRWIN, 2013; IRWIN & WINTERTON, 2011; JOHNSTON, 2014; NOTZ, 2005; WHITESIDE, MILLS & MCCALMAN, 2012). As IRWIN (2003) asserts, "in order to grasp the contexts underpinning diversity, and ... insights into conditions and causes" a thorough understanding and matching of study settings and people situated within the study sample become important (p.299). [28]

Arguing that research background and collected data are related to each other, VAN DEN BERG (2005) also suggests evaluating background information about data collection methods and collected "raw" data (i.e., interviews, interview audiotapes, and transcripts); background characteristics of researcher/s and study participants; and information about the data collection site, time of data collection, and data collection settings. Additionally, Harry VAN DEN BERG and his colleague, Martyn HAMMERSLEY (2010), have emphasized the importance of accessing information about how study participants were selected and recruited and any other relevant information that may be helpful for a secondary data analyst to fully recreate the meaning and background of the original study. [29]

Evaluation of the quality and sufficiency of a qualitative dataset for secondary analysis also includes consideration of dataset timeliness and the possibility of further follow-up with the sample by the investigator of the initial study. Because the re-analysis is completed after the original study, data may be outdated due to changing theoretical and conceptual lenses, as well as historical events (LESKE, 1990; MORIARTY et al., 1999). If the original research had time-sensitive questions and aimed to portray historical events at the time of data collection, the secondary analysis may fail to provide an understanding of a research problem in the present day. Pamela HINDS and her colleagues (1997) note that a research problem interacts with and is affected by its context and that contexts change over time: "An analysis of a dated qualitative data set, in which the phenomena of interest may have since taken on different forms, is unlikely to provide a new or expanded understanding of the phenomena" (p.415). Therefore, it may be necessary for the secondary analyst to choose another more recent dataset or less time-sensitive data. [30]

The possibility of follow-up with primary study participants is especially important when the data quality is problematic, meaning data are outdated or missing (CORTI & BACKHOUSE, 2005; WHITESIDE et al., 2012). Although follow-up with subjects appears challenging due to numerous reasons, such as outdated or missing contact information, complete deidentification of study participants, original agreement that subjects will never be contacted again, etc., "in principle, it is wise to leave oneself the freedom to recontact on one's own behalf, or to ask informants whether they would be willing to be contacted for a different project" (THOMPSON, 2000, p.8). An opportunity to reconnect with subjects can be created when a primary researcher intentionally includes it in the consent form, sets up a flexible participation schedule of data collection, collects additional contact information, and/or tracks study participants during the study (WOOLARD et al., 2004). These procedures help not only facilitate the outcomes of a study, but also clarify characteristics of the data collection process and participant recruitment. [31]

A summary of these criteria for evaluating data for secondary analysis are found in the rubric below (Table 1). They allow for assessing: 1. the fit and relevance of preexisting qualitative data to secondary research; 2. the general quality of data; 3. the trustworthiness of the original dataset; and 4. dataset timeliness.

Table 1: Example of assessment rubric: Evaluation criteria in a secondary analysis of qualitative data. Click here to download the PDF file [32]

The rubric is designed as a diagnostic guide to identify criteria that could inform an assessment of the quality and suitability of preexisting data for secondary analysis. It helps remove the anxiety that comes with a researcher having to guess about what the dataset should or should not include, and makes it easier for one to justify the use of the preexisting data. If used for data archiving or dataset assessment, the rubric also provides consistency of assessment when more than one dataset is used for secondary analysis purposes. [33]

When using the rubric, one should note that each dataset is unique and has its own story. Evaluation of previously collected data is, therefore, the process of learning more about this story, and, specifically, about the original research, its background and settings, and participants' perspectives and how they were collected. When evaluation of preexisting data centers only on the rubric presented in this article, its results can lead to a risk of missing the overall meaning of the data. Furthermore, relying on the assessment rubric only, without taking into consideration the unique needs of the secondary analysis study, might turn the assessment process into a deficiency approach to secondary analysis, focusing on data deficiencies, limitations and errors. Therefore, the use of the rubric should be accompanied by critical evaluation of the general story presented by data, along with the consideration of secondary research questions and objectives. [34]

In their recent writing, Sarah IRWIN and Mandy WINTERTON (2011) suggest that an increasing capacity of research to generate abundant, rich, in-depth, and accessible data creates a profound opportunity to conduct efficient and cost-effective secondary inquiry. Asking additional research questions that have not been asked in the original study can reveal new knowledge, question current findings, or support existing scholarly findings and theories. The method also allows for the development, extension, and exploration of a phenomenon in a flexible and unobtrusive way. Particularly, the overall goal of this method is to contribute to the specific body of knowledge by providing an alternative perspective on topics without any intrusion into vulnerable populations (JOHNSTON, 2014; ROBERTS, 1996). As Barney GLASER (1963) suggests, "secondary data analysis can help save time, money, career, degrees, research interests, vitality and talent, self images and myriads of data from untimely, unnecessary and unfortunate loss" (p.14). [35]

In this article I have discussed advantages and limitations of qualitative secondary analysis. Yet successful qualitative secondary analysis is most effective when used with high-quality, relevant, rich, and complex datasets. As suggested in this article, careful examination and assessment of preexisting qualitative data can increase the validity and reliability of secondary analysis research findings. A systemic approach to the assessment of the quality and sufficiency of data used for secondary analysis can help qualitative researchers select an appropriate dataset and prepare already collected data for future archiving and re-use. [36]

To systematically evaluate the quality and sufficiency of preexisting qualitative data, an assessment rubric, presented in this article, describes in detail major parameters of data in respect to their fit and relevance for secondary research questions and aims; completeness, sufficiency, and accuracy; trustworthiness; and timeliness. The rubric can be used as a tool to begin examining and learning about previously collected data. It allows for knowledge construction in regards to the original study and recognition of possible variations between the original and secondary analysis research questions. While the rubric provides evaluation criteria, which emerged from extensive literature review, it is theoretical in nature and needs to be practically tested. Overall, the rubric clarifies expectations for the quality and sufficiency of preexisting data, as well as saves time in data assessment and helps researchers think critically and broadly about research purpose, data context, and meaning. [37]

Andranovich, Gregory D. & Riposa, Gerry (2012). Data collection in the urban setting. In John Goodwin (Ed.), Sage secondary data analysis: Ethical, methodological and practical issues in secondary analysis (pp.129-149). Thousand Oaks, CA: Sage.

Bishop, Libby (2007). A reflexive account of reusing qualitative data: Beyond primary/secondary dualism. Sociological Research Online, 12(3), http://www.socresonline.org.uk/12/3/2.html [Accessed: January 17, 2016].

Bishop, Libby (2014). Re-using qualitative data: A little evidence, on-going issues and modest reflections. Studia Socjologiczne, 3(214), 167-176.

Boris, Lenore L. (2015). Hearing the voices of HIV positive women in Kenya: Secondary analysis of interview data using dialogic/performance analysis. Doctoral dissertation, University of Wisconsin-Milwaukee, USA, https://dc.uwm.edu/etd/860 [Accessed: July 10, 2015].

Bornat, Joanna (2003). Secondary analysis in reflection: Some experiences of re-use from an oral history perspective. Families, Relationships and Societies, 2(2), 309-317.

Bornat, Joanna (2005). Recycling the evidence: Different approaches to the reanalysis of gerontological data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), Art. 42, http://dx.doi.org/10.17169/fqs-6.1.504 [Accessed: May 19, 2016].

Briassoulis, Helen (2010). Online petitions: New tools of secondary analysis? Qualitative Research, 10(6), 715-727.

Broom. Alex; Cheshire, Lynda & Emminson, Michael (2009). Qualitative researchers' understandings of their practice and the implications for data archiving and sharing. Sociology, 43(6), 1163-1179.

Bulmer, Martin (1979). Concepts in the analysis of qualitative data. The Sociological Review, 27(4), 651-677.

Burstein, Leigh (1978). Secondary analysis: An important resource for educational research and evaluation. Educational Researcher, 7(5), 9-12.

Bustamante-Gavino, Maria Irma; Rattani, Salma & Khan, Kausar (2011). Women's empowerment in Pakistan—definitions and enabling and disabling factors: A secondary data analysis. Journal of Transcultural Nursing, 22(2), 174-181.

Coleman, James S.; Campbell, Ernest Q.; Hobson, Carol J.; McPartland, James; Mood, Alexander M.; Weinfeld, Frederic D. & York, Robert L. (1966). Equality of educational opportunity. Report No OE-38001, National Center for Educational Statistics, U.S. Department of Health, Education, and Welfare, https://files.eric.ed.gov/fulltext/ED012275.pdf [Accessed: September 3, 2017].

Corti, Louise & Backhouse, Gill (2005). Acquiring qualitative data for secondary analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(2), Art. 36, http://dx.doi.org/10.17169/fqs-6.2.459 [Accessed: April 5, 2015].

Corti, Louise & Bishop, Libby (2005). Strategies in teaching secondary analysis of qualitative data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), Art. 47, http://dx.doi.org/10.17169/fqs-6.1.509 [Accessed: April 5, 2015].

Corti, Louise & Thompson, Paul (2004). Secondary analysis of archived data. In Clive Seal, Giampietro Gobo, Jaber F. Gubrium & David Silverman (Eds.), Qualitative research practice (pp.297-311). Thousand Oaks, CA: Sage.

Dargentas, Magdalini (2006). Secondary analysis and culture of disputation in European qualitative research. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 7(4), Art. 9, http://dx.doi.org/10.17169/fqs-7.4.197 [Accessed: April 5, 2015].

du Plessis, Emmerentia & Human, Susara P. (2009). Reflecting on "meaningful research": A qualitative secondary analysis. Curationis, 32(3), 72-79.

Dworkin, Rosalind (2012). Alternative data sources. In John Goodwin (Ed.), Sage secondary data analysis: Ethical, methodological and practical issues in secondary analysis (Vol.4, pp.84-104). Thousand Oaks, CA: Sage.

Ebbinghaus, Berhnard (2005). When less is more: Selection problems in large-N and small-N cross-national comparisons. International Sociology, 20(2), 133-152.

Elashoff, Janet D. & Snow, Richard E. (1971). Pygmalion reconsidered: A case study in statistical inference: Reconsideration of the Rosenthal-Jacobson data on teacher expectancy. Worthington, OH: Jones Pub Co.

Elliott, Diana (2015). Secondary data analysis. In Frances K. Stage & Kathleen Manning (Eds.), Research in the college context: Approaches and methods (2nd ed., pp.175-184). New York, NY: Routledge.

Fielding, Nigel (2000). The shared fate of two innovations in qualitative methodology: The relationship of qualitative software and secondary analysis of archived qualitative data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), Art. 22, http://dx.doi.org/10.17169/fqs-1.3.1039 [Accessed April 5, 2015].

Gladstone, Brenda M.; Volpe, Tiziana & Boydell, Katherine M. (2007). Issues encountered in a qualitative secondary analysis of help-seeking in the prodrome to psychosis. The Journal of Behavioral Health Services & Research, 34(4), 431-442.

Glaser, Barney G. (1963). Retreading research materials: The use of secondary analysis by the independent researcher. The American Behavioral Scientist, 6(10), 11-14.

Glass, Gene V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.

Gleit, Carol & Graham, Barbara (1988). Secondary data analysis: A valuable source. Nursing Research, 38(6), 380-381.

Goodwin, John & O'Connor, Henrietta (2009). Through the interviewer's lens: Representations of 1960s households and families in a lost sociological study. Sociological Research Online, 14(4), http://www.socresonline.org.uk/14/4/6.html [Accessed: May 15, 2016].

Guba, Egon G. & Lincoln, Yvonna S. (1981) Effective evaluation: Improving the usefulness of evaluation results through responsive and naturalistic approaches. San Francisco, CA: Jossey-Bass.

Hadfield, Lucy (2010). Balancing on the edge of the archive: The researcher's role in collecting and preparing data to deposit. Timescapes Working Paper Series, 2, 60-74.

Hakim, Catherine (1982). Secondary analysis in social research: A guide to data sources and methods with examples. London: George Allen & Unwin.

Hammersley, Martyn (2010). Can we re-use qualitative data via secondary analysis? Notes on some terminological and substantive issues. Sociological Research Online, 15(1), http://www.socresonline.org.uk/15/1/5.html [Accessed: May 15, 2016].

Harris, Howard (2001). Content analysis of secondary data: A study of courage in managerial decision making. Journal of Business Ethics, 34, 191-208.

Hinds, Pamela S.; Vogel, Ralph J. & Clarke-Steffen, Laura (1997). The possibilities and pitfalls of doing a secondary analysis of a qualitative data set. Qualitative Health Research, 7(3), 408-424.

Irwin, Sarah (2013). Qualitative secondary data analysis: Ethics, epistemology, and context. Progress in Development Studies, 13(4), 295-306.

Irwin, Sarah & Winterton, Mandy (2011). Debates in qualitative secondary analysis: Critical reflections. Timescapes Working Paper Series, 4, 2-23.

Jacobson, Ann F.; Hamilton, Patti & Galloway, James (1993). Obtaining and evaluating data sets for secondary analysis in nursing research. Western Journal of Nursing Research, 15(4), 483-494.

Johnston, Melissa P. (2014). Secondary data analysis: A method of which the time has come. Qualitative and Quantitative Methods in Libraries, 3, 619-626.

Kaufman, Roger A.; Guerra, Ingrid & Platt, William A. (2006). Practical evaluation for educators: Finding what works and what doesn't. Thousand Oaks, CA: Sage.

Kelder, Jo-Anne (2005). Using someone else's data: Problems, pragmatics, and provisions. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), Art. 39, http://dx.doi.org/10.17169/fqs-6.1.501 [Accessed: April 5, 2015].

Kidd, Pamela; Scharf, Ted & Veazie, Mark (1996). Linking stress and injury in the farming environment: A secondary analysis of qualitative data. Health Education & Behavior, 23(2), 224-237.

Kuzel, Anton J. & Like, Robert C. (1991). Standards of trustworthiness for qualitative studies in primary care. In Peter G. Norton, Moira Stewart, Fred Tudiver, Martin J. Bass & Earl V. Dunn (Eds.), Primary care research: Traditional and innovative approaches (pp.138-158). Newbury Park, CA: Sage.

Leske, Jane S. (1990). Evaluating sources for secondary analysis. Heart & Lung, 19(5), 537-539.

Lichtman, Marilyn (2005). Review: Janet Heaton (2004). Reworking qualitative data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(3), Art. 15, http://dx.doi.org/10.17169/fqs-6.3.40 [Accessed: May 15, 2016].

Lipset, Seymour Martin & Bendix, Reinhard (1959). Social mobility in industrial society. New Brunswick, NJ: Transaction Publishers.

Long-Sutehall, Tracy; Sque, Magi & Addington-Hall, Julia (2011). Secondary analysis of qualitative data: A valuable method for exploring sensitive issues with an elusive population?. Journal of Research in Nursing, 16(4), 335-344.

MacLean, Lynne M.; Meyer, Mechthild & Estable, Alma (2004). Improving accuracy of transcripts in qualitative research. Qualitative Health Research, 14(1), 113-123.

Mason, Jennifer (2007). "Re-using" qualitative data: On the merits of an investigative epistemology, Sociological Research Online, 12(3), http://www.socresonline.org.uk/12/3/3.html/ [Accessed: February 20, 2016].

Mauthner, Natasha S.; Parry, Odette & Backett-Milburn, Kathryn (1998). The data are out there, or are they? Implications for archiving and revisiting qualitative data. Sociology, 32(4), 733-745.

Mazzocchi, Mario (2008). Statistics for marketing and consumer research. London: Sage.

McLeod, Julie & Thomson, Rachel (2009). Researching social change. London: Sage.

Medjedovic, Irena (2011). Secondary analysis of qualitative interview data: Objections and experiences. Results of a German feasibility study. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(3), Art. 10, http://dx.doi.org/10.17169/fqs-12.3.1742 [Accessed: February 14, 2016].

Miles, Matthew B. & Huberman, Michael A. (1994). Qualitative data analysis (2nd ed.). Thousands Oaks, CA: Sage.

Milner, Murray Jr. (1972). The illusion of equality: The effects of educational opportunity on inequality and conflict. San Francisco, CA: Jossey-Bass.

Mitchell, Fraser (2015). Reflections on the process of conducting secondary analysis of qualitative data concerning informed choice for young people with a disability in transition. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 16(3), Art. 10, http://dx.doi.org/10.17169/fqs-16.3.2300 [Accessed: February 14, 2016].

Moldanado, Swarnalatha A. (1991). Secondary analysis: Expanding survey research by faculty members. Nurse Educator, 16(4), 4-5.

Moriarty, Helen J.; Deatrick, Janet A.; Mahon, Margaret M.; Feetham, Suzanne L.; Carroll, Ruth M.; Shepard, Margaret P. & Orsi, Allen J. (1999). Issues to consider when choosing and using large national databases for research of families. Western Journal of Nursing Research, 21(2), 143-153.

Mosteller, Frederick & Moynihan, Daniel (1972). On equality of educational opportunity. Science and Society, 36(4), 469-476.

Murphy, John W. & Schlaerth, Christian A. (2010). Where are your data? A critique of secondary data analysis on sociological research. Humanity & Society, 34, 379-390.

Notz, Petra (2005). Secondary qualitative analysis of interviews. A method used for gaining insight into the work/life balance of middle managers in Germany. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), 1-8, Art. 34, http://dx.doi.org/10.17169/fqs-6.1.506 [Accessed: December 5, 2015].

O'Connor, Henrietta & Goodwin, John (2010). Utilizing data from a lost sociological project: Experiences, insights, promises. Qualitative Research, 10(3), 283-298.

Parry, Odette & Mauthner, Natasha S. (2005). Back to basics: Who re-uses qualitative data and why?. Sociology, 39(2), 337-342.

Rew, Lynn; Koniak-Griffin, Deborah; Lewis, Mary Ann; Miles, Margaret & O'Sullivan, Ann (2000). Secondary data analysis: New perspective for adolescent research. Nursing Outlook, 48(5), 223-229.

Roberts, Brian A. (1996). Issues in methodology: Secondary analysis in qualitative research: Thoughts on data obsolescence. Bulletin of the Council for Research in Music Education, 130, 44-51.

Rosenthal, Robert & Jacobson, Lenore (1968). Pygmalion in the classroom. The Urban Review, 3(16), 16-20.

Rosenthal, Robert & Rubin, Donald B. (1971). Pygmalion reaffirmed. Cambridge, MA: Harvard University.

Rush, Kathy L.; Watts, Wilda E. & Stanbury, Janice (2011). Mobility adaptations of older adults: A secondary analysis. Clinical Nursing Research, 20(1), 81-100.

Sales, Esther; Lichtenwalter, Sara & Fevola, Antonio (2006). Secondary analysis in social work research education: Past, present, and future promise. Journal of Social Work Education, 42(3), 543-558.

Sandelowski, Margarete (2011). When a cigar is not just a cigar: Alternative takes on data and data analysis. Research in Nursing & Health, 34, 342-352.

Savage, Mike (2005). Revisiting classic qualitative studies. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), 1-20, Art. 31, http://dx.doi.org/10.17169/fqs-6.1.502 [Accessed: May 22, 2016].

Shenton, Andrew K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for Information, 22, 63-75.

Sherif, Victoria (2016). The voices of rural high school youth: Qualitative secondary analysis and youth leadership modeling. Doctoral dissertation, University of Kentucky, USA, https://uknowledge.uky.edu/edsc_etds/17 [Accessed: August 1, 2016].

Smith, Emma (2008). Using secondary data in educational and social research. London: Open University Press.

Stouffer, Samuel A.; Suchman, Edward, A.; DeVinney, Leland, C.; Star, Shirley, A. & Williams, Robin M. Jr. (1949). The American soldier: The adjustment during army life. Vol. I. The Annals of the American Academy of Political and Social Science, 265(1), 173-175.

Szabo, Vivian & Strang, Vicki (1997). Secondary analysis of qualitative data. Advances in Nursing Science, 20(2), 66-74.

Thompson, Paul (2000). Re-using qualitative research data: A personal account. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), Art. 27, http://dx.doi.org/10.17169/fqs-1.3.1044 [Accessed May 22, 2016].

Thorne, Sally (1998). Ethical and representational issues in qualitative secondary analysis. Qualitative Health Research, 84(4), 547-555.

Trzesniewski, Kali H.; Donnellan, Brent M. & Lucas, Richard E. (2011). Secondary data analysis: An introduction for psychologists. Washington, DC: American Psychological Association.

van den Berg, Harry (2005). Reanalyzing qualitative interviews from different angles: The risk of decontextualization and other problems of sharing qualitative data. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 6(1), Art. 30, http://dx.doi.org/10.17169/fqs-6.1.499 [Accessed: May 22, 2016].

Vartanian, Thomas P. (2011). Secondary data analysis. New York: Oxford University Press.

Webb, Sidney & Webb, Beatrice (1920). History of trade unionism. London: Longmans, Green.

Whiteside, Mary; Mills, Jane & McCalman, Janya (2012). Using secondary data for grounded theory analysis. Australian Social Work, 65(4), 504-516.

Windle, Pamela E. (2010). Secondary data analysis: Is it useful and valid?. Journal of PeriAnesthesia Nursing, 25(5), 322-324.

Woolard, Robert H.; Carty, Kathleen; Wirtz, Philip; Longabaugh, Richard; Nirenberg, Ted D.; Minugh, Allison P.; Becker, Bruce & Clifford, Patrick R. (2004). Research fundamentals: Follow-up with subjects in clinical trials: Addressing subject attrition. Academic Emergency Medicine, 11(8), 859-866.

Yardley, Sarah J.; Watts, Kate M.; Pearson, Jennifer & Richardson, Jane C. (2014). Ethical issues in the reuse of qualitative data: Perspectives from literature, practice, and participants. Qualitative Health Research, 24(1), 102-113.

Victoria SHERIF, Ph.D., is a post-doctoral research associate in the Department of Educational Leadership Studies at the University of Kentucky, USA. Her research centers around issues related to qualitative secondary analysis. She is also interested in examination of youth leadership, its development and practice, as well as leadership of school superintendents to elevate their voices in various educational and socio-economic settings.

Educational Leadership Studies
University of Kentucky
111 Dickey Hall
Lexington, KY
USA 40502

Sherif, Victoria (2018). Evaluating Preexisting Qualitative Research Data for Secondary Analysis [37 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 19(2), Art. 7, http://dx.doi.org/10.17169/fqs-19.2.2821.

Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (FQS)

ISSN 1438-5627

Creative Commons Attribution 4.0 International License