KWALON Conference: Is Qualitative Software Really Comparable? Reflections on "the Experiment": An "Expert" View

As an experienced sociologist, methodologist and qualitative analyst who has trained and consulted on a range of CAQDAS packages since the middle 1990s, I came to this conference with well-formed views on this topic which are evident in a number of my publications (Di GREGORIO & DAVIDSON, 2008; DAVIDSON & Di GREGORIO, in press). I am an enthusiastic yet not uncritical advocate of the use of CAQDAS as well as new emerging digital tools to support qualitative analysis (Di GREGORIO, 2010). However, I learned the craft of qualitative analysis in the early 1970s before the development of software tools.

I have used four of the five software packages which were part of the "experiment." The one which I had not used (nor had heard about) was Cassandre. I have, in the past, experimented with putting the same dataset in ATLAS.ti, NVivo and MAXqda to satisfy my own curiosity about differences among those packages. However, I would not consider my own attempts as a true experiment as I was well aware about how having done an analysis in one software package influences how I approach it in another. In fact, I confess that I have extreme doubts whether it is possible to conduct such an experiment. However, I was curious to see the outcome of this "experiment." I was also aware that I arrived at the conference with pretty fixed ideas about this issue and that my perception of this event could be very different from someone who was a complete novice or who had some experience of working with one package.

All the developers (or their representatives) were given the same research question and the same dataset. However, they were free to choose their analysis strategy, the tools in their software and the proportion and type of data from the set to analyze. However, in the instructions is the statement: "It is obvious that we would want you to include the whole dataset in your analysis." In practice, everyone involved in the experiment commented on the size of the corpus of data they were given. Given their time constraints, they all adopted different strategies for managing the dataset which were not related necessarily to their software package. Suzanne FRIESE had only a week to work on the dataset by herself in ATLAS.ti and decided to focus on only 11 pieces of data but chose data of different types (text, audio, video) from the two time periods (see FRIESE, 2011). Fiona WILTSHIER decided to take advantage of the teamwork capacities of NVivo and while they imported all the data and divided up the coding, the analysis they reported on was based on the textual material which Fiona commented was due to time constraints as opposed to limitations of the software (see WILTSHIER, 2011). While Anne KUCKARTZ was unable to attend the conference due to the volcanic ash cloud closing down European airspace, the abstract about MAXqda's experiment comments that while they put all the data in the software, they quickly realized they did not have the time to do a thorough analysis and decided to focus on one topic—responsibility (see KUCKARTZ & SHARP, 2011). David WOODS decided to focus on the audio and video data even though Transana can handle transcripts without audio/video. He also decided to use Transana's collaborative function to divide up the analysis work with a transatlantic colleague (see DEMPSTER & WOODS, 2011). Only Christophe LEJEUNE's Cassandre imposed a software limitation in that it can only handle textual material (see LEJEUNE, 2011).

Given the different strategies adopted by the software representatives for managing the corpus were determined by time and manpower constraints, I do not believe we can call this exercise an experiment. Furthermore, these decisions could skew a novice's perceptions of the capabilities of a particular package. For example, ATLAS.ti allows teamworking but that feature was "hidden" as Suzanne chose not to use it (probably due to the lack of an available co-analyst). Both Fiona for NVivo and David for Transana highlighted their packages capacity for teamwork as they decided to get help with their analyses. A novice's perception could easily have been that ATLAS.ti could only handle a very small dataset while NVivo could handle a large dataset. However, despite the differences in choices on how to manage and utilize the dataset what was interesting was that all the analysts, regardless of software package and regardless of approach to analysis, came up with very similar answers to the research questions.

My view before attending this conference was that while there are differences between the various CAQDAS packages, they are not significant. The analysis is done by the analyst and the software supplies tools to support that analysis. Certain things may be easier to do in one package over another but usually the analysts can find "workarounds" to do what they want. In addition, we are in a period of rapid development and software developers are looking at each others' products and often match or improve a rival's new feature with the next new version. Furthermore, I have never felt that tools could be categorized as supporting a particular form of qualitative analysis. I feel that the link with the more popular brands of CAQDAS e.g. ATLAS.ti and NVivo and grounded theory have been overstated—a point I discuss in DAVIDSON and di GREGORIO (in press). At the same time, as a trainer, I was aware that the numerous CAQDAS packages on offer are confusing to the novice. What criteria should one use when choosing a CAQDAS package? I can see how support for a particular approach to an analysis would be a logical key criterion for someone new to the area to question.

All the presenters were both expert users of their software package and experienced qualitative analysts. They all very clearly described their analysis process and that process was similar across all the packages. KUCKARTZ and SHARP's abstract description of the process used in MAXqda describes succinctly this process:

The tools they used to organize, explore and display their data varied across the software packages. Many of these tools, although they are different in design, perform exactly the same function. An analogy I would make is with a bottle opener for wine. There are a variety of designs for that tool—the traditional wine waiter's tool, the screwpull design and the twin prong cork puller. However, they all perform the same function—to remove the cork from the bottle. In the same way, the organizing tools may look and work slightly differently across the different packages, but they all perform the same function—to manage the dataset in order to find the data you want and retrieve just the parts of the data you may want to focus on at any one time. However, it is the analyst who decides how to organize organise the data. The analyst needs to be skilled in not only the approach to qualitative analysis they choose to adopt but also in the software package they choose in order to know how best to organize their data in that software in order to retrieve what they will need for their analysis.

There were variations in some of the tools. Cassandre stood out in offering a range of exploratory and display tools that are not available in the others. Unlike the others (which support multi-media), it focuses just on textual analysis (and it supports only text saved as Unicode-UTF 8 text). Its tools focus on the analyst identifying keywords which the software retrieves and assigns in a notional family which can be compared through visual display with other notional families (see LEJEUNE, 2011). Although it has a semi-automatic coding tool, the researcher remains in control—deciding, upon retrieval of text, whether or not a key word should be included in a family. However, the process that LEJEUNE described was similar to the others—an iterative process of identifying the questions to be addressed, using the tool to access the data that could illuminate those questions, and through a process of exploration, retrieval and comparison develop the analysis.

Transana stood out because it was originally designed to support video analysis. Its tools to support video/audio analysis have been highly refined and far surpass the capabilities of ATLAS.ti, MAXqda and NVivo in that area. In particular, the ability to have multiple transcripts linked to a video or multiple video views of the same event is unique in this range of software. However, the analysis process that David WOOD and his colleague described was similar to the others. He described four analytical passes through the data. The tools he used— transcribing and organizing, coding, visualizing and searching the data for patterns—may have looked different from the others but they shared the functions of organizing and exploring the data (see DEMPSTER & WOODS, 2011).

As I mentioned earlier, all the presenters came up with very similar answers to the research questions they were asked to address. And that is the clue. These experienced qualitative analysts were guided by the research questions and adopted appropriate methods to answer those questions. They used the software package with which they were most familiar. The defining difference between the packages was not methodological but the type of data they support. As the corpus included a wide range of data of different types, the analysis could be covered not only by the multi-media packages of ATLAS.ti, MAXqda and NVivo but the textual analysis package Cassandre and the audio/video specialist package Transana. It is the analyst that directs the analysis in the software, not the other way round. However, it is true that the data is displayed differently in the various packages and I feel that research could be done to see how this impacts on both the analyst and the analysis. My hunch is that the more options you have to display the data, the better the analyst can "see" and explore patterns.

My advice to novices is to 1. get a firm grounding of the various approaches to analyzing qualitative data and 2. learn one package very well. The choice of package depends firstly, on what colleagues in your area are using so you can tap into a support network and secondly, on what type of data you are likely to use.

Davidson, Judith & di Gregorio, Silvana (in press). Qualitative research and technology: In the midst of a revolution. In Norman K. Denzin & Yvonna S. Lincoln (Eds.), Handbook of qualitative research (4th ed.). Thousand Oaks: Sage

Dempster, Paul G. & Woods, David K. (2011). The economic crisis though the eyes of Transana. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(1), Art. 16, http://nbn-resolving.de/urn:nbn:de:0114-fqs1101169.

di Gregorio, Silvana (2010). Using Web 2.0 tools for qualitative analysis: An exploration. Proceedings of the 43rd Annual Hawaii International Conference on System Sciences (CD-ROM), January 5-8 2010, Computer Society Press.

di Gregorio, Silvana & Davidson, Judith (2008). Qualitative research design for software users. Maidenhead, Berks.: Open University Press/McGraw-Hill.

Kuckartz, Anne M. & Sharp, Michael J. (2011). Responsibility: A key category for understanding the discourse on the financial crisis—Analyzing the KWALON data set with MAXQDA 10. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(1), Art. 22, http://nbn-resolving.de/urn:nbn:de:0114-fqs1101222.

Lejeune, Christophe (2011). From normal business to financial crisis ... and back again. An illustration of the benefits of Cassandre for qualitative analysis. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(1), Art. 24, http://nbn-resolving.de/urn:nbn:de:0114-fqs1101247.

Silvana di GREGORIO received her PhD in social policy and administration from the London School of Economics and Political Science. She has worked in several applied research settings including the Nuffield Centre for Health Services Studies, University of Leeds, and the Department of Social Policy, Cranfield University, where she was involved in numerous practitioner-research studies. She was Director of Graduate Research Training at Cranfield School of Management during the 1990s where she developed her interest in methodological issues, particularly looking at the affordances of software to support the analysis of qualitative data. In 1996, she resigned her position at Cranfield to focus on consulting and teaching on a range of packages that support qualitative analysis. SdG Associates is her consulting business. She is coauthor with Judith DAVIDSON of "Qualitative Research Design for Software Users" (2008), which addresses both methodological and practical issues related to working with qualitative data analysis software packages—regardless of which brand of package is used. She is also exploring the use of Web 2.0 tools to support the analysis of qualitative data. She is on the Advisory Board of the European Chapter of the Merlien Institute, which promotes innovations in qualitative research.

di Gregorio, Silvana (2011). Comment: KWALON Conference: Is Qualitative Software Really Comparable? Reflections on "the Experiment": An "Expert" View. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(1), http://nbn-resolving.de/urn:nbn:de:0114-fqs1101C35.

Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (FQS)

ISSN 1438-5627

Creative Commons Attribution 4.0 International License