Volume 1, No. 3, Art. 7 – December 2000

Confidentiality and Informed Consent: Issues for Consideration in the Preservation of and Provision of Access to Qualitative Data Archives

Louise Corti, Annette Day & Gill Backhouse

Abstract: This paper highlights the main issues concerned with preserving fieldwork "contracts", such as informed consent agreements, as they relate to the conduct of research and the archiving of qualitative data. We pay particular attention to the techniques and efficacy of anonymisation and, outline methods of gate-keeping for access to data. Our discussions are based on seven years experience of Qualidata, the ESRC Qualitative Data Archival Resource Centre in dealing with a wide range of qualitative data, including interviews with public figures, and the raw material arising from some of the most classic empirical studies in the UK.

Key words: qualitative data, archiving, qualitative research, secondary analysis, confidentiality, informed consent, data access, gate-keeping, Qualidata

Table of Contents

1. Introduction

2. The Ethical Position

2.1 Informed consent

3. The Legal Position

3.1 Duty of confidentiality

3.2 Criminal activities

4. Maintaining Informed Consent Agreements in Archived Data

4.1 Solutions for enabling short-term re-use of qualitative data

4.2 Options for protecting confidentiality in qualitative data archives

4.3 Providing access to data—anonymisation

4.3.1 Is anonymisation appropriate for every dataset and if so, to what degree?

4.3.2 What is involved in anonymising data?

4.3.3 Anonymisation of particular kinds of datasets

4.4 Warnings about anonymisation

5. Are there any Qualitative Datasets that are Impossible to Archive?

6. Confronting and Dispelling Researchers' Worries: Negotiating Data Deposit

7. Conclusion

Acknowledgement

References

Authors

Citation

 

1. Introduction

Qualidata, the ESRC Qualitative Data Archival Resource Centre, has undertaken considerable consultation within the research community, as well as liaising with potential depositors of data, concerning the issues of confidentiality and informed consent (see CORTI 2000, and the Qualidata short description). These have undoubtedly been the most frequent causes of concern in the archiving of data. Data archivists have a deep concern both for the rights of participants and the professional integrity and peace-of-mind of researchers, and therefore both the issues of confidentiality and informed consent must be addressed in the context of archiving qualitative material. [1]

In order to place these issues within a legal framework, Qualidata worked with the Economic and Research Council to commission a report. This report sought to provide the research community with a clearer picture of the main legal consideration relating to confidentiality, informed consent and copyright. The consultancy took the form of a literature review and, interviews and focus groups, from representatives from within the social science research community. [2]

This paper highlights the main issues and solutions in preserving fieldwork "contracts", such as informed consent agreements, as they relate to the conduct of research and the archiving of qualitative data. [3]

2. The Ethical Position

The research community has long recognised the importance of respecting the rights of research participants. As a consequence, both the various professional and commercial organisations within the field of social science research have ethical guidelines and rules of conduct. These include:

All have much in common in their recommendations, and reflect the general principles prevalent within the social science research community. The guidelines are meant, primarily, to inform members about the ethical judgements they need to make rather than to impose standards. Few, however have sought to address issues such as interviewing in difficult circumstances, researching sensitive topics and the archiving of qualitative data. The ultimate responsibility for ethical decisions relating to a research project has traditionally been placed with the researcher. Professional researchers are expected to protect the well-being of participants in addition to maintaining the integrity of their profession. [5]

2.1 Informed consent

Research should, as far as possible, be based on participants' freely volunteered informed consent. This implies a responsibility to explain fully and meaningfully what the research is about and how it will be disseminated. Participants should be aware of their right to refuse to participate; understand the extent to which confidentiality will be maintained; be aware of the potential uses to which the data might be put; and in some cases be reminded of their right to re-negotiate consent. [6]

However, the issue as to what extent participants can ever be fully informed is a much disputed one. Explaining the details of a research project and the intentions of the study intentions requires is a prerequisite before entering into fieldwork, but we should never assume that all participants have a detailed appreciation of the nature and aims of academic research. Finally, consent alone does not absolve the responsibility of researchers to anticipate and guard against potential harmful consequences for participants. [7]

3. The Legal Position

3.1 Duty of confidentiality

In UK law, as it stands today, there exists a "duty of confidentiality". It has not been established by Parliamentary Act but has been developed through case law. In addition, the laws relating to contracts may come into play. If an explicit statement of agreement has been made concerning the extent of the confidentiality to be afforded to the supplier of confidential information, this may constitute a contract. A duty of confidentiality can also arise without an explicit statement of this kind. It may be established in two situations: when confidential information is passed, in confidence, to the confidant (the receiver of the information); and when it has been supplied in circumstances in which the confidant might reasonably suppose it to be confidential. [8]

However, this duty of confidentiality may be negotiable in that it requires the confidant not to disclose information unless authorised to do so and only in ways agreed. Authorisation is based upon an undertaking or understanding about the ways in which the information will be used (for example if I tell you a secret, I expect you not to pass it on), for what purposes, and to whom it will be disclosed. A breach of confidentiality may occur if the information is used in any other way. Although undertakings can be verbal, there are advantages in a written record. [9]

3.2 Criminal activities

There is no legal obligation to disclose information received relating to criminal activities unless legal proceedings or an investigation are underway. Even then, the confidant will only be guilty of perverting the course of justice if a researcher deliberately evades questioning. Researchers are therefore unlikely to be under a legal duty to disclose unless actually approached by the police with regards to the specific information or case in question. Participants should be aware of this before they reveal possibly incriminating information. [10]

4. Maintaining Informed Consent Agreements in Archived Data

We must attempt, at all times, to guarantee promises of confidentiality made to research participants, where possible. Data Protection legislation was established to ensure that personal information, where desirable, should be kept confidential and be stored in a secure manner according to the provisions of the Act. Whilst not all data subjects may be concerned about their anonymity, others are. For those subjects who wish to remain anonymous, for archiving we must seek to anonymise identifying information about them. [11]

However, even for the original researchers, adhering to guarantees of anonymity at this stage in the research process may become problematic. The first publication arising out of analyses of qualitative data may, perhaps, be the first time facts or stories about the subjects under study are "outed". This can be a most challenging time for researchers—The rich nature of qualitative data lends itself to descriptions of the interviewees, their lives and their surroundings, and as such dilemma is presented to the researcher in how much detail to reveal. Is it really possible to completely disguise a workplace or a village or the central characters in the drama? The situation is even more complex when we begin to think about other researchers re-using data they have not collected themselves. How can we ensure that respondents are suitably protected? Can we trust re-users to act responsibly? What measures are available for enabling qualitative data archivists to provide access to data? How realistic is anonymisation? [12]

It must be pointed out here that Qualidata is still seeking answers from the qualitative research community in order to reconcile the difference in informed consent practices for survey research and for qualitative studies. Qualitative researchers often feel the need to enter into sometimes extended and complex informed consent agreements with research subjects. For social surveys however, prior to the Data Protection Act at least, respondents were rarely asked if they consented to their responses data being coded up and stored in a database—this fact was taken for granted by the investigating team by virtue of a verbal guarantee of anonymity promised to survey respondents before data collection. The majority of large-scale social surveys are archived—preserving statistics has been on national agendas for many years. In order to rationalise the discrepancies in seeking informed consent between the methods, we should perhaps establish which practice is legally "correct". Ethically, consent for the long-term storage of and access to data use should be obtained in all fieldwork circumstances, where this is possible. Covert research, seen as a powerful research tool of the 1960s and 70s is now generally frowned upon—it is viewed as an infringement of people's rights. However, the argument for making better use of publicly funded research, by exploiting data further than originally intended (as is the practice for survey data) is often considered to be in the public interest. The main problem here is the lack of any UK case law which establishes whether archiving data is "legal" or not. [13]

4.1 Solutions for enabling short-term re-use of qualitative data

In trusting new researchers to use data soon after the data have been collected, we trust that they will respect the rights of participants. The overriding difference between the primary and secondary analyst is a familiarity with the nature of the relationship between the investigators (or those delegated to carry out the fieldwork) and the participants—the agreements set up may well have included verbal promises. The following discussion concerning short-term use is framed within the context of gaining access to data within the lifetime of the participants. [14]

Naturally, as time passes the exposure of information which may pose an apparent, or real, threat to research subjects, becomes less of a concern. Inevitably, participants die, and the details contained within the material, become a historical record. In 50-70 years time, it would not usually be problematic to allow access to the originals. [15]

How are we to decide who can have access and who cannot? First, we must place a significant portion of the initial responsibility for allowing people to have access to data with the original investigators (although additional measures should always be in place to ensure that they have the correct information). For any potentially archivable dataset, Terms and Conditions of Future Access are established in careful negotiation with the depositor(s). Below we outline Qualidata's procedures and criteria used for drawing up these Terms and Conditions. Second, re-users given access to data should be made aware of any specific informed consent agreements and guarantees of anonymity made with research subjects. Users should be asked to respect and abide by these, in the same way that the original investigators writing up their first piece of analysis should have done. [16]

4.2 Options for protecting confidentiality in qualitative data archives

Qualidata has established a number of options aimed at enhancing the protection of confidentiality for participants who have contributed personal information to a study, where this has been a prerequisite. [17]

These are:

A period of closure is imposed on a collection before access is allowed. For some interviews, such as those with public figures, closure may be the only solution to enabling data to be accessible in the future. As repositories are often resistant to accepting material that cannot be used for a long period of time, the dataset would have to be judged to be of exceptional value. Occasionally, parts of datasets are closed, for example a set of 15 of 20 interviews from a study may be closed for 25 years, but the other five may be immediately accessible. The decision to impose a period of closure is typically negotiated by the depositor and/or participant. In rare cases where the investigators of a set of potentially sensitive material cannot be traced, Qualidata will make an informed decision about closure. [19]

4.3 Providing access to data—anonymisation

In this chapter, we discuss the realities and practice of anonymising qualitative data. There is almost nothing written on this topic, with the exception of an interesting discussion paper drafted by ROCK (1999) on "Policy and Practice in the Anonymisation of Linguistic Data". [20]

Anonymising qualitative data, where practical, typically involves:

4.3.1 Is anonymisation appropriate for every dataset and if so, to what degree?

As we have seen, not every dataset will require deletion of identifying details. Other options for making a given dataset available should be considered. Gate-keeping is increasingly being used to allow restricted access to data. If anonymisation is deemed to be necessary, then it is important to arrive at an appropriate level of anonymisation. In some cases it can be difficult to disguise the identity of participants without introducing an unacceptable distortion into the data. Indeed, removal of too much detail can lessen the potential of data for re-use. The level depends on the history and nature of the study and each case may present a unique set of concerns. In some cases, revealing the names of regions and towns may not be problematic, in other cases the consequences of disclosure could be damning. [22]

4.3.2 What is involved in anonymising data?

Qualidata has gathered much expertise in the anonymisation of datasets and we are able to provide advice about this process. Since our first large-scale anonymisation project in 1996 we have drawn up a set of procedures which, so far, has been reliable. Anonymisation is carried out for both paper-based and machine-readable text and is always done in close collaboration with investigators. [23]

The following basic strategies are used by Qualidata for anonymisation:

4.3.3 Anonymisation of particular kinds of datasets

As qualitative data archivists we acquire a range of datasets, in a range of formats, and in varying states of "anonymity". We have classified datasets into three categories and below discuss how we would approach the anonymisation of each type. [25]

a) Word-processed data

Sometimes, investigators have used pseudonyms and have paid attention to a system of anonymisation which they hope will protect the anonymity of participants. We always discuss the validity and reliability of their system and may suggest that identifying details not already disguised in the documents should be altered). [26]

In many cases, particularly for older datasets, transcripts have not been anonymised, other than surnames and addresses having being removed. We discuss with the depositor what level of anonymity is required to meet the conditions of informed consent for the study. If data are to be anonymised we establish which terms need to be replaced by pseudonyms and what these pseudonyms should be. The most preferable schema is where the names match those used in resulting publications based on the data. Transcripts should always be skim-read to detect whether any other types of information might have damaging implications if revealed. In cases where Qualidata has undertaken the anonymisation process, machine-readable documents are edited in MS word with the Track Changes option set to Highlight Changes. Once the changes are agreed by all the key parties with concerns (eg. depositors, participants, sponsors, Qualidata) the changes are confirmed. The documents are typically preserved as rich text and/or ASCII files. [27]

b) Paper documents

There are two options here: either identifiers on the originals are tippexed or deleted with pen, or the originals are photo-copied and the copy anonymised with tippex or pen (there may be a case for retaining the originals under closure where the depositor will allow this); or the paper is scanned, and if possible, optically read into ASCII text. Anonymisation is then carried out as for a) above. As scanning technology and software advances we are tending towards the scanning option, although it may only be possible to do this for larger collections with additional devoted funds. [28]

c) Audio and video recordings

Clearly, it is very difficult to anonymise audio recordings, and usually inappropriate to censor video-recordings. As a result, either transcripts are deposited without the original recordings, or access to the recordings can be closed for a period of time, or have additional restricted access. In all cases up until now, we have not archived video recordings unless explicit consent for archiving has been obtained from the subjects. [29]

4.4 Warnings about anonymisation

Finally, we highlight a number of issues that should be borne in mind by qualitative data archivists before agreeing to undertake the anonymisation of a qualitative dataset:

5. Are there any Qualitative Datasets that are Impossible to Archive?

On this point we would have to say yes. All social science researchers have a responsibility not to expose their participants to potential injury, whether through making accessible confessions of illegal activity, opening them up to libel suits, or putting them at risk of harm, scandal or ridicule. Holding data in a "public" place could present a risk—under these circumstances, some qualitative research material may be intrinsically impossible to archive. [31]

6. Confronting and Dispelling Researchers' Worries: Negotiating Data Deposit

In this section we reveal the concerns of the UK research community. We show how the options for archiving discussed in the previous sections can help counter some of the key worries. Our experience of liaising and negotiating with around 1500 qualitative researchers over a period of seven years, has highlighted a number of reasons as to why some researchers show great scepticism about sharing and re-using qualitative data. Since 1995 we have published documents on this topic which discuss the kinds of worries together with suggested solutions to try and overcome them (CORTI 1999). We have always figured that the best way to dispel the (initial) rising tide of scepticism about archiving qualitative data in the UK was to introduce the problems into the debating arena as quickly as possible. [32]

The concerns expressed all have one thing in common—they have arisen largely from an unfamiliarity with any kind of established culture in social science of secondary analysis of qualitative data. Few contemporary social scientists have either shared their own data or used archived data, the exception being oral historians, who, by virtue of their use of the life-story method, have sought to preserve and exploit the long and rich narratives arising out of their research. Few qualitative researchers in the past have experience of using survey data, and are generally not aware of the huge potential of exploiting data collected by others. [33]

Naturally, the most common concern is that surrounding confidentiality. Other concerns are based on fear, insecurity, ignorance or arrogance, which are alluded to above, are discussed in CORTI in this issue. [34]

Concern No. 1

I don't understand anything about archiving-if I do agree to share my data it I would like to have some kind of control while I'm still alive, or at least, still of sound mind.

Qualidata offers web pages on Guidelines for Depositing Qualitative Data which set out details of requirements for depositing. Elucidating the process of archiving, from setting conditions of re-use to preparing and documenting data, is a vital stage at the pre-depositing stage. In 1996 Qualidata prepared a document called "Journey of a Qualitative Dataset" which took readers through the entire process, summarised as a single page flowchart and allowed them to take a peek "behind the scenes" at Qualidata. [35]

Potential depositors should be made aware at an early stage that they have the option to retain some degree of "control" over the materials once they are archived-by setting out terms and conditions for access and re-use. Indeed, many depositors, often for good reason, do prefer to maintain some degree of gate-keeping to "their" data, and, many have only agreed to archive when they are satisfied that the data will be used in a manner they deem to be appropriate. Some demand to see drafts of papers or reports submitted for publication (but, as may be expected monitoring this may be difficult). [36]

Concern No. 2

I am unsure whether I can share this data.

I gave promises to the participants that their identity would be completely protected.

Or

I did not ask, explicitly, participants' permission to archive their contributions.

As we have seen in an earlier section, the complete protection of anonymity that researchers sometimes offer their participants is often untenable—a first publication which a journalist then seizes upon may undermine this promise with a misguided stroke of a pen. In essence it is impossible to promise total anonymity. In contrast, we have found that when depositors have recontacted participants in a study to ask for permission to archive, the majority seem to be in favour, even though this wasn't mentioned at the time of fieldwork. Our experiences tell us that, providing their contribution is not abused, for example, their identifying characteristics are not cited (if they choose them not be), they are happy for serious scholars of the future to look at the raw materials. Most people do believe that research is for the public good, and that their contribution will be used in some way to create a better informed society, and even go some way towards implementing policy changes. A minority of participants who have been recontacted have asked for a period of closure for their interviews. [37]

Although recontacting is an option, it can be time consuming to trace people and, depending on what participants were told at the time of fieldwork, it may be unethical to recontact them. [38]

Finally, it is clear that interviews with public figures should be treated carefully. Not only are the interviews often carried out under "non-attributable" agreements, such as Chatham House rules (Royal Institute of International Affairs) in the UK political sphere, but also the interviewees are more likely to own copyright in the words they have provided. Furthermore, as public figures are more likely to want to publish their own memoirs, sometimes with financial reward, researchers should take care to clarify the use of quotations from the interviewee. Qualidata has produced a set of Guidelines on Copyright for Researchers. [39]

So, how does Qualidata approach interviews with public figures? First of all, these interviews are, often without exception, of excellent quality, in they are often collected by interviewers of the highest calibre, and that a broad range aspects of life about these figures is revealed. The research and historical value is undisputable. Anonymisation is usually not a feasible option, not only would it destroy the salience of the interview, but also it is impossible. Putting together even a few minor details could reveal the identity of the person. Recontacting "elite" interviewees to gain permission to archive is really the only possible option for archiving the interview, other than the more severe option of closure of interviews.

Example: In 1998 Qualidata was offered a set of interviews conducted, in 1984-85, with figures who had been left-wing student political activists in the 1960s. These interviews formed part of a larger project which explored the memories, motivations and experiences of those involved in the politically radical, student movements of the late 1960s in six of the West's industrialised countries. Ironically perhaps, many of the participants later went on to become members of government or high-ranking officials in other political or public organisations in the UK. Removing identifying information, such as names and places, would not have disguised informants who could be readily identified from descriptions of the activities they had engaged in. Qualidata provided the administrative support to enable the investigator to recontact most of the interviewees. The fact that many were in public office or prominent figures made re-contacting them relatively straightforward. Fifteen out of 18 agreed that their contributions could be archived, with their names attributed, in a prestigious academic repository, based at the London School of Economics in the UK. [40]

Concern No. 3

I am really concerned about asking participants for permission to archive their interviews.

Since 1996, researchers with Economic and Research Council (ESRC) funding are expected to offer their data (all kinds) for archiving as part of their contracts. Naturally, with this obligation in mind, investigators have a responsibility, where practicable, to try to gain consent from participants in their study. In this light, many investigators have had to rethink negotiations about informed consent and have had to be prepared to discuss with their participants, at some stage, access to data beyond their own team and the life of the project. [41]

In spite of the great worry expressed by researchers in confronting participants about further use of their contributions, the task has been, overall, achieved with great success. Indeed, feedback since 1996, suggests that many participants do not have a problem with this concept. They have spared time to offer information, and for qualitative research this may have been hours, and expect use to be made of it. These findings suggest that some investigators may be taking an unduly paternalistic stance towards "protecting their research subjects" and should, where possible, perhaps seek to discuss issues of usage of the data in more depth with respondents. [42]

Exceptions to this last statement are where the prospective participants are children or vulnerable adults. Qualidata offers specific advice in these circumstances, based on two documents: Guidelines for Researchers Interviewing Children and for Research with People with Learning Difficulties: Confidentiality and Informed Consent (ROLPH 2000). In these cases, it is a legal requirement to establish whether the child or "vulnerable" adult has the "capacity" to understand the meaning of informed consent, and if not, that a nominated "responsible" adult should determine whether participation in the research project, and subsequent archiving of the material, will not have adverse affects on the research subject. [43]

7. Conclusion

This paper has demonstrated that ethical considerations are of paramount importance in the archiving of qualitative research data. In this paper we hope to have demonstrated that some of the fears expressed by researchers about sharing data can be alleviated—there are robust techniques available for safeguarding research participants. [44]

In order for data to be archived in a relatively seamless way, ethical and data preparation issues need to be addressed up-front at the start of new research projects, preferably prior to data collection. Researchers need to explain in clear and understandable language what they intend to use data for, and where possible, gain consent in writing. For medical research in the UK and, for all research in the US, it is mandatory to get a written agreement for consent prior to any interview. Unfortunately, in the UK, many Medical Research Ethics Committees (MRECs, http://dialspace.dial.pipex.com/mrec/ [broken link, March 02]) have little knowledge or experience of qualitative research which prevents a proper consideration of archival issues. Feedback from researchers who have had to gain permission from MRECs to conduct a study suggests that MRECs sometimes place extreme demands on researchers. These demands have included gaining consent for each step of the research and ensuring data are destroyed on completion of a project, without exploring other avenues to "protect" participants. Qualidata is currently contributing to a review of MRECs procedures, being carried out by the National Health Service Research & Development Executive, in such a way that will help elucidate the process of qualitative research and the value of sharing data top Committees. [45]

Finally, we end on a note about European Data Protection legislation which may have far reaching consequences for data archiving (see The European Union Directive 95/46/EC for Data Protection, http://www.privacy.org/pi/intl_orgs/ec/eudp.html [Broken link, FQS, August 2005]). On the face of it the Act appears to make archiving research data an illegal activity without the data collector or person storing data having obtained prior permission to store and allow access to data. Permission should result from the donor having been provided with explicit details of how data will be re-used, and whether it will be stored in an anonymous form. Whilst we can feel relatively happy that we are allowed to use data in anonymised form, archivists are pressing for a more liberal interpretation of the law to ensure that within the law, there are clearly defined exemptions of other more "sensitive" archives for research purposes. The main point in our favour is that there is little or no case law, at least in the UK, which has unearthed any complaints by research participants about misuse of their contributions. [46]

Acknowledgement

Qualidata is supported by the Economic and Social Research Council (ESRC) of the UK.

References

Corti, L. (2000). Progress and problems of preserving and providing access to qualitative data for social research-the international picture of an emerging culture (58 paragraphs). Forum Qualitative Sozialforschung / Forum: Qualitative Social Research [On-line Journal], 1(3), Art. 2. Available at: http://www.qualitative-research.net/fqs-texte/3-00/3-00corti-e.htm.

Rock, F.E. (1999). Policy and Practice in the Anonymisation of Linguistic Data. Unpublished Report, Department of Linguistics, University of Birmingham, UK.

Rolph, S. (2000). Research with People with Learning Difficulties: Confidentiality and Informed Consent, http://www.essex.ac.uk/qualidata/forms/ldiffs.htm.

Authors

Louise CORTI is currently the Deputy Director and Manager of Qualidata, the ESRC Qualitative Data Archival Resource Centre, based at Essex. In January 2001 she will be taking up the post of Director of User Service of the UK Data Archive, where alongside the duties of that role, she will retain an overall responsibility for qualitative data archives. In the past she has taught sociology, social research methods and statistics, and spent six years working on the design, implementation and analysis of the British Household Panel Study at the University of Essex. She is interested in both qualitative and quantitative aspects of social research.

Annette DAY was appointed as Qualidata's first Researcher Support Officer in 1998, a post dedicated to providing advice and support to ESRC researchers on all aspects of preparing qualitative research data for archiving at the grant application stage, during the project and through to the deposit of data. She contributed much to the development of Qualidata's sets of guidelines on ethical issues, and on guidelines for interviewing children. She left Qualidata in January 1999 to become an oral history archivist at the Museum of London.

Gill BACKHOUSE holds a professional qualification in social work and has trained in social research methods. She has experience of working in both social work and social research over a number of years. She has been working as Researcher Support Officer for Qualidata since March 1999.

Contact:

Qualidata
University of Essex
Colchester CO4 3SQ, UK

Tel.: +44 1206 873058 / 297

E-mail: quali@essex.ac.uk
URL: http://www.essex.ac.uk/qualidata/

Citation

Corti, Louise; Day, Annette & Backhouse, Gill (2000). Confidentiality and Informed Consent: Issues for Consideration in the Preservation of and Provision of Access to Qualitative Data Archives [46 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(3), Art. 7, http://nbn-resolving.de/urn:nbn:de:0114-fqs000372.

Revised 7/2008

Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (FQS)

ISSN 1438-5627

Creative Common License

Creative Commons Attribution 4.0 International License