Observation of Online Communities: A Discussion of Online and Offline Observer Roles in Studying Development, Cooperation and Coordination in an Open Source Software Environment

Abstract: This paper addresses the application of observation to online settings with a special focus on observer roles. It draws on a study of online observation of a virtual community, i.e. an open source software (OSS) community. The paper examines general and specific advantages and disadvantages of the observer roles in online settings by relating these roles to the same roles assumed in offline settings. The study suggests that under the right circumstances online and offline observation may benefit from being combined as they complement each other well. Quality issues and factors important to elicit trustworthy observational data from online study settings, such as OSS communities, are discussed. A proposition is made concerning how threats to credibility and transferability in relation to online observation (i.e. lack of richness and detail, risk of misunderstandings) can be diminished, while maintaining the level of dependability (which is potentially high due to a greater degree of anonymity and "isolation" in online settings). The paper thus suggests that the less participative the researcher's online observer role is, the more s/he should consider introducing offline data collection techniques rather than adopting a more participative role in the observed online setting. This methodological discussion forms the basis for making a well-considered choice of online observer role rather than passively sliding into a role assigned by the setting.

Key words: online data; observation; observer roles; mailing lists; open source software communities

4. An Illustrative Case Study of Online and Offline Observation of an Open Source Software Community

5. Discussion: Anonymity and Identity in Online Observation and their Effect on the Research Quality

This paper discusses online and offline observer roles in the study of open source software (OSS) development, cooperation and coordination. It meets the call for research into how OSS development is organized (KRISHNAMURTHY, 2005), especially in relation to the cooperation and coordination mechanisms used to manage distributed development teams (CROWSTON, 2005). Methodologically, it positions itself in a grounded, ethnographic research tradition (LIN, 2005). The inadequacy of traditional theories in explaining OSS (BITZER & SCHRÖDER, 2006a) testifies to the need for proper methodologies for studying the OSS phenomenon. As argued by CROWSTON, LI, WEI, ESERYEL and HOWISON (2007), the qualitative research method "aims at building understanding of, rather than just measuring, development practices" (p.564). In line with this argument, the paper focuses on qualitative research methods and in particular on a qualitative participant observation technique, as we find it most suitable for investigating social processes and practices in OSS development. [1]

The OSS phenomenon has been held up as one of the primary phenomena of the nascent networked information economy (BENKLER, 2006), which has given rise to new and effective, large-scale cooperative efforts through peer production of information, knowledge and culture. This new type of collaboration (BENKLER, 2006; FRIEDMAN, 2006; TAPSCOTT & WILLIAMS, 2006) is made possible by the availability of new, low-cost collaborative infrastructure of the Internet, which has been the critical tool for facilitating the OSS development (BITZER & SCHRÖDER, 2006b). Differently put, virtual communities, which are the natural context for OSS development, cooperation and coordination, are ideal settings for studying innovative ways of product development (VON HIPPEL, 2001; VON HIPPEL & VON KROGH, 2003). [2]

Observation as a method of data collection and analysis enables the researcher to gather data across perspectives, time and in the phenomenon's natural setting (BABBIE, 1986; PATTON, 2002). It may therefore reveal implicit problems and offer important insight into and information about informal aspects of interactions and relations, which can be difficult to obtain through, for example, interviews. The data collection technique in online observation is grounded in interactive Internet-based (i.e. virtual) communities. Data gathered through online observation is not merely disembodied exchanges of text. If online communities are capable of existing and growing (e.g. open source communities) when individuals electronically gather, exchange and share information and knowledge, cooperate and develop information and communication technology-based products, then it could be argued that the "texts" they exchange must contain information on the various aspects of the relations between the communicating individuals. This, in turn, can constitute the foundation for studying power relations, work coordination, cooperation, product development, conflict, work culture and many other aspects of professional work-related processes in virtual communities. [3]

Computer mediated communication (CMC) has been studied from various perspectives and with a variety of methods relevant to the focus of this paper, including, among others: ethnographic accounts of specific virtual places (KLINE & MYERS, 1999), analysis of intra-organizational networks (COHEN & SPROULL, 1995), laboratory experiments comparing face-to-face and CMC (KIESLER, SIEGEL & McGUIRE, 1984; DUBROVSKY, KIESLER & SETHNA, 1991), electronic surveys used to investigate work organization in OSS communities (LAKHANI & VON HIPPEL, 2003) and observation in textual online settings (MARKHAM, 1998; HINE, 2000; LEANDER & McKIM, 2003; HOURAHINE & HOWARD, 2004). Much research has addressed the complexities and issues related to the adoption of offline social science-research methodologies to online settings (e.g. JONES, 1995, 1997, 1998; MANN & STEWART, 2000; DOMÍNGUEZ et al., 2007). Surprisingly, less specific work has been done on issues relating to the researcher/observer roles in online observation. Participation and isolation as the central design characteristics of the different observer roles are likely to have distinct implications in online compared with offline conditions. As the Internet becomes more and more integrated into our everyday lives, the importance of applying and adapting research methodologies to virtual communities increases as well. It is important to emphasize that the Internet is interactive by nature (TSOUKAS, 1997), which makes it imperative to understand the assimilation of the observer into the observed field in an online setting. For example, although the role of an investigator observing websites may, at first, be perceived as a complete observer role, the existence of personalized dynamic interactive websites like Amazon.com, which configure the website based upon earlier interaction, makes it questionable whether a complete observer role is attainable in similar cases. Consequently, the interactivity of the Internet may lead to unnoticed and unintended changes in observer roles. [4]

This paper systematically compares qualities of online and offline observation methods with the help of a case study of an OSS community. It is thus a meta-study of observation as a research method. In particular, the paper discusses issues connected to the application of observation in general and the observer roles in online mailing lists in particular. Mailing lists have been chosen, since they are the primary communication media in OSS communities. As such, they intimately reflect the nature of organizational practices, process and structures of those communities. This paper draws upon a study of an OSS community to illustrate and discuss general and specific implications of applying this method in online environments. This case study employed both offline and online observation, and it thus illustrates the role of the complete observer in an online setting and the role of the observer-as-participant in an offline setting. The study revealed the issues of anonymity and identity control as particularly relevant for judging the quality of research based on observation method. Online observation roles are thus elaborated and discussed in relation to anonymity and identity control in online settings, and their implications for dependability, credibility, transferability and confirmability—the widely acknowledged criteria for assessing the trustworthiness of qualitative research (LINCOLN & GUBA, 1985). Overall, the paper finds that studies may benefit much from combining offline and online methods of observation, as each contributes to the uncovering of different aspects of collaboration in online communities, and each offers different levels of analytical detail. [5]

The next section introduces a qualitative participant-observation technique and considers this technique in relation to research of online settings and mailing lists in particular. Section 3 outlines four specific offline observer roles and discusses how they can be applied in studying OSS communities. An illustrative case study of online and offline observations of an OSS community is presented in Section 4. Based on the findings and insights from the case study, Section 5 discusses and elaborates on the implications of aspects of anonymity and identity control on the four quality criteria in relation to the observation roles in online settings. Lastly, Section 6 presents the conclusion and implications for further research on online observer roles. [6]

The literature distinguishes between two overall types of observation: 1. unstructured observation and 2. structured observation. The former is a qualitative technique often unstructured in nature, and it seeks to uncover the subtle nuances of meaning in the behavior of social actors; the latter is a quantitative technique concerned with the frequency of actions (SAUNDERS, LEWIS & THORNHILL, 2000). The degree of predetermined structure is hence a particularly noteworthy difference between these two types of observation (ROBSON, 2002). Another related difference concerns their degree of formality. Whereas a formal approach to observation entails a large degree of structure and coding schemes containing predetermined categories for recording the observations, the informal approach involves less structure and allows the researcher substantial flexibility and freedom regarding data collection and recording, as no categories are predetermined (ibid.). Although unstructured observation often makes use of the informal approach, it can at the same time be systematic—not by virtue of its predetermined observational categories, but by virtue of its logical inference system (BØLLINGTOFT, 2005). This paper focuses on the qualitative unstructured observation technique applicable in a particular type of online setting: mailing lists. [7]

Online observation refers to textual exchanges that can be of both a synchronous (i.e. simultaneous, such as chat) and an asynchronous nature (i.e. non-simultaneous, such as e-mail). It takes place within mailing lists (discussions lists), chat, wikis, blogs with their commenting systems, and other interactive social media platforms. This paper is interested in asynchronous textual exchanges on mailing lists as a site for observational research, because mailing lists are the primary communication media for hundreds of thousands of production communities, such as OSS communities (GUTWIN et al., 2004; SHIHAB et al., 2010). In spite of this empirical limitation, our paper is indeed relevant for both the pre-Web 2.0 and the Web 2.0 types of interactive social media, because both are capable of facilitating participatory user-centered collaboration, as the example of OSS so finely proves. [8]

A mailing list has been defined as "a group of email addresses that can be contacted by sending a single message to one address: the list's address" (MANN & STEWART, 2000, p.12). It is a discussion group where a particular topic is discussed via e-mail distribution. Some mailing lists have restricted access, as participants need to be moderated before joining the exchange of messages. [9]

Accepting that the Internet as a technology has limitations as a communication medium and as an arena for qualitative methods of data collection and analysis (MANN & STEWART, 2000), this paper focuses on one of the ways the Internet lends itself to social inquiry through qualitative observation. Qualitative research uses many methods to collect contextually situated, rich and descriptive data with the purpose of gaining an understanding of human experience or relationships within a culture or a system (SILVERMAN, 2000). Observation is one of the ways to generate qualitative findings. It has been characterized as "fieldwork descriptions of activities, behaviour, actions, conversations, interpersonal interactions, organizational or community processes, or any other aspect of observable human experience" (PATTON, 2002, p.4). This definition is also applicable to the online environment, which allows the researcher to study a multiplicity of social phenomena through a variety of online communication media. [10]

Acquiring a basic understanding of a particular phenomenon is often accomplished in isolation from the natural context, which prevents the researcher from obtaining a complete picture of that phenomenon. This situation is remedied by the method of data collection and analysis presented here, which enables the researcher to gather data from multiple perspectives at different times and in the phenomenon's natural setting (BABBIE, 1986; PATTON, 2002). Using CMC as a means of studying human behavior and interaction has been found to minimize the constraints of time and space (MANN & STEWART, 2000). Thus, geographical distance is overcome, and individuals who would otherwise not be reached are significantly easier to get into touch with using this approach. In this case, CMC is seen as a tool for studying the virtual trace and outline of individuals, their representations (MARKHAM, 1998). However, it can also be viewed as a place to meet and interact with others, or as a way of being, since Internet users express their self through text (ibid.). CMC can accordingly be studied as a cultural sphere where individuals develop specific forms of communication, practices and/or specific identities. Embarking on a research journey into these new social dimensions generates a set of questions about how existing (offline) research methods will apply. Considering vast research populations, multiple sites of interaction and often anonymous informants, these methods ought to be adapted. [11]

To fully grasp online observation as a data collection technique, we present and discuss observer roles in relation to offline observations. In general, observation can be either 1. overt or 2. covert (STAFFORD & STAFFORD, 1993). The research is considered overt when the researcher makes his/her intentions and objectives known and obtains permission to observe a situation, i.e. the subjects are thus aware that they are being observed/studied (ibid.). The research is considered covert when the researcher becomes an insider, i.e. the subjects are unaware of the researcher's identity and consider him/her as a group member (JORGENSEN, 1989). In between these two extreme roles, i.e. covert (complete insider) and overt (complete outsider), the investigator can assume hybrid observer roles, i.e. be an outsider or an insider to different degrees. Accordingly, four general options for (offline) observational approaches are found in the literature. They differ in terms of the degree of the observer assimilation into the observed field and fall into the following categories:1. the complete participant, 2. the participant-as-observer, 3. the observer-as-participant and 4. the complete observer (GOLD, 1958; SPRADLEY, 1980; BABBIE, 1986; JORGENSEN, 1989; STAFFORD & STAFFORD, 1993; FRANKFORT-NACHMIAS & NACHMIAS, 1996). [12]

The complete participant assumes an investigative role by becoming a full member of the observed group without revealing his/her own identity. One of the main reasons for choosing this observational approach is to avoid unduly influencing the observed setting (BABBIE, 1986), i.e. when individuals are aware that they are being observed, they may alter their behavior because to observe is to inevitably interact at some level. The data are also expected to be of greater value as the researcher can obtain more accurate information having been a member of the group for an extended period (STAFFORD & STAFFORD, 1993). However, this observer role raises ethical concerns, as the researcher is not honest about his/her real identity. Furthermore, another two issues in relation to this role are that: 1. the researcher may be so self-conscious about accidentally disclosing his/her true identity that s/he may fail to perform convincingly in the chosen role, and 2. the researcher may "go native," i.e. the informants' views become the researcher's and thereby the observer role is violated (GOLD, 1958, p.220). [13]

This observer role is also applicable in online settings. For example, a researcher studying the authority structure and power mechanisms in OSS communities would become a participant in and contributor to such a community. Depending on the type of mailing lists (restricted or public access) the researcher can, more or less easily, act as a participant by raising and answering questions as part of the mailing list dialogue. A basic requirement is, of course, that the researcher is familiar with the particular topic of the discussions. This provides the researcher with an opportunity to reach a deeper understanding of the investigated phenomena and uncover aspects that s/he would not be able to uncover through other means, e.g. interviews. [14]

The participant-as-observer conducts the observations while acting as a full group member, but lets the subjects/informants know that they are under observation. Two potential problems may apply to this role according to GOLD (1958, p.221): 1. through interaction and subsequent identification with the researcher, the informant may become "too much of an observer," and 2. the researcher may over-identify with the informant, lose perspective, and thus "go native." These observations can be either informal or formal. [15]

For instance, the participant-as-observer studying an online OSS community would develop relationships with informants (i.e. other community members) over time. This would make him/her inclined to spend more time participating in than observing the interaction in the community. In formal observations s/he would, for example, conduct planned online interviews with the informants and make more or less structured observations of the communication on the community's mailing lists. On the other hand, there would be instances where s/he would engage in informal conversations and discussions and thereby make informal and unscheduled observations of informants online. Such informal observation could also include social events connected to the community conferences. [16]

The observer-as-participant assumes the role of an observer through social interaction with the informants, but does not pretend to be an actual group member (BABBIE, 1986). In this role, the researcher's relationship to the informants remains strictly research-related and does not develop into a relationship (ADLER & ADLER, 1994). [17]

This role may also be put into practice in online settings, for example by observing the mailing list interaction of an OSS project in order to identify and learn more about the kinds of project roles assumed and exhibited by those on the list. Taking up the observer-as-participant role would involve more observation than participation and often short, structured one-visit interviews. Because of this brief and perhaps also superficial contact with the informants, there is an augmented risk of misunderstandings (GOLD, 1958). On the other hand, as pointed out by PEARSALL (1970), this role enjoys two advantages: 1. informants may be more willing to talk to "attentive strangers" than to individuals whom they know better, and 2. there is less "temptation either for the observer to go native or for the natives to try to include him permanently in their lives" (p.342). [18]

Finally, the complete observer avoids influencing the observed activities, keeping a distance to the observed interactions, thus approximating the traditional idea of the "objective" observer (ADLER & ADLER, 1994). This role is the common hands-off role where no interview is conducted. [19]

An example of the application of this role in online environment may be the investigator who observes how tasks are assigned in an OSS community. Hence, the investigator would follow the community's mailing lists and observe without taking part in the group's activities. The investigator would observe, for example, whether the tasks are assigned to the developers by others and, in that case, by whom, or whether the developers choose tasks on their own (i.e. self-assignment). The Internet creates the ideal conditions for unobtrusive observation as demanded by the complete observer role. However, this detachment from the observed interaction increases the risk of misunderstanding the observed and it involves a greater possibility of ethnocentrism (GOLD, 1958; BABBIE, 1986). It can be difficult to overcome one's prejudices against others' behavior, customs, etc. without the option of two-way dialogues. [20]

4. An Illustrative Case Study of Online and Offline Observation of an Open Source Software Community

In a recent study, VUJOVIC (2007) examined how CMC affects work performance and coordination in an OSS community, and how such effects are dealt with. The OSS community investigated in this study is called TYPO3, and it has been public since the year 2000 and it is published and distributed under the GNU GPLv3¹⁾. The TYPO3 system is a small to midsize enterprise class Content Management System (CMS) offering out-of-the-box operation²⁾ with standard modules, such as news listing and archiving, events calendar, indexed search within PDF and Word files, e-commerce, e-mail newsletter facility, forum, guestbook, etc. TYPO3 is aimed at two different groups: authors as well as administrators and content managers. [21]

This project was chosen as an information-rich case, which could be studied in-depth in accordance with the study's research objective. According to the documentation available on the TYPO3 homepage, TYPO3 began experiencing fast growth in 2003 when the number of registered developers began to double on a yearly basis. This made it a suitable case for our study. Furthermore, its size and growth can be assumed to place an even greater demand on the communication within the project. TYPO3's Core Team members have played a central role in the community, since they contribute most of the source code and administer the design and development of the project on a voluntary basis. At the time when the study started, approximately half of these Core Team members (i.e. nine individuals) made up the project's Research & Development (R&D) Committee. This committee was the project's central coordination body as their responsibilities included supervising and coordinating development of the software; providing knowledge, contacts and financial support; and supervising and supporting community-driven teams. The R&D Committee members have been at the same time members of and contributors to the project's other teams and working groups. They were therefore found to be the most appropriate and crucial source of data for the purposes of this study. The informants were all male, all aged between mid 20s to late 30s, except one who was around 20. The youngest one was a student, while the rest of them were software developers and consultants. [22]

To examine how CMC affects work performance and coordination in an OSS community, and how such effects are dealt with, the study employed: observation of a two-day face-to-face meeting of the project's R&D Committee, observation of the interaction and communication within the R&D Committee on their mailing list (otherwise not open for public) during a period of five months and phone interviews with some of the R&D Committee's members. [23]

First, more than 18 hours were spent over a two-day period observing face-to-face (FTF) meetings between the R&D Committee members. This allowed the researcher to become familiar with the project, their discussion topics, issues they deal with, decision-making processes, problem-solving, etc. During the breaks, a number of short conversations were initiated. Detailed notes were taken after each break. The observations, together with the conversations, provided the researcher not only with an overview of the issues regularly addressed by the members, but it also accentuated various communication issues and thus provided important input to the subsequent identification of themes to be built into the semi-structured interviews. The investigator assumed the role of the observer-as-participant (GOLD, 1958; STAFFORD & STAFFORD, 1993; SAUNDERS et al., 2000), i.e. the investigator unobtrusively observed the Committee members and their communication during the meetings. However, during the breaks, the members approached the investigator who then engaged in short conversations, which were then used to diminish the probability of misunderstanding the observed (GOLD, 1958; BABBIE, 1986). Thus, it was difficult to maintain the role of the complete observer—the benefits and shortcomings related to this will be discussed in the findings section. Although it was highly useful to observe these face-to-face meetings, an online OSS community also needed to be studied in its natural environment. [24]

As a result, the investigator contacted the project leader who subsequently discussed with the rest of the R&D Committee, whether the investigator could also observe their communication on the mailing list. They all agreed to grant her access to the mailing list because they perceived their work as highly valuable and hoped that the research results would assist them in coping with certain organizational issues they were experiencing. Hence, the group members were all fully aware that the observer was a researcher and they also knew the purposes of her study. The researcher herself never participated in the mailing-list communication, thus trying to observe unnoticed to avoid influencing the group's communication in any way. Hence, she assumed the role of a complete observer in this online setting. Nonetheless, knowing that one is being observed is already an influence; however, its effects were unlikely to be significant, since the communication revolved around technical issues and discussions about coordination, to which the researcher did not have anything to contribute. [25]

Online observation consisted of reading all postings to the mailing list of the project's R&D Committee for a five-month period. Reading these messages allowed the researcher to become familiar with the wider scope, content and frequency of discussions; the contributions and roles of different individuals; work coordination and delegation; and to obtain a deeper insight into communication aspects. [26]

Although vast amounts of online communication were gathered, which were a potential source of rich data, the process also included a number of challenges. One of these challenges was that some of the things "said" in the online conversations (i.e. in the mailing list) contained implicit meanings or messages that were not entirely clear or self-evident to the researcher/observer because the contributors used various communication media, not only the one observed by the researcher. The online communication media used by the Committee members (besides the mailing list) included IRC, instant messaging and VoIP (Voice over IP, e.g. Skype) for private discussions. Other types of communication included phone conversations and face-to-face meetings. Thus, the researcher found that observing online communication exclusively through only one communication channel (i.e. the mailing list) sometimes was not enough. If the researcher does not have access to the conversations that take place through other communication media, the interpretation of data may therefore sometimes be more difficult and it may easily entail misunderstandings. Nonetheless, it was not feasible to access all private conversations occurring spontaneously among the members. However, including face-to-face observation and phone interviews made it possible to overcome the interpretation challenges mentioned. [27]

Another challenge identified was that in virtual communication settings body language, facial expressions, tone of voice, etc. are important but at the same time absent sense-giving dimensions. The investigator, for example, noticed clues to, e.g. facial expressions and/or emotions, through emoticons. However, these could not fully replace the various meanings an individual's facial expressions, tone of voice and body language can convey. [28]

These two factors (restricted access to communication/interaction and reduced cues and social presence) were found to increase the risk of making the potentially rich online observational data poor in two ways: firstly by overlooking important issues (in the communication taking place) and secondly by misinterpreting statements. This problem was, however, reduced because of the (offline) observation of face-to-face meetings undertaken before the online observation. It had been highly useful to get to know the individuals "behind the e-mail addresses" in person prior to the online observations, as it had provided extra insight into their personality and character as well as additional project-related information that made it easier to detect and identify various implicit aspects in the mailing list communication relevant to further inquiry. At this point, phone interviews were employed. In general, the triangulation of data sources increased the accuracy and trustworthiness of the findings by offsetting the risk of overlooking important issues and misinterpretation of statements in online data. To put it another way, the obstacles related to restricted access to communication/interaction (taking place among the R&D Committee members through other communication channels) and reduced cues and social presence in the online observation of the mailing list were significantly offset by the use of offline observation and phone interviews. [29]

Qualitative techniques were used to analyze the data (EISENHARDT, 1989; MILES & HUBERMAN, 1994; STRAUSS & CORBIN, 1998) with an overall focus on work practices and with specific attention to grounded theory concepts. The grounded theory approach was chosen because it emphasizes the actor's own emergent interpretations and meanings (GLASER & STRAUSS, 1967) and involves the possibility of discovering the unanticipated (VAN MAANEN, 1998). This process was based on a comparative analysis (LOCKE 2001; STRAUSS & CORBIN, 1998), which helped discover latent patterns (GLASER, 2002) in the observational data (i.e. the two-day meeting and mailing list) and the phone interviews. Raw data were purposively selected in order to identify concepts and relationships relevant to the questions raised (STRAUSS & CORBIN, 1998). The interviews, observation notes (from the face-to-face meetings) and mailing list data were analyzed to develop categories for information of interest. After each category was named, the data were reread to ensure that the codes were correct (i.e. that they describe what is actually going on in the selected part of the data). Furthermore, the data were iteratively re-examined to trace confirmatory or contradictory data fragments. The category was discarded or revised if no other instances emerged. On the other hand, categories that occurred repeatedly were refined by adding descriptors. After identifying and refining numerous categories, the next step was to understand how they were related to each other in order to explore underlying connectedness as an aspect of causality (STRAUSS & CORBIN, 1998). Hence, they were organized into an explanatory scheme to ascertain how different categories fitted into a coherent picture of theoretical logic. [30]

The case study of online and offline observation in an OSS community illustrates the investigator's intention of becoming acquainted with OSS development primarily through observation in the natural context, namely the mailing list of the core developer group. The investigator acted as a complete observer as this role was the best alternative considering not only the possible (undesired) influence on the observed setting, but also her lack of technical knowledge and skills, which prevented her from assuming one of the fully participative roles. In other words, the complete observation was a learning process for the researcher. [31]

Throughout the research process, the investigator discovered the value and the usefulness of combining offline and online observation in order to minimize the disadvantages related to the online observation. This process involved two different observer roles, i.e. observer-as-participant and complete observer. The findings indicate that a researcher has better opportunities to proactively manage the observer role online than in an offline setting. For example, during the offline observations some committee members approached the researcher to provide her with their interpretations of what was going on at the meeting. One of them, for instance, felt that he was not being involved in decision-making and another one was distrustful of one particular member. While this can reveal valuable information, it can also increase the risk of taking sides and not remaining neutral when becoming an insider or a confidant. Here, a reflexive analysis of the context and the research process, including the techniques of decentering and recentering, were crucial for the research quality (BREUER & ROTH, 2003). These techniques involved creating a self-critical meta-perspective on the researcher's practices and reflecting upon and rearranging the research process (ibid.). [32]

Overall, the findings in the case study support the assumption of some similarity between the online and offline observer roles. Both as an offline and online complete observer, the researcher is observing from a distance and is isolated from the phenomena (i.e. no direct contact is permitted). In both roles/settings, the advantages are that the complete observer role most closely approximates the traditional ideal of the "objective" observer, while the potential disadvantages are lack of richness and detail, potential for misunderstanding and inaccuracy and the possibility of ethnocentrism, when one's prejudices cannot be confronted in two-way dialogues. However, the potential for incurring these disadvantages was found to be much greater in online than in offline observation due to the restricted access to communication/interaction, reduced social presence and reduced cues. [33]

In addition, this study found that it was more difficult to assume and maintain the complete observer role in an offline rather than in an online setting, where the actualized role became the observer-as-participant role. In this regard, the dissimilarities between the online and offline complete observer roles could be used to understand this need for role modification. Firstly, the social setting in an offline observation has a greater impact on the actualized observer role than in an online setting (as the above-mentioned example illustrates). In the offline setting in the presented case, direct contact and interplay were established immediately upon the investigator's arrival at the scene of the meeting, and she was no longer able to have the same level of control over participation. In the online setting, the observed social interaction was textual, which made it easier for the researcher to maintain a distanced complete observer role. Secondly, in an online setting, the researcher's presence and identity are more easily hidden than in an offline setting. This applies both in the case of covert and overt observation, which makes it easier for the complete observer role to remain "invisible." For instance, during the online observation the researcher was never addressed nor contacted. In contrast, in the offline setting in the present study, a covert observation was not possible and the observed actors then chose to interact with the researcher. [34]

To sum up, the case study illustrates that online and offline observation can be productive complementary methods whereby the researcher obtains more fine-grained control over identity and anonymity. This may improve dependability because in an online setting the researcher also enjoys more control over the communication process than in an offline setting as she can revise and reflect on her statements before articulating them. The credibility and transferability of the findings in an online setting can, nonetheless, be feeble because of potential lack of richness and detail and the potential for misunderstanding and inaccuracy. Offline observation can, to some degree, strengthen the credibility and transferability by allowing more direct contact or interplay with the observed group members. Based on this recognition, the argument can be made that combining offline and online observation may significantly improve the quality of the research. [35]

5. Discussion: Anonymity and Identity in Online Observation and their Effect on the Research Quality

Besides the fact that the online observation of mailing lists and forums is generally easier and involves less complicated access (for covert or overt observation) than many offline settings, the findings discussed above suggest that the aspects of anonymity and identity differ between online and offline observation, which has implications for credibility, transferability, dependability and confirmability. This section will elaborate on the implications of these dimensions with respect to the four observation roles in an online setting only. [36]

Online observation allows a high degree of anonymity for all observer roles, when needed. MARKHAM (2004) notes that the anonymity granted by certain types of CMC allows individuals to create alternate identities online and the ability to disappear from interaction at any time. Hence, online settings enable participants to form their identities detached from their "real-world bodies" (TURKLE, 1995). MARKHAM (2004) furthermore argues that this leads to a feeling of increased control, e.g. control over the content and form of the message, over the presentation of oneself and over others' perceptions of the self. Identity thus seems to become a more controllable element in virtual rather than physical settings. This indicates that researchers can vary and adapt their observation roles more online than offline. Certain characteristics, which are found to disturb the observed interaction or to keep the researcher from accessing the setting, can remain unknown to the observed subjects. For instance, characteristics such as age, gender and race are invisible online, unless the researcher chooses to reveal them (MANN & STEWART, 2000). [37]

As a result, researchers have a high degree of control over their online identities, i.e. what and how much about themselves to reveal when they are assuming the online roles of complete participant, participant-as-observer, observer-as-participant and complete observer. However, this degree of control will also depend on the type of CMC utilized. In public forums and mailing lists, full control can be attained. The control over identity is less in a private/restricted mailing list setting, where the researcher can only get access by requesting permission, which would usually entail that the subjects will get somewhat acquainted with the researcher or, at least, with his/her purposes of research/observation. [38]

Moreover, the control aspect is also pertinent to the possibility of revising and reflecting upon the researchers' statements before articulating them, thus controlling the meaning of their statements. This is important as the researchers' statements influence the images they create of themselves in the observed settings. [39]

ANGROSINO and MAYS de PÉREZ (2003) note that in offline observation studies "... the quality of what is recorded becomes the measure of usable observational data (because it can be monitored and replicated) rather than the quality of the observational data itself (which is, by definition, idiosyncratic and not subject to replication)" (p.676). In online observations of textual exchanges in mailing lists, the observational data are equivalent to the recorded data, since the social interaction and behavior exist in a written form. Data are hence easily separated from interpretation, which is rarely the case in offline observation. This has a positive impact on the dependability factor in online observation making. [40]

Dependability is greatest in the online complete observer role, as the researcher's presence is most often "invisible" and the observer does not interact with the subjects. In the other three roles, dependability is likely to be less because the researcher interacts with the subjects and may thus cause changes in the social process in the observed setting. Furthermore, the problem of "going native" is also present in these roles (but mainly in the roles of participant-as-observer and complete participant). This problem may affect the data interpretation, also. [41]

In addition, in the role of observer-as-participant, the investigator has more control over the communication process. Especially asynchronous online communication, such as mailing lists, provides that opportunity. In general, when the group members are aware of the online presence of an investigator (both in the role of observer-as-participant and participant-as-observer) is likely to disturb them less than offline/physical presence. [42]

The researcher evidently seeks to strike a delicate balance between observation and participation when collecting data (ANGROSINO & MAYS de PÉREZ, 2003). In effect, when discussing the credibility and transferability of the findings, the interaction and the relationships between the researcher and the objects studied are of utmost importance. [43]

A distinction has to be made between two interrelated potential sources of misunderstanding and inaccuracy in online observation: one is the lack of shared physical context in CMC, the other is the lack of subjective involvement in the observed interactions. The former is directly related to the inherent technical aspects of CMC (and non-existent in offline observation), while the latter is connected to the degree of participation, i.e. the observer role of the researcher (and the same holds true in offline observation). However, both of these problematic sources indicate that the less the researcher participates, the greater will be the risk of misunderstanding and inaccurate observation. In consequence, this raises questions pertaining to the discussion on whether observational objectivity is desirable as a goal. [44]

A potential third source of misunderstanding and inaccuracy is the fact that observation of only one online communication channel (e.g. a mailing list) may be insufficient for understanding what is actually going on in the observed online setting. It is possible that the subjects also use other communication channels (e.g. using other CMC media, via face-to-face communication and by telephone) besides the observed one, which may impede the uncovering of all relevant details and issues and prevent a coherent understanding of the phenomenon studied. The more the investigator participates in the observed group activities, the greater is the possibility to firstly learn about these other communication channels, and secondly to inquire about the matters discussed there and thirdly to obtain access to the communication that takes place in those other channels. [45]

In the literature about offline participant observation, participation and observation are viewed as competing and conflicting objectives (JORGENSEN, 1989). In consequence, the more the researcher participates in the observed setting/activities, the less s/he is able to observe. This problem is diminished in online observations based on textual exchanges, such as those in mailing lists, forums and IRC. Participating in communication and interaction in the observed setting allows the researcher to log the synchronous communication of IRC as well as the asynchronous communication of mailing lists. Consequently, the researcher can even use these logs to assess his/her own influence on the interaction in the observed setting. For instance, communication and activities where the researcher is only the observer could be compared with those where s/he acts as a participant. Although our empirical work did not deal with this, existing literature and our experience indicate that this comparison could improve the general trustworthiness of results. [46]

Lastly, which of the four online observer roles is chosen in a particular study will, as one would expect, depend on the research purposes and the phenomenon under study. However, as regards the study of online environments such as OSS communities, a proposition can be made about how threats to credibility and transferability in relation to online observation (i.e. lack of richness and detail, risk of misunderstandings) can be diminished, while maintaining the level of dependability (which is potentially high due to a greater degree of anonymity and "isolation" in online settings). The less involved the researcher's online observer role is, the more s/he should consider introducing offline data collection techniques rather than adopting a more participative role in the observed online setting (Figure 1). For instance, in studying OSS communities through a complete online observer role, the researcher should avoid becoming more participative (unless required for the purposes of research), because the dependability level might be compromised. Rather, obtaining the necessary details and eliminating potential sources of misunderstanding should take place through additional data collection via offline interviews and/or observations. Most OSS communities allow for such types of data collection, since these communities also often have a vibrant offline dimension—conferences, workshops and other types of face-to-face meetings are part of their community lives. Going offline will naturally increase the researcher's degree of participation in the community while decreasing his/her degree of anonymity. However, the researcher will avoid influencing the social process in the observed online setting, which is of primary relevance for the study. The most desirable scenario for data collection in these types of online communities would thus be to complete online observations in the role of a complete observer first and subsequently collect the necessary data offline.

Figure 1: How to combine the researcher's online observer role with offline research techniques in order to diminish threats to credibility and transferability [47]

The above proposition is best suited to "non-sensitive" online settings such as OSS development communities where participants are open about their membership and contributions. In some vulnerable online environments where participants carefully guard their identities, the possibility of offline data collection is unlikely. [48]

Online observation is a method employed to study interactions in virtual communities in their natural setting. The multitude of activities (e.g. social interaction, production and development processes) that may be observed in these communities makes them relevant research-wise. With this in mind, it is essential to explicitly consider online observer roles in order to ensure the methodological quality of such studies. [49]

In online observations, the observational data are equivalent to the recorded data, since the social interaction and behaviors exist in a written form. Consequently, data are easily separated from interpretation, which is rarely the case in offline observation. This is likely to have a positive impact on dependability in online observation making. However, the paper identifies other aspects with a potentially negative impact on the accuracy of data and interpretation in online observation: firstly communication between the community members under investigation taking place through other online and offline communication channels (e.g. IRC, instant messaging, telephone and face-to-face meetings) than the one observed, secondly lack of shared physical presence (e.g. reduced social cues) and finally the degree of social involvement (i.e. participation in the observed group's activities). Their impact should be considered in order to elicit trustworthy data from virtual communities like OSS communities. [50]

Offline and online observations should therefore not be viewed as competing methods of inquiry, but as each other's supplements. Depending on the phenomenon investigated, a study may benefit a lot from combining the two methods of observation, since each contributes different aspects and provides different levels of detail. Finally, this paper proposes that during the online observation of settings such as OSS communities, threats to credibility and transferability can be diminished without reducing the objectivity level, through the introduction of offline data collection techniques rather than by adapting a more participative role in the observed online setting. [51]

This paper suggests that online and offline observer roles complement each other in the study of virtual communities. However, several limitations of this study call for further research. First, we need to become familiar with the importance of the sequence of the offline and online observations. In other words, it should be determined which approach is most appropriate from the inception of the study. Moreover, the consequences of a particular order of sequencing should be examined in relation to anonymity, identity control and their effect on trustworthiness criteria. [52]

Second, it should be analyzed to what extent offline observation uncovers the same aspects of the phenomenon studied as online observation does. FALLOWS (2002) found that American workers perceive e-mail as a means of relaying facts, but 85% of workers using e-mail prefer to have face-to-face conversations when they are dealing with workplace problems, and fewer than 6% consider e-mail effective in these cases. Other studies have arrived at similar conclusions, i.e. when facts are in focus, e-mail is the preferable medium (HANSON, 1999). This suggests that different CMC methods are likely to convey different types of content, which may have important implications in regard to online observation studies and whether such studies should be combined with offline observation (when possible), or some other types of data sources. [53]

Third, an additional avenue for further research could be to examine the implications for observer roles in virtual communities that are based upon asynchronous and synchronous textual exchanges such as Wikipedia, Facebook, LinkedIn or MySpace, which are often used as exemplars of Web 2.0. Because of openness, peer production and global interaction and cooperation—characteristics of Web 2.0—participation is critical (RASK, 2008), and therefore it is likewise important to discuss the different observer roles in order to understand observation of online communities other than OSS environments. [54]

Finally, online observer roles ought to be investigated with regard to websites where it is open to debate whether complete observation is possible as the Web is interactive by nature. An example could be the observation of dynamic interactive websites like Amazon.com that configure the website based upon earlier interaction, which is explicitly stated with the welcome message "Hello, Morten Rask. We have recommendations for you." Also the discussion of online and offline observer roles is important because every website has a covert sender (LEVINE, LOCKE, SEARLS & WEINBERGER, 2000), where the interface is the website that acts as "the face between faces" (POSTER, 1996). [55]

1) The GNU General Public License (GNU GPL) "is the most popular and well-known example of the type of strong copyleft license that requires derived works to be available under the same copyleft. Under this philosophy, the GPL is said to grant the recipients of a computer program the rights of the free software definition and uses copyleft to ensure the freedoms are preserved, even when the work is changed or added to. This is in distinction to permissive free software licenses, of which the BSD licenses are the standard examples", http://en.wikipedia.org/wiki/GNU_General_Public_License [Accessed: June 6, 2011]. <back>

Adler, Patricia A. & Adler, Peter (1994). Observational techniques. In Norman K. Denzin & Yvonna S. Lincoln (Eds.), Handbook of qualitative research (pp.377-392). Thousand Oaks: Sage.

Angrosino, Michael V. & Mays de Pérez, Kimberly A. (2003). Rethinking observation: From method to context. In Norman K. Denzin & Yvonna S. Lincoln (Eds.), Handbook of qualitative research (pp.673-702). Thousand Oaks: Sage.

Babbie, Earl R. (1986). The practice of social research. Belmont, CA: Wadsworth Publishing Co.

Benkler, Yochai (2006). The wealth of networks. New Haven: Yale University Press.

Bitzer, Jürgen & Schröder, Philipp J.H. (2006a). The economics of open source software development: An introduction. In Jürgen Bitzer & Philipp J.H. Schröder (Eds.), The economics of open source software development (pp.1-13). Amsterdam: Elsevier.

Bitzer, Jürgen & Schröder, Philipp J.H. (2006b). The impact of entry and competition by open source software on innovation activity. In Jürgen Bitzer & Philipp J.H. Schröder (Eds.), The economics of open source software development (pp.219-246). Amsterdam: Elsevier.

Bøllingtoft, Anne (2005). The bottom-up incubator: A collaborative approach to (entrepreneurial) organizing? Ph.D. Thesis, Department of Management and International Business, Aarhus School of Business.

Cohen, Michael D. & Sproull, Lee S. (Eds.) (1995). Organizational learning. Thousand Oaks, CA: Sage.

Crowston, Kevin; Li, Qing; Wei, Kangning; Eseryel, U. Yeliz & Howison, James (2007). Self-organization of teams for free/libre open source software development. Information and Software Technology, 49, 564-575.

Domínguez, Daniel; Beaulieu, Anne; Estalella, Adolfo; Gómez, Edgar; Schnettler, Bernt & Read, Rosie (2007). Virtual ethnography. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 8(3), http://nbn-resolving.de/urn:nbn:de:0114-fqs0703E19 [Accessed: June 6, 2011].

Dubrovsky, Vitaly; Kiesler, Sara & Sethna, Beheruz N. (1991). The equalization phenomenon: Status effects in computer-mediated and face-to-face decision making groups. Human Computer Interaction, 6, 119-146.

Eisenhardt, Kathleen M. (1989). Building theories from case study research. Academy of Management Review, 14(4), 532.

Frankfort-Nachmias, Chava & Nachmias, David (1996). Research methods in the social sciences. London: St Martin's Press.

Friedman, Thomas L. (2006). The world is flat: The globalized world in the Twenty-first Century (rev. and exp. ed.). London: Penguin.

Glaser, Barney G. & Strauss, Anselm L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine Publishing Company.

Gold, Raymond L. (1958). Roles in sociological field observations. Social Forces, 36, 217-223.

Gutwin, Carl; Penner, Reagan & Schneider, Kevin (2004). Group awareness in distributed software development. Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, November 6-10, 2004, Chicago, Illinois, USA.

Hanson, Ward (1999). Principles of Internet marketing. Cincinnati, OH: South-Western College Publishing.

Hourahine, Ben & Howard, Melanie (2004). Money on the move: Opportunities for financial service providers in the "third space". Journal of Financial Services Marketing, 9, 57-67.

Jones, Steves (1995). Cybersociety: Computer mediated communication and community. Newbury Park, CA: Sage.

Jones, Steves (1998). Cybersociety 2.0: Revisting computer mediated communication and community. Newbury Park, CA: Sage.

Jorgensen, Danny L. (1989). Participant observation: A methodology for human studies. Newbury Park, CA: Sage.

Kiesler, Sara; Siegel, Jane & McGuire, Timothy W. (1984). Social psychological aspects of computer-mediated communication. American Psychologist, 39, 1123-1134.

Kline, Heinz K. & Myers, Michael D. (1999). A set of principles for conducting and evaluating interpretive field studies in information systems. MIS Quarterly, 23, 67-93.

Lakhani, Karim L. & von Hippel, Eric (2003). How open source software works: "Free" user-to-user assistance. Research Policy, 32, 923-943.

Leander, Kevin M. & McKim, Kelly K. (2003). Tracing the everyday "sitings" of adolescents on the Internet: A strategic adaptation of ethnography across online and offline spaces. Education, Communication & Information, 3, 211-240.

Levine, Rick; Locke, Christopher; Searls, Doc & Weinberger, David (2000). The Cluetrain Manifesto: The end of business as usual. Cambridge, MA: Perseus Books.

Lincoln, Yvonne S. & Guba, Egon G. (1985). Naturalistic inquiry. New York: Sage.

Mann, Chris & Stewart, Fiona (2000). Internet communication and qualitative research: A handbook for researching online. London: Sage.

Markham, Annette N. (1998). Life online: Researching real experience in virtual space. Thousand Oaks, CA: Sage.

Markham, Annette N. (2004). Internet communication as a tool for qualitative research. In David Silverman (Ed.), Qualitative research: Theory, method and practice (pp.95-124). London: Sage.

Miles, Matthew B. & Huberman, A. Michael (1994). An expanded sourcebook: Qualitative data analysis. Thousand Oaks, CA: Sage.

Patton, Michael Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.

Pearsall, Marion (1970). Participant observation as rote and method in behavioral research. In William J. Filstead (Ed.), Qualitative methodology: Firsthand involvement with the social world (pp.340-352). Chicago: Markham.

Poster, Mark (1996). Postmodern virtualities. In Mike Featherstone & Roger Burrows (Eds.), Cyberspace/cyberbodies/cyberpunk: Cultures of technological embodiment (pp. 79-95). Thousand Oaks, CA: Sage.

Robson, Colin (2002). Real world research: A resource for social scientists and practitioner-researchers. Oxford: Blackwell.

Saunders, Mark; Lewis, Philip & Thornhill, Adrian (2000). Research methods for business students. Harlow: Prentice Hall.

Shihab, Emad; Bettenburg, Nicolas; Adams, Bram & Hassan, Ahmed E. (2010). On the central role of mailing lists in open source projects: An exploratory study. In Kumiyo Nakakoji, Yohei Murakami & Eric McCready (Eds.), New frontiers in artificial intelligence (pp.91-103). Berlin: Springer.

Silverman, David (2000). Doing qualitative research—A practical handbook. London: Sage.

Spradley, James P. (1980). Participant observation. New York: Holt, Rinehart and Winston.

Stafford, Marla Royne & Stafford, Thomas F. (1993). Participant observation and the pursuit of truth: Methodological and ethical considerations. Journal of the Market Research Society, 35, 63-76.

Strauss, Anselm L. & Corbin, Juliet M. (1998). Basics of qualitative research—techniques and procedures for developing grounded theory. London: Sage.

Tapscott, Don & Williams, Anthony D. (2006). Wikinomics—How mass collaboration changes everything. New York: Penguin Group.

Tsoukas, Haridimos (1997). The tyranny of light: The temptations and the paradoxes of the information society. Futures, 29(9), 827-843.

Turkle, Sherry (1995). Life on the screen: Identity in the age of the Internet. New York: Simon & Schuster.

Van Maanen, John E. (1998). Different strokes: Qualitative research in the administrative science quarterly from 1956 to 1996. In John E. Van Maanen (Ed.), Qualitative studies of organizations (pp.ix-xxxii). Thousand Oaks, CA: Sage.

von Hippel, Eric (2001). Innovation by user communities: Learning from open-source software. MIT Sloan Management Review, Summer, 82-86.

von Hippel, Eric & von Krogh, Georg (2003). Open source software and the "private-collective" innovation model: Issues for organization science. Organization Science, 14(2), 209-223.

Vujovic, Sladjana (2007). Computer-mediated and face-to-face communication: The quest for social cohesion in an OSS development community. Det Danske Ledelsesakademi (DDL) conference, December 10-11, 2007, Aarhus, Denmark.

Sladjana V. NØRSKOV is an Assistant Professor at the Department of Business Administration, Business and Social Sciences, Aarhus University in Denmark. She received her Ph.D. from Aarhus School of Business. Her research interests include user-centered innovation processes, community governance and new organizational forms. She studies these issues using mainly theory building from cases.

Department of Business Administration,Business and Social Sciences, Aarhus UniversityHaslegaardsvej 10, DK-8210 Aarhus VDenmark

Morten RASK is Associate Professor at the Business Administration, Business and Social Sciences, Aarhus University in Denmark. His research focuses on the global dimension of sustainable business models, organizing industrial marketing, purchasing & supply management, e-markets, e-commerce and m-commerce. He has contributed to Electronic Markets, European Journal of Purchasing & Supply Management, First Monday, Marketing Intelligence & Planning, European Journal of International Management, Journal of Teaching in International Business, Energy Policy and to the Danish Management and Business Economics. He holds a Ph.D. from Aalborg University in Denmark.

Department of Business Administration,Business and Social Sciences, Aarhus UniversityHaslegaardsvej 10, DK-8210 Aarhus VDenmark

Nørskov, Sladjana V. & Rask, Morten (2011). Observation of Online Communities: A Discussion of Online and Offline Observer Roles in Studying Development, Cooperation and Coordination in an Open Source Software Environment [55 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 12(3), Art. 5,
http://nbn-resolving.de/urn:nbn:de:0114-fqs110358.

Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (FQS)

ISSN 1438-5627

Creative Commons Attribution 4.0 International License