University of Sheffield University of Nijmegen

v.lally@shef.ac.uk m.delaat@ped.kun.nl

ABSTRACT

This paper, which reports work in progress, considers the methodological difficulties of investigating learning and tutoring processes of an advanced learning community, within a collaborative virtual professional development environment. Results are presented from content analysis. NVivo was used to assist the coding process with two schemas. The first schema attempts to probe the social co-construction of knowledge by analysing the social, cognitive and metacognitive contributions to an online learning event. In the second schema the presence and patterns of tutoring processes within the group is investigated. These patterns of group and individual activity for both learning and tutoring are then analysed. The interactions between these are also considered. In conclusion, consideration is given to the prospects for this type of approach as a means of adding value to our understandings of the complexity of the relationship between tutoring and learning in virtual professional development environments. The methodological issues of using complimentary methods of analysis to probe participant dynamics and non-expressed thinking are also considered.

Keywords

Content Analysis, Advanced Learning Communities, Methodology, NVivo

Aproaches to researching Networked Learning: the methodological Challenge

At the recent (January 2002) Fifth International Conference on Computer Support for Collaborative Learning, in Boulder, Colorado , one of the stated aims was to articulate a new paradigm for ‘a distinctive form of learning research’. Surprisingly, perhaps, a browse through the conference proceedings soon reveals that, despite this, only a small minority of more than 50 long papers focused on the issues and practicalities of researching learning in networked environments. In some ways this was disappointing and perplexing, given the stated aim. At the same time it is understandable. The challenges to be faced in researching learning are at once attractive, but also formidable. In our work with advanced learning communities (in the Dutch Police and on the E-Learning MEd at the University of Sheffield) learning is mediated by the virtual learning environments (VLEs) First Class and WebCT. This can create the comforting feeling, for unwary researchers interested in learning processes, that the transcripts of discussions taking place in the VLEs contain easily accessible and significant evidence of learning among the participants. There is no manual transcription to undertake, and it’s clear who said what, and when. Initially then, the problem can be easily understood in terms of analytical tools: reach for content analysis of the written messages. This was our own first approach. Immediately, however, we were confronted with a range of problems. Content analysis is cumbersome and time consuming. The choice of coding categories is a complex issue in itself. What does one do about those aspects of learning that are not expressed, and therefore not amenable to content analysis? Does the content analysis reveal the actual patterns of participation? The emerging reality of this work, for us, is that the nature of learning interactions among participants in advanced learning communities is sometimes very complex and multi-dimensional. It is not easy to research using any single method; increasingly we think that no one method, even when refined, may be up to the task. This has been the stimulus for the three papers in this symposium: to explore a multi-method approach to understanding learning among members of these communities, and in so doing attempt to reveal and understand a richness of learning beyond the capability of any one of the methods, when used by itself. In this first paper we share some of our findings from the application of computer assisted content analysis to asynchronous discussion transcripts. In the second symposium paper in this volume de Laat looks at a combined approach using social network analysis and content analysis. In the third paper Lally explores the use of ‘critical event recall’ to probe learning that is not expressed in the actual text records used as data for the other two methods. In future work we aim to move towards a more coherent synthesis of these methods. However, this is the longer term aim of our research collaboration.

Theoretical Basis Of Learning, Teaching And Content Analysis

In previous work we have attempted to explore a range of aspects of collaborative learning and begun to develop analytical frameworks in order to understand the complex tutoring and learning processes that are occurring in advanced learning communities. In the analysis presented in this paper we are interested in gaining insight into collaborative knowledge construction and tutor processes in a collaborative learning environment through the use of two compatible coding schemas. The students featured here were professional educators following a Master’s Programme in E-Learning (a continuing professional development programme) . Our analysis is based upon work conducted by these students, and a tutor, in the first ‘workshop’ of this programme. In the main project we were interested to explore the relationship between individual and group processes as they relate to knowledge construction and tutor processes, as these developed over time. In this paper we report on the individual analyses only. Many authors in attempting to define cognition in groups (group mediated cognition or gmc), have suggested that, in a group meeting, the situation itself may exert a strong mediating effect on individual cognitive and conceptual processes. The thinking of individuals is influenced by the group in which they are working. The merger of intellectual and social processes may be a fundamental feature of group mediated cognition. A second key feature is the tension between the conceptual structure or understanding (of the problem or ideas under discussion) of the group and that of the individuals within it. This tension is the driving force for the collective processing of the group. In this process interaction between individuals, as well as their shared and individual cognitions, are the key aspects of co-construction of knowledge, meaning and understanding. However, the situation is further complicated because the participants in these learning processes are also engaged in tutoring processes. The students in the Master’s Programme in E-Learning are a sophisticated group of professionals. They are engaged in learning processes and also in tutoring processes as they support each other and the group as a whole. Tutoring processes in this course are not the exclusive domain of the designated tutors

In order to probe collaborative knowledge construction and tutoring in this learning environment we ‘coded’ the contributions made to a 10 week discussion used two coding schemas. The first was used to investigate knowledge construction. This included four main categories: cognitive activities used to process the learning content and to attain learning goals; metacognitive knowledge and metacognitive skills; affective activities (used to cope with feelings occurring during learning ), and miscellaneous (used to score all other units including social talk). Basically, we hoped to be able to reveal something of the participants’ thinking, as expressed in their message contributions, while they were undertaking the collaborative task. The second schema, adapted from , is used to probe ‘tutor’ processes. This includes three main sub-categories: design and organisation, facilitating discourse , and direct instruction. The work on cognitive apprenticeship by Collins , Rogoff’s model of apprenticeship in thinking and Vygotsky’s scaffolding analogies provide some of the theoretical basis for these categories. Basically, we hoped to reveal something of the ways in which the participants were supporting each other’s learning while undertaking the task. The choice of coding schemas is an important one for this type of work. It could be argued that a more ‘grounded’ approach, using categories that emerge from a reading of the messages, would provide a more ‘authentic’ summary of the intentions of the participants. In our view this is a valid and important way of approaching the analysis. However, we wanted to connect with some of the conceptual and theoretical ideas about learning and tutoring in the literature rather than create de novo categories. At the same time we hoped to be able to share our analyses with colleagues in other contexts by supporting the use of publicly available schemas as a basis for comparison.

Analysing Networked Teaching And Learning with computer assisted content analysis

In the process of analysing tutoring and learning processes, messages from a learning event need to be coded and analysed. As described above. The central purpose of coding is to extract, generalise and abstract from the complexity of the original messages in order to find significant themes and develop theories about the situation that illuminate it. This is a balance between oversimplification, resulting in the loss of subtlety and insight into complex processes, and over-coding where the themes and trends are still obscured by too many sub-categories. We used computer assisted data analysis software (CAQDAS) to achieve this. The main advantages of such an approach include: partial automation of the coding process, with increased speed of coding; a wider range of ways to search, re-code and interrogate the coded data (in this case messages), including visual coding. We used NVivo 1.1 for this work, and set up the categories in our two schemas as ‘nodes’ within the NVivo system. Each message was imported as a text file and given a ‘time-stamp’ to indicate when it was posted in the original discussions in WebCT. It was also given other ‘descriptors’ including who authored the message, and the gender of the author. Once all the messages had been coded and described we used the search facility in NVivo to carry out two analyses. The results in this paper are based on one of these analyses: a search, by individual, for her contributions within each category of the coding schemas. In the second analysis, reported elsewhere {Lally, 2002 #1345}, we looked at tutoring and learning processes for the whole group over time, in order to try to understand how the relative proportions of learning and tutoring processes changed over the lifetime of the group’s work.

Results

The following tables (one to six) give the results of our analyses of individual contributions to the workshop, using the two coding schemas. Tables one to three show percentages of units of meaning coded for learning processes for eight individuals, including the tutor. This is a sample from three phases of the activity. The total number of

 

Table 1 Units of Meaning Coded for Learning Type in the First Phase

 

Table 2 Units of Meaning Coded for Learning Type in the Middle Phase

 

Table 3 Units of Meaning Coded for Learning Type in the Ending Phase

messages from the activity was approximately 1000. Our sample consisted of 10 per cent of these massages, spread equally between the beginning, middle and end phases of online activity. Tables four to six show percentages of units of meaning coded for tutoring processes for the same individuals.

 

Table 4 Units of Meaning Coded for Tutoring Type in the Beginning Phase

 

Table 5 Units of Meaning Coded for Tutoring Type in the Middle Phase

 

Table 6 Units of Meaning Coded for Tutoring Type in the Ending Phase

Discussion and Conclusions

Our use of two coding schemas to code contributions to the learning set discussions, one for tutoring processes, and one for learning processes, was an attempt to interrogate key aspects of these processes for individuals within the group. In this section we would like to offer some analysis of the results of this coding process, and then to comment on the possibilities and limitations of the method of coding we have employed. Focusing firstly on learning (Tables one to three), the codings suggest some clear patterns, over the three phases of the activity. In the beginning phase, 60 percent of the learning processes are cognitive, and 20 per cent are metacognitive (table 1). This is the phase of activity when the task of carrying though a collaborative project, on an aspect of networked learning, is being conceptualised by the group. In the middle phase, however, this relationship changes. The cognitive activity rises to 81 per cent of all coded units of meaning while the metacognitive drops to 14 per cent. All of the participants are thinking, and discussing the concepts of the task itself (table 2). By the ending phase the cognitive discussions have dropped back to 42 per cent of all units of meaning. Metacognitive discussion has also dropped; in this case to 20 per cent of all units. At the same time, miscellaneous discussion has increased considerably, from 1.3 per cent in the middle phase, to 28.8 per cent in the concluding phase. As the group members complete their project they may be moving away from thinking around the task and start to discuss and reflect upon it in a range of ways. At the same time, distinctive individual profiles are discernible in these coding percentages. For example, Andrea is a student participant who makes extensive contributions to learning processes throughout the activity, although at lower levels in the middle phase. Katie, on the other hand, makes very few explicit contributions to any learning process. Margaret makes extensive contributions during the middle phase, but much less at other times. The tutor during this activity is Brian (denoted by * in the tables). During the activity he contributes at a low level in the beginning phase; makes no contribution in the middle phase and a low level of contribution at the end of the activity.

Secondly, we attempted to focus on tutoring processes occurring in the discussions (tables 4, 5 and 6). In this programme, based upon an advanced learning community of professionals the activities that we have described as tutoring: direct instruction, facilitation, and curriculum organisation, are not the exclusive domain of the designated (*) university tutor. They are activities used and employed by all members of the group at different times. Our decision to try to probe these processes using a second coding schema was based on our own awareness of the strong interrelationship between ‘tutoring’ and ‘learning’ that may occur in such groups. It is an important feature of this kind of collaborative on-line work. The results of these tutoring codings are shown in tables four to six. Once again, some patterns are discernible. The level of direct instruction remains at a low level throughout the activity. This is not surprising given that the group was engaged in a collaborative activity that drew on their own professional resources and other material to which they were directed before the activity commenced. Therefore, there was little need for anyone to provide this during the period of work that we analysed. The level of instructional design, on the other hand, is highest in the early phase. This may arise from the group’s need to help each other to organise and plan for the activity. In the middle phase this tutor process decreases as facilitation by group members increases. Facilitation continues at a high level into the ending phase, where instructional design increases again as the group members plan and prepare to review their work. Brian (the tutor, denoted by * in the tables) engages in high levels of tutor processes in the early and ending phases, but is less present in this role in the middle phase. Some of the students, for example Andrea, are engaged in high levels of tutor processes. Indeed, Andrea sustains these processes in the middle phase when Brian’s contributions are relatively low. The role undertaken by Charles is interesting because his tutor contributions are highest in the beginning phase and then decline. During this change his engagement in learning processes remains steady. These two processes do not always follow each other in their patterns of change.

The methodological issues raised by this type of study are considerable. The use of schemas to ‘code’ the messages is an attempt to ‘categorise’, and to some extent quantify the meanings embedded in the exchanges between participants. However, this is a considerable task. Because the total number of messages was around 1000 we had to ‘sample’ these in order to make the coding a manageable task. Hence the exchanges were sampled during the first ten days of the group’s work, during the middle ten days and for a further ten days at the end (the three phases in tables one to six). This sampling approach was used so that the sampled episodes of work retained meaning and coherence over time. This was important to us because we wanted to look at the development of tutoring and learning processes in the group, over time, as well as at individual totals. Our theoretical approach to learning has taken us in the direction of attempting to investigate both the individual processes and group processes, as well as the possible interactions between them. Furthermore, the coding schemas required to capture the complexity of the activities were necessarily complex in themselves. There was a total of 60 categories and sub-categories. Some passages of text could have been coded using more than one category, because of the multiplicity of meanings that could be inferred from the text. At these points we had to make judgments about this and agree them in coding conversations between the two researchers. Given these difficulties, the use of coding in this way is still only a partial solution to the methodological challenges we identified at the beginning of this paper. Coding of discussions in the social space that was created in WebCT, for use during the workshop, was not undertaken. Yet this space was a place where ideas were discussed, and relationships built that supported the group’s work in the more formal group space (or forum). Furthermore, coding provides little insight into two other key aspects of the group’s processes: individual thinking that was not expressed in text messages, and participant dynamics, that is the patterns of reading of messages posted to the forum. These issues are addressed in the two accompanying papers of this symposium.

Educational Value of the Study

In this paper we attempted to study both learning processes and tutoring processes, and the relations between them, within a group of collaborating professionals in an educational environment. We have presented the results of an approach to content analysis of messages exchanged during a professional development activity. This analysis has enabled the tentative identification of patterns of individual and group learning during the activity. It has also allowed us to discern different individual roles in tutoring processes among these professionals. We have tentatively attempted to relate these to learning processes. We suggest that these analyses have added considerable value to our understanding of tutoring and learning processes by professionals in an advanced learning community within an online Master’s Programme. They show, for example, how participants may operate quite differently, and yet within discernable patterns, some being strong facilitators, while others offer little support to their collaborators. There are many other implications in terms of differentiated patterns of working that we hope to articulate in future work. Through this approach we contend that it may be possible to gain deeper insights into how professionals collaborate successfully to develop their own practice, and into the complexity of the interactions between individual and group processes during these collaborations. At the same time, we have indicated that the analysis of such complex interactions in advanced learning communities presents a strong methodological challenge for researchers. The use of coding schemas, for example, is beset with difficulties, some of which we have outlined. The need to complement coding analyses with other complimentary forms of analysis is clearly also very important in order to more fully understand the richness of these learning interactions. This paper, and the other two contributions to this symposium represent an attempt to articulate a methodological synthesis that might be up to the challenge. This work will be further developed in future publications.

ACKNOWLEDGMENTS

We would like to thank the students and tutors of the E-Learning MEd at the University of Sheffield. We are also very grateful to David McConnell for providing valuable and positive criticism and feedback on earlier versions of aspects of the research presented here.

This paper describes work in progress and is part of an ongoing collaboration between the authors at the University of Sheffield (UK) and Nijmegen (Netherlands).

 

REFERENCES

Anderson, T., Rourke, L., Garrison, D. R., & Archer, W. (2000). Assessing Teaching Presence in a Computer Conference Context. Retrieved 27/3/01, 2001, from the World Wide Web: http://www.atl.ualberta.ca/cmc/publications.html

Barrett, E., & Lally, V. (1999). Gender differences in an on-line learning environment. Journal of Computer Assisted Learning, 15, 48-60.

Collins, A., & Brown, J. S. (1991). Cognitive Apprenticeship: Making Things Visible. American Educator, 15(3), 38-46.

CSCL. (2002). Computer Support for Collaborative Learning: Foundations for a CSCL Community. Retrieved 27/2/02, 2002, from the World Wide Web: http://newmedia.colorado.edu/cscl/

De Laat, M., De Jong, F., & Simons, R.-J. (2001). Supporting Self Regulation of Learning Activities in Online Communities of Practice (poster paper). Paper presented at the First European Conference on Computer Supported Collaborative Learning, Maastricht, Netherlands.

De Laat, M., De Jong, F., & Ter Huurne, J. (2000, June 26 2000.). Supporting a Community of Practice: The Role of Workers as Learners. Paper presented at the Ed-Media, Montreal, Canada.

Goldstein, L. S. (1999). The Relational Zone: The Role of Caring Relationships in the Co-Construction of Mind. American Educational Research Journal, 36(3), 647-673.

Lally, V. (2001). Analysing Teaching and Learning in a Networked Collaborative Learning Environment: Issues and Work in Progress. In C. W. Day & D. van Veen (Eds.), Educational Research in Europe Yearbook 2001 (pp. 219-237). Leuven: Garant/EERA.

Lally, V., & de Laat, M. (2002). Cracking the Code: Learning to Collaborate and Collaborating to Learn in a Networked Environment. In G. Stahl (Ed.), Computer Support for Collaborative Learning (pp. 160-168). Hillsdale, New Jersey: Lawrence Erlbaum.

Lally, V. E., & Barrett, E. (1999). Building a Learning Community On-Line: towards socio-academic interaction. Research Papers in Education, 14(2), 147-163.

Lave, J. (1988). Cognition in Practice. Cambridge: Cambridge University Press.

Lave, J. (1996). Teaching, as Learning, in Practice. Mind, Culture, and Activity, 3(3), 149-164.

Lave, J., & Wenger, E. (1991). Situated Learning: legitimate peripheral participation. Cambridge: Cambridge University Press.

Levine, J. M., Resnick, L. B., & Higgins, E. T. (1996). Social Foundations of Cognition. Annual Review of Psychology, 44, 585-612.

Lipman, M. (1991). Thinking in Education. Cambridge: Cambridge University Press.

Moll, L. C., Tapia, J., & Whitmore, K. P. (1993). Living Knowledge: the social distribution of cultural resources for thinking. In G. Salomon (Ed.), Distributed Cognitions: psychological and educational considerations (pp. 139-163). Cambridge: Cambridge University Press.

Qualitative Solutions and Research. (1999). QSR NUD*IST Vivo (NVivo) (Version 1.1). London: Sage.

Resnick, L. B. (1991). Shared Cognition: thinking as social practice. In L. B. Resnick & J. M. Levine & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 1-20). Arlington VA: American Psychological Association.

Rogoff, B. (1995). Apprenticeship in Thinking. New York: Cambridge University Press.

Salomon, G., & Perkins, D. N. (1998). Individual and Social Aspects of Learning. Review of Research in Education, 23, 1-24.

Scardamalia, M., & Bereiter, C. (1994). Computer Support for Knowledge Building Communities. Journal of the Learning Sciences, 3(3), 265-283.

Smith, J. B. (1994). Collective Intelligence in Computer-Based Collaboration. New Jersey: Lawrence Erlbaum Associates.

Stahl, G. (Ed.). (2002). Computer Support for Collaborative Learning: Foundations for a CSCL Community. Hillsdale New Jersey: Lawrence Erlbaum.

University of Sheffield. (2001). MEd in E-Learning (formerly Networked Collaborative Learning) [Web Site]. School of Education. Retrieved 31/07/01, from the World Wide Web: http://www.shef.ac.uk/uni/projects/csnl/medindex.htm

Veldhuis-Diermanse, A. E., & Biemans, H. J. A. (2000). Is CSCL an adequate tool to reach a deep level of academic learning? In M. De Laat (Ed.).

Vygotsky, L. S. (1978). Mind in Society: the development of higher psychological processes (V. J.-S. M. Cole, S. Scribner and E. Souberman, Trans.). Cambridge MA: Harvard University Press.

Wegerif, R., Mercer, N., & Dawes, L. (1999). From social interaction to individual reasoning: an empirical investigation of a possible socio-cultural model of cognitive development. Learning and Instruction, 9(6), 493-516.

Wertsch, J. V. (1991). A Sociocultural Approach to Socially Shared Cognition. In L. B. Resnick & J. M. Levine & S. D. Teasley (Eds.), Perspectives on Socially Shared Cognition (pp. 85-100). Arlington VA: American Psychological Association.

University of Nijmegen, Department of Education

m.delaat@ped.kun.nl

ABSTRACT

The aim of this paper is to study interaction patterns among the members of a community of practice within the Dutch police organization and the way they share and construct knowledge together. The online discourse between 46 members, using First Class, formed the basis for this study. Social Network Analysis and content analysis were used to analyze the data. The results show that the interaction patterns between the members are rather centralized and that the network is relatively dense. Most of the members are involved within the discourse but person to person communication is still rather high. Content analysis revealed that discourse is focused on sharing and comparing information.

Keywords

Social Network Analysis, content analysis, networked expertise, community of practice, organizational learning

introduction

The purpose of this symposium is to present a number of papers in which collaborative interactions in a networked learning setting are being studied. The focus of this contribution is on the use of network and content analysis to study the interaction patterns and quality of the discourse in an online community of practice. The aim of this paper is to study the nature of networked expertise within an organization, and the way its members share and construct knowledge together.

Organizations are more and more confronted with the problem of creating and managing their knowledge in order to respond flexible to changes that occur in their working environment. Organizations are transforming into learning organizations. In a learning organization, workers are stimulated to share and develop knowledge together. The learning potential of expertise networks has become a matter of interest and social and cultural aspects of learning have become important to understand and foster their learning ). In organizations workers tend to form networks of expertise to facilitate individual learning, collaboration and to discuss work related problems together ). Sometimes these networks transform into communities of practice. In a community of practice (COP), participants, who share a common interest for the field they work in, come together to help out each other, solve problems, and share and create knowledge collaboratively.

The term communities of practice is related to the notion of Lave and Wenger (1991) who describe learning as legitimate peripheral participation in various communities. According to Brown and Duguid (1991) Workplace learning can best be understood, then, in terms of the communities being formed or joined and personal identities being changed. The central issue in learning is becoming a practitioner not learning about practice. This approach draws attention away from abstract knowledge and situates it into the practices and communities in which knowledge takes on significance.

A community of practice therefore is a group of people informally bound by a shared practice related to a set of problems [...] they typically solve problems, discuss insights, share information, talk about their lives, and ambitions, mentor and coach on each other, make plans for community activities, and develop tools and frameworks that become part of the common knowledge of the community. Over time these mutual interactions and relationships build up a shared body of knowledge and a sense of identity. They constitute an informal, social structure initiated by members and reflecting on their collective learning (Wenger, 1999, p. 4).

A community of practice defines itself along three characteristics (see Wenger, 1998; 1999):

What it is about – A joint enterprise as understood and continually renegotiated by its members

How it functions - Mutual engagement that bind members together into a social entity

What capability it has produced - The shared repertoire of communal recourses (routines, sensibilities, artefacts, vocabulary, styles, etc) that members have developed over time.

Large organizations, like the Dutch police force, deal with a wide range of specialized knowledge, which needs to be updated and adapted to new situations. However, criminal investigators responsible for maintaining and innovating their work practice often don’t work together in a physically shared space. They discuss work related problems by telephone, arrange meetings, etc. To facilitate this sharing and development of knowledge, providing communities of practice with ICT-tools can be an advantage in bringing people together. It offers the possibility to collaborate online independent of time and space. Computer supported collaborative learning (CSCL) tools, like groupware, make it possible for criminal investigators to join communities of practice and participate in their own time and at their own pace.

One of the programs that support this kind of collaboration is First Class. First Class is a kind of groupware designed specifically to share, create and manage knowledge together by writing messages. The participants operate in a shared workspace in which they read and write messages. A message is a contribution that can contain text, pictures and attachments. Not only sharing knowledge is important. The participants also have the possibility to go into something thoroughly by writing reactions to previously written messages and therefore elaborate on the knowledge that is already in the database. By working together participants develop greater competence in a particular subject area, using what group members already know as an important component and co-constructing plans of action to extend that knowledge. The created knowledge thus can be seen as a social product.

This study focuses on the exchange of information through a CSCL-environment (First Class) within the Dutch police organization. The members of this network frequently exchange information and discuss work related problems together. Their shared interest for drugs issues in criminal investigation resulted in the establishment of a shared practice. This network can be characterized as a community of practice because of voluntary engagement, existence of this network over time (two years), and realization of a shared practice ). According to Lave and Wenger (1991) learning is a form of becoming an active participant in a community through legitimate peripheral participation. ‘Participation in the cultural practice in which any knowledge exists is an epistemological principle of learning’ , p. 98.). The way people participate in expertise networks provides insight in the process of learning. A CSCL environment provides ideal possibilities to study interaction patterns between the members of a network. Most discourse programs keep log-files in which information is stored about the engagement of the participants in the discussion. This information can be scrutinized using social network analysis ). Social network techniques can be used to describe patterns of relationships between individuals. A network is described as a set of nodes and the set of ties connecting these nodes ). Insight in communication patterns within a certain network alone is not enough. Also the content of the discourse must be taken into account ). This way information can be gathered about the quality of the learning and the social construction of knowledge.

Our goal is to study interaction patterns among the members of the COP and the quality of the discourse between its members.

method

An existing community of practice within the Dutch police organization was studied to analyze the interaction patterns and quality of the discourse. They used the program First Class as a communication tool in which the discourse took place. In this study we focus on the following questions:

1. How active are the members in the discourse?

2. Who are central participants in the discourse?

3. How dense is the participation within the network?

4. What is the quality of the discourse?

Subjects and procedure

Communities of practice can’t be built they emerge. Therefore we followed an existing community of practice within the Dutch police organization to analyze their activities. This COP consists of 46 members who are affiliated with or conducting drugs related investigations. The Dutch police force is divided over 26 regions and from every region one or two members participate in this COP. Therefore the members of this community are divided all over the Netherlands and decided to use First Class as an electronic environment to discuss work related problems, exchange information and to maintain their expertise. First Class is a communication forum that facilitates an asynchronous discourse. The members operate in a shared workspace in which they can read and write messages that are stored in a central database. This database is accessible through an authorized modem connection with a secured server. The data that was analyzed during this study is from the period of January till June 2001.

Instruments

The way people participated and interacted with each other provides information about the activities of such a community. First Class generates log-files through which information about the activity of the members can be obtained. It is possible to receive information about the amount of messages people have written, and how many times and by whom a certain message was read.

The information retrieved from First Class can be treated as relational data and stored away in matrix to analyze interaction patterns. Social network analysis (SNA) is used to analyze the social structure of the COP. For this purpose we focused on the cohesion of the network ). This way we gain information about the interaction between the members of the COP. Several techniques have been used.

We conducted centrality measures to find the central participants within the network. For each participant this has been done using Freeman’s degree. Freeman’s degree calculates the activity of individual members in the COP. With degree the network activity of individual members can be indicated. This can be done by calculating the indegree and outdegree centrality measures. Indegree centrality is a form of degree centrality that counts only those relations with a focal individual reported by other group members and is therefore not based on self reports as is outdegree centrality (Borgatti, Everett, & Freeman, 2000). In this study indegree measures provide information about the amount of people that read a message from a certain person. Outdegree gives an indication of the amount of messages a person has sent to other individual members or to the entire network. Secondly we conducted a density analysis to describe the overall linkage between the participants. Density of a network is defined as the number of lines in a network divided by the maximum number of all possible lines ). This varies between 0 and 100%.

Finally multi dimensional scaling (MDS) has been used to visualize the interaction between the members. The idea behind MDS is that of using the concept of space and distance to map relational data ). The intensity of the engagement in the network is used as a measurement of closeness. The more people interact with each other the closer they are on the MDS map. The UCINET program developed by (2000) was used to conduct the analysis.

To assess the quality of the discourse the coding scheme developed by Gunawardena et al. (1997) was applied. This coding scheme is designed to examine the negotiation of meaning and social construction of knowledge in CSCL-environments. This scheme consists of five phases, reflecting the complete process of negotiation, however where there is less disagreement within the community the process may conclude at one of the earlier phases (p. 413). The first phase is one of sharing and comparing information. The members exchange opinions, ask questions, and provide descriptions about the topic of their discussion. The second phase concerns the discovery and exploration of dissonance or inconsistency among ideas, concepts or statements. In this stage the members try to identify their areas of disagreement, ask and answer questions to further clarify the topic of discussion. In the next phase the members are negotiating or constructing knowledge by making new proposals, integrating or accommodating knowledge, or try to compromise. In the fourth phase these newly constructed statements are being tested against personal experience, collected data, or acceptance in their culture. This leads to the final phase in which statements of agreements are being made, or applications of newly constructed meaning will be applied.

For this study we followed Gunawardena’s (1997) approach to code a message as a whole because this way you can describe the process of social knowledge construction between the members of the COP.

results

1. How active are the members in the discourse?

In the period of January till June the participants wrote 233 messages to the entire network, with an average of 5,07 (SD 6,72; min 0, max 32) messages per person. 14 members of the community did not write any message to the whole group. In total the written messages were read 7253 times with an average of 157,67 (SD 79,11; min 1, max 237) per person.

2. Who are central participants in the discourse?

Closer examination of the nature of the discourse showed that most participants sent new messages. There were hardly any responses sent as a reply to a certain message. The discussion threads did not go deeper than level 1. However it turned out that often a new message was sent as a response to a message that was being sent earlier. Also in some cases responses were just sent directly to another member instead of to the network as a whole. These messages were counted as a sent message as well. Because the lack of clear interaction in terms of sent and received comments we decided to create a matrix of sending and reading comments to conduct the SNA analysis.

Centrality measures are being conducted to find central actors in a network. This can be done using Freeman’s degree (table 1). With degree the level of activity of individual members in a network can be indicated. This can be done by calculating the indegree and outdegree measures. In this study indegree provides information about the amount of people that read a message from a certain person. Outdegree gives an indication of the amount of messages a person has read from others.

The results of the indegrees are rather interesting. It shows that 14 members did not send any message, but that the members who sent messages were all rather active being read by the other participants. For example participant ‘ed’ who sent 1 message has an indegree of 26. Two high extremes are measured for 2 participants (rs & ew), they have an idegree of 746 and 937 respectively. High indegrees indicates that others are, for some reason, very often reading messages from this particular participant.

The same tendency is shown by the outdegrees. High outdegrees indicates that a participant actively creates connections by reading messages from other members within the network. Eight (dv,as,dw,re,od,ov,ek,uj) members, who did not write any message, were still very involved within the exchange of information through the COP. They read most of the messages that were sent to the COP. This means that they are willing to keep up with the discourse. Six members have both low indegrees as outdegrees (ek, sc, az, ev, el, eb) this means they hardly participated within the discourse at all. Overall we can say that most members of this network have strong links, and are therefore actively involved with keeping track with the discourse.

Table 1: Engagement of members in the network

3. How dense is the participation within the network?

To get an indication of the overall linkage of members in the network we conducted density calculations. This gives an indication of the level of engagement in the network. Density calculations indicate how active the members are involved in the discourse. In the case of sending and reading the messages that were exchanged through First Class the COP had a density of 57%.

A MDS (figure 1) was calculated to visualize the patterns of interaction between the members. The amount of messages the members of this network have sent and read indicates how close members are situated on the MDS map. The stress value, that indicates the quality of the MDS map was 0,048 which was satisfactory.

Figure 1: Interaction pattern within the network

4. What is the quality of the discourse?

The 233 messages were coded with the Gunawardena scheme to analyze the social construction of knowledge. Fifty five of the messages could not be coded. The content of these messages were social talks and did not contribute to the discourse as such. The results of the 178 coded messages are shown in table 2.

Table 2: Social construction of knowledge (Gunawardena, et al., 1997)

Most of the communication (72%) between the members of the COP, remains in the phase of sharing, or comparing information (phase 1). This is somewhat expected because the reason for being member of this COP is sharing information and discuss work related problems. Therefore a lot of new trends related to drugs cases are shared between its members. This results in the exchange of descriptions and opinions sometimes being supported with examples of other members or questions about clarification of the written statements. Different experiences sometimes lead to discussion in which participants seek to further explore or advance statements of other members. 20% of the written messages are of phase 2. 8% of the messages were coded as phase 3. No statements of the other levels (4 and 5) were found. This means that the discussion forum is mainly used to discuss regional developments and check whether this might be a national trend.

conclusion and discussion

This study indicates that the interaction patterns between the members of this network are rather centralized, all the members sort of gel around the more active members of this COP. There are no subgroups within this COP, and most of the members are somehow involved within the discourse. The results have shown that some members are more passively engaged in this COP. This might be attributed to the culture of the police organization, where there traditionally is a lot of face to face communication. People tend to share information and solve problems through their personalized networks. The content of the discourse is mainly focused on sharing information sometimes leading into the construction of new knowledge. Testing and developing new applications through a CSCL environment is a relatively new development, which has yet to be explored.

Although the network is rather dense (many strong links between the members were found) there are also weak links between members that are less active. According to (1999) it will be interesting to conduct further research on the effect that strong and weak links in a network might have on the process of knowledge construction. ‘Strong links tend to mediate redundant information because they tend to occur among small groups of actors in which everyone knows what the others know. Strong links allow for construction of in-depth expertise within a community but do not provide new information because a significant part of the interaction is redundant by nature. In contrast to strong links, weak links, support functioning of a knowledge organization by transmitting information to and from the network’ (p. 10).

The interaction patterns within this network are centralized and the members are relatively well engaged in this COP, but the quality of the discourse in terms of social knowledge construction remains mainly in the phase of sharing information. However the members of this COP want to use First Class not just as a tool for sharing information. Their intention is to recognize drugs related trends throughout the whole country and to develop collaboratively an approach to meet those new developments. This involves not just processes of sharing information, also discussion and negotiation resulting in construction of knowledge are necessary to maintain and develop their expertise. A suggestion to stimulate this process of knowledge construction is to form small subgroups around topics of interest (in which in-depth knowledge can constructed) and use all the members of the COP for feedback on their results and input of new trends and information.

references

University of Sheffield

v.lally@shef.ac.uk

ABSTRACT

The aim of this paper is to explore a ‘third way’ methodology for complementing the methodological approaches outlined in the two accompanying papers by Lally and De Laat, and De Laat. The methodology discussed in this paper is Critical Event Recall, and is based upon the research of Kagan and others into the stimulation of recall of learning events using video records of those events. In a collaborative networked learning environment it is possible to derive detailed understandings of the nature of group interactions and their relationship to learning by undertaking social network analysis. The use of systematic content analysis also provides much insight into the cognitive, affective and social processes in which the group is engaged. In this paper I describe a preliminary attempt to use a form of stimulated critical event recall to probe the non-expressed cognitive, affective and social processes of individuals and group. The ways in which this data may then be used to re-interpret the analyses derived from network and content analysis is considered. Finally, the potential of this triangulated approach as a rich method of analysis of networked collaborative learning interactions is considered and evaluated.

Keywords

Critical Event Recall Collaborative Learning Methodology

squaring the circle

A recent international conference on Computer Support for Collaborative Learning (CSCL 2002) in Boulder, Colorado attempted to articulate a new paradigm for a distinctive form of learning research. My attendance at the conference helped me to focus on this issue and clarify some of my own thinking about the problems of researching learning in computer supported collaborative environments. In my own research collaborations with Maarten De Laat of the University of Nijmegen we have previously explored the use of content analysis to probe learning in on-line events (see, for example, the paper by Lally and De Laat in this volume). One of the emerging conclusions of this work is our realization of the complexity of learning interactions we are trying to probe. This is particularly so when the learners are career professionals collaboration in an established advanced learning community. The CSCL conference reminded me that there is presently little work that is attempting to analyse learning in such virtual communities. It also stimulated me to reflect that content analysis cannot, by itself, provide the complete picture. The need for some triangulation became very apparent; a new paradigm of learning research in virtual environments will require a lot more work yet, before the circle can be squared. The focus of the three papers in this symposium is on current research in progress that is probing learning interactions between participants in networked collaborative learning events using three methods:

In previous work we have explored a range of aspects of collaborative learning and begun to develop analytical frameworks in order to understand the complex tutoring and learning processes that are occurring. In the analysis presented in this paper we are interested in gaining deeper insight into collaborative knowledge construction and tutor processes in collaborative learning environments through the use of critical event recall. I had previously been introduced to a form of ‘stimulated’ event recall by Jon Scaife at the University of Sheffield (UK). This was Interpersonal Process Recall (IPR), a process developed by Norman Kagan, commencing at Michigan State University in the early 1960s . A broader theoretical and practical overview has been provided by Tuckwell . The basis of IPR, as it was developed by Kagan and others, is the realization that humans store vast amounts of information, feeling, impressions and ideas about the events, or ‘interpersonal processes’, in which they have participated. Because of the speed at which human interactions occur much of the detail of these processes is soon ‘forgotten’, and not available for subsequent reflection. However, Kagan and his co-workers discovered that it is possible to use video records of events to assist participants in recalling many details of the thoughts, feelings, and even physical responses, that happened at the time of the recorded event. IPR has subsequently been used in many contexts including supervision training, team building, teaching and teaching research, counselling, and therapy in order to investigate and illuminate interpersonal processes. Kagan developed the role of the ‘inquirer’ as a central feature of IPR. The function of the inquirer is to assist recallers in articulating their recollections of actual events by helping them to stay focused upon their thoughts and feelings at the time of the original event, as distinct from their subsequent thoughts and feelings about it. In this way it is possible for recallers to become conscious of the original thoughts and feelings that they experienced during the event. These may not have been previously articulated by them, and as a result of the recall process they become aware of considerable detail that helps in reflecting and gaining deeper understanding of what was going on in the original processes (depicted in the video). The original events may have occurred in the recent past, or may have taken place months, or even years previously. Jon and I first used IPR together to help schoolteachers to analyse the teaching and learning processes occurring among the pupils in their classrooms . Other researchers have developed variations of this approach to stimulated recall, independently of Kagan’s work, to investigate learning processes, for example, of students during the viewing of audio-visual learning materials .

Critical event recall in networked environments

My experiences of using IPR in face to face learning contexts, using video to stimulate recall by participants, helped me to become aware of and understand the complexity of the individual and group processes that occur in educational settings. The processes that participants recall include their worries, concerns, motivation, thinking, understanding, plans and intentions, observations and conclusions in relation to the event. When groups of participants engage in mutual or shared recall of events in which they have been present together they can gain enormous insight into each others behaviour and learning. In a sensitively guided recall this can be a real benefit for the future learning of the group, as well as the individuals within it. The recall enables the articulation of many previously unexpressed aspects of learning.

In networked collaborative learning environments such as the Sheffield E-Learning M.Ed. Programme students and tutors are working in advanced learning communities with many complex learning interactions occurring simultaneously. There are no video records of these events but full textual records remain available on the WebCT server used to host the programme. The use of these records as a stimulus to recall of critical learning events occurring during the Programme workshops suggests itself as a way of investigating those aspects of processes not actively expressed during the events. We are not aware of any previous examples of the use of stimulated recall using text to investigate learning in networked collaborative environments. Our intention in undertaking this type of work is to gain a fuller understanding of learning processes than is possible using content analysis alone.

This short paper reports some preliminary work with critical even recall. We have adopted two approaches to recall and these will be evaluated as the results are analysed. In the first approach participants are presented with summary analyses of learning events. This gives an overview of the patterns of learning and tutoring within the event. In the second approach we use the full text of learning events. In both approaches participants were presented with these items in advance of the recall sessions so that they could familiarise them selves with the summary analyses and full text of the events.

Results of a preliminary analysis

In this paper I shall present some preliminary results from a recall event with the tutor of an online learning event (all names have been changed to protect confidentiality of all participants). Brian was provided with summary analyses from a ten week collaborative learning event involving himself and seven course participants. These analyses were the result of coding the event, that had taken place 14 months previously. The details of the coding procedure are available in Lally and De Laat (in this volume) and in Lally and De Laat . In summary, messages contributed to the activity were coded for their tutoring content and their learning content in three phases of the work. The percentages of all the units of meaning that were concerned with categories of learning or tutoring, in all the messages, were then calculated. An example of these summary analyses, for the beginning phase (first 10 days) of the event is shown in tables one and two. Summary analyses were also provided to Brian for the middle and end phases of the event (not shown in the tables).

The recall event with Brian and myself, using all six tables of summaries, occupied approximately 45 minutes. I spent an initial phase of the recall session helping Brian to clarify the meaning of the percentage figures in the tables and the way in which they had been calculated. For example, we discussed the meaning of the coding categories listed on the left hand side of the tables, and I provided Brian with examples of the kinds of utterances that they represented.

The recall event was very loosely structured by myself, and tended to follow a natural pattern arising from the structure of the six tables we were using for the recall process. However, it quickly became apparent that Brian wanted to focus on individuals in the group, including himself, rather than learning, tutoring, phases of work or other aspects of the experiences. After articulating patterns of individual behaviour, gleaned from the summary tables, he began to recall his impressions, at the time of the event, of the learning and tutoring behaviours of the participants. At several points in the recall process I checked with Brian that he was in fact recalling how he felt and thought at the time of the original event, rather than what he thought about these behaviours now, i.e. at the time of the recall session. He confirmed that this was the case throughout the recall event (and two subsequent events not reported here).

Starting with himself, Brian commented that the tables showed him to be much more active at the start and end of the learning event, and much less so in the middle phase:

‘[I made a] conscious decision: [I had] a personal policy to be there at the start and the end’

Here Brian was recalling a decision he made at the time to be much more visible to the other participants at both ends of the event, but to withdraw to a large extent in the middle phase of work. He went on to explain to me his thinking behind this strategy, and how it related to the way in which he wanted to give space to participants to work together and express their own ideas, not dominated by him. This revealed strategic thinking about his role as a tutor. The pattern indicated in the summary tables had stimulated his recall of the decisions he made at the time of the original event.

 

Table 1 Example of Summary Analysis for Units of Meaning Coded for Learning in the Beginning Phase of the Learning Event

 

Table 2 Example of Summary Analysis for Units of Meaning Coded for Tutoring in the Beginning Phase of the Learning Event

Brian then went on to recall the behaviours of others in the group. Andrea was a significant participant in the event, and Brian recalled several impressions of her, stimulated by the figures about her participation contained in the summaries:

Brian recalled his own thinking at the time about Bill’s pattern of working as one in which he (Bill) was unsure about his own purposes and the medium at the beginning of the event. For this reason his level of participation, for both learning and tutoring, was low. As he clarified his own purposes for being involved Bill’s participation increased and he was the second highest participant for learning by the end (having been the lowest at the beginning), and third highest participant for tutoring at the end (having been equal lowest at the beginning).

Towards the end of the recall session Brian recalled that at the time he had been much concerned about the unevenness of participation within the group:

possibilities and implications for future use of critical event recall

The aim of this paper was to explore a ‘third way’ methodology for complementing the methodological approaches outlined in the two accompanying papers in this symposium by Lally and De Laat, and De Laat. The methodology discussed in this paper is Critical Event Recall, and is based upon the research of Kagan and others into the stimulation of recall of learning events using video records of those events. The brief recall episode presented here is based on a single recall session with the tutor in the featured group. Summary content analyses of the texts of a learning event were used as the basis for the stimulation of recall of critical events in the work of the group. The most important finding of this study is that recall of important details of the tutor’s thinking at the time of the original event is possible using summary analyses of this kind. The recollections presented here indicate that the tutor engaged in many reflective and analytical observations about his own facilitation of the group and the behaviours of individuals within it, yet much of this thinking was not directly observable in the transcripts of the group’s work. The tutor was making careful judgements about when and how to intervene based upon his interpretations of the needs and behaviours of individuals, the needs of the group, and his own largely unarticulated (in the group forum) values about the nature and purposes of collaborative learning. In conclusion, then, we argue that critical event recall has the potential to access aspects of learning and tutoring processes that are not directly available in group discussion transcripts. Furthermore, this tool can complement content analysis in an important way by using its results to probe ‘the thinking behind the text’ in collaborative work within advanced learning communities in networked environments.

In future studies we will report on the use of critical event recall that employs full transcripts of learning and tutoring events as the stimulus, and on the combination of content analysis, critical event recall and social network analysis to the same learning event. The aim of this work is to move towards a more complete understanding: to square the circle of analysis of complex on-line learning in advanced communities.

ACKNOWLEDGMENTS

This work part of an ongoing international co-operation between Vic Lally at the University of Sheffield (UK) and Maarten De Laat at the University of Nijmegen (Netherlands). We would like to thanks Erica McAteer and Andy Tolmie for participating in discussions about the key ideas for this work; students and staff of the M.Ed. in E-Learning at the University of Sheffield.

REFERENCES

Barrett, E., & Lally, V. (1999). Gender differences in an on-line learning environment. Journal of Computer Assisted Learning, 15, 48-60.

De Laat, M., De Jong, F., & Simons, R.-J. (2001). Supporting Self Regulation of Learning Activities in Online Communities of Practice (poster paper). Paper presented at the First European Conference on Computer Supported Collaborative Learning, Maastricht, Netherlands.

De Laat, M., De Jong, F., & Ter Huurne, J. (2000, June 26 2000.). Supporting a Community of Practice: The Role of Workers as Learners. Paper presented at the Ed-Media, Montreal, Canada.

Kagan, N. (1984). Interpersonal Process Recall: Basic Methods and Recent Research. In D. Larsen (Ed.), Teaching Psychological Skills (pp. 229-244). Monterey, California: Brooks Cole.

Kagan, N., & Kagan, H. (1991). IPR - A Research/Training Model (offprint). In P. N. Dewrick (Ed.), Practical Guide to Using Video in the Behavioural Sciences. Canada: Wiley.

Lally, V. (2001). Analysing Teaching and Learning in a Networked Collaborative Learning Environment: Issues and Work in Progress. In P. Dillenbourg & A. Eurelings & K. Hakkarainen (Eds.), European Perspectives on Computer Supported Collaborative Learning (pp. 397-405). Maastricht: Maastricht McLuhan Institute.

Lally, V., & de Laat, M. (2002). Cracking the Code: Learning to Collaborate and Collaborating to Learn in a Networked Environment. In G. Stahl (Ed.), Computer Support for Collaborative Learning (pp. 160-168). Hillsdale, New Jersey: Lawrence Erlbaum.

Lally, V., & Scaife, J. A. (1995). Towards a Collaborative Approach to Teacher Empowerment. British Educational Research Journal, 21(3), 323-338.

Lally, V. E., & Barrett, E. (1999). Building a Learning Community On-Line: towards socio-academic interaction. Research Papers in Education, 14(2), 147-163.

McConnell, D. (1985). Learning from Audiovisual Media: Assessing Students' Thoughts by Stimulated Recall. Journal of Educational Television, 11(3), 177-187.

Popping, R. (2000). Computer Assisted Text Analysis. London: Sage.

Stahl, G. (Ed.). (2002). Computer Support for Collaborative Learning: Foundations for a CSCL Community. Hillsdale New Jersey: Lawrence Erlbaum.

Tuckwell, N. B. (1980). Stimulated Recall: Theoretical Perspectives and Practical and Technical Considerations (Occasional Paper Series 8-2-3). Alberta, Canada: Centre for Research in Teaching.

University of Sheffield. (2001). M.Ed. in E-Learning (formerly Networked Collaborative Learning) [Web Site]. School of Education. Retrieved 31/07/01, from the World Wide Web: http://www.shef.ac.uk/uni/projects/csnl/medindex.htm