Supporting collaborative learning and e-discussions using artificial intelligence techniques

Abstract. An emerging trend in classrooms is the use of networked visual argumentation tools that allow students to discuss, debate, and argue with one another in a synchronous fashion about topics presented by a teacher. These tools are aimed at teaching students how to discuss and argue, important skills not often taught in traditional classrooms. But how do teachers support students during these e-discussions, which happen at a rapid pace, with possibly many groups of students working simultaneously? Our approach is to pinpoint and summarize important aspects of the discussions (e.g., Are students staying on topic? Are students making reasoned claims and arguments that respond to the claims and arguments of their peers?) and alert the teachers who are moderating the discussions. The key research question raised in this work: Is it possible to automate the identification of salient contributions and patterns in student e-discussions? We present the systematic approach we have taken, based on artificial intelligence (AI) techniques and empirical evaluation, to grapple with this question. Our approach started with the generation of machine-learned classifiers of individual e-discussion contributions, moved to the creation of machine-learned classifiers of pairs of contributions, and, finally, led to the development of a novel AI-based graph-matching algorithm that classifies arbitrarily sized clusters of contributions. At each of these levels, we have run systematic empirical evaluations of the resultant classifiers using actual classroom data. Our evaluations have uncovered satisfactory or better results for many of the classifiers and have eliminated others. This work contributes to the fields of computer-supported collaborative learning and artificial intelligence in education by introducing sophisticated and empirically evaluated automated analysis techniques that combine structural, textual, and temporal data. p. 1

FreeStyler (Hoppe & Gaßner, 2002; http://www.collide.info/index.php/FreeStyler/) p. 2

Evidence from the computer-supported collaborative learning (CSCL) literature (Dillenbourg, Baker, Blaye, & O’Malley, 1995), as well as from p. 2

educational psychology (Cohen, 1994; Salomon & Globerson, 1989), suggests that fruitful collaboration does not occur spontaneously. p. 3

One approach to address this that has been explored by CSCL researchers is the notion of “collaboration scripts,” instructions for groups of learners on what activities to execute, when they need to be executed, and by whom they need to be executed in order to foster knowledge acquisition and learning (O’Donnell, 1999; Kollar, Fischer, & Hesse, 2003; Weinberger, Ertl, Fischer, & Mandl, 2005; Diziol, Rummel, Spada, & McLaren, 2007). Another approach is to provide a software agent that can coach and/or tutor the collaborating students (Constantino-Gonzalez & Suthers, 2002; Walker, McLaren, Rummel, & Koedinger, 2007). p. 3

A third approach is to include an artificial student in the collaboration whose responsibility it is to provide student-like contributions and, at the same time, peer coaching (Vizcaíno, 2005). p. 3

(Salmon, 2004) p. 3

(Gil, Schwarz, & Asterhan, 2007), need technology to ameliorate this situation. In particular, our aim has been to use technology to summarize what is occurring in student e-discussions and alert teachers to critical aspects and events of those discussions. p. 3

The key idea is to analyze student contributions and e-discussions using machine learning (Witten & Frank, 2005; Han & Kamber, 2006), shallow text processing (Rosé et al., 2008), and case- based graph matching (McLaren, 2003). The automated analysis is used to alert teachers to important patterns in the e-discussions as they grapple with simultaneous, synchronous e-discussions. The underlying approach of ARGUNAUT leverages the structure of the argument graphs, the textual contributions of the students, and the temporal sequence of those contributions. p. 3

Research Question 1: Is it possible to automate the identification of salient contributions and patterns in student e-discussions? p. 3

This question is still open within the educational technology research community; to our knowledge there have been no successful approaches to analyzing graphical e-discussions in an automated fashion. Furthermore, the use of such automated analysis to support teacher e-moderation is also novel. p. 3

Research Question 2: Do automatically identified contributions and patterns in student ediscussions help teachers in e-moderating simultaneous, synchronous e-discussions? p. 4

there have been two published studies (Wichmann, Giemza, Krauß, & Hoppe, 2009; Schwarz & Asterhan, in press) that have at least preliminarily shown the benefits of ARGUNAUT’s e- moderation approach (i.e., use of its most basic tools, not including the automated analysis), as well as a small study, discussed in this paper, that explores a teacher’s use of ARGUNAUT’s automated analysis tools. p. 4

(for a review of Belvedere, and other approaches to the support of argumentation with software tools, see Scheuer et al., 2010). p. 4

“Alerts”1 are a special, more focused type of Awareness Display that are visualized as colored circles next to the object to which they refer (e.g., students, contributions). “Shallow alerts” are those that can be computed in a straightforward fashion (e.g., alerts that indicate inactive students or the use p. 5

1 Note that an earlier conference paper (McLaren et al., 2007) refers to “alerts” as “awareness indicators.” We use “alerts” in this paper, since this term more aptly describes the end-user purpose of the technology and is favored by pedagogical experts. The term “awareness indicators,” on the other hand, is a more technical term indicating the core analysis technology. p. 5

of profanity in contributions). “Deep alerts,” the primary focus of the current paper, are those that are generated using more sophisticated artificial intelligence-based techniques. There are three types of deep alerts in the ARGUNAUT system: • Shape-level alerts reflect characteristics of individual contributions (e.g. whether a particular contribution contains a “reasoned claim”); • Paired-shape alerts reflect characteristics of pairs of linked contributions (e.g., whether two shapes constitute a “contribution followed by counterargument”); and • Cluster alerts reflect characteristics of arbitrary sets of two or more (although typically not more than five) linked contributions (e.g., a sequence of shapes that constitute a “chain of opposition” in which two students argue back and forth). p. 6

A total of eleven deep alerts have been created, evaluated, and incorporated into the current ARGUNAUT system. p. 6

While a teacher uses the Moderator’s Interface to monitor the simultaneous, synchronous3 e- discussions (the logged representation of these e-discussions are called “discussion maps”), two dedicated components are employed to analyze the discussion maps online and provide the teacher with alerts. The Shallow Loop is an integrated component of the Moderator’s Interface, providing shallow alerts in a local fashion. The Deep Loop is a Web Service-based independent component that provides deep alerts. The Classifier Proxy provides deep alerts as a local proxy for the remote Classification Service (in the middle of Figure 4, between the online and offline processes), which offers a set of classifiers each of which is capable of computing one deep alert. Classifiers are developed by an AI engineer using the Classifier Development Environment (shown on the right side of Figure 4) in an offline process and are then deployed to the Classification Service for online, run- time use. The AI-based Classifier Development Environment unfolds as two sub-components: The Machine Learning Training System is used to “train” classifiers targeted at shapes and paired-shapes using many annotated examples, in a supervised learning fashion (Witten & Frank, 2005; Han & Kamber, 2006). The Case-Based Graph Matching System is used to develop classifiers targeted at clusters using a small set of cluster examples. It is possible for teachers (and researchers) to add to the body of examples of both components through the Annotation Service shown on the border between the online and offline systems (since it can be used in both online and offline fashion), which sends new annotations to an Annotation Database. The Annotation Service is aimed at providing new examples for already existing classifiers to improve their performance. The Classification Service and Annotation Service constitute the interface between the online and offline process and are collectively called the “Classifier Web Service.” p. 7

The offline process of developing new classifiers, shown on the right side of Figure 4, takes annotated discussions, translates them to a form suitable for either machine learning or the case-based graph matching algorithm (different translators are used for the different approaches), and, with the intervention of an AI Engineer, generates new classifiers that are subsequently accessed at run time to first classify actions of the students in real time and then alert teachers (i.e., the “deep alerts”).4 p. 7

There is nothing inherent in the tools to preclude their use in an asynchronous fashion. However, the pedagogical approach discussed in this paper is focused on synchronous, in-classroom discussion. See (McAlister, Ravenscroft, & Scanlon, 2004; Schwarz & Asterhan, in press) for a discussion of and relative advantages of synchronous versus asynchronous communication. 4 There are numerous other functions of the Moderator’s Interface that are not discussed in this paper. For more details see Harrer, Ziebarth, Giemza, and Hoppe (2008) and Wichmann et al. (2009). p. 7

Pedagogical experts5 on the ARGUNAUT project have annotated (offline) components of many past discussions as to whether, for instance, students made reasoned claims, were on topic, or were engaged in raising and answering questions. This corresponds to the process shown at the bottom right of Figure 4. These annotations were used to train (or provide examples for) the current live set of ARGUNAUT classifiers to identify meaningful types of contributions (or groups of contributions) in new e-discussions. The annotations and subsequent classifications are based on structural, textual, and temporal elements of the student contributions. The specific way that these elements are used to generate the classifiers is explained in detail in the following section. p. 8

A key to our research approach was to methodically evaluate the different levels of the discussion maps, starting with single contributions, moving to pairs of linked contributions, and, finally, focusing on clusters (i.e., two or more connected contributions). p. 9

That is, at the outset of our work we had a high degree of belief that we could create classifiers that would work at the single contribution level (due to the prior success of, for instance, Rosé et al., 2008), but were less sure at the paired level, and even less so at the cluster level. Second, we believed that the classification results of the lower levels could possibly be used as attributes in the solutions at the more complex levels. That is, we believed the classification of single contributions might be usable as inputs to or attributes of the classification process of the paired contributions entailing those single contributions (and subsequently to clusters, as well). Finally, we realized that the simpler structures would be easier for our pedagogical experts to annotate, meaning we would much sooner have data to evaluate and experiment with by starting with the simpler structures. p. 9

Across all levels, we knew that we needed language- processing techniques, given that a central feature of each student’s contribution is the text that he or she types. Some of the issues inherent in language analysis, well known to the natural language community, include references that participants make to other’s contributions, use of prior knowledge brought to bear by participants, and implications that span across different contributions. Due to the success that Carolyn Rosé and her students had with a comparable corpus of argumentation data (Rosé et al., 2008; Dönmez, Rosé, Stegmann, Weinberger, & Fischer, 2005), but with a different purpose (i.e., they were, at least initially, more interested in helping human coders (semi)-automatically annotate data, rather than in doing online automated analysis and also were focused primarily on classifying single contributions), we chose TagHelper as a means to extract meaningful, yet shallow, attributes of text. Given text strings, TagHelper performs stemming (i.e., finding the roots of words), extracts textual attributes such as unigrams and bigrams (single words and pairs of words occurring in the text), part-of-speech bigrams (paired grammatical structures such as noun-verb, adjective-noun) and punctuation, and filters out words that are not likely to contribute to a classifier’s performance such as stop and rare words. The various attributes extracted by TagHelper were then used, in conjunction with structural and temporal attributes of the discussion maps, as descriptive attributes of the student contributions in our machine learning and case-based graph matching approaches. p. 9

At the shape and paired-shape level, we pursued a classic machine learning approach (McLaren et al., 2007; Scheuer & McLaren, 2008), using an off-the-shelf data mining tool called RapidMiner (formerly called YALE, Mierswa, Wurst, Klinkenberg, Scholz, & Euler, 2006), since (a) such an approach is often successful in finding patterns in data when annotated data with well-defined categories is available (our pedagogical experts had defined some categories, were in the process of defining others, and we knew they could annotate the data) and (b) the unit of analysis is consistent between the annotated data and the subsequently classified data. By (b) we mean that we knew we p. 9

could annotate both individual and paired contributions that could then be used to train classifiers for, respectively, individual and paired contributions (at run-time). Put another way, we knew that at run- time we could precisely identify the structures (i.e., shapes and paired shapes) for which we had developed pre-trained classifiers and could apply the classifiers to the invariant structures. However, at the cluster level, in which an arbitrary and varying number of contributions (but typically five or less) might represent a particular category, we knew we could not (directly) apply a supervised learning approach. Thus, at that level, we investigated the idea of using example clusters, that is, sub-graphs of discussion maps, as a means for searching for similar clusters in new maps (Mikátko & McLaren, 2008; McLaren et al., 2009). We call this a case-based graph-matching approach. It is inspired both by case-based reasoning (Kolodner, 1993; McLaren, 2003) and the query-by-example approach used in database languages (Zloof, 1977). In the following sections, we describe, in more detail, how we developed and applied the various AI techniques to the different levels of the discussion maps to create the classifiers that support the deep alerts of ARGUNAUT, as shown in Figure 4. p. 10

The coding process was influenced by theoretical notions of dialogism in the context of critical reasoning and argumentation (Schwarz & Glassner, 2007; Schwarz & De Groot, 2007; Wegerif, 2006). The ARGUNAUT pedagogical experts looked for dialogic and argumentation aspects, even at the individual contribution level. All but two of the coding categories were identified through a bottom-up analysis of a relatively small subset of the 84 discussion maps, with single and groups of recurring contributions identified as potentially representing interesting, important, or problematic phenomena. Two of the cluster categories, Deepening and Widening, were selected in top-down fashion; they are interactional categories that indicate creative reasoning and the emergence of new ideas (De Laat, Chamrada, & Wegerif, 2008; Wegerif, 2007; McLaren et al., 2009). A Deepening occurs when students provide further argumentation for an on-going perspective (Baker, Andriessen, Lund, van Amelsvoort, & Quignard, 2007). A Widening occurs, on the other hand, when a student (or students) attempts to diverge from the current perspective by either questioning it or presenting a new perspective (Wegerif, 2007). New perspectives enable participants in a dialogue to see things in a new way and expand their understanding, thus a widening move in a debate is also a creative move. p. 10

Originally, we were interested in all of the categories shown in Table 1, but after initial machine learning experiments, we focused both our annotation efforts and machine learning experiments on only two categories, Topic Focus and Reasoned Claim. Two categories, Critical Evaluation of Opinions and Summary, were dropped due to low inter-rater reliability. The other three categories (i.e., Task Management, Request for Clarification, Intertextuality) led to weaker machine learning results (i.e., Kappas well under .60) most likely because of imbalanced class distributions and too few p. 11

examples for one class, two problems well known for their detrimental effects on machine learning (Japkowicz & Stephen, 2002; Weiss, 2004). p. 12

The shape and paired-shape classifiers were developed in two steps: First, the annotated data was translated into a format amenable to standard machine-learning algorithms. This is indicated in Figure 4 by the “Deep Loop Data Translator(s)” box. Second, experiments with a multitude of machine- learning techniques (and parameters) were carried out in order to derive the most effective classifiers for each category. The “Machine Learning Training System” box of Figure 4 represents this process. p. 13

Attributes used for machine learning at the shape level p. 13

Note that temporal attributes (e.g., shape1 was created before shape2, the text of shape1 was written before the text of shape2) were also extracted at the first (structural) and second (TagHelper) extraction step. While the shape level does not use such attributes, the paired-shape level does; this is discussed below. The RapidMiner toolkit, freely available software that supports interactive experimentation with a wide range of machine learning algorithms (Mierswa et al., 2006), then accepted the training data, which now consisted of the derived structural, textual, and temporal attributes and was used to experiment with different machine learning algorithms, different parameters, and so on. Once a suitable algorithm was found, a classifier was generated, one that could also be accessed by the Classifier Web Service. (The right side of Figure 5 is explained later in the paper.) p. 14

The shape and paired-shape experiments described in this paper extend those reported in McLaren et al. (2007) and Scheuer and McLaren (2008). p. 14

In the experiments reported here, we used the attribute sets of the best classifiers from the earlier experiments and systematically experimented with algorithms that have been shown to be effective for text categorization tasks: Support Vector Machines (SVM) (Joachims, 1998), Naïve Bayes (McCallum & Nigam, 1998) and Boosted Decision Trees (Boosted DT) (Schapire & Singer, 2000). We also tried the Decision List algorithm, since this led to the best results for some of our classifiers (Scheuer & McLaren, 2008). Unlike the earlier experiments, we also used attribute selection, specifically, chi- squared attribute selection of the top 100 attributes. Since SVMs typically cope well with high dimensional input spaces, we also tested SVMs without attribute selection. Finally, we used SVM with cost balancing activated, meaning that the relative weights (or misclassification costs) of the two classes were adapted to the class distribution during SVM training. In our earlier experiments we achieved increased performance using this option, presumably because of the skewed class distributions in our data set, as described above. p. 15

An acceptability threshold of 0.8, or at least 0.7 (Rosé et al., 2008; Krippendorff, 1980), is recommended in content analysis. Given that in our approach we have teachers will be aware of the possibility of possible misclassifications by the classifiers and the need to use their own judgment, we considered a slightly more generous interpretation sufficient. Thus, we used an acceptability threshold of 0.61, which means, according to Landis and Koch (1977)10, a “substantial” agreement between a machine-learned classifier and a gold standard. p. 15

The right side of Figure 5 depicts the automated online process used to analyze discussions by means of machine-learned classifiers that have been derived in the offline process on the left side of Figure 5. Starting from the top right, a discussion is provided (by the Moderator's Interface) in an XML format called “GraphML,” which represents the current discussion state. The discussion is processed in two steps in order to map shapes (and paired-shapes) into the attribute space on which the classifier was previously trained. In a first step the “Structural Extraction Program” segments the input into the respective analysis units (shapes or paired-shapes) and structural attributes are extracted (e.g., on the p. 16

shape level: shape type, the # of undirected links, the # of in-links, # of out-links, as shown in Table 2). In a second step the TagHelper software extracts text attributes. This process uses the Text Attributes file to ensure that the extracted text attributes are the same as those used during training. (As on the left side of Figure 5, temporal attributes can also be derived at either the first or second extraction step, at least for paired shapes). Shapes (and paired-shapes) are now represented as attribute vectors that comply with the format expected by the machine-learned classifiers (the “Classifier Applier” in Figure 5). The classifier analyzes these attribute vectors, determines the classes of the shapes (and paired-shapes) and returns the results (alerts) to the Moderator’s Interface. p. 17

Note, however, that any participant in a discussion can create the link between two shapes, meaning therefore that connected shapes don’t necessarily imply interaction between students. In practice, however, the second student almost always creates the link to the first student’s shape. p. 19

Clusters presented an additional challenge because: the example there are two branches, 1st to 2nd contribution and 1st to 3rd contribution. 3rd shape text (Claim), Student 1: “Also, if people are able to learn about other cultures, etc., then they may not feel that they need to actually experience the culture for themselves, e.g., by visiting other countries, possibly creating a greater divide, because the global community would interact less in person, only through technology.” (Link between 1st and 3rd shape is “other”) Deepening Providing further argumentation for a perspective that is part of the current discussion. 1st shape text (Claim), Student 1: “Yes, but how are we to overcome that problem? As technology is advancing all the time, and computers are becoming more and more common place in learning resources,….. how are we to deal with the problem of certain individual not having a computer to use at home…?” 2nd shape text (Idea), Student 2: “Its hard to say… An idea would be to only use ICT learning when done in a school setting where everyone has acess to the same materials” (Link between 1st and 2nd shape is “other”) 3rd shape text (Idea). Student 3: “places such as schools and colleges, etc … would probly argue that that as a solution to this problem they do provide students with ict facilitys, however i still do not think this works as often they do not have enought facilities to accomadate the number of students.” (Link between 1st and 3rd shape is “other”) 30 (14 maps) • • • The data is more complex (i.e., a larger combination of structure and text), The data is noisier (i.e., more student mistyping and mistakes), Annotated data is much scarcer, due to the difficulty of coding clusters. We explored several approaches, including, unsupervised learning (Jain, Murty, & Flynn, 1999), pattern mining (Srikant & Agrawal, 1996), pre-defined pattern rules (Harrer, Hever, & Ziebarth, 2007) and supervised clustering (Finley & Joachims, 2005), but ultimately designed and developed an approach that best fit the problem characteristics: DOCE (Detection of Clusters by Example) (Mikátko & McLaren, 2008). DOCE, a case-based graph-matching algorithm, is based on the idea of using cluster examples to find similar clusters in other discussions and is inspired by case-based reasoning (Kolodner, 1993; McLaren, 2003), analogical mapping (Forbus, Gentner, & Law, 1994), and database query-by-example (Zloof, 1977). An example cluster, also called a “model graph,” is selected and used as a search query for similar clusters across other discussion maps, called “input p. 24

graphs.” The output of the algorithm is a list of matching clusters in the input graph(s), sorted according to a similarity rating. The DOCE algorithm is summarized in Figure 9. First, the example cluster (model graph) and the discussion map (input graph) are parsed from the XML file format used by the Moderator’s Interface for representing a snapshot of the discussion. Both graphs are preprocessed as follows: (1) an adjacency matrix representing the structure of the graph is constructed and (2) each contribution and link in the discussion graph is characterized by an attribute vector, extracted from attributes such as shape/link type, text length, link direction and whether the same user created two linked shapes. As with the shape and paired-shape classifiers, TagHelper (Rosé et al., 2008) further enriches the attribute vectors with additional information from the text analysis of contributions (e.g., unigrams, bigrams, part-of-speech bigrams). As per Figure 5, a Text Attributes file is generated by TagHelper and used by the DOCE algorithm. (Note that this file is the same one used by the shape-level and paired-shape classifiers). Additionally, DOCE extends the attribute vectors of shapes (links) with shape and paired- shape classifications.14 In the next step, DOCE compares the attribute vectors of vertices/edges in the model and input graphs by calculating their distance in a manner similar to unsupervised learning algorithms. The proximity is pre-computed for each pair of model/input objects and stored in the similarity matrices. Finally, an inexact graph matching method based on a customized version of the edit distance algorithm (Wong, You, & Chan, 1990; Tsai & Fu, 1979) is employed to find clusters with the highest structural and content similarity to the model graph. Similar algorithms have been successfully used for various purposes, such as computer vision (Gregory & Kittler, 2002) and retrieving relevant principles from ethics cases (McLaren, 2003). p. 25

Can multiple example clusters of the same type, provided as input to DOCE, lead to even better results in retrieving relevant clusters than a single example cluster? Such a data fusion technique, combining the preferences and results of several “experts,” has been successfully used in information retrieval (IR) (Aslam & Montague, 2001) and machine learning (boosting, bagging) (Han & Kamber, 2006). For example, in IR, the same query is often submitted to several search engines and the results combined. Prior work has shown improvements in recall and precision using such a technique (Aslam & Montague, 2001). p. 28

Our Cluster Experiment #1 results were first reported in Mikátko and McLaren (2008) p. 28

An important contribution of our work has been the emphasis on accounting for structural, textual, and temporal aspects of the contributions made by students to e-discussions within graphical tools such as Digalo and FreeStyler. To our knowledge, no prior research on automated analysis has attempted to cover all of these aspects of online discussions – and particularly not with an emphasis on real-world use to support a teacher in the classroom, as we have on ARGUNAUT – and thus our research makes a novel contribution to both the fields of artificial intelligence in education and computer-supported collaborative learning. p. 33

This could be tested with Wizard of Oz approach, having a human simulating the markup. p. 35

simultaneously? Put another way: are the deep alerts relevant and reliable enough to support teachers? p. 35

Put another way: are the deep alerts relevant and reliable enough to support teachers? p. 35

On the other hand, we have already anticipated some shortcomings in the deep alerts and have some clear ideas about how they might be extended to better support teachers (Scheuer & McLaren, 2008). Most of the patterns found by the classifiers, especially shape and paired-shape classifications, are more descriptive than alerting in nature, that is, they point to general discourse patterns whose occurrence is not problematic per se. For instance, while teachers generally want their students to stay on-topic, should they intervene on every off-topic contribution? Such an intervention pattern would probably interrupt the flow of the discussion and cause more harm than good. But what if eight of ten total contributions are off-topic? In such a discussion something clearly is wrong and the teacher should probably intervene. These examples show that fine-grained patterns in isolation may have limited significance. However their aggregation may point to critical general situations, suggesting the need for teacher action. We have incorporated some simple aggregating indicators in the Moderators’ Interface to help with this; for instance, the Moderator’s Interface displays the total number of positive classifications per discussion, such that teachers can identify undesirable overall discussions at a glance (e.g., a discussion containing predominantly off-topic contributions). We envision going beyond simple aggregations, developing models on top of the classifications. The goal is to provide teachers with a clearer and more global picture by qualitatively summarizing the discussions. Such a model might define numeric conditions such as “If more than 30% of all links are part of argument- counterargument clusters then the discussion qualifies as controversial.” p. 36

in the Group Leader Tutor (Israel & Aiken, 2007) that classifies the collaborative effort of a group along three dimensions by aggregating and combining counts of collaborative skills that have been displayed. Alternatively, one could use inductive inference. Labels might be assigned to complete discussions, and a machine-learned model computed that infers discussion-level classifications from shape and paired-shape classifications. p. 37

One way to detect larger patterns, without using a simple aggregation approach, is a knowledge-engineering approach, in which complex graphical patterns are explicitly pre-defined based on shape and paired-shape classifications. A Graph Grammar formalism, such as described in Pinkwart, Ashley, Lynch, & Aleven (2008), may be an option here. p. 37

The closest research to ours is the work of Rosé and colleagues, who developed the text analysis tool, TagHelper (Rosé et al., 2008), also used in our work. p. 37

More recently, they developed an approach to providing dynamic support to dyads collaborating on a problem-solving task (Kumar, Rosé, Wang, Joshi, & Robinson, 2007) p. 37

In contrast to our approach, their analysis results are not displayed to human teachers but are instead used to trigger automatic interventions, sent directly to students. p. 37

Goodman and colleagues (2005) also have applied a machine-learning approach to support collaborative problem solving in the EPSILON system. Peer groups work together on a problem in the domain of object modeling techniques (OMT). Their collaboration takes place within a shared whiteboard, similar to the shared workspaces of ARGUNAUT, in which diagrams (e.g., class diagrams) are constructed. Peers communicate via a text chat with a sentence opener interface; an agenda tool supports task management. In contrast to ARGUNAUT, dialogues are focused rather narrowly on a single domain (OMT) and on the coordination of task-related activities. The system p. 37

evaluates aspects concerning domain (e.g., domain knowledge of peers), task (e.g., progress in solving the task) and, similar to our objectives, possible problems in the collaboration process (e.g., unanswered questions). The sentence opener interface plays a critical role; it is used to automatically assign a dialogue act classification to each chat contribution. These dialogue acts are used as a meta- level description of the discourse and serve as attributes for machine-learning analyses, bypassing the complicated task of natural language (or text) processing that we tackle in our work. The analysis approach of Goodman et al. (2005) differs to ours in other respects: our analysis is based on graphical argument diagrams, in which the relations between contributions are made explicit, that is, we know existing relations and their types; they analyze sequences of classified chat messages in which neither relations nor their types are explicitly represented. We apply automated shallow language processing techniques to approximate the content of messages; for the same purpose, they use simple human- defined keyword lists. Another difference is how the analysis results are used: We support students indirectly via a moderator, whereas they follow more of an ITS-style approach: Some of the results are displayed immediately to the peers via meters, while direct support is provided by means of an artificial peer agent that verbally interacts with the participants. p. 38

Soller’s work, also in the context of EPSILON, was concerned with the analysis of chat conversations that accompanied activities in a problem-solving environment (Soller, 2004). The chat tool was enhanced by a sentence-openers interface to structure users’ communication (the same one used by Goodman et al.) and to make automated analysis feasible, similar in some respects to how the pre-defined shapes and link types of ARGUNAUT make automated analysis tractable. The analysis of EPSILON aimed at identifying episodes in which students communicated their knowledge to their peers (“knowledge sharing episodes”) and episodes in which they failed to do so (“knowledge sharing breakdowns”). She computed machine-learned classifiers, more specifically, Hidden Markov Models (HMMs), from data annotated as knowledge sharing episodes and knowledge sharing breakdown episodes. In a second step, she investigated reasons for knowledge sharing breakdowns using multidimensional scaling (MDS) and clustering techniques. In this way she identified different patterns (clusters) of successful knowledge sharing and knowledge sharing breakdowns. The key differences to our work are: (1) Soller analyzes sequences of actions, whereas we analyze sub-graphs within a discussion map that are sometimes sequential, sometimes parallel, (2) the textual content of contributions is not analyzed in Soller’s work; it relies exclusively on dialogue acts, which are automatically inferred from the selected sentence openers, (3) she uses a sequential learning approach that can be applied to sequences of arbitrary length whereas our ML approach is currently restricted to single shapes and pairs, (4) a sequential learning approach captures dependencies between contributions naturally (because it assumes by design a sequential dependencies); our ML approach with flat representations requires a pre-processing step to encode sequential (and other) dependencies explicitly in its attribute space. On the other hand HMMs cannot be used to analyze graphical structures, such as the ones we are working with in ARGUNAUT. Also, Soller’s approach requires a pre-segmentation of knowledge-sharing sequences; a restriction that introduces a new potential source of error in actual practice. Furthermore, our machine learned classifiers are more reliable: Soller reports an accuracy of 74%, which we re-calculated to a Kappa of 0.47, considerably lower than our best six machine-learned classifiers. p. 38

Ravi and Kim (2007) analyzed threads in a technical discussion board in order to call an instructor’s attention to threads that contain unanswered questions. They developed two classifiers (linear SVMs), one for detecting questions (QC), and a second for detecting answers (AC). They achieved accuracies of 88% (QC) and 73% (AC). Similar to our approach, they used shallow language p. 38

attributes, although somewhat more elaborated ones than the ones we used (e.g., in addition to uni- and bigrams they also use tri- and quadrograms). In addition to the classifiers they implemented a rule- based thread profiler that assigns one of four typical profiles to threads (accuracies vary between 70% and 93%). A follow-up to this work, Kim et al. (2008) describe PedaBot, a threaded discussion system that scaffolds students’ discussions by retrieving messages from past discussions that are possibly relevant to the current context. Discussions are about technical topics (e.g., operating systems). The system is based on a text corpus (extracted from two text books), which has been analyzed to (1) determine relevant topics (corresponding to book chapters and sections), (2) determine relevant technical terms (corresponding to glossary entries) and (3) develop a topic profiler (a classic information retrieval term frequency–inverse document frequency (tf-idf) classifier that assigns topics to messages based on technical terms contained within them). The topic profiler, in conjunction with a further application of tf-idf term weighting, can then be used to fetch possibly relevant messages from prior discussions. The key differences between our approach and theirs are: (1) their analytical procedure is enhanced with domain knowledge (topics and technical terms) and hence is very domain- specific; our approach, on the other hand, while likely tied somewhat to our “domain” is less strictly so, (2) the output of PedaBot and its predecessor is also more specific (i.e., a concrete message instead of an abstract classification) and (3) the development of the topic profiler did not require annotation efforts. Finally, as with the other prior work already mentioned, the Kim et al. work does not specifically take into account structure and temporal attributes to support its classification approach, as do both our machine learning and DOCE approaches. p. 39

Jeong (2003; 2005; 2006) has developed a software tool called the Discussion Analysis Tool (DAT) that uses sequential analysis to capture and model sequences of speech acts. DAT models an online threaded conversation as a network of transitional probabilities, called a transitional state diagram, building the diagram from pre-labeled data. For example, in one diagram generated by DAT from real data a “challenge” act occurred with a probability of 0.52 after an “argument” was made, while an “explanation” followed an “argument” with a probability of 0.08. DAT has been used, for instance, to evaluate the interactions that are most likely to promote critical thinking (Jeong, 2003) and the effects of supportive language (e.g., I agree, ask questions) on subsequent group interactions (Jeong, 2006). DAT can also be used to evaluate whether threaded conversations deviate from a norm; it creates a Z-score matrix to show probabilities that were significantly higher or lower than expected in one state diagram compared to another. While Jeong shares our goal of analyzing collaborative, and argumentative, discussions, DAT is intended more as a post-hoc analysis tool for supporting various types of group interaction studies, and not for online “live” analysis. In particular, Jeong’s system does no language analysis; it depends on human post-hoc coding (or real-time labeling) to identify the individual acts. p. 39

Sequential pattern mining algorithms (Agrawal & Srikant, 1995), which discover frequently occurring sequences of activities in a dataset, are an intriguing possibility for finding salient patterns of student collaboration. Such algorithms have, for instance, been used to account for both language and contextual attributes – to a reasonable degree of accuracy – in classifying email messages (Carvalho & Cohen, 2005). Sequential pattern mining algorithms have also been tried in collaborative learning scenarios. For instance, Kay, Maisonneuve, Yacef, and Zaïane (2006) used a variant of the Generalized Sequential Pattern Algorithm (GSP) (Srikant & Agrawal, 1996) to identify common interaction patterns in a source repository and Wiki log data from student software development projects. The authors’ goal was to build tools that can flag student interactions that are indicative of problems, so that the tools can later be used to assist students in recognizing problems in new p. 39

interactions. However, in contrast to log records, which contain simple events such as “file X was modified,” the ARGUNAUT discussion maps contain natural language and highly complex graph structures, thus making the Srikant and Agrawal approach far less likely to succeed. Furthermore, the Srikant and Agrawal approach is an unsupervised learning approach, searching for frequently occurring patterns, whereas our classifiers target predefined and pedagogically relevant categories. Furthermore, Rosé et al. (2008) were able to show that designing or selecting appropriate linguistic and contextual features may be more important than using sophisticated supervised sequential learning algorithms, such as those used in approaches like Carvalho and Cohen (2005). A more manual approach to pattern mining was also evaluated on the ARGUNAUT project (Harrer et al., 2007). Their tool was designed for mining user-specified sequences of actions in the discussions (i.e., pattern rules), such as “create shape”/“add link”/“modify text.” The tool was able to detect some commonly occurring patterns. However, this algorithm, which does exact-matching, was unable to detect patterns that differed in subtle and imprecise ways from one another. A key objective of our work on the DOCE algorithm was to develop a tool that could perform such inexact matching. p. 40

REFERENCES Agrawal, R., & Srikant, R. (1995). Mining sequential patterns. In P.S. Yu & A.L.P. Chen (Eds.) Proceedings of the 11th International Conference on Data Engineering (ICDE '95), (pp. 3–14). Washington, D.C.: IEEE Computer Society. Andriessen, J.E.B., & Schwarz, B.B. (2009). Argumentative design. In N. Muller Mirza and A.-N. Perret Clermont (Eds.) Argumentation and Education: Theoretical Foundations and Practices. (pp. 145-174). Dordrecht, Heidelberg, London, New York: Springer. Aslam, J.A., & Montague, M. (2001). Models for metasearch. In D.H. Craft, W.B. Croft, D.J. Harper, & J. Zobel (Eds.) Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. (pp. 276-284). New York: ACM Press. Asterhan, C.S.C., Wichmann, A., Mansour, N., Wegerif, R., Hever, R., Schwarz, B.B., & Williams, M. (2008). Argunaut deliverable D6.3: Evaluation report on the pedagogical content of the Argunaut system (revised). Unpublished Manuscript. http://www.argunaut.org/ARGUNAUT_-_D6-3.zip Baker, M., Andriessen, J., Lund, K., van Amelsvoort, M., & Quignard, M. (2007). Rainbow: A framework for analyzing computer-mediated pedagogical debates. International Journal of Computer-Supported Collaborative Learning, 2, 315-357. Ben-David, A. (2006) What's wrong with hit ratio? IEEE Intelligent Systems, 21(6), 68-70. p. 41

Carvalho, V.R., & Cohen, W.W. (2005). On the collective classification of email “speech acts.” In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 345–352). New York: ACM Press. Coghlan, M. (2001). eMODERATION – Managing a new language? Net*Working 2001 Conference – from Virtual to Reality, Brisbane, October 2001. http://michaelcoghlan.net/mc/ESL_WORK/ARTICLES_PRESENTATIONS/nw2001/emod_newlang.htm. Cohen, E.G. (1994). Restructuring the classroom: Conditions for productive small groups. Review of Educational Research, 64(1), 1-35. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. Constantino-Gonzalez, M.A., & Suthers, D. (2002). Coaching collaboration in a computer-mediated learning environment. In G. Stahl (Ed.) Proceedings of the Conference on Computer Support for Collaborative Learning: Foundations for a CSCL Community (CSCL-02). (pp. 583-584). Mahwah, NJ: Lawrence Erlbaum Associates. De Groot, R., Drachman, R., Hever, R., Schwarz, B., Hoppe, U., Harrer, A., De Laat, M., Wegerif, R., McLaren, B.M., & Baurens, B. (2007). Computer supported moderation of e-Discussions: The ARGUNAUT approach. In C. A. Chinn, G. Erkens & S. Puntambekar (Eds.), Mice, Minds and Society, Proceedings of the Conference on Computer Supported Collaborative Learning (CSCL-07), Vol 8 (pp. 165-167). International Society of the Learning Sciences, Inc. ISSN 1819-0146. De Laat, M., Chamrada, M., & Wegerif, R. (2008). Facilitate the facilitator: Awareness tools to support the moderator to facilitate online discussions for networked learning. In Proceedings of the 6th International Conference on Networked Learning (pp. 80-86) Lancaster: University of Lancaster. De Laat, M., & Wegerif, R. (2007). Perspectives/rules to evaluate discussions. Argunaut public deliverable D5.1. http://www.argunaut.org/. Dillenbourg, P., Baker, M., Blaye, A., & O'Malley, C. (1995). The evolution of research on collaborative learning. In P. Reimann & H. Spada (Eds.) Learning in humans and machines: Towards an interdisciplinary learning science (pp. 189–211). Oxford: Elsevier/Pergamon. Diziol, D., Rummel, N., Spada, H., & McLaren, B.M. (2007). Promoting learning in mathematics: Script support for collaborative problem solving with the Cognitive Tutor Algebra. In C.A. Chinn, G. Erkens & S. Puntambekar (Eds.) Mice, minds and society. Proceedings of the Computer Supported Collaborative Learning Conference (pp. 39-41). International Society of the Learning Sciences, Inc. Dönmez, P., Rosé, C., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with automatic corpus analysis technology. In T. Koschmann, D.D. Suthers, T.-W. Chan, (Eds.) The next 10 years! Proceedings of the Conference on Computer Supported Collaborative Learning (CSCL-05) (pp. 125-134) Mahwah, NJ: Lawrence Erlbaum. Finley, T., & Joachims, T. (2005). Supervised clustering with Support Vector Machines. In L. De Raedt & S. Wrobel (Eds.) Proceedings of the 22nd International Conference on Machine Learning (pp. 217-224). New York: ACM. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378– 382. Forbus, K.D., Gentner D., & Law, K. (1994). MAC/FAC: A model of similarity-based retrieval. Cognitive Science, 19, 141-205. Garey, M.R., & Johnson, D.S. (1979). Computers and Intractability: A Guide to the Theory of NP- Completeness. New York: WH Freeman & Co. Gil, J., Schwarz, B.B., & Asterhan, C.S.C. (2007). Intuitive moderation styles and beliefs of teachers in CSCL- based argumentation. In C.A. Chinn, G. Erkens & S. Puntambekar (Eds.) Mice, minds and society. Proceedings of the Computer Supported Collaborative Learning Conference (pp. 219-228). International Society of the Learning Sciences, Inc. Goodman, B., Linton, F., Gaimari, R., Hitzeman, J., Ross, H., & Zarrella, G. (2005). Using dialogue features to predict trouble during collaborative learning. User Modeling and User-Adapted Interaction, 15, 85-134. p. 42

Gregory, L., & Kittler, J. (2002). Using graph search techniques for contextual colour retrieval. In T. Caelli, A. Amin, R.P.W. Duin, M.S. Kamel, & D. de Ridder (Eds.) Proceedings of the Joint IAPR International Workshop on Structural, Syntactic, and Statistical Pattern Recognition (pp. 186-194). New York: Springer. Han, J., & Kamber, M. (2006). Data Mining: Concepts and Techniques. San Francisco: Morgan Kaufmann. Harrer, A., Hever, R., & Ziebarth, S. (2007). Empowering researchers to detect interaction patterns in e- Collaboration. In R. Luckin, K.R. Koedinger, & J Greer (Eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007) (pp. 503-510). Amsterdam: IOS Press. Harrer, A., Ziebarth, S., Giemza, A., & Hoppe, H.U. (2008). A framework to support monitoring and moderation of e-discussions with heterogeneous discussion tools. In P. Díaz et al (Eds.) Proceedings of the 8th IEEE International Conference on Advanced Learning Technologies (ICALT-2008) (pp. 41-45). Washington, D.C.: IEEE Computer Society. Hever, R., De Groot, R., De Laat, M., Harrer, A., Hoppe, H.U., McLaren, B.M., & Scheuer, O. (2007). Combining structural, process-oriented and textual elements to generate alerts for graphical e-discussions. In C.A. Chinn, G. Erkens & S. Puntambekar (Eds.) Mice, minds and society. Proceedings of the Computer Supported Collaborative Learning Conference (pp. 286-288). International Society of the Learning Sciences, Inc. Hoppe, H.U., & Gaßner, K. (2002). Integrating collaborative concept mapping tools with group memory and retrieval functions. In G. Stahl (Ed.) Proceedings of the Conference on Computer Support for Collaborative Learning: Foundations for a CSCL Community (pp. 716-725). Mahwah, NJ: Lawrence Erlbaum Associates. Israel, J., & Aiken, R. (2007). Supporting collaborative learning with an intelligent web-based system. International Journal of Artificial Intelligence in Education, 17, 3-40. Jain, A.K., Murty, M.N., & Flynn, P.J. (1999). Data clustering: A review. ACM Computing Surveys (CSUR), 31 264-323. Japkowicz, N., & Stephen, S. (2002). The class imbalance problem - A systematic study. Intelligent Data Analysis, 6, 429-450. Jeong, A. (2003). The sequential analysis of group interaction and critical thinking in online threaded discussions. American Journal of Distance Education, 17 (1), 25-43. Jeong, A. (2005). A guide to analyzing message-response sequences and group interaction patterns in computer- mediated communication. Distance Education, 26(3), 367-383. Jeong, A. (2006). The effects of conversational styles of communication on group interaction patterns and argumentation in online discussions. Instructional Science, 34(5), 367-397. Joachims, T. (1998). Text categorization with Support Vector Machines: Learning with many relevant features. In C. Nedellec & C. Rouveirol (Eds.) Proceedings of the 10th European Conference on Machine Learning (ECML-98) (pp. 137-142). Berlin: Springer. Kay, J., Maisonneuve, N., Yacef, K., & Zaïane, O. (2006). Mining patterns of events in students’ teamwork data. In C. Heiner, R. Baker and K. Yacef (Eds.) Proceedings of the Workshop on Educational Data Mining at the 8th International Conference on Intelligent Tutoring Systems (ITS 2006) (pp. 45–52). http://www.educationaldatamining.org/ITS2006EDM/EDMITS2006.html Kim, J., Shaw, E., Ravi, S., Tavano, E., Arromratana, A., & Sarda, P. (2008). Scaffolding on-line discussions with past discussions: An analysis and pilot study of PedaBot. In B. Woolf, E. Aimeur, R. Nkambou, S. Lajoie (Eds.) Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS-08) (pp. 343-352). Berlin: Springer. Klein, P., Tirthapura, S., Sharvit, D., & Kimia, B. (2000) A tree-edit-distance algorithm for comparing simple, closed shapes. In D. Schmoys (Ed.) Proceedings of the Eleventh Annual ACM-SIAM Symposium on Discrete Algorithms (pp. 696-704). San Francisco: Society for Industrial and Applied Mathematics Kollar, I., Fischer, F., & Hesse, F. W. (2003). Cooperation scripts for computer-supported collaborative learning. In B. Wasson, R. Baggetun, H.U. Hoppe & S. Ludwigsen (Eds.) Proceedings of the Computer Support for Collaborative Learning (pp. 59-61). Bergen, Norway: InterMedia, University of Bergen. Kolodner, J. (1993). Case-Based Reasoning. San Francisco: Morgan Kaufmann Publishers. p. 43

Krippendorff, K. (1980). Content Analysis: An Introduction to its Methodology. Thousand Oaks, CA: Sage Publications. Kumar, R., Rosé, C.P., Wang, Y.-C., Joshi, M., & Robinson, A. (2007). Tutorial dialogue as adaptive collaborative learning support. In R. Luckin, K. R. Koedinger, & J. Greer (Eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007) (pp. 383-390). Amsterdam: IOS Press. Landis J.R., & Koch G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. Lingnau, A., Harrer, A., Kuhn, M., & Hoppe, H.U. (2007). Empowering teachers to evolve media enriched classroom scenarios. Research and Practice in Technology Enhanced Learning, 2, 105-129. McAlister, S., Ravenscroft, A., & Scanlon, E. (2004). Combining interaction and context design to support collaborative argumentation using a tool for synchronous cmc. Journal of Computer Assisted Learning, 20 (3), 194–204. McCallum, A., & Nigam K. (1998). A comparison of event models for Naïve Bayes text classification. In M. Sahami, M. Craven, T. Joachims, & A. McCallum (Eds.) AAAI-98 Workshop on Learning for Text Categorization, (pp. 41–48). Menlo Park, CA: AAAI Press. Also technical report WS-98-05, Carnegie Mellon University, Pittsburgh, USA. McLaren, B.M. (2003). Extensionally defining principles and cases in ethics: An AI model. Artificial Intelligence, 150, 145-181. McLaren, B.M., & Ashley, K.D. (2001). Helping a CBR program know what it knows. In D.W. Aha and I. Watson (Eds.) Proceedings of the Fourth International Conference on Case-Based Reasoning (ICCBR-01) (pp. 377-391). Berlin: Springer-Verlag. McLaren, B.M., Scheuer, O., De Laat, M., Hever, R., De Groot, R., & Rosé, C.P. (2007). Using machine learning techniques to analyze and support mediation of student e-discussions. In R. Luckin, K.R. Koedinger, & J. Greer J. (Eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007) (pp. 141-147). Amsterdam: IOS Press. McLaren, B.M., Wegerif, R., Mikátko, J., Scheuer, O., Chamrada, M., & Mansour, N. (2009). Are your students working creatively together? Automatically recognizing creative turns in student e-discussions. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.) Proceedings of the 14th International Conference on Artificial Intelligence in Education (AIED-09), (pp. 317-324). Amsterdam : IOS Press. Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). YALE: Rapid prototyping for complex data mining tasks. In M. Craven & D. Gunopulos (Eds.) Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2006) (pp. 935-940). New York: ACM Press. Mikátko, J. (2007). Using machine learning techniques to analyze and recognize complex patterns of student e- discussions (M.Sc. Thesis). Charles University, Prague. Mikátko, J., & McLaren, B.M. (2008). What’s in a cluster? Automatically detecting interesting interactions in student e-discussions. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 333-342). Berlin: Springer. O’Donnell, A.M. (1999). Structuring dyadic interaction through scripted cooperation. In A. M. O’Donnell & A. King (Eds.) Cognitive Perspectives on Peer Learning (pp. 179-196). Mahwah, NJ: Erlbaum. Pinkwart, N. (2003). A plug-in architecture for graph based collaborative modelling systems. In H.U. Hoppe, F. Verdejo & J. Kay (Eds.) Proceedings of the 11th International Conference on Artificial Intelligence in Education (AIED 2003) (pp. 535-536). Amsterdam: IOS Press. Pinkwart, N., Ashley, K.D., Lynch, C., & Aleven, V. (2008). Graph grammars: An ITS technology for diagram representations. In H. Chad Lane, & D. Wilson (Eds.) Proceedings of the 21st International FLAIRS conference (pp. 433-438). Menlo Park, CA: AAAI Press. Ravi, S., & Kim, J. (2007). Profiling student interactions in threaded discussions with speech act classifiers. In R. Luckin, K.R. Koedinger, & J. Greer (Eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007) (pp. 357-364). Amsterdam: IOS Press. p. 44

Rosé, C., Wang, Y.-C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in CSCL. International Journal of Computer-Supported Collaborative Learning, 3(3), 237-271. Salmon, G. (2004). E-moderating: The Key to Teaching and Learning Online (2nd Ed.). London: Routledge Falmer. Salomon, G., & Globerson, T. (1989). When teams do not function the way they ought to. International Journal of Educational Research, 13, 89-100. Schapire, R.E., & Singer, Y. (2000). BoosTexter: A boosting-based system for text categorization. Machine Learning, 39, 135-168. Scheuer, O., Loll, F., Pinkwart, N., & McLaren, B.M. (2010). Computer-supported argumentation: A review of the state of the art. International Journal of Computer-Supported Collaborative Learning. 5(1), 43-102. Scheuer, O., & McLaren, B.M. (2008). Helping teachers handle the flood of data in online student discussions. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds) Proceedings of the 9th International Conference on Intelligent Tutoring Systems (ITS-08) (pp. 323-332). Berlin: Springer. Schwarz, B.B., & Asterhan, C.S.C. (in press). E-moderation of synchronous discussions in educational settings: A nascent practice. To appear in The Journal of the Learning Sciences. Schwarz, B.B., & De Groot, R. (2007). Argumentation in a changing world. International Journal of Computer- Supported Collaborative Learning, 2, 297-313. Schwarz, B.B., & Glassner, A. (2007). The role of floor control and of ontology in argumentative activities with discussion-based tools. International Journal of Computer-Supported Collaborative Learning, 2(4), 449– 478. Sebastiani, F. (2002). Machine learning in automated text categorization. ACM Computing Surveys, 34, 1-47. Soller, A. (2004). Computational modeling and analysis of knowledge sharing in collaborative distance learning. User Modeling and User-Adapted Interaction: The Journal of Personalization Research, 14, 351-381. Srikant, R., & Agrawal, R. (1996). Mining sequential patterns: Generalizations and performance improvements. In P.M.G. Apers, M. Bouzeghoub, & G. Gardarin (Eds.): Advances in Database Technology - EDBT'96, Proceedings of the Fifth International Conference on Extending Database Technology, Lecture Notes in Computer Science 1057 (pp. 3-17). Berlin: Springer. Suthers, D.D., Connelly, J., Lesgold, A., Paolucci, M., Toth, E.E., Toth, J., & Weiner, A. (2001). Representational and advisory guidance for students learning scientific inquiry. In K. D. Forbus, & P. J. Feltovich (Eds.) Smart Machines in Education: The Coming Revolution in Educational Technology (pp. 7- 35). Menlo Park, CA: AAAI/MIT Press. Tsai, W.H., & Fu, K.S. (1979). Error-correcting isomorphisms of attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, 9, 757-768. Vizcaíno, A. (2005). A simulated student can improve collaborative learning. International Journal of Artificial Intelligence in Education, 15, 3-40. Walker, E., McLaren, B.M., Rummel, N., & Koedinger, K.R. (2007). Who says three’s a crowd? Using a Cognitive Tutor to support peer tutoring. In R. Luckin, K.R. Koedinger, & J. Greer (Eds.) Proceedings of the 13th International Conference on Artificial Intelligence in Education (AIED 2007) (pp. 399-406). Amsterdam: IOS Press. Wegerif, R. (2006). A dialogic understanding of the relationship between CSCL and teaching thinking skills. International Journal of Computer Supported Collaborative Learning, 1(1), 143-157. Wegerif, R. (2007). Dialogic, Education and Technology: Expanding the Space of Learning. New York, NY: Kluwer-Springer. Weinberger, A., Ertl, B., Fischer, F., & Mandl, H. (2005). Epistemic and social scripts in computer-supported collaborative learning. Instructional Science, 33(1), 1-30. Weiss, G.M. (2004). Mining with rarity: A unifying framework. SIGKDD Explorations, 6, 7-19. Wichmann, A., Giemza, A., Krauß, M., & Hoppe, H.U. (2009). Effects of awareness support on moderating multiple parallel e-discussions. In C. O’Malley, D. Suthers D., P. Reimann, & A. Dimitracopoulou (Eds.) Computer Supported Collaborative Learning Practices: Proceedings of the 9th International Conference p. 45

on Computer Supported Collaborative Learning(pp. 646-650). International Society of the Learning Sciences, Inc. Witten, I.H., & Frank, E. (2005). Data Mining: Practical Machine Learning Tools and Techniques (2nd Ed.). San Francisco: Morgan Kaufmann. Wong, A.K.C., You, M., & Chan, S.C. (1990). An algorithm for graph optimal monomorphism. IEEE Transactions on Systems, Man and Cybernetics, 20, 628-638. Zloof, M.M. (1977). Query-by-example: A data base language. IBM Systems Journal, 16, 324-343. p. 46