70 1541-1672/07/$25.00 © 2007 IEEE IEEE INTELLIGENT SYSTEMSPublished by the IEEE Computer Society

I n t e l l i g e n t E d u c a t i o n a l S y s t e m s

COMET: A CollaborativeTutoring System for Medical Problem-Based LearningSiriwan Suebnukarn, Thammasat University Dental School

Peter Haddawy, Asian Institute of Technology

C linical reasoning is the cognitive process by which physicians synthesize clinical case

information, integrate it with their knowledge and experience, and diagnose or man-

age the patient’s problem.1 Problem-based learning is a method often used to characterize

medical problems as a context for students to acquire knowledge and clinical-reasoning

skills. PBL instructional models vary, but the generalapproach involves student-centered, small-group, col-laborative problem-solving activities. The main argu-ments for using collaborative problem solving in med-ical PBL include the wider range of ideas generatedand the higher quality of discussion that ensues. In ad-dition, students obtain training in consultation skillsand group clinical problem solving, which are impor-tant for a successful clinical medicine practice.

However, effectively implementing PBL in the clin-ical curriculum is difficult. First, no standards existfor PBL tutoring,2 and properly trained tutors are lack-ing. In addition, effective PBL requires the tutor togive students a high degree of personal attention. Inthe current academic environment, where resourcesare becoming scarcer and costs must be reduced, pro-viding such attention isn’t necessarily feasible. Con-sequently, medical students often don’t get as muchfacilitated PBL training as they might need or want.

Intelligent technologies can enrich medical train-ing by providing environments that maximize learn-ing while minimizing patient risks. Most research onintelligent medical-training systems has focused onparticular domains—for example, pathology to trainstudents in feature perception and disease classifi-

cation.3 Little or no research has addressed a generaldomain-independent framework for intelligent med-ical tutoring, and none has addressed intelligent med-ical tutoring in group settings.

We’ve developed a collaborative intelligent tutor-ing system for medical PBL called COMET (Collab-orative Medical Tutor). COMET uses Bayesian net-works to model the knowledge and activity ofindividual students as well as small groups. It appliesgeneric tutoring algorithms to these models and gen-erates tutorial hints that guide problem solving. Anearly laboratory study shows a high degree of agree-ment between the hints generated by COMET andthose of experienced human tutors.4 Evaluations ofCOMET’s clinical-reasoning model and the group rea-soning path provide encouraging support for the gen-eral framework.5

Medical problem-based learningPBL is designed to challenge learners to build up

their knowledge and develop effective clinical-reasoning skills around practical patient problems.1

It’s typically carried out in three phases:

• Problem analysis: In group discussion, students

COMET models

both individual

and group knowledge

and learning activity.

Students using the

system achieved

significantly greater

improvements in

clinical-reasoning

skills than students

in human-tutored

sessions.

evaluate the patient problem presented tothem exactly as they would a real patient,attempting to determine the possibleunderlying anatomical, physiological, orbiochemical dysfunctions and to enumer-ate all possible causal paths (hypothesesand their causal relations) that wouldexplain the progression of the patient’sproblems.

• Self-directed study: Students work indi-vidually outside the tutorial session, ad-dressing any open issues identified in thefirst phase through relevant learning re-sources such as literature, laboratories, andspecialists.

• Synthesis and application of newlyacquired information: Students analyzedata and wrap up the problem collabora-tively after they return from their self-study period.

One main issue in PBL is the tutor’s role.Like a good coach, a tutor needs enough com-mand of whatever the learners are working onto recognize when and where they most needhelp.2 The ideal tutor should be an expert inboth learning content and learning process,which is rare among human tutors. The tutorintervenes as little as possible, giving hints onlywhen the group appears stuck or off track. Inthis way, the tutor avoids making the studentsdependent on him or her for their learning.

A collaborative medical tutorCOMET aims to provide an experience that

captures the tutor’s essential role in facilitat-ing PBL groups. The current version sup-ports the initial PBL problem analysis phase,where the interactions are intensive betweenstudents and the system as well as among thestudents in the group.

We’ve implemented the system as a Javaclient/server combination that works over theInternet or local area networks and supportsany number of simultaneous PBL groups.Students in a COMET PBL group can be in dis-parate locations but share the same environ-ment over the network. There’s no technicallimit on the number of students in a COMET

PBL group, but for practical pedagogical rea-sons, a group typically consists of no morethan eight students.

Figure 1 shows the basic system compo-nents. The four primary components are sim-ilar to any typical intelligent tutoring systemdomain clinical-reasoning model (or domainmodel), student clinical-reasoning model (orstudent model), pedagogic module, and inter-face, which includes the student multimodalinterface and the medical-concept repository.

Our prototype incorporates substantial do-main knowledge about head injuries, stroke,and heart attack. These domains are quite dif-ferent. The knowledge used to reason abouthead injury is primarily anatomical, while

that used to reason about strokes and heartattacks is primarily physiological. Further-more, the pathophysiology of the latter twodiseases is more dynamic. The system imple-mentation is modular, and the tutoring algo-rithms are generic. So, adding a new scenariorequires only adding the appropriate modelrepresenting how to solve a particular case,such as a peptic-ulcer domain model. Thestudent model is a probabilistic overlay ofthe domain model. COMET constructs it atruntime by instantiating the nodes that defineeach student’s knowledge and activity.

Multimodal interfaceLive human-tutored medical PBL ses-

sions are facilitated by the use of a white-board or similar device during problemanalysis and hypothesis development. Par-ticipants can use the whiteboard to edit andexpand ideas. The students are encouragedto express all their ideas in this process andto challenge each other’s ideas as they pro-ceed under the tutor’s guidance. The stu-dents use the knowledge and skills theyalready have, aided by handy referencessuch as a medical dictionary and a fewappropriate textbooks on anatomy, physiol-ogy, and the like. COMET emulates this PBLenvironment by incorporating a multimodalinterface (see figure 2), which integrates textand graphics in a rich communication chan-

JULY/AUGUST 2007 www.computer.org/intelligent 71

Medical-concept repository

Pathophysiologyterms and images

Anatomical termsand images

Web server

Java client-server applicationModel-View-Controller design

Domain clinical-reasoning model

Individual student modelGroupreasoningalgorithms

Individual student modelTutoring strategies

Tutoring module

Tutoringimages

Tutoringdialogues

Bayesian network clinical-reasoning modeling

InternetIntranet

Outp

ut

Inpu

t

Student interface

Figure 1. The COMET system overview.

nel between the students and the system aswell as among students in the group.

The hypothesis board (figure 2, lower pane)provides the central shared group workspace.Students can create hypothesis nodes andcausal links between them. Hypothesis nodelabels are available in the medical-conceptrepository. The system has access to all infor-mation entered on the board. Students cancommunicate with others in the group by typ-ing into the chat pane (figure 2, middle leftpane). COMET has no language processing ca-pabilities, so it can’t use the chat pane text assystem input. However, the discussion pane(figure 2, upper left pane) supports collabora-tive interaction by displaying tutoring hintsand student chat dialogues. In the image pane(figure 2, upper right pane), COMET displaysimages relevant to the group discussion’s cur-rent focus. All students see the same imageand anything that other students sketch orpoint to on it.

Along with facilitating system input,COMET’s medical-concept repository helpsstudents better understand the relationshipsbetween domain concepts. The repositorystores all valid and invalid hypotheses rele-vant to all problem scenarios, so the systemdoesn’t need to process typed text input.

Domain and student clinical-reasoning model

Generating appropriate tutorial actions re-

quires a model of the students’understandingof the problem domain and of the problemsolution. However, as in human-tutored PBLsessions, COMET must provide an unrestrictedinteraction style that gives students the free-dom to solve the patient case without havingto explain the rationale behind their actions.This complicates the modeling task becausethe system must infer the student’s level ofunderstanding from a limited number of ob-servations. To deal with the resulting uncer-tainty, we selected Bayesian networks as ourmodeling technique. To construct the model,we used information from various sources. Weobtained the generic network’s structure fromresearch articles and textbooks on clinical rea-soning and PBL. For each problem scenario,we consulted medical textbooks and expertPBL tutors (details on the network’s structureare available elsewhere5).

The structure contains two types ofinformation:

• the problem solution’s hypotheses andcausal links (see figure 3, right half) and

• how students derive the hypotheses (seefigure 3, left half).

We represent the hypothesis structure fol-lowing Paul Feltovich and Howard Barrows’model,6 which defines three illness featurecategories: enabling conditions, faults, andconsequences. The right half of figure 3

shows five possible faults associated with thesingle consequence, chest pain: myocardialinfarction, angina, musculoskeletal injury,gastrointestinal disorder, and stress. Athero-sclerosis is the enabling condition of My-ocardial infarction and Angina. The remain-ing hypothesis nodes are consequences ofmyocardial infarction. Each hypothesis nodehas parent nodes, which have a direct causalimpact on it. For example, right heart failurehas parents pulmonary congestion andmyocardial infarction. All hypothesis nodeshave two states, indicating whether the stu-dent knows that the hypothesis is valid forthe case. For every hypothesis that directlycauses another hypothesis (for example, ath-erosclerosis and myocardial infarction), anode (for example, Link_14) represents thecausal link between them. The two hypoth-esis nodes (atherosclerosis, myocardial in-farction) are that link node’s parents. The in-tuition is that the student can’t create the linkunless someone has first created both hy-potheses. Each link node has two states, indi-cating whether the student creates a causallink between two hypotheses.

The derivation of hypotheses (figure 3, lefthalf) is represented in terms of three kinds ofnodes: goals, general medical knowledge, andapply actions. Every hypothesis node has aunique Apply node as one of its parents. TheApply node represents a medical concept ap-plied to a goal to derive the hypothesis. Partof the modeling process is to come up with adiscrete medical concept that underlies a uni-que hypothesis. For example, the Apply_13node indicates that the student can use knowl-edge of the vessel lumina occlusion medicalconcept to infer that myocardial infarction isa consequence of atherosclerosis. Each hypo-thesis node thus has a conditional-probabilitytable specifying the probability of the hypoth-esis being known conditioned on whether theparent hypotheses are known and whether thestudent can apply the appropriate piece ofknowledge to determine the cause-effect rela-tionship. The conditional-probability tablesfor the Apply nodes are simple AND gates.

We obtained each resulting network’s con-ditional-probability tables by learning fromPBL session transcript data. The data for thisstudy consisted of tutorial session tape record-ings and photographs for head injury, stroke,and heart attack scenarios at Thammasat Uni-versity Medical School. The study included15 groups of third-year medical students.Each group consisted of eight students withdifferent backgrounds. We presented each

I n t e l l i g e n t E d u c a t i o n a l S y s t e m s

72 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Figure 2. The COMET student interface includes windows for discussion, chat, images,and hypotheses development.

group with the head injury, stroke, and heartattack cases and asked them to construct pos-sible case hypotheses, under a tutor’s guid-ance. After the sessions, we analyzed the tapeand the whiteboard results to determinewhether they mentioned each goal, concept,hypothesis, and link. We used the expecta-tion-maximum learning algorithm providedby the Hugin Researcher software to learneach node’s conditional probabilities.7

Individual and collaborative student modeling

We instantiate each student’s domain clin-ical-reasoning model by entering that stu-dent’s medical background knowledge as ev-idence. For example, if a student has abackground in thoracic anatomy, we wouldinstantiate the thoracic organ node. All stu-dents have basic knowledge in anatomy, phy-siology, and pathology before they encounterthe PBL tutorial sessions, so we assume thatas soon as one student in a group creates ahypothesis in the domain model, every stu-dent knows that hypothesis. As hypothesesare created during a session, COMET instan-tiates them in each student model. The dif-ferences in background knowledge, repre-sented in the individual student models,result in differences in the likelihoods of yet-to-be-created hypotheses.

Because the students work in a group, thesystem must identify a causal path linking en-abling conditions, faults, and consequencesthat it can use to focus group discussion, par-ticularly when the discussion seems to bediverging in different directions. Intuitively,

we would like to identify a path that has muchof the whole group’s attention and at least onemember’s focused attention. COMET uses analgorithm that identifies what we call thegroup path by determining the probabilities ofthe various candidate paths, given thehypotheses the students have created thus far.5

Generating tutorial hintsOur automated tutor guides the tutorial

group to construct possible case hypothesesby asking specific open-ended questions. Itaims to give hints when the group appears tobe stuck, off track, collaborating poorly, orproducing erroneous hypotheses. To do this,the tutor requires knowledge of both theproblem domain and the problem-solvingprocess. From our study of the tutoring-ses-sion transcripts, we identified and imple-mented seven hint strategies that humantutors commonly use:

1. Focus group discussion. Group membersmight suggest various valid hypotheseswithout focusing on any given causalpath. When such lack of focus becomesapparent, COMET intervenes by directingthe students to focus on one hypothesesin the group path.

2. Promote open discussion. If a studentproposes a hypothesis that’s not on thecurrent group reasoning path, COMET pro-vides positive feedback by encouragingthe student to relate the hypothesis to thecurrent discussion’s focus.

3. Deflect uneducated guessing. If a stu-dent creates an incorrect causal link,

COMET points this out and encouragesthe student to correct the error.

4. Avoid jumping critical steps. If a stu-dent creates a link that jumps directlyfrom one hypothesis to a downstreamconsequence, leaving out intermediatehypotheses, COMET asks the student forthe more direct consequences.

5. Address incomplete information. Afterthe students have finished elaboratingall hypotheses on the group path,COMET identifies another path for themto work on.

6. Refer to experts in the group. If the stu-dents still don’t respond correctly afterCOMET provides a general and then amore specific hint, the system determinesthe student most likely to know theanswer and refers directly to him or her.

7. Promote collaborative discussion. Ifone student dominates the discussion,COMET asks for input from other stu-dents. If a student doesn’t contributeafter a certain number of hypotheseshave been mentioned, COMET solicitsinput from that student.

We used the interaction log and both thestructure and the probabilities of the Bayesian-network models as input to the algorithms wedeveloped to generated each hint type. Allstrategies except strategies 3 and 7 use boththe structure and the probabilities of theBayesian network models. Strategy 3 usesonly the model’s structure, while strategy 7uses only a count of the number of inputsfrom each student. Strategies 1, 2, and 5 use

JULY/AUGUST 2007 www.computer.org/intelligent 73

12.2118.79

01

Cause of chest pain

14.9185.09

01

Chief complaint

Goal

21.5778.43

01

Reduced perfusion

Concept

3.1598.85

01

Vessel lumina occlus

15.9484.06

01

Thoracic organ

32.2967.71

Apply 12

Apply

14.6685.34

Apply 13

27.6172.39

Apply 14

01

01

01

0.00100.00

Atherosclerosis

Hypothesis

0.00100.00

Myocardial infarction

01

01

0.00100.00

Chest pain01

0.00100.00

Link 1401

17.1882.82

Indigestion01

39.7260.28

Gastrointestinal dis.01

28.0271.98

Stress01

Angina01

35.2364.77

01

Cardiac output decr.01

47.7552.25

Link 1701

52.4147.59

Link 1501

28.0971.91

Link 13

Link

01

16.7883.22

57.8742.13

Right heart failure

Left heart failure

Pulmonary congestion01

01

01

Musculoskeleto injury41.2558.75

19.1290.88

62.2437.76

01

01

Figure 3. Part of the Bayesian network clinical-reasoning model of a heart attack scenario. The complete network contains 194nodes. The model contains five types of nodes: (a) goal, concept, apply, (b) hypothesis, and link.

the group path discussed earlier. Strategies1–5 have general and specific versions.COMET first gives a general hint using theparent goal node of the hypothesis it hasdetermined the students should focus on. Ifthe students either don’t respond or respondincorrectly, the system uses the more spe-cific parent medical-concept node. If the stu-dents still can’t come up with the correcthypothesis, COMET refers directly to the stu-dent in the group most likely to know theanswer. If this doesn’t work, the systemidentifies this topic as a learning objectivefor study outside the session (tutoring strat-egy details appear elsewhere4).

Figure 4 presents a transcript showing theinteraction with the system after the studentsread the heart attack problem scenario. Thesystem selects hint strategies and content onthe basis of student inputs to the hypothesisboard.

EvaluationCOMET has been tested empirically in both

the laboratory and the field. In an early lab-oratory study, we compared the tutoring hintsgenerated by COMET with those of experi-enced human tutors.4 On average, 74.17 per-cent of the human tutors used the same hintas COMET. The most similar human tutoragreed with COMET 83 percent of the time,and the least similar tutor agreed 62 percentof the time. Our results show that COMET’shints agree with the hints of the majority ofhuman tutors with a high degree of statisticalagreement (kappa index = 0.773).

We also compared the focus of group activ-ity chosen by COMET with that chosen by

human tutors. COMET agrees with the major-ity of human tutors with a high degree of sta-tistical agreement (kappa index = 0.823).

We tested the modeling approach’s valid-ity with student models in the areas of headinjury, stroke, and heart attack. Receiver-operating-characteristic curve analysis showsthat the models are highly accurate in pre-dicting individual student actions.5

In this field study, we demonstrate COMET’seffectiveness in imparting clinical-reasoningskills and medical knowledge to students bycomparing student clinical-reasoning gainsobtained using COMET versus those obtainedfrom human-tutored PBL sessions.

Study designTo evaluate the system’s overall impact on

student learning, we designed a study to testthe hypothesis that a COMET tutorial will resultin student clinical-reasoning gains similar tothose obtained from a session with an expe-rienced human PBL tutor. We recruited 36second-year medical students from Tham-masat University Medical School—that is,they didn’t yet have PBL experience in strokeand heart attack. We applied stratified randomsampling to divide the students into sixgroups according to their background knowl-edge. We compared three student groupstutored by COMET with three other groups tu-tored by experienced human tutors (see figure5). Initial training in the use of COMET re-quired about 15 minutes. Three medical PBLtutors participated in the study. Rather thanproviding answers or explanations, PBLtutors intervene actively and opportunisti-cally, posing open-ended questions and giv-

ing hints only when the group appears to begetting stuck or off track. The tutors had atleast five years’experience in conducting theStroke and Heart Attack course at Thammasat.

The study had a pretest/posttest controlgroup design. After the pretest session, stu-dents participated in a two-hour problem-solving session. We asked each group to enu-merate a network of hypotheses and causallinks collaboratively explaining the stroke andheart-attack scenarios. The human-tutoredgroups created the network on a classroomwhiteboard. We counted all valid hypothe-ses and links created in each group. A strokeand heart attack expert verified the validity ofhypotheses and links. We assessed all stu-dents on their clinical reasoning before andafter the PBL tutorial session to determineeach student’s reasoning gains. Then wecompared the clinical-reasoning gains be-tween the two groups.

We used the Clinical-Reasoning Problemapproach to assess clinical reasoning.8 EachCRP consisted of a clinical scenario that a spe-cialist physician vetted for clinical accuracyand realism. The pretest used two heart attackcases and two stroke cases to measure each stu-dent’s initial ability to solve the problems. Twodifferent posttest cases in each area measuredtheir ability to generalize the clinical reason-ing acquired from the PBL tutorial session tonew related cases. We asked participants toname the two diagnoses they considered mostlikely. We also asked them to list the case fea-tures they regarded as most important in for-mulating their diagnoses, to indicate whetherthese features were positively or negativelypredictive, and to give each one a weight. Fig-ure 6 presents an example CRP student result.

To establish reference scores, we asked 10volunteer general practitioners to completeboth sets of CRPs. We selected GPs becausethey have experience with a broad range ofundifferentiated clinical presentations thatencompass all medical practice areas. In thisrespect, they provide the most appropriatestandard against which to compare medicalstudents who graduate with a sound back-ground in general medicine but without anyspecialist knowledge.

ResultsThe GP results showed no statistically sig-

nificant differences between pretest andposttest scores, indicating that the tests wereof approximately equal difficulty (see table 1).The GP scores varied from 88.20 to 91.50,indicating that the questions weren’t trivial.

I n t e l l i g e n t E d u c a t i o n a l S y s t e m s

74 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Figure 4. A transcript showing students’ interaction with COMET after they read theheart attack problem scenario.

Students: Gastrointestinal disorder,Smoking, Hypertension, Angina, Myocar-dial infarction, Chest pain (Students in thegroup gradually create six hypotheses onthe board, while analyzing the problem.)

Tutor: What is the consequence of Myo-cardial infarction? (Strategy 1: COMET

focuses group discussion by identifyingwhich causal path to focus on, findingthe highest-probability nonmentionedhypothesis along the path (left heart fail-ure) and providing a hint using its parentgoal node (consequence of myocardialinfarction).)

Student: Cardiac output decreased(Cardiac output decreased is a node

along the group reasoning path but notthe node that COMET wants the group tofocus on.)Tutor: Think about decrease in myocar-dial contractility. (Strategy 1: COMET givesthe next hint using the medical-conceptparent node of left heart failure.)Student: Right heart failure (Right heartfailure is a node along the group reason-ing path but not the node that COMET

wants the group to focus on.)Tutor: It seems we have a problem. Nida,can you help the group? (Strategy 7:Nida is called on because she has thehighest probability among the studentsof knowing the left-heart-failure node.)

We used Cronbach’s alpha to measure test reli-ability. For the students’ pretest and posttestscores, Cronbach’s alpha was 0.901 and 0.921,respectively. A reliability coefficient of 0.80or higher is commonly considered acceptable.

We used the Wilcoxon test for nonpara-metric data and matched pairs to examinethe differences between pretest and posttestscores on the same group. We used theMann-Whitney test for unmatched data todetect any differences between COMET andhuman-tutored groups. Table 2 shows almostidentical pretest mean scores for the COMET

and human-tutored students, and the Mann-Whitney test confirms no significant differ-ences between COMET and human-tutoredgroups. The posttest mean scores were sig-nificantly higher than the pretest scores forboth COMET and human-tutored students(Wilcoxon, p = 0.000), indicating that sig-nificant learning occurred. But the averageposttest score for the COMET students (65.12)was significantly higher than that for thehuman-tutored students (59.46) (Mann-Whitney, p = 0.011), indicating that studentswere learning more in the COMET sessionsthan in the human-tutored sessions. Compar-ing the number of hypotheses and links cre-ated by each group shows that, on average,the COMET groups created more hypothesesand links (stroke: hypotheses = 45.33, links= 48.33; heart attack: hypotheses = 40.67,links = 44.67) compared to the human-tutoredgroups (stroke: hypotheses = 35, links = 40;heart attack: hypotheses = 33.33, links = 38).

We didn’t expect the clinical-reasoninggains for COMET-tutored students to be higherthan those for human-tutored students. This isparticularly true in light of our earlier study

showing that, on average, 74 percent of hu-man tutors used the same hint strategy andcontent as COMET.4 We believe the explanationlies primarily in the 26 percent disagreement.Human tutors often give up after providing ageneral hint, jumping right to identifying thehypothesis as a learning objective. In contrast,COMET is more relentless in pushing the stu-dents, always following the sequence of gen-eral hint, specific hint, referral to an expert,and finally identifying a learning objective.It’s generally agreed that students should gen-erate as many hypotheses as possible in a PBLsession, leaving only the truly difficult issuesas learning objectives.

DiscussionMost cognitive tutors, such as those devel-

oped by Cristina Conati,Abigail Gertner, andKurt Vanlehn3 and by Rebecca Crowley andOlga Medvedeva,9 produce hints when stu-dents request help and bug messages whenthey err. Every entry a student makes in theproblem-solving interface receives immediatefeedback, whether it’s correct or incorrect—for example, green and red in ANDES.9 Fol-lowing PBL tutorial principles, however, stu-dents generally don’t ask the tutor when theyget stuck, and the tutor doesn’t say whetherthe students’ idea is right or wrong. COMET

therefore has no button for the student to askfor help and doesn’t indicate whether the stu-dent’s entry is correct or incorrect. To recog-nize when and where the group needs helpand to give hints that will help them continuetheir discussion, COMET uses its strategies to

JULY/AUGUST 2007 www.computer.org/intelligent 75

Figure 6. An example Clinical-Reasoning Problem for chest pain.

(b)(a)

Figure 5. (a) Students tutored by COMET and (b) students tutored by experienced human tutors.

focus group discussion (strategy 1) and topromote collaborative discussion (strategy 7);these strategies help the group continue itsdiscussion on a productive track. The strate-gies to avoid jumping the critical steps (strat-egy 4) and deflect uneducated guessing (strat-egy 3) are forms of bug messages.

Most strategies have general and specificversions. In COMET, the general hint uses theparent goal node of the hypothesis that thestudents should focus on; the more specifichint—triggered when there’s no studentresponse or an incorrect response—uses themore specific parent medical-concept node.Generating general and specific hints fromBayesian network nodes is consistent withANDES. In ANDES, students who don’t knowwhat to do after receiving the general hintcan select a follow-up question by clickingon the three buttons:

• “Explain further” gives slightly more spe-cific information about the propositionrepresented by the node;

• “How do I do that?” finds the lowest prob-

ability child node, assuming that it’s thenode the student is most likely to be stuckon; and

• “Why?” displays a canned description ofthe rule used to derive the node.

In COMET, the system instead gives a spe-cific hint based on the medical concept of thenode for the hypothesis the student is mostlikely to know. If the student still can’t comeup with the hypothesis of interest, the systemrefers the student to experts in the group (strat-egy 6). As we described earlier, if this doesn’twork, COMET identifies this topic as a learn-ing objective for study outside the session.

Researchers have shown that studentsinteracting with an intelligent tutoring

system that lacks natural language under-standing (NLU) achieve learning gains half-way between those of students in the usualclassroom setting (lowest) and students inter-acting with a human tutor (highest).10 This dif-

ference is attributed to the conversations be-tween tutors and students. Research on next-generation tutoring systems is therefore ex-ploring the use of natural languages as a keyto bridging the performance gap. BecauseNLU technology isn’t yet powerful enough toreliably monitor student discussion either ver-bally or as free text, we avoided it in our prag-matic approach to building COMET’s studentinterface. Although we haven’t yet formallyevaluated the interface, the learning gain itsstudents showed in comparison to the human-tutored students suggests how we might buildeffective teaching systems that don’t try toslavishly mimic noncomputer human-intelli-gence-based teaching situations.

It would nevertheless be useful to add somelanguage-processing capabilities to the stu-dent-modeling module. The system could theninterpret student interactions in the chat tooland both track and comment on the discussion.The ability to communicate with the systemthrough anatomical sketches would also beuseful. Students commonly make sketcheswhen analyzing a PBL scenario. COMET shouldsupport this form of communication beyondproviding a whiteboard; it should also be ableto parse the sketches so that the tutor can fol-low and comment on the students’ problem-solving process. The model for each problemscenario required about one person-month tobuild, so creating the domain model isn’t triv-ial. It involves significant expert knowledgethat will ultimately require developing anauthoring system that employs medicalresources like the Unified Medical LanguageSystem’s semantic network (www.nlm.nih.gov/pubs/factsheets/umls.html) to help createnew cases.

Finally, COMET currently supports single-session group PBL. However, PBL typicallyoccurs over several days, with students car-rying out individual learning tasks and bring-ing their learned knowledge back to thegroup. Support for this aspect of PBL,including integration with the whole medicalcurriculum, remains to be addressed.

AcknowledgmentsWe thank Hugin Expert for letting us use the

Hugin Researcher software. Thanks to ThammasatUniversity Medical School for their participationin the data collection and system evaluation.

References

1. H. Barrows, “A Taxonomy of Problem-BasedLearning Methods,” Medical Education, vol.20, 1986, pp. 481–486.

I n t e l l i g e n t E d u c a t i o n a l S y s t e m s

76 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Table 1. Mean scores for all Clinical-Reasoning Problems.*

General practitioner’s Student’s score (standard deviation)score (standard COMET Human tutor

CRPs deviation) n = 10 n = 18 n = 18

Pretest 1.1 88.70 (0.78) 33.56 (0.48) 34.56 (0.80)

1.2 91.50 (0.78) 34.00 (0.77) 34.78 (0.64)

1.3 88.20 (0.53) 38.72 (0.52) 38.61 (0.80)

1.4 89.80 (1.10) 39.17 (0.51) 38.22 (0.83)

Posttest 2.1 89.50 (1.07) 62.44 (0.59) 58.11 (0.46)

2.2 87.70 (1.40) 62.94 (0.46) 58.67 (0.56)

2.3 90.60 (0.83) 67.56 (0.47) 60.00 (0.65)

2.4 89.50 (1.04) 67.56 (0.46) 61.05 (0.56)

*CRPs 1.1, 1.2, 2.1, and 2.2 are chest pain cases; CRPs 1.3, 1.4, 2.3, and 2.4 are stroke cases.

Table 2. Mean CRP scores for each cohort.

Mean score (standard deviation)

Cohort Pretest Posttest

COMET (1) 36.38 (0.70) 65.12 (0.69)

COMET (2) 36.58 (0.74) 64.33 (0.57)

COMET (3) 36.12 (0.76) 65.92 (0.69)

COMET (all) 36.36 (0.42) 65.12 (0.38)

Human tutor (1) 36.83 (0.64) 60.96 (0.51)

Human tutor (2) 37.42 (0.89) 58.63 (0.41)

Human tutor (3) 35.38 (0.70) 58.79 (0.55)

Human tutor (all) 36.54 (0.44) 59.46 (0.31)

2. M. Das et al., “Student Perceptions of TutorSkills in Problem-Based Learning Tutorials,”Medical Education, vol. 36, 2002, pp.272–278.

3. R. Crowley and O. Medvedeva, “An Intelli-gent Tutoring System for Visual Classifica-tion Problem Solving,” Artificial Intelligencein Medicine, vol. 36, 2006, pp. 85–117.

4. S. Suebnukarn and P. Haddawy, “A BayesianApproach to Generating Tutorial Hints in aCollaborative Medical Problem-Based Learn-ing System,” Artificial Intelligence in Medi-cine, vol. 38, 2006, pp. 5–24.

5. S. Suebnukarn and P. Haddawy, “ModelingIndividual and Collaborative Problem-Solv-ing in Medical Problem-Based Learning,”User Modeling and User-Adapted Interac-tion, vol. 16, 2006, pp. 211–248.

6. P.J. Feltovich and H.S. Barrows, “Issues ofGenerality in Medical Problem Solving,” Tu-torials in Problem-Based Learning: A NewDirection in Teaching the Health Professions,H.G. Schmidt and M.L. De Volder, eds., VanGorcum, 1984, pp. 128–142.

7. S. Lauritzen, “The EM-Algorithm for Graph-ical Association Models with Missing Data,”Computational Statistics and Data Analysis,vol. 1, 1995, pp. 191–201.

8. M. Groves, I. Schott, and H. Alexander,“Assessing Clinical Reasoning: A Method to

Monitor Its Development in PBL Curricu-lum,” Medical Teacher, vol. 5, 2002, pp.507–515.

9. C. Conati, A. Gertner, and K. VanLehn,“Using Bayesian Networks to Manage Uncer-tainty in Student Modeling,” J. User Model-ing and User-Adapted Interaction, vol. 12,2002, pp. 371–417.

10. J.R. Anderson et al., “Cognitive Tutors: Les-sons Learned,” J. Learning Sciences, vol. 4,1995, pp. 167–207.

For more information on this or any other com-puting topic, please visit our Digital Library atwww.computer.org/publications/dlib.

JULY/AUGUST 2007 www.computer.org/intelligent 77

T h e A u t h o r sSiriwan Suebnukarn is an assistant professor at Thammasat UniversityDental School and an adjunct faculty member of computer science and infor-mation management at the Asian Institute of Technology in Bangkok. Herinterests lie in intelligent medical training systems, medical and dental infor-matics, clinical skill training, and problem-based learning. She received herPhD in computer science and information management from the Asian Insti-tute of Technology and her DDS from the Prince of Songkhla University.Contact her at the Thammasat Univ. Dental School, Pathumthani, 12121,Thailand; ssiriwan@tu.ac.th.

Peter Haddawy is a professor in the Asian Institute of Technology’s Com-puter Science and Information Management program, the vice president forthe institute’s academic affairs, and a senior scientist at the University ofWisconsin-Milwaukee. His research interests include decision-theoreticproblem solving, probabilistic reasoning, preference elicitation, and e-com-merce and medical applications. He received his PhD in computer sciencefrom the University of Illinois at Urbana-Champaign. Contact him at theAsian Inst. of Technology, School of Eng. and Technology, Pathumthani,12120, Thailand; haddawy@ait.ac.th.

intelligent@computer.org

QUESTIONS?COMMENTS?

IEEE IntelligentSystems wants tohear from you!

EMA

IL

QUESTIONS?COMMENTS?

IEEE IntelligentSystems wants tohear from you!