Made with Protégé: An Intelligent Medical Training SystemOlga Medvedeva, Eugene Tseytlin, and Rebecca Crowley

Center for Pathology Informatics, University of Pittsburgh School of Medicine, Pittsburgh PA

The SlideTutor Project

Pedagogic Knowledge Representation

References

Acknowledgements

http://slidetutor.upmc.edu

“I hear and I forget, I see and I remember, I do and I understand” – Chinese Proverb Learning by doing is an extremely effective instructional method, but training novices in practice environments is not an option in high-risk domains such as Air Traffic Control and Medicine. Intelligent Tutoring Systems provide environments for learning by doing, which have proven to be highly effective, but have not yet been applied in domains that are knowledge-rich.

Q: Can Knowledge-Based Systems (KBS) be used as underlying component for developing such a training environment?

The SlideTutor project is developing a KBS based Intelligent Training System for learning microscopic diagnosis. We are simultaneously examining this question from four perspectives: technical, cognitive, design, and educational. This demonstration and poster highlights some of our progress in each of these dimensions.

Cognitive & developmental model

The design of SlideTutor is based on a developmental cognitive model of visual classification problem solving derived from our own empirical studies of classification problem solving in microscopic diagnosis, as well as numerous other studies of expertise in visual and non-visual domains.

Derived Principles

Architecture

Based on the implications of our cognitive and developmental model, we designed our system, to:

(1) determine both correct and incorrect student actions, determine the general class of error that has been made, and leave open the specific instructional response to this error. (2) reason with the student, in order to provide correct feedback as intermediate solutions are developed. For example, the system should accept hypotheses and even diagnoses based on an early, incomplete or incompletely refined set of evidence. When additional evidence is identified, the system should require that students revise their diagnoses or statements of diagnostic support.

(3) reason both forwards (data to hypothesis) and backwards (hypothesis to data) so that it can support both strategies among students.

Three important considerations led to our need for a highly modular architecture: (1) deep structural similarities between all visual classification tasks, (2) need for a completely separate and therefore flexible pedagogic model, and (3) requirement for parametric testing of learning gains. We chose to use the UPML Component model as the basis for our architecture.

Domain Knowledge Representation

On the expert model side, we used Protégé to implement and slightly extend Motta et al’s representation of classification problem solving. Projects for Case and Domain include a thirds project which contains features, attributes, and values common to both the case and knowledge representation. Classes and instances from these KB populate the Jess-based Dynamic Solution Graph.

Dynamic Solution Graph

To create our virtual slide case projects, we created a Protégé plugin that utilizes the same Domain KB as the Tutor. When a new slide is authored, the author indicates the diagnosis and FEATURE_SPECIFICATION from the KB that represents the current case. The authoring system automatically fills instances of ATTRIBUTE_VALUE_PAIR and OBSERVABLE. When authors draw around particular visual features they choose the correct OBSERVABLE from the list of those associated with the FEATURE_SPECIFICATION, which is saved in combination with the location as a LOCATED_OBSERVABLE. We have created an initial set of 30 cases in Subepidermal Vesicular Dermatitides.

A Protégé Plugin for Authoring

The Dynamic Solution Graph (DSG) is a directed acyclic graph that models the current problem state and all valid-next-steps. It exists only in Jess Working Memory. The DSG generates the initial problem state at the beginning of each problem, using knowledge derived from the Domain, Task and Case models. After each student action, the graph structure is updated by a set of abstract problem solving methods that may add or delete nodes and arcs, or change the node state. Any individual state of the DSG represents only the current problem state and all valid next-steps. But taken together, sequential DSG states define one path through the problem space – the path taken by the student. The graph representation supports the ability to reason in both directions (design principle 3) and the dynamic nature of the graph enables the system to reason with the student (design principle 2).

Instructional Layer

Tutor Interfaces

Like our domain knowledge, pedagogic knowledge is modeled in Protégé and then loaded into Jess, using a modified version of JessTab. Examples of bug scenarios for Hypothesis Evaluation are shown below:

The instructional layer is composed of Jess Rules instantiated with pedagogic knowledge, which implement the strategies derived from our cognitive and developmental model.

We have developed two student interfaces that can be used by students and we are currently testing their effect on learning gains. The case-focused (CF) tutor uses simple graphical relationships between features and hypotheses. The knowledge-focused (KF) tutor presents an interactive algorithmic interface which is designed to co-construct knowledge with students.

We are still in the midst of our first formative evaluation of the CF and KF interfaces. Early results from the CF tutor are very promising, demonstrating that students gain and retain on test which assess individual skills, and general diagnostic skills.

We gratefully acknowledge contributions made to this project by: Elizabeth Legowski, Girish Chavan, Katsura Fujits, Maria Bond; students in the Interactive Design Course Winter, 2004; and the developers of Protégé, Jess, JessTab, SpaceTree, and JGraph.

ITS Flexibility/Constraint Pedagogic Design Requirement (from Table 1) Implementation of Instructional Layer interaction with DSGLocation of features must be indicated. Help s tudents connect visual features to symbolic feature

names (1.4)Student indicated location must match location derived from Case and s tored in DSG evidence node for feature

Features can be added in any order, but hints provide optimum order specified in diagnostic algorithm .

Allows more novice s tudents to fully explore feature space (1.6) but encourages more advanced s tudents to search for features most efficiently (1.8)

Any valid-next-s tep is allowed but best-nest-s tep follows most efficient sequence.

Attributes can be added in any order, but hints suggest refining features immediately after it is asserted.

Encourage feature refinement at the time it is asserted to help more novice s tudents learn the salient refinements (1.9)

Any valid-next-s tep is allowed but best-next-s tep after feature identification is feature refinement.

Hints take s tudents through all features before suggesting hypotheses . But only one supporting feature must be identified before a hypothesis can be asserted and even accepted as diagnosis . If diagnosis is made incorrect by addition of features or refinem

Encourage complete feature articulation among novices (1.5). Permit hypotheses consis tent with any one feature (1.12) in order to allow students to explore relationships to other hypotheses (1.17). Allow students to jump to diagnosis as long as region of

Best-next-s teps complete feature identification before suggesting hypotheses . Hypotheses can only be added to DSG when preceeded by one or more supporting features . Evidence cluster must support relationship between all previously asserted features and di

Features do not need to be refined before hypothesis can be asserted or accepted as the diagnosis .

Allow more advanced s tudents to reason without fully refined features (1.10). Permit sequences in which new hypotheses require re-examination of feature refinement (1.11)

Attribute Value nodes are not required in default pedagogogic model.

All diseases with feature specifications matching the case data must be asserted as hypothesis and diagnoses

Encourage hypotheses that are consis tent with all of features (1.14), and help s tudents learn sets of hypotheses that share s imilar features (1.15)

Problem is not complete until all Diseases matching the FEATURE SPECIFICATION are asserted.

E1 Feature indicated as supporting hypothesis does not support that hypothesis because feature does not matchE2 Feature indicated as refuting hypothesis does not support that hypothesis because feature does not matchE3 Feature indicated as supporting hypothesis does not support that hypothesis because one or more attribute value pairs do not matchE4 Feature indicated as refuting hypothesis does not support that hypothesis because one or more attribute value pairs do not matchE5 Feature previously indicative of supporting hypothesis now does not support hypothesis because attribute value pairs have been addedE6 Feature previously indicative of refuting hypothesis now does not refute hypothesis because attribute value pairs have been addedE7 Hypothesis previously supported by one feature is no longer supported by any feature because attribute value pairs have been addedE8 Diagnosis does not fit with feature(s) found so far E9 Diagnosis fits with some features that have been identified but not other features that have been identifiedE10 Diagnosis now inconsis tent with identified feature because new feature added E11 Diagnosis now inconsis tent with identified feature because new attribute value pairs of feature added

Hypothesis Evaluation

Learning gains

0

10

20

30

40

50

60

70

80

90

100

Percentage

Multiple Choice Score 56 77 78

Pre Test Post Test Retention Test0

10

20

30

40

50

60

70

80

90

100

Percentage

Case Diagnosis Score 14.1 12.5 56.9 8.3 29.8 48.1

Tutored UnTutored Tutored UnTutored Seen Unseen

Pre Test Post Test Retention Test