Is the Script-Concordance Test a Valid Is the Script-Concordance Test a Valid Instrument for Assessment of Intra-Instrument for Assessment of Intra-operative Decision-making Skills?operative Decision-making Skills?

Brent Zabolotny1, Robert Gagnon2, Bernard Charlin2 and Sarkis Meterissian1

1Division of General Surgery and Center for Medical Education, McGill University

2Unit for Research and Development in Health Sciences Education, University of Montreal

The Basics of Medical ReasoningThe Basics of Medical Reasoning

Medical reasoning is a hypothetico-deductive process (Ellstein 1978, Barrows et al

1982) involving:

1. Early generation of hypotheses

2. Oriented data collection

3. Decision-making judgements

Problem-solving in Clinical MedicineProblem-solving in Clinical Medicine

Well-defined problems are rare:Clear data, clear goals, clear solutionsTechnical knowledge

Ill-defined problems are common:Uncertain data, unclear goals, different solutions

Professional knowledge

Donald A Schön, The Reflective Practitioner, 1983

What is an Expert?What is an Expert?

Experts can store, use and retrieve knowledge more efficiently than novices

Experts possess elaborated networks of knowledge fitted to the tasks they regularly do

These networks are called SCRIPTS(Feltovich 1983, Schmidt 1990, Charlin 2000)

SCRIPT TheorySCRIPT Theory

In clinical situations, clinicians fill their working memory with knowledge related to each relevant hypothesis

Then a deductive process is used to seek information to either accept or refute each respective hypothesis

A test has been developed that measures this deductive-reasoning process – The Script Concordance Test (SCT)

Script Concordance TestScript Concordance Test A challenging context

– Authentic clinical situation

– Not enough data or conflict

- Relevant options A Likert scale

– Capture opinion Scoring system

– Takes into account variability of experts

– Aggregate scoring method

The Qualities of a Good SCTThe Qualities of a Good SCT There must be at least 50-60 items to achieve a

reliability coefficient (Cronbach alpha) of 80 (r = .80)

Questions should not lead to a single answer (they must not be clearcut) otherwise the SCT would really be an MCQ

Conversely, the answers should not be too widely spread thus indicating a vague question

Evidence that this works?Evidence that this works?

PurposePurpose

To develop and validate a script-concordance test for assessment of clinical reasoning (ie: intra-operative decision-making) in General Surgery Residents.

Methods: SCT FormatMethods: SCT Format

Methods: Example (1)Methods: Example (1) A 50 year old male sun worshipper presents with a 2 cm. dark pigmented lesion on the R calf. Initial punch biopsy revealed a 2.3 mm. Breslow thickness superficial spreading melanoma. In your management:

If you were planning… and found… the planned management is:

1) Only a wide excision with 2 cm margins

A palpable inguinal node in the operating room

-2 -1 0 +1 +2

2) A wide local excision plus only a sentinel node biopsy of the groin nodes

The hot/blue node to be clearly black with tumor

-2 -1 0 +1 +2

3) A wide local excision plus a superficial node dissection for a palpable groin node

Involvement of Cloquet’s node at frozen section

-2 -1 0 +1 +2

Methods:Example (2)Methods:Example (2) A 35 year old female of Chilean descent presents with an upper GI bleed. Endoscopy reveals a gastric cancer and CT scan fails to reveal any distant spread. In your management:

If you were planning… and found… the planned management is:

4) A subtotal gastrectomy A single peritoneal implant biopsy proven adenocarcinoma

-2 -1 0 +1 +2

5) A subtotal gastrectomy Multiple palpable nodes along both the greater/lesser curvature

-2 -1 0 +1 +2

6) A total gastrectomy That a 5 cm. proximal margin with the esophagus was achievable with a subtotal resection

-2 -1 0 +1 +2

Methods: Scoring an SCTMethods: Scoring an SCT

The scoring grid of an SCT should be derived by administering it to 10 experts (aggregate scoring method)

Based on their responses some questions (if too vague or too clearcut) can be eliminated

Scoring grid takes into account variability due to the uncertainty of the clinical situation

The Scoring GridThe Scoring Grid

-2 -1 0 +1 + 2

Number of experts choosing answer 0 0 5 4 1

Score 0 0 5/10 4/10 1/10

Transformed score 0 0 5/5 4/5 1/5

Credit per item 0 0 1 0.8 0.2

MethodsMethods

Participants: 36 out of the 40 McGill surgical residents wrote the exam

Panel of experts: 10 board-certified general surgeons

MethodsMethods

Statistical Analysis:- Reliability: Cronbach alpha coefficient- Analysis of item-to-total correlation used to select

the best items for the final analysis- Construct validity tested with a one-way ANOVA

with post-hoc comparisons test and planned contrasts

- All p values at alpha<5% were considered significant

ResultsResults

Developed 196 questionsFace and content validity assessed by 3

independent Board-certified General Surgeons

Final exam consisted of the best 100 questions

ResultsResults

Initial test: 100 items Cronbach alpha : 0.70

Optimisation of the items (elimination of items with an inter-item correlation less than 10%) : final test used for analysis = 62 items

Cronbach alpha of the optimised test: 0.85

ResultsResultsExamination Scores (62-item test) by Resident Level

Resident Level

n Mean Score (%)

Std Deviation (%)

R1 8 52.5 9.96

R2 6 62.5 5.12

R3 9 68.3 9.19

R4 6 75.7 9.61

R5 7 68.0 6.44

ResultsResults

40.00

45.00

50.00

55.00

60.00

65.00

70.00

75.00

80.00

R1

R2

R3

R4

R5

DiscussionDiscussion

The SCT developed in this study demonstrated:

1. excellent internal reliability

2. Good construct validity

Discussion:Discussion:

Questions:1. Correlation with in-training evaluations?2. Correlation with oral exam scores?3. Ideal number of panel members?4. Makeup of the panel members?

subspecialists vs general surgeons5. Can the SCT identify residents having

difficulty with clinical reasoning?

More on More on

Web site:

http://www.cme.umontreal.ca/tcs

E-mail :

Sarkis.meterissian@muhc.mcgill.ca

bernard.charlin@umontreal.ca

ResultsResults

Evaluation of the exam:

- 64% found the SCT tested their intra-operative decision-making skills

- 50% found the exam initially difficult but 79% found that the exam became easier as they better understood the format

- 71% found that the SCT exam was a valuable and educational examination tool