Blueprinting and drafting examination questions, Liz Norman, ANZCVS Exam Writing Workshop, November...
-
Upload
liz-norman -
Category
Education
-
view
2.058 -
download
1
description
Transcript of Blueprinting and drafting examination questions, Liz Norman, ANZCVS Exam Writing Workshop, November...
Blueprinting and drafting
examination questions
Liz Norman
Massey University
Australian and New Zealand College of Veterinary Scientists,
Exam Writing Workshop, November 2013
Validity
Validity
The exam must elicit proper evidence of the
abilities of the candidate and we must evaluate the
evidence properly
Validity
• Not a property of a test, but relates to the
interpretations or actions made on the basis of
test scores
• Eg:
“The award of Fellowship of the College implies that the
awardee has sufficient knowledge and experience in a
particular area of veterinary science to entitle him/her to be
acknowledged as a specialist or consultant in that area"
Chain of inferences
1. Fellows have sufficient knowledge, skills, attitudes and judgement to be considered specialists
2. The things we can measure in examinations are things that are necessary to be a Fellow
3. A particular examination (all 4 components) is representative of all that we could measure in an examination
4. The examination score is a measure of achievement in the examination
5. The passing score of 70% correctly separates someone with sufficient knowledge, skills, attitude and judgement to be a Fellow from someone without
Specialist practice domain
All the
knowledge, skills, a
ttitudes and
judgements you
need to be a
specialist
Fellowship domain
All the
knowledge, skills, att
itudes and
judgements you
need to be a fellow
Content domain
Everything we could
possibly examine
Credentials domain
Everything that is
examined in the
credentials
document
Examination domains
What is examined in
a particular
examination in a
particular year
Why we need to blueprint
• Its important that we sample representatively
from the content domain (all that it is possible to
examine)
• This is so we can extrapolate performance to the
whole content domain
Content of questions
Content
• Scope of topics
• Other scopes eg species mix
Subject guidelines
• Represent the entire content domain (all that is
examinable).
• Specify both scope (breadth) and level (depth)
of the knowledge, skills, attitudes and
judgements required.
• For Fellowship some aspects are assessed in
the credentials documents.
Content
• Scope of topics
• Other scopes eg species mix
The candidate will have a detailed knowledge of:
The aetiology, pathogenesis and pathophysiology of
cardiac, renal, respiratory, alimentary, musculoskeletal,
endocrine, ophthalmological and neurological organ
dysfunction in the cat and the dog.
Level of questions
Level - depth
• Surface – deep
• Critical thinking
• Fact recall – applied
• Blooms taxonomy
• SOLO taxonomy
Critical thinkers
• seek reasons.
• attempt to be well-informed
• acknowledge credible sources
• consider alternative points of view
• withhold judgement until they have sufficient
evidence
• seek to be as precise as possible
Fact recall vs applied
Fact recall:Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)
Applied (higher order)Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact
From: Peitzman et al. (1990). Academic Medicine, 65(9), S59-60.
create
evaluate
analyse
apply
understand
rememberRecall
Application
Blooms taxonomy
Blooms instructional verbs
• Create: compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare.
• Evaluate: judge, appraise, evaluate, rate, compare, revise, assess, estimate
• Analyse: distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise.
• Apply: interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch.
• Understand: translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell
• Remember: define, repeat, record, list, recall, name, relate, underline.
SOLO taxonomy
http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/
Prestructural Question may be rephrased as the answer; almost completely
misses the point of the question.
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural Able to list as well as describe distinct aspects of a response (such
as being able to describe aetiology, clinical features, management
of thrombotic stroke) but unable to explicitly explain causes for
observations; unable to present cause-effect relationships.
Relational Able to describe multiple aspects of a process and additionally
explain or elaborate observations into cause-effect relationships;
able to compare similarities and differences between apparently
distinct phenomena. This level is taken as suggesting that the
learner has understood.
Extended
abstract
Highly developed; able to explain mechanisms of phenomena and
apply this information to a novel context — able to develop novel
hypotheses, theories, and deduce principles; creative thinking.
Prakash et al. (2010) Adv Physiol Educ, 34(3):145-149
Prestructural
Unistructural Paraphrase, define, identify, count, name, recite, follow simple
instructions, calculate, reproduce, arrange, recognise, find, note,
seek, sketch, pick
Multistructural Combine, classify, structure, describe, enumerate, list, do
algorithm, apply method, account for execute, formulate, solve,
conduct, prove, complete, illustrate, express, characterise
Relational Analyse, compare, contrast, integrate, relate, explain causes,
apply theory (to its own domain), argue, implement, plan,
summarize, construct, design, interpret (some senses), structure,
conclude, substantiate, exemplify, derive, adapt
Extended
abstract
Theorise, generalise, hypothesise, predict, judge, transfer theory
(to new domain), assess, evaluate, interpret (some senses),
critically reflect, predict, criticise, reason
Potter & Kustra (2012). http://www1.uwindsor.ca/ctl/system/files/PRIMER-on-Learning-Outcomes.pdf
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
surface
deep
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
recall
application
Prestructural
Unistructural Able to identify, list, name, enumerate but does not describe,
explain, relate or elaborate multiple aspects of a response
Multistructural
Relational
Extended
abstract
Quantitative change
Qualitative change
Blueprinting
Blueprinting
• Mapping to scope and level
Scope - breadth
PathophysiologyInvestigation and
diagnosis
Treatment and
management
Gastrointestinal P1Q1 P1Q1, P2Q4
Cardiovascular P1Q4 P2Q2 P2Q2
Nervous P1Q3, P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
Breadth across species
species number of Qs percent
small animal 15 52%
farm 5 17%
horse 4 14%
exotic 3 10%
lab 1 3%
all 1 3%
Level - depth
PathophysiologyInvestigation and
diagnosis
Treatment and
management
recallhigher
orderrecall
higher
orderrecall
higher
order
Gastrointestinal P1Q1P1Q1, P2
Q4
Cardiovascular P1Q4 P2Q2 P2Q2
NervousP1Q2,
P2Q1
Endocrine P1Q3 P2Q3
Musculoskeletal P2Q5
Question difficulty
Should questions to be difficult?
• Norm referenced vs standards based grading
Question difficulty
• Type of content
eg:
– Name the two most significant muscles involved for
mastication.
– Explain the physiological actions of insulin
– Explain the physiological actions of ghrelin
Question difficulty
• Type of content
• Level of task
– Complexity
– Abstractedness
Question difficulty
• Type of content
• Level of task
– Complexity
– Abstractedness
• Structure
Structure
1. Explain the aetiopathogenesis of hepatic lipidosis in cats. (12 marks)
2. The following drugs are commonly used in the management of hepatic lipidosis in cats. For each drug describe its mechanism of action AND any adverse effects or precautions for use.
a) Ursodeoxycholic acid. (3 marks)
b) S-adenosyl methionine (3 marks)
3. Outline the principles of nutritional management and fluid therapy for a cat with idiopathic hepatic lipidosis(12 marks)
Question difficulty
• Type of content
• Level of task
– Complexity
– Abstractedness
• Structure
• Resources
Question difficulty
• Type of content
• Level of task
– Complexity
– Abstractedness
• Structure
• Resources
• Wording
Written, oral, practical
Written Paper 1:
…designed to test the candidate’s knowledge of the principles of the subject as described in the learning outcomes….. Answers may cite specific examples where general principles apply, but should primarily address the theoretical basis underlying each example.
Written Paper 2:
…designed to (a) test the candidate’s ability to apply the principles of the subject to particular cases/problems or tasks, and to (b) test the candidates’ familiarity with current practices and current issues that arise from activities within the discipline…
Written, oral, practical
Practical examination
…designed to test practical aspects of the learning outcomes. …written answers …pertaining to case-based material presented. Candidates must demonstrate deep understanding and practical application of equipment used in the subject… Patient images or videos, radiographs, ultrasound images, CT studies, MR images, ECG recordings, EMG recordings, NCV recordings, cytology images as well as copies of pathology, radiography or other ancillary test reports are likely to be used during this examination.
Oral Examination:
…may cover case-based material or more general discussion, including areas of recent interest or controversy in the subject. …designed to test practical aspects of the learning outcomes. …be able to demonstrate the required level of knowledge …and to support their opinions with citations of the veterinary literature….. Images and pathology reports are likely to be used….
What’s different about oral Qs
• Thinking on your feet
• Can’t revisit or edit
• Tests verbal communication skills as well as
knowledge
• Interactive – can prompt and redirect, can ask
candidate to explain their answer
Question length
Question length
• More shorter questions?
Better content coverage but if not careful could have
less higher order Qs
Easier to mark