Blueprinting and drafting questions

Liz NormanMassey University

What is a blueprint?

"blueprint, n.". Something which acts as a plan, model, or template

OED Online http://www.oed.comRailway and harbour report, Vancouver, B.C. by City of Vancouver Archives, Attribution License

Why we need to blueprint

Its important that we sample representatively from the content domain (all that it is possible to examine)

This is so we can generalise performance to the whole content domain

Constructing blueprints• Table or grid with at least 2 dimensions• Dimensions could include:

– Learning outcomes– Diseases or presenting problem– Body system– Species– Knowledge domain: anatomy, pathophysiology, diagnosis,

management, prognosis, epidemiology, investigations, ethics & law– Skills: interpretation, reasoning, decision making, evaluation, problem

solving– Cognitive level: factual recall, analysis and interpretation of data,

problem solving– Degree of abstraction: principles (theoretical), applied (concrete)– Location of question in the examination components

Constructing blueprints

• Begin by considering the curriculum• What knowledge, skills, and attitudes are

examinable?• Subject guidelines

Body system by knowledge domain

  Written paper 1

Written paper 2

Oral examination

Practical examination

LO1 X X X

LO2 X X

LO3 X X X

LO4 X

Body system by knowledge domain by exam component

  Pathophysiology Investigation and diagnosis

Treatment and management

Gastrointestinal P1Q1 P1Q1, P2Q4 OQ3

Cardiovascular  P1Q4 P2Q2, OQ1 P2Q2

Nervous P1Q3, P2Q1

Endocrine P1Q3 OQ2 P2Q3

Respiratory P2Q5

Proportion of exam by species

species number of Qs percent

small animal 15 52%

farm 5 17%

horse 4 14%

exotic 3 10%

lab  1 3%

all 1 3%

Body system by knowledge domain by cognitive level

Pathophysiology Investigation and diagnosis Treatment and management

recall analyse, interpret

problem solve recall analyse,

interpretproblem

solve recall analyse, interpret

problem solve

Gastrointestinal P1Q1 P1Q1

P2Q4 OQ3

Cardiovascular P1Q4 P2Q2 OQ1 P2Q2

Nervous P1Q2 P2Q1

Endocrine P1Q3 OQ2 P2Q3

Respiratory P2Q5

Body system by knowledge domain by subject of question

 Written paper 2 Pathophysiology Investigation and diagnosis Treatment and management

Gastrointestinal Investigation of exocrine pancreatic insufficiency

Cardiovascular  Investigation of atrial fibrillation

Management of cardiomyopathy

Nervous Diagnosis of L2-3 spinal neoplasia

Endocrine Management of diabetic ketoacidosis

Respiratory Diagnosis of feline asthma Management of feline asthma

Weighting questions

Impact Weight Frequency Weight

Less important 1 Rarely seen 1

Essential 2 Relatively common 2

High impact 3 Very common 3

McLaughlin, K., Lemaire, J., & Coderre, S. (2005). Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach, 27(6), 544-547

Drafting questions

Creating appropriate tasks

• Questions are actually tasks• We need the tasks we assess to be things that

Members or Fellows need to be able to do.• This ensures we can validly extrapolate from

performance on the exam, to performance in the Membership or Fellowship domain.

Creating appropriate tasks

• Emphasis should be on assessing the candidate’s – understanding– skills in analysing and interpreting information– skills in problem solving– judgement

• Not recall of information

Recall-higher order classification

Fact recall Questions capable of being answered by reference to one paragraph in a text or notes (or several paragraphs for questions requiring recall of several facts)

Applied (higher order)

Questions that require the use of facts or concepts, the solution of a diagnostic or physiologic problem, the perception of a relationship, or other process beyond recalling discrete fact

Peitzman, S. J., Nieman, L. Z., & Gracely, E. J. (1990). Comparison of "fact-recall" with "higher-order" questions in multiple-choice examinations as predictors of clinical performance of medical students. Academic Medicine, 65(9), S59-60.

Knowledge-understanding classification

Knowledge knowing about; a body of coherent facts; can be thought of as right or wrong.

Understanding knowing how and why; the meaning of facts; the theory that links facts and provides meaning; how sense is made of facts to enable them to be applied to analysis, synthesis, evaluation; to be able to explain why particular facts or skills are applicable to a particular situation; to know which fact to apply when; to be able to create new knowledge or modify or adapt an idea to a new situation.

Wiggins, G., & McTighe, J. (2005). Understanding by design (2nd ed.). Alexandria, VA, USA: Association for Supervision and Curriculum Development.

SOLO taxonomy

http://pamhook.com/2012/01/20/creating-solo-taxonomy-symbols-in-many-colours/

Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.

Prestructural Question may be rephrased as the answer; almost completely misses the point of the question.

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural Able to list as well as describe distinct aspects of a response (such as being able to describe aetiology, clinical features, management of thrombotic stroke) but unable to explicitly explain causes for observations; unable to present cause-effect relationships.

Relational Able to describe multiple aspects of a process and additionally explain or elaborate observations into cause-effect relationships; able to compare similarities and differences between apparently distinct phenomena. This level is taken as suggesting that the learner has understood.

Extended abstract

Highly developed; able to explain mechanisms of phenomena and apply this information to a novel context — able to develop novel hypotheses, theories, and deduce principles; creative thinking.

Prakash, et al. (2010). Adv Physiol Educ, 34(3), 145-149.

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended abstract

Quantitative change

Qualitative change

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended abstract

surface

deep

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended abstract

recall

application

Prestructural

Unistructural Able to identify, list, name, enumerate but does not describe, explain, relate or elaborate multiple aspects of a response

Multistructural

Relational

Extended abstract

knowledge

understanding

Prestructural

Unistructural Paraphrase, define, identify, count, name, recite, follow simple instructions, calculate, reproduce, arrange, recognise, find, note, seek, sketch, pick

Multistructural Combine, classify, structure, describe, enumerate, list, do algorithm, apply method, account for execute, formulate, solve, conduct, prove, complete, illustrate, express, characterise

Relational Analyse, compare, contrast, integrate, relate, explain causes, apply theory (to its own domain), argue, implement, plan, summarize, construct, design, interpret (some senses), structure, conclude, substantiate, exemplify, derive, adapt

Extended abstract

Theorise, generalise, hypothesise, predict, judge, transfer theory (to new domain), assess, evaluate, interpret (some senses), critically reflect, predict, criticise, reason

Biggs, J. B., & Tang, C. S.-K. (2011). Teaching for quality learning at university (4th ed.). Maidenhead UK: McGraw-Hill.

Specifying tasks• Examples of instructional verbs

– Compare: to find similarities between things, or to look for characteristics and features that resemble each other.

– Contrast: to find differences or to distinguish between things.– Discuss: to present a detailed argument or account of the subject

matter, including all the main points, essential details, and pros and cons of the problem, to show your complete understanding of the subject.

– Define: to provide a concise explanation of the meaning of a word or phrase; or to describe the essential qualities of something.

– Explain: to clarify, interpret, give reasons for differences of opinions or results, or analyse causes.

– Illustrate: to use a picture, diagram or example to clarify a point.

Specifying scope

• Do you want a general answer about this condition that applies to all possibilities or one that specifically applies to this case?

• Do you want an answer to include – all possible options?– those that are currently available in this region of the

world?– those that are available under cost constraints?

• Writing marking scheme helps you get the scope right!

Specify boundaries of the answer

• Speciese.g. “in both dogs and cats…”

• Quantities and amountse.g. “Provide 5 reasons why…”

• With reference toe.g. “ With reference to the published research from ..”

• Timeeg: “in the first 24 hours”

• Part of the questionEg: “for one of your differentials….”

Construct irrelevance

When the skills, abilities or knowledge required to answer examination questions is not part of the domain we are trying to assess.

What are we actually assessing?

Important sources of construct irrelevance

• Unclear tasks – because the candidate may do something other than what we intended them to do.

• Unanticipated and irrelevant difficulty and demand

• Insufficient time to complete the task (fully)

Controlling task difficulty and demand

• Tasks need to be at an appropriate level of difficulty

• They don’t need to be “tricky” in order to be difficult

Some factors that affect demand and difficulty

• Type of operation– Simple steps vs synthesis, interpretation, evaluation

• Degree of novelty

Question 1:Explain the physiological actions of insulin.

Question 2:Explain the physiological actions of ghrelin.

Some factors that affect demand and difficulty

• Type of operation• Degree of novelty• Number of components or ideas involved

Question 1:Compare and contrast the clinical signs of hypoadrenocorticism with those of hyperadrenocorticism in dogs.

Question 2:Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs.

Question 3:Compare and contrast the clinical signs of diabetes mellitus with those of hyperadrenocorticism in dogs and cats.

Question 4:Compare and contrast the clinical signs of diabetes mellitus with those of hypoadrenocorticism in dogs and cats.

Some factors that affect demand and difficulty

• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be

generated by the candidate

For EACH ECG, make a recommendation for management of the horse and explain your reasoning.

Some factors that affect demand and difficulty

• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be

generated by the candidate• The question wording and any images, diagrams,

or tables provided

Some factors that affect demand and difficulty

• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be

generated by the candidate• The question wording and any images, diagrams,

or tables provided• Degree of abstraction

Discuss the concept of quality of life in terms of biological function, “feelings” and natural existence.

Describe how both classical conditioning and operant conditioning are involved in cows confidently entering the milking shed and letting down.

Outline and discuss a conceptual framework for differentiating between ryegrass varieties available in the New Zealand market place and for defining possible strengths and weaknesses of a particular cultivar. Your conceptual framework should reflect the various options currently employed by plant breeders in developing new cultivars. (10 marks)

Some factors that affect demand and difficulty

• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be

generated by the candidate• The question wording and any images,

diagrams, or tables provided• Degree of abstraction• Response strategy – simple, stepwise, integrated

Describe the skull. (30 marks)

Question 1Discuss the use of insulin for the treatment of diabetes mellitus in cats (25 marks)

Question 2a) Describe the advantages and disadvantages of insulin therapy for

diabetes mellitus in cats (10 marks)b) Indicate the dose and frequency of administration of insulin you

would prescribe to a newly diagnosed cat with diabetes mellitus. (5 marks)

c) Describe the recommendations you would make for the frequency and timing of feeding in relation to insulin dosing in cats with diabetes mellitus (10 marks).

Some factors that affect demand and difficulty

• Type of operation• Degree of novelty• Number of components or ideas involved• Whether resources are provided or need to be

generated by the candidate• The question wording and any images, diagrams, or

tables provided• Degree of abstraction• Response strategy – simple, stepwise, integrated• Guidance provided

You have been contacted by a farmer producing Pacific oysters (Crassostrea gigas) intertidally, in a bay containing a number of oyster farms. The farmer is concerned with the amount of dead shell they are seeing during the current grading. Explain how you would approach this scenario. (20 marks)

Include in your answer how the information you could gather might influence your assessment, what differential diagnoses you consider and detail how you might further investigate potential causes and what advice you would provide.

Pacing examinations

• Reading time: 40 words per minute• Writing time: 20 words per minute

• Marking schemes help you check the pacing