Test Writing: Test Writing: Moving Away from Moving Away from Publisher MaterialPublisher Material

Bruce Butterfras, MS Ed., L.P.Bruce Butterfras, MS Ed., L.P.Assistant Professor – Dept. of Emergency Health Assistant Professor – Dept. of Emergency Health

SciencesSciences

University of Texas Health Science Center at San University of Texas Health Science Center at San AntonioAntonio

ObjectivesObjectives

Explain the importance of an Explain the importance of an EMS curriculum unique to your EMS curriculum unique to your programprogram

Identify the steps involved in Identify the steps involved in creating good examscreating good exams

Create and evaluate exams for Create and evaluate exams for your coursesyour courses

WhyWhy Customize My Customize My Course?Course?

EMT/Paramedic National Standard EMT/Paramedic National Standard CurriculumCurriculum From 1971 to 2000 From 1971 to 2000 Contained specific knowledge and skills Contained specific knowledge and skills

objectivesobjectives EMS Education Agenda for the Future EMS Education Agenda for the Future

(2000)(2000) National EMS Education StandardsNational EMS Education Standards ““The EMS NSC, with their detailed declarative The EMS NSC, with their detailed declarative

material, limit instructor flexibility and the material, limit instructor flexibility and the ability to adapt to local needs and resources”ability to adapt to local needs and resources”

Publisher ResourcesPublisher Resources

Most EMS textbooks come with instructor Most EMS textbooks come with instructor resourcesresources

These often include chapter objectives, These often include chapter objectives, lesson plans, PowerPoint presentations lesson plans, PowerPoint presentations and test banks to be used to construct and test banks to be used to construct examsexams

When selecting a textbook these additional When selecting a textbook these additional resources should be a strong considerationresources should be a strong consideration

Disclaimer: I have helped create Publisher material including PowerPoint presentations and test questions

Publisher ObjectivesPublisher Objectives

Publishers create objectives based on Publishers create objectives based on the National EMS Education Standardsthe National EMS Education Standards

The National EMS Education Standards The National EMS Education Standards are intentionally broad, so specific are intentionally broad, so specific objectives vary among the publishersobjectives vary among the publishers

Publishers’ objectives are by necessity Publishers’ objectives are by necessity generic in order to be applicable to all generic in order to be applicable to all areas of the countryareas of the country

Often specific to their textbook as wellOften specific to their textbook as well

Publisher Lesson PlansPublisher Lesson Plans

Publisher provided Lesson Plans are Publisher provided Lesson Plans are also relatively generic in order to also relatively generic in order to work for programs around the work for programs around the countrycountry

These lesson plans usually focus on These lesson plans usually focus on presentation of material (that is presentation of material (that is found in the textbook) to students in found in the textbook) to students in a standard (ie. lecture oriented) a standard (ie. lecture oriented) educational modeleducational model

Publisher PresentationsPublisher Presentations

Publisher provided PowerPoint Publisher provided PowerPoint presentations highlight the material presentations highlight the material presented in the textbookpresented in the textbook

This This cancan allow students to follow along in allow students to follow along in the textbook as the lecture progressesthe textbook as the lecture progresses

Publisher provided PowerPoint Publisher provided PowerPoint presentations can also be:presentations can also be: ReorganizedReorganized to fit your way of teaching the to fit your way of teaching the

subjectsubject Added to Added to with additional materials from your with additional materials from your

own research on the topicown research on the topic

Modifying PresentationsModifying Presentations

Publisher provided Publisher provided PowerPoint PowerPoint presentations can presentations can also be:also be: Deleted from Deleted from if you if you

feel there is material feel there is material included that doesn’t included that doesn’t need to be discussedneed to be discussed

Spiced up Spiced up to make to make them more appealing them more appealing to studentsto students

Publisher Test BanksPublisher Test Banks

Publisher provided test banks serve Publisher provided test banks serve primarily to determine if your primarily to determine if your students remember what they read students remember what they read in the textbookin the textbook

The questions almost always refer to The questions almost always refer to a particular page in the textbooka particular page in the textbook

You need to test your students on You need to test your students on what you taught them and not just what you taught them and not just what is in the bookwhat is in the book

So What Do I Do?So What Do I Do?

To make your class more effective, you need To make your class more effective, you need to modify publisher materials or create your to modify publisher materials or create your ownown

Go through the objectives and modify them Go through the objectives and modify them to meet your needs. Add, delete, or edit to meet your needs. Add, delete, or edit objectives to fit your particular courseobjectives to fit your particular course

Create your own presentations or edit those Create your own presentations or edit those provided to make them fit your courseprovided to make them fit your course

Use other teaching techniques besides Use other teaching techniques besides lecture!lecture!

Can I write a good exam Can I write a good exam myself?myself?

Finally create your own exams with Finally create your own exams with questions you create to test what your questions you create to test what your students have learnedstudents have learned

Start by asking what type of exam will this Start by asking what type of exam will this be be A short quiz to see if the students read their A short quiz to see if the students read their

bookbook A longer exam covering several chapters worth A longer exam covering several chapters worth

of materialof material A final exam testing mastery of the entire A final exam testing mastery of the entire

coursecourse

Where do I start?Where do I start?

Start by making a Blueprint for the Start by making a Blueprint for the examexam Identify what information will be testedIdentify what information will be tested Select an exam length appropriate for Select an exam length appropriate for

the information and the purpose of the the information and the purpose of the examexam

Determine how many questions will be Determine how many questions will be included from each sectionincluded from each section

Am I testing what I Am I testing what I should?should?

Exams should test if the students Exams should test if the students can meet the course objectivescan meet the course objectives

So each test question should be tied So each test question should be tied to an individual objectiveto an individual objective

Think about the objective as you Think about the objective as you write each question and make sure write each question and make sure the question you write actually tests the question you write actually tests the objectivethe objective

What question format is What question format is best?best?

There are many types of test There are many types of test questionsquestions

You need to choose the one (or ones) You need to choose the one (or ones) that best fit your situation (and/or that best fit your situation (and/or your students)your students)

EssayEssay Questions Questions What are the advantages and What are the advantages and

disadvantages of Essay questions?disadvantages of Essay questions?

What about other types What about other types of questions?of questions?

Short Answer / Fill in the BlanksShort Answer / Fill in the Blanks Name one advantage of a short answer Name one advantage of a short answer

question.question. Fill in the blank exams are good for testing Fill in the blank exams are good for testing

______ of presented materials.______ of presented materials. True / False True / False questionsquestions

Is this statement true or false:Is this statement true or false: True / false questions are a favorite of students True / false questions are a favorite of students

because they are always clearly written and because they are always clearly written and give students a 50/50 chance of getting the give students a 50/50 chance of getting the answer right.answer right.

Can I use Matching Can I use Matching questions?questions?

MatchingMatching Questions Questions

1. Matching questions 1. Matching questions make this easy:make this easy:2. Matching questions are 2. Matching questions are poor at:poor at:3. Matching questions are 3. Matching questions are good for:good for:4. Matching questions are 4. Matching questions are difficult in this respect:difficult in this respect:

a. Creatinga. Creatingb. Evaluating b. Evaluating understandingunderstandingc. Gradingc. Gradingd. Testing d. Testing recallrecall

What about Multiple What about Multiple Choice?Choice?

Multiple Choice Multiple Choice QuestionsQuestions How many of you primarily or exclusively use How many of you primarily or exclusively use

multiple choice exams? – Why?multiple choice exams? – Why? Multiple choice questions are:Multiple choice questions are:

A. The best choice for testing recall of informationA. The best choice for testing recall of information B. Easy to write as a test of advanced B. Easy to write as a test of advanced

understandingunderstanding C. Capable of evaluating higher levels of thinkingC. Capable of evaluating higher levels of thinking D. A poor choice for evaluating general knowledgeD. A poor choice for evaluating general knowledge

Writing Multiple Choice Writing Multiple Choice QuestionsQuestions

Since this is probably the most Since this is probably the most popular type of question used for popular type of question used for EMS exams, let’s look at how to EMS exams, let’s look at how to construct a good MC questionconstruct a good MC question

Multiple choice questions have three Multiple choice questions have three partsparts A ‘Stem’A ‘Stem’ A ‘Correct answer’A ‘Correct answer’ Several ‘Distractors’ (incorrect answers)Several ‘Distractors’ (incorrect answers)

Multiple Choice Multiple Choice QuestionsQuestions

Start with a Stem tied to one of the Start with a Stem tied to one of the objectives being testedobjectives being tested

Try to create a stem that requires more Try to create a stem that requires more than just recall (memorization of facts)than just recall (memorization of facts)

Scenarios work well for these types of Scenarios work well for these types of questionsquestions

Include enough information to choose Include enough information to choose the correct answer and possibly the correct answer and possibly additional irrelevant information additional irrelevant information

Multiple Choice Multiple Choice QuestionsQuestions

Write the Correct AnswerWrite the Correct Answer Tie it to the objective and make sure that Tie it to the objective and make sure that

there is only one correct answerthere is only one correct answer Avoid ‘All of the Above’ and ‘None of the Avoid ‘All of the Above’ and ‘None of the

Above’Above’ You can use ‘Both A and B’ type answers You can use ‘Both A and B’ type answers

if you have enough choices (usually more if you have enough choices (usually more than 4)than 4)

Consider writing a ‘feedback’ statement Consider writing a ‘feedback’ statement to explain why this answer is correctto explain why this answer is correct

Multiple Choice Multiple Choice QuestionsQuestions

Now write the DistractorsNow write the Distractors Make each distractor plausibleMake each distractor plausible Try to use answers that might be given Try to use answers that might be given

by students who do not have a good by students who do not have a good grasp of the materialgrasp of the material

Make sure there is something in the Make sure there is something in the distractor that makes it incorrectdistractor that makes it incorrect

Consider writing a ‘feedback statement’ Consider writing a ‘feedback statement’ telling why this answer is not correcttelling why this answer is not correct

Multiple Choice Multiple Choice QuestionsQuestions

Distractors (Distractors (continuedcontinued)) Make sure both distractors and correct answer Make sure both distractors and correct answer

sound correct (read each question out loud sound correct (read each question out loud with each answer option)with each answer option)

Make all answer choices almost the same Make all answer choices almost the same lengthlength

Shortest or longest answer choices often stand outShortest or longest answer choices often stand out If necessary reword the stem to make the If necessary reword the stem to make the

answers fitanswers fit Avoid ‘NOT’, ‘EXCEPT’, etc. in stems if Avoid ‘NOT’, ‘EXCEPT’, etc. in stems if

possiblepossible

Multiple Choice Multiple Choice QuestionsQuestions

Now consider Now consider alphabetizingalphabetizing the answer the answer choiceschoices Prevents the tendency to write 2 distractors, Prevents the tendency to write 2 distractors,

the right answer and then another distractor the right answer and then another distractor so the correct answer is most likely “C”so the correct answer is most likely “C”

Include enough questions to test the Include enough questions to test the objectives covered in the materialobjectives covered in the material

Somewhere within at least one exam Somewhere within at least one exam there should be a question for each there should be a question for each course objectivecourse objective

Building the ExamBuilding the Exam

Use a variety of easy and difficult questions Use a variety of easy and difficult questions Some recall, some higher level thinkingSome recall, some higher level thinking

This will allow you to identify students at This will allow you to identify students at different levels of understandingdifferent levels of understanding

It will also allow you to anticipate the It will also allow you to anticipate the gradesgrades

Aim for a mean grade of 85% and half Aim for a mean grade of 85% and half should be above and half belowshould be above and half below If they score below a 70% they need If they score below a 70% they need

remediationremediation

Testing the ExamTesting the Exam

When you finish your exam, test it outWhen you finish your exam, test it out Give the exam to your fellow program Give the exam to your fellow program

instructors instructors See what grades they make & listen to their See what grades they make & listen to their

feedbackfeedback Don’t expect their grades to be perfect – they Don’t expect their grades to be perfect – they

didn’t sit through your classdidn’t sit through your class Consider also giving the exam to a few Consider also giving the exam to a few

recent graduates and listen to their recent graduates and listen to their feedbackfeedback

Giving the ExamGiving the Exam

Finally you should be ready to give the examFinally you should be ready to give the exam Consider giving computerized exams if possibleConsider giving computerized exams if possible

Allow about one minute per questionAllow about one minute per question Listen to the student’s feedback afterwardsListen to the student’s feedback afterwards

If there is another way to interpret your If there is another way to interpret your question, they will find it and point it out to question, they will find it and point it out to you!you!

Make adjustments to the questions based on Make adjustments to the questions based on student feedbackstudent feedback

Keep statistics on each questionKeep statistics on each question

Evaluating the ExamEvaluating the Exam

Each time before you teach the class Each time before you teach the class review your exams and make adjustmentsreview your exams and make adjustments

Keep your own ‘test bank’ of questions in Keep your own ‘test bank’ of questions in case you need to create a ‘make-up’ examcase you need to create a ‘make-up’ exam

It will also allow you to use different It will also allow you to use different exams each time you teach the classexams each time you teach the class

Always be open to suggestions for Always be open to suggestions for improvementimprovement

ConclusionConclusion

Publisher provided materials are a great Publisher provided materials are a great starting point, but don’t rely on them starting point, but don’t rely on them exclusively for teaching your classexclusively for teaching your class

Multiple choice questions can make great Multiple choice questions can make great exams so practice making them exams so practice making them whenever you canwhenever you can

Constantly reevaluate your courses and Constantly reevaluate your courses and strive to make them better. The life your strive to make them better. The life your student may eventually save may be student may eventually save may be yours!yours!

Selected ReferencesSelected References National EMS Education Agenda for the future: National EMS Education Agenda for the future: http://

www.nhtsa.gov/people/injury/ems/edagenda/final/agenda6-00.htm

Cason, D. McKenna K. Cason, D. McKenna K. Foundations of Education: An EMS Foundations of Education: An EMS Approach 2Approach 2ndnd edition edition Clifton Park, New York; Delmar Clifton Park, New York; Delmar Cengage Learning (2013) PrintCengage Learning (2013) Print

My Contact Information:My Contact Information: Butterfras@uthscsa.edu Office phone - 210-567-8719Office phone - 210-567-8719