3/18/2013

1

Susan Sanders, DNP, RN, NEA-BC

Vice President, Kaplan Nursing

Test Construction &

Item-Writing

Objectives

1. Examine a test blueprint construction

2. Describe best practices for writing multiple choice

questions

3. Compare lower and higher level questions

4. Analyze test questions and edit to create higher level

questions

Exams

• Should require that students think

• Questions should be a mix of low and high order

• Short-answer essay questions test the ability of

students to analyze information and draw

conclusions.

• Take-home essay question written specifically to

enhance critical thinking. Questions that go beyond

the lecture material forcing the use of lecture notes

and book to answer a complicated question.

Education Instruction

• Planning

• Teaching

• Assessment

Brookhart, S.M . & Nitko, A.J. (2008)

3/18/2013

2

Covey’s Time Management grid

Urgent Not Urgent

Important Urgent & Important Not Urgent but

Important

Not Important Urgent but Not

Important

Not Urgent & Not

Important

Effective Assessment – 5 principles

• Identify the learning targets (obj/competencies)

• Match assessment technique to learning targets

• Meet the students’ needs

• Use multiple assessment techniques

• Remember assessment limitations when interpreting results

Brookhart & Nitko (2008)

2 Methods for Measurement

• Norm Referenced – compares student’s scores with

others

• Criterion Referenced – compares student’s scores

with preset criterion

Tests

• Use to assess student’s knowledge and skills prior to

instruction

• Determine student’s grades

• Selecting students for admission

• Curriculum and program evaluation (accreditation)

3/18/2013

3

Underlying Concepts

PURPOSE OF A TEST

Provide information about an individual's achievement

of a course objective or mastery of an area of nursing

content

Evaluation – 2 major roles

• Formative

Obj and competencies met?

Is further learning needed?

• Summative

End of course

Quality of students’ achievement

Test Plan

• What – defined by objectives and course content

• How – directed by the test plan or blueprint

• Schedule

• Length of test

Test Plan Grid

1st Term 2ndTerm 3rd Term 4th Term

Knowledge

Questions 25% 20% 10% 10%

Comprehension

Questions 50% 40% 15% 10%

Application and

Analysis

Questions

25% 40% 75% 80%

3/18/2013

4

TEST ITEM BLUE-PRINT NET COURSE: _______ NURSING PROCESS (NP) CLIENT NEEDS 1. Safe, Effective Care Environment TEST/ UNIT: _________ A. Coordinated Care B. Safety & Infection Control DATE: ______________ 2. Health Promotion & Maintenance 3. Psychosocial Integrity 4. Physiological Integrity COGNITIVE LEVEL A. Basic Care & Comfort R- Remember B. Pharmacological Therapies U-Understand C. Reduction of Risk Potential Apply D. Physiological Adaptation Analyze

Q# NP COGNITIVE LEVEL CLIENT NEEDS

R U Apply Analyze 1A 1B 2 3 4A 4B 4C 4D

1

2

3

4

TOTAL

Item Writing

• An effective item:

• Discriminates students who understand

the content from those who do not.

• Focuses on important information.

Step 1 - Select Objective

• Select a test objective:

• Based on specific behavioral objectives of the course.

• Specify one task, skill, or concept that is

behaviorally specific, result oriented, and

within the student's area of knowledge or

responsibility.

Level of Question Difficulty

Passing Level

Questions

(line of minimum competency)

3/18/2013

5

Remembering:

Can the student recall or remember the

information?

define, duplicate, list, memorize, recall,

repeat, reproduce state

Understanding:

Can the student explain ideas or

concepts?

classify, describe, discuss, explain, identify,

locate, recognize, report, select, translate,

paraphrase

Applying:

Can the student use the information in a

new way?

choose, demonstrate, dramatize, employ,

illustrate, interpret, operate, schedule,

sketch, solve, use, write.

Analyzing:

Can the student distinguish between the

different parts?

appraise, compare, contrast, criticize,

differentiate, discriminate, distinguish,

examine, experiment, question, test.

http://www.odu.edu/educ/roverbau/Bloom/blooms_taxonomy.htm

Step 2 – Write the Stem

Write the question stem based on the behavioral objective.

Step 2 – Write the Stem

Bad -Joe, aged 25, is admitted to the hospital with a

ruptured appendix. His temperature is 101°F and WBC

is 15,000. During his post-op recovery phase Joe

receives TPN and complains of thirst and frequent

urination. This complaint most likely indicates:

Step 2 – Write the Stem

Good - The nurse cares for a patient receiving TPN.

The patient reports increased thirst and frequent

urination. These symptoms MOST likely indicate the

following complication.

3/18/2013

6

Question Stem

Should Should not:

Contain a subject and a verb Contain irrelevant information

CAPITALIZE, italicize, or underline

qualifier word to support the critical

thinking process

“Teach" information

Always be a patient and a nurse

Step 3 – Write the Answers

• Write and validate the correct answer

“I should remain on complete bedrest.”

“I should eat a low-protein diet.”

• Create distracters from errors or misconceptions.

“My abdomen is large because I eat a lot, there’s nothing wrong

with my liver.”

Answer Choices

Should Should not:

Eliminate unnecessary words. Contain key words or phrases related

to information in the stem.

Enable students to choose the correct

answer because they know the

information tested, not because of the

way answer choices are written.

Vary in length, vocabulary, or syntax.

Be mutually exclusive. One answer

choice should not be included in

another.

Distracters

• Can be diagnostic and provide information about the

student's lack of knowledge.

• Can be a predictable error, a common

misconception, a clinical error, an incomplete but

related idea, or a common procedural mistake.

3/18/2013

7

Distracters (cont)

Should Should not:

Be realistic and plausible. Be close in meaning to the correct

response.

Be the same length and grammatically

similar.

Be direct opposites of distracters or

the correct answer.

Be consistent with the correct answer. Be absolute: such as always or never.

Be homogenous.

Be inviting and well written, but not tricky!

• Step 1 – Select Objective

• Step 2 – Write the Stem

• Step 3 – Write the Answers

To Review:

Question Construction Concepts

Level of Question Difficulty

Passing Level

Questions

(line of minimum competency)

Knowledge (Remembering) Level

• Requires recall or recognition of information

previously learned.

• Question requires no understanding or judgment.

• Good for testing factual information.

• Limited value for testing nursing competence.

3/18/2013

8

Knowledge Level (cont)

The nurse cares for a patient during the first 24 hours after

a percutaneous liver biopsy. Which of the following is a

complication of that procedure?

1. Nausea, and vomiting.

2. Constipation.

3. Pain at the biopsy site.

4. Hemorrhage.

Comprehension (Understanding) Level

• Evaluates a student’s understanding of the

information.

Comprehension Level (cont)

The nurse cares for a patient after a percutaneous liver biopsy. The nurse understands that hemorrhage is a complication of this procedure for which of the following reasons?

1. There are several large blood vessels near the liver.

2. The liver cells are bathed with a mixture of venous and arterial blood.

3. The test is performed on patients with elevated enzymes.

4. The procedure requires a large piece of tissue to be removed.

Application Level

• Requires students to apply their knowledge in

a situation.

• Tests ability to use learned concepts in specific

situations.

• This is where you are asking the student to

make some sort of judgment.

3/18/2013

9

Application Level (cont)

The nurse observes care for the client post liver biopsy and

should intervene if which of the following is observed?

1. The patient is lying on their right side.

2. The patient complains of pain at the biopsy site.

3. The patient is ambulating 1 hour post procedure.

4. The patient is fed a regular diet.

Analysis Level

• Requires students to break down information to

obtain the meaning of the material or demonstrate

how it is organized or structured.

• Requires students to analyze the situation, interpret

data, set goals, establish nursing diagnoses, set

priorities, and determine actions.

Analysis Level (cont)

Which of the following symptoms, if observed by the nurse

24 hours after a patient’s percutaneous liver biopsy,

indicate a complication from the procedure?

1. Anorexia, nausea, and vomiting.

2. Abdominal distention and discomfort.

3. Constant pain at the biopsy site.

4. Pulse 112, BP 100/60, respirations 24.

Creating Higher Level Questions

• In the stem of the question, give clinical assessment

data rather than the diagnosis. This requires students to

analyze the data to answer the question.

• Example:

Describe symptoms of pulmonary edema rather than

state "client with pulmonary edema..."

3/18/2013

10

Creating Higher Level Questions (cont)

• Write questions that include integrated concepts.

• Examples:

Diabetic woman in postpartum situation

Pediatric patient in surgery

Elderly patient receiving medication

General Rules

• Avoid bias by gender.

• Use a description of the patient without a proper name.

• Write questions at appropriate grade reading level. (PN

textbooks 10th – 11th grade level; RN textbooks 11th -

13th: varies by book).

• Write as a team or review each other’s questions.

• Write in the active voice and put a patient and nurse in

stem.

General Rules, cont.

• Don’t try to write the test in one day.

• Write one or two questions after each class.

• Randomly distribute the correct response among

the alternatives throughout the test.

• Allow 1 minute to respond to each item.

• Utilize a test plan.

Question Formats

• Multiple-choice questions with 4 possible answer choices

• Alternate question types

o Multiple response

o Ordered response

o Hot spot

o Calculation

o Chart exhibit

o Fill in the blank

3/18/2013

11

Level of Question Difficulty

Passing Level

Questions

(line of minimum competency)

The LPN/LVN cares for a patient diagnosed with heart failure

(HF). Which assessment is consistent with left-sided heart

failure?

1. Dependent edema.

2. Shortness of breath.

3. Anorexia.

4. Jugular vein distension.

Help Suspend End Exam ? Next

Calculator

The LPN/LVN cares for a patient diagnosed with left-sided heart

failure. The nurse understands the cause of pulmonary

congestion is

1. The right ventricle does not completely empty.

2. The left ventricle does not fill with an adequate

amount of blood.

3. The heart rate increases due to stimulation of the

sympathetic nervous system.

4. The heart enlarges.

Help Suspend End Exam ? Next

Calculator

The LPN/LVN cares for clients in the long-term care facility. The

LPN/LVN notes the client gained 7 pounds in one week. Which

response by the LPN/LVN is BEST?

1. Contact the RN.

2. Determine if the client is eating more than

three meals a day.

3. Contact the dietician.

4. Encourage the client to balance rest and

activity.

Help Suspend End Exam ? Next

Calculator

3/18/2013

12

References

• Bastable, S. (2008). Nurse as educator. Jones and Bartlett,

Sudbury, MA.

• Benner, P., Sutphen, M., Leonard, V., & Day, L. (2010).

Educating nurses: A call for radical transformation. Jossey-

Bass, San Francisco, CA.

• Billings, D. & Halstead, J. (2009). Teaching in nursing: A guide

for faculty. Saunders, St. Louis, MO.

• Brookhart, S.M . & Nitko, A.J. (2008). Assessment and grading

in classrooms. Upper Saddle River, NJ: Pearson Education.

• Chance, P. (1986). Thinking in the classroom: A survey of

programs. New York: Teachers College, Columbia University.

• Clifton, S., Schriner, C. (2010). Assessing the quality of multiple-

choice test items. Nurse Educator, 35 (1):12-16

References continued:

• Gronlund, N.E. & Brookhart, S. (2009). Writing instructional

objectives (8th ed.). Upper Saddle River, NJ: Allyn & Bacon

• McDonald, M.E. (2007). The nurse educator’s guide to

assessing learning outcomes. Jones & Bartlett Publ.

• Oermann, M. & Gaberson, K. (2009). Evaluation and testing in

nursing education. Springer Publishing. 3rd Ed.

• Overbaugh, R., Schultz, L. Retrieved June 12, 2010

http://www.odu.edu/educ/roverbau/Bloom/bloomCollins, J.

(2006). Writing multiple choice questions for continuing medical

education activities and self- assessment modules.

RadioGraphics, 26, 543-551. Retrieved June 2, 2010 from:

http://www.arrs.org/uploadedFiles/ARRS/Publications/writingMul

tipleChoiceHandout.pdf

• s_taxonomy.htm

References continued:

• Paul, Richard. (2009). Critical thinking in every domain of knowledge

and belief. The Critical Thinking Community, Foundation for Critical

Thinking.

• Su, W. (2007). Writing context-dependent items that reflect critical

thinking learning outcomes. Nurse Educator, 32 (1):11-15.

• Su, W., Osisek, P., Motgomery, C., Pellar, S. (2009). Designing

multiple choice test items at higher cognitive levels. Nurse Educator,

34 (5):223-227.

• University of North Carolina School of Nursing. Center for Instructional

Technology & Educational Support. Guide to creating multiple choice

tests. Retrieved June 2, 2010 from

http://docs.google.com/viewer?a=v&q=cache:o6f4dmZT8SYJ:nursing.u

nc.edu/cites/coursedev/mctests.pdf+unc+school+of+nursing+multiple+c

hoice+tests&hl=en&gl=us&pid=bl&srcid=ADGEESh1NqijkGcOH0xlxqV

_pFvWngwk9GeoloCFHXjQXz9llnVhFGOX9dChyEEJfElevxGKgRHiZE

DuqRtrJbJ8wZm09bUDfZwZAfUovt1sRVsovSZp