Download - Writing Good Evaluation Questions

Transcript
Page 1: Writing Good Evaluation Questions

Writing Good Evaluation Questions

Page 2: Writing Good Evaluation Questions

Selected Response Constructed ResponseTrue/False Fill-in-the-blankMatching EssayMultiple Choice/Response

Types of Questions

Page 3: Writing Good Evaluation Questions

Part figuring out answer

Part figuring out assessmentLess of this

More of this

Page 4: Writing Good Evaluation Questions

(True or False)The physical relationship between most petroleum products and purely aqueous solutions is generally such that physical interaction and diffusion of the two is severely limited.

(True or False)As a general rule, oil and water do not mix.

Which is easier to process for the student? Why?

Page 5: Writing Good Evaluation Questions

(True or False)The physical relationship between most petroleum products and purely aqueous solutions is generally such that physical interaction and diffusion of the two is severely limited.

(True or False)As a general rule, oil and water do not mix.

(A) keeps the language simple. Unless the purpose of the test is to survey the extent of the students’ vocabulary, there is no point in using words that are unfamiliar to students or in phrasing questions so that they are difficult to understand. Students will be justifiably angry and frustrated if they get answers wrong because they could not understand what was being asked rather than because they did not know the right answer.

Page 6: Writing Good Evaluation Questions

(True or False)The area of a rectangle is found by multiplying the length by the width.

(True or False)A 2’ x 4’ rectangle has an area of 8ft2.

Which asks the student to demonstrate at a higher cognitive level? Why?

Page 7: Writing Good Evaluation Questions

(True or False)The area of a rectangle is found by multiplying the length by the width.

(True or False)A 2’ x 4’ rectangle has an area of 8ft2.

(B) Ask students to apply, rather than simply recall, information.If students can apply the information they learned, it is a safe bet they have committed it to memory. It does not follow, however, that simply because students have memorized information they can also apply it. This being the case, it is better to aim questions at application rather then simple recall.

Page 8: Writing Good Evaluation Questions

5) The number of square feet in a room nine feet long and twelve feet wide is:

A) 3. B) 21. C) 81. D) 108. E) 144.

6) At $2.00 per square foot, what would it cost to carpet the room described in question #5?

A) $6 B) $42 C) $162 D) $216 E) $288

Here are two successive questions on an assessment. What is our opportunity for improvement?

Page 9: Writing Good Evaluation Questions

5) The number of square feet in a room nine feet long and twelve feet wide is:

A) 3. B) 21. C) 81. D) 108. E) 144.

6) At $2.00 per square foot, what would it cost to carpet the room described in question #5?

A) $6 B) $42 C) $162 D) $216 E) $288

Make sure that each item is independent.Check questions to be sure that one question does not provide a clue to some other question or that the answer to one question is not crucial to another.Any student missing #5 probably missed #6. Nothing gained by linking questions… combine.

Page 10: Writing Good Evaluation Questions

1)

2)

3)

4)

5)

6)

7)

8)

A B C D

What is the answer to #8?

Page 11: Writing Good Evaluation Questions

1)

2)

3)

4)

5)

6)

7)

8)

A B C D

(D) Do not establish or follow a pattern for correct responses. Regardless of how clever an answer pattern is, some student will eventually discover it and compromise the test results. (You know someone has done it, or it wouldn’t be a rule!)

Page 12: Writing Good Evaluation Questions

The acronym USDA stands for:

A) United States Dairy Association.B) United States Department of Agriculture.C) United States Department of Administration.D) United States Dietitians’ Association.

What is our opportunity for improvement?

Page 13: Writing Good Evaluation Questions

The acronym USDA stands for:

A) United States Dairy Association.B) United States Department of Agriculture.C) United States Department of Administration.D) United States Dietitians’ Association.

(B) Do not include trick or trivial questions. Sometimes evaluators are tempted to ask questions that require extended effort for correct interpretation that deal with unimportant points. Trick or trivial questions not only reduce the validity and reliability of tests, but they may have a powerful negative effect if they antagonize students.This should be something important and relevant… something the participant should know rather than looking up in a reference document.

Page 14: Writing Good Evaluation Questions

How should you respond to a question from a studentonce the assessment has begun?

Page 15: Writing Good Evaluation Questions

Do not answer questions after the test has started unless it is done publically.Sometimes individual students will seek further clarification of a question as they work through the test. If additional information is provided to that student, the test results may be biased, since the student will have had access to direct help when others did not. It is wise to make a general announcement prior to each test concerning your reluctance to answer questions during a test. This, plus careful proofreading of test prior to their administration, should forestall most questions. IF it does become necessary to answer a question about some test item, assume that other students may be equally confused about it and call everyone’s attention to the clarification. Remember however, that interrupting students during a test destroys their train of thought and thus should be avoided whenever possible.

Page 16: Writing Good Evaluation Questions

12) John Adams was:

A) the second president of the United States.B) the third president of the United States.C) the fourth president of the United States.D) none of the above.

13) The second president of the United States was:

A) John Adams.B) Thomas Jefferson.C) James Madison.D) none of the above.

How are these two questions different?

Page 17: Writing Good Evaluation Questions

12) John Adams was:

A) the second president of the United States.B) the third president of the United States.C) the fourth president of the United States.D) none of the above.

13) The second president of the United States was:

A) John Adams.B) Thomas Jefferson.C) James Madison.D) none of the above.

Stem

Distractors

Answer

If the stem does its job properly, it gives the student an idea of what is sought before reading the options. In #12, the student doesn’t know what is sought until they’ve read each option. Furthermore, more time is consumed because the student must read the repetitive wording – inefficient testing.In Question #13, the stem is clear and concise and gives the student sufficient time to begin thinking about the answer.

Page 18: Writing Good Evaluation Questions

13) The second president of the United States was:

A) John Adams.B) Thomas Jefferson.C) James Madison.D) none of the above.

What could be done to improve this question?

Page 19: Writing Good Evaluation Questions

13) The second president of the United States was:

A) John Adams.B) Thomas Jefferson.C) James Madison.D) none of the above.

Make distractors reasonable and plausible.

Distractors should be equally appealing to someone who is purely guessing. Someone vaguely familiar with the early presidents would recognize the names and eliminate option D, thereby increasing their odds of guessing a right answer from 1 in 4 to 1 in 3.

Page 20: Writing Good Evaluation Questions

The area of a circle with a radius of 3 feet is:

A) 9.42 ft2.B) 18.85 ft2.C) 28.27 ft2.D) 29.6 ft2.

3’

Why are these distractors plausible?

Page 21: Writing Good Evaluation Questions

The area of a circle with a radius of 3 feet is:

A) 9.42 ft2.B) 18.85 ft2.C) 28.27 ft2.D) 29.6 ft2.

3’

One way to construct good distractors is based on the type of errors most students are likely to make. In this case, each distractor is an incorrect calculation based on different errors.

= *3= 2*3= *32

= 2*3

Page 22: Writing Good Evaluation Questions

15) To make an effective presentation, the presenter should:

A) identify the audience’s needs in advance.B) meet everyone in person after the presentation.C) summarizes the main points in a title slide.D) looks at the slide to focus attention on it.

What is our opportunity for improvement?

Page 23: Writing Good Evaluation Questions

15) To make an effective presentation, the presenter should:

A) identify the audience’s needs in advance.B) meet everyone in person after the presentation.C) summarizes the main points in a title slide.D) looks at the slide to focus attention on it.

Make sure that unintentional clues are not provided.

Sometimes a student can eliminate an option simply because it is grammatically incorrect. For example a stem ending with … “An” vs “a”Incorrect tenses or forms of words can provide clues…

Page 24: Writing Good Evaluation Questions

16) “Lake effect snow storms” are most likely to occur when:

A) the temperature drops.B) snow is created when a cold wind blows over the warmer water in the lake causing the evaporating water to crystalize.C) it gets cold.D) it gets warm.E) it has recently rained.

What is our opportunity for improvement?

Page 25: Writing Good Evaluation Questions

16) “Lake effect snow storms” are most likely to occur when:

A) the temperature drops.B) snow is created when a cold wind blows over the warmer water in the lake causing the evaporating water to crystalize.C) it gets cold.D) it gets warm.E) it has recently rained.

Another example of unintentional clues. Distractors that are consistently longer or shorter than the other options.The use of “always” or “never” is usually a big giveaway, as is the user of “all of the above” or “none of the above”

Page 26: Writing Good Evaluation Questions

19) The second president of the United States was:

A) John Adams.B) Warren Harding.C) Thomas Jefferson.D) James Madison.

18) Who was the second president of the United States?

A) John AdamsB) Warren HardingC) Thomas JeffersonD) James Madison

Which punctuation is correct?

Page 27: Writing Good Evaluation Questions

19) The second president of the United States was:

A) John Adams.B) Warren Harding.C) Thomas Jefferson.D) James Madison.

18) Who was the second president of the United States?

A) John AdamsB) Warren HardingC) Thomas JeffersonD) James Madison

If the stem is a complete question, it should end with a question mark. The answer and the distractors would be capitalized, but not have periods unless they were complete sentences.If the stem is part of a statement that will be completed by one of the choices, it should end with a colon. Colons precede lists and the choices constitute a list. If the choices complete the statement, each should begin with a lower case letter and end with a period because the stem plus the choice will make a complete statement.

Page 28: Writing Good Evaluation Questions

19) Who was the second president of the United States?A) John AdamsB) Warren HardingC) Thomas JeffersonD) James Madison

What is our opportunity for improvement?

Page 29: Writing Good Evaluation Questions

19) Who was the second president of the United States?

A) John AdamsB) Warren HardingC) Thomas JeffersonD) James Madison

Format… Skip a line between the stem and the choices to make it easier for students to visually separate the two components. Also keep both the stem and the choices on the same page, if there is not enough room, move the whole question to a new page.

Page 30: Writing Good Evaluation Questions

(True or False)If every teacher were given his or her own computer, most of the difficult problems currently limiting wider use of computer-assisted instruction would be solved.

What is the opportunity for improvement?

Page 31: Writing Good Evaluation Questions

(True or False)If every teacher were given his or her own computer, most of the difficult problems currently limiting wider use of computer-assisted instruction would be solved.

“Most of the difficult problems”? How many is that? 99%? 52%?True False items are most appropriate for lower-level cognitive skills. (Keep in mind it is a 50/50 guess) Although True-False items are often misused, it is possible to construct true-false items that cover the analysis level of Bloom’s Taxonomy.Other points to keep in mind when writing True/False items:•Be sure every item is wholly true or wholly false.•Whenever possible, avoid such terms as ‘generally’ or ‘usually’. (Although not as much as a giveaway as “always” or “never”, they are still open to varying interpretations.)•Be sure that items are not dependent upon insignificant facts. Make sure each item asks something of importance and worth remembering.•Be sure that correct items are not consistently longer or shorter than incorrect items.•Avoid the use of double negatives, but if you use any negative at all, emphasize it.

Page 32: Writing Good Evaluation Questions

Connect the state with its state bird.

___Connecticut A) Robin

___Nevada B) Mountain Bluebird

___Idaho C) Bluebird

___Ohio D) Cardinal

___Oregon E) Western Meadowlark

___Mississippi F) Mockingbird

___Texas G) Chickadee

___Wisconsin H) Lark Bunting

___Minnesota I) Common Loon

___North Dakota J) Brown Thrasher

___South Carolina K) Great Carolina Wren

___Iowa L) Eastern Goldfinch

What is our opportunity for improvement?

Page 33: Writing Good Evaluation Questions

Connect the state with its state bird.

___Connecticut A) Robin

___Nevada B) Mountain Bluebird

___Idaho C) Bluebird

___Ohio D) Cardinal

___Oregon E) Western Meadowlark

___Mississippi F) Mockingbird

___Texas G) Chickadee

___Wisconsin H) Lark Bunting

___Minnesota I) Common Loon

___North Dakota J) Brown Thrasher

___South Carolina K) Great Carolina Wren

___Iowa L) Eastern Goldfinch

Matching Items are used most easily to measure low-level cognitive skills such as recall or comprehension. Link People to dates, capitals to states, labels for charts, etc.In this question, neither list is in alphabetical order so students must continually scan the list if they know the answer. If the list is 1-1 match, student can guess right answers by eliminating some options. So…

Page 34: Writing Good Evaluation Questions

Connect the state with its state bird.

___Connecticut A) Bluebird

___Idaho B) Brown Thrasher

___Iowa C) Cardinal

___Minnesota D) Chickadee

___Mississippi E) Common Loon

___Nevada F) Eastern Goldfinch

___North Dakota G) Great Carolina Wren

___Ohio H) Lark Bunting

___Oregon I) Mockingbird

___South Carolina J) ) Mountain Bluebird

___Texas K) Robin

___Wisconsin L) Western Meadowlark

1. Sort both lists chronologically2. Add 1-many options… in this case, some birds are used multiple times for different

states, and some not at all. Alternatively, have more choices than matches. This prevents guessing by elimination. (Note this in the instructions too!)

3. Ensure all items are around one topic… (seems obvious, but again…)4. Shorten list… usually no more than 10

Page 35: Writing Good Evaluation Questions

Fill-in-the-Blank vs. Essay

How is grading different for these two items?

Page 36: Writing Good Evaluation Questions

Fill-in-the-Blank vs. Essay

For completion items, write items that can be completed with a single word or short phrase. When students are required to ‘fill in’ more than a few words, the grading of the item is no longer black and white, and it ceases to be a completion item.

Page 37: Writing Good Evaluation Questions

The first World War began in ____________.

What is our opportunity for improvement?

Page 38: Writing Good Evaluation Questions

The first World War began in ____________.

State? Country? Continent? Year? Decade?

Be sure that only one word or phrase can correctly complete the sentence.

Page 39: Writing Good Evaluation Questions

21) ______________ was the second president of the United States.

22) The second president of the United States was ______________.

Which completion item is easier for the student?

Page 40: Writing Good Evaluation Questions

21) ______________ was the second president of the United States.

22) The second president of the United States was ______________.

Put blanks near the end of the sentence so the student is guided toward the correct response.

Page 41: Writing Good Evaluation Questions

The letter from the 1776 General Congress to King George that severed all political connections between America and Britain is known as the_________________ ___ ___________________.

What is our opportunity for improvement?

Page 42: Writing Good Evaluation Questions

The letter from the 1776 General Congress to King George that severed all political connections between America and Britain is known as the_____________ ______________ _______________.

Make all blanks the same length. Making the length of the lines correspond to the words, gives away hints for students who are guessing.

Page 43: Writing Good Evaluation Questions

__________ __________ wrote _________ ___________.

What is our opportunity for improvement?

Page 44: Writing Good Evaluation Questions

__________ __________ wrote _________ ___________.

No more than 2 blanks in an item. The more blanks in an item, the greater the chance the student will be unable to determine the question.

Page 45: Writing Good Evaluation Questions

1. In-depth sampling of students’ knowledge of a specific topic.

2. Students must interpret, evaluate, and organize data; draw conclusions; make inferences; and express thoughts coherently.

3. Emphasizes higher-level cognitive skills such as synthesis and evaluation.

4. Easier to construct than objective tests.

Benefits and Risks of Essays

1. Teacher fatigue, subconscious biases, and other extraneous variables can affect grade.

2. Essay tests inherently biased in favor of students who can write quickly, neatly, and effectively.

3. Reduced reliability and validity because only a few questions are asked.

4. Longer time to grade.5. Increased possibility of

bluffing.

Advantages Disadvantages

Page 46: Writing Good Evaluation Questions

M. A. Lorber and W. D. Pierce (1990). Objectives, Methods, and Evaluation for Secondary Teaching (3rd Edition). Englewood Cliffs, NJ: Prentice Hall.