Transcript

Writing Good Evaluation Questions

Selected Response Constructed ResponseTrue/False Fill-in-the-blankMatching EssayMultiple Choice/Response

Types of Questions

Part figuring out assessmentLess of this

More of this

(True or False)The physical relationship between most petroleum products and purely aqueous solutions is generally such that physical interaction and diffusion of the two is severely limited.

(True or False)As a general rule, oil and water do not mix.

Which is easier to process for the student? Why?

(True or False)The physical relationship between most petroleum products and purely aqueous solutions is generally such that physical interaction and diffusion of the two is severely limited.

(True or False)As a general rule, oil and water do not mix.

(A) keeps the language simple. Unless the purpose of the test is to survey the extent of the students’ vocabulary, there is no point in using words that are unfamiliar to students or in phrasing questions so that they are difficult to understand. Students will be justifiably angry and frustrated if they get answers wrong because they could not understand what was being asked rather than because they did not know the right answer.

(True or False)The area of a rectangle is found by multiplying the length by the width.

(True or False)A 2’ x 4’ rectangle has an area of 8ft2.

Which asks the student to demonstrate at a higher cognitive level? Why?

(True or False)The area of a rectangle is found by multiplying the length by the width.

(True or False)A 2’ x 4’ rectangle has an area of 8ft2.

(B) Ask students to apply, rather than simply recall, information.If students can apply the information they learned, it is a safe bet they have committed it to memory. It does not follow, however, that simply because students have memorized information they can also apply it. This being the case, it is better to aim questions at application rather then simple recall.

5) The number of square feet in a room nine feet long and twelve feet wide is:

A) 3. B) 21. C) 81. D) 108. E) 144.

6) At \$2.00 per square foot, what would it cost to carpet the room described in question #5?

A) \$6 B) \$42 C) \$162 D) \$216 E) \$288

Here are two successive questions on an assessment. What is our opportunity for improvement?

5) The number of square feet in a room nine feet long and twelve feet wide is:

A) 3. B) 21. C) 81. D) 108. E) 144.

6) At \$2.00 per square foot, what would it cost to carpet the room described in question #5?

A) \$6 B) \$42 C) \$162 D) \$216 E) \$288

Make sure that each item is independent.Check questions to be sure that one question does not provide a clue to some other question or that the answer to one question is not crucial to another.Any student missing #5 probably missed #6. Nothing gained by linking questions… combine.

1)

2)

3)

4)

5)

6)

7)

8)

A B C D

What is the answer to #8?

1)

2)

3)

4)

5)

6)

7)

8)

A B C D

(D) Do not establish or follow a pattern for correct responses. Regardless of how clever an answer pattern is, some student will eventually discover it and compromise the test results. (You know someone has done it, or it wouldn’t be a rule!)

The acronym USDA stands for:

A) United States Dairy Association.B) United States Department of Agriculture.C) United States Department of Administration.D) United States Dietitians’ Association.

What is our opportunity for improvement?

The acronym USDA stands for:

A) United States Dairy Association.B) United States Department of Agriculture.C) United States Department of Administration.D) United States Dietitians’ Association.

(B) Do not include trick or trivial questions. Sometimes evaluators are tempted to ask questions that require extended effort for correct interpretation that deal with unimportant points. Trick or trivial questions not only reduce the validity and reliability of tests, but they may have a powerful negative effect if they antagonize students.This should be something important and relevant… something the participant should know rather than looking up in a reference document.

How should you respond to a question from a studentonce the assessment has begun?

A) the second president of the United States.B) the third president of the United States.C) the fourth president of the United States.D) none of the above.

13) The second president of the United States was:

How are these two questions different?

A) the second president of the United States.B) the third president of the United States.C) the fourth president of the United States.D) none of the above.

13) The second president of the United States was:

Stem

Distractors

If the stem does its job properly, it gives the student an idea of what is sought before reading the options. In #12, the student doesn’t know what is sought until they’ve read each option. Furthermore, more time is consumed because the student must read the repetitive wording – inefficient testing.In Question #13, the stem is clear and concise and gives the student sufficient time to begin thinking about the answer.

13) The second president of the United States was:

What could be done to improve this question?

13) The second president of the United States was:

Make distractors reasonable and plausible.

Distractors should be equally appealing to someone who is purely guessing. Someone vaguely familiar with the early presidents would recognize the names and eliminate option D, thereby increasing their odds of guessing a right answer from 1 in 4 to 1 in 3.

The area of a circle with a radius of 3 feet is:

A) 9.42 ft2.B) 18.85 ft2.C) 28.27 ft2.D) 29.6 ft2.

3’

Why are these distractors plausible?

The area of a circle with a radius of 3 feet is:

A) 9.42 ft2.B) 18.85 ft2.C) 28.27 ft2.D) 29.6 ft2.

3’

One way to construct good distractors is based on the type of errors most students are likely to make. In this case, each distractor is an incorrect calculation based on different errors.

= *3= 2*3= *32

= 2*3

15) To make an effective presentation, the presenter should:

A) identify the audience’s needs in advance.B) meet everyone in person after the presentation.C) summarizes the main points in a title slide.D) looks at the slide to focus attention on it.

What is our opportunity for improvement?

15) To make an effective presentation, the presenter should:

A) identify the audience’s needs in advance.B) meet everyone in person after the presentation.C) summarizes the main points in a title slide.D) looks at the slide to focus attention on it.

Make sure that unintentional clues are not provided.

Sometimes a student can eliminate an option simply because it is grammatically incorrect. For example a stem ending with … “An” vs “a”Incorrect tenses or forms of words can provide clues…

16) “Lake effect snow storms” are most likely to occur when:

A) the temperature drops.B) snow is created when a cold wind blows over the warmer water in the lake causing the evaporating water to crystalize.C) it gets cold.D) it gets warm.E) it has recently rained.

What is our opportunity for improvement?

16) “Lake effect snow storms” are most likely to occur when:

A) the temperature drops.B) snow is created when a cold wind blows over the warmer water in the lake causing the evaporating water to crystalize.C) it gets cold.D) it gets warm.E) it has recently rained.

Another example of unintentional clues. Distractors that are consistently longer or shorter than the other options.The use of “always” or “never” is usually a big giveaway, as is the user of “all of the above” or “none of the above”

19) The second president of the United States was:

18) Who was the second president of the United States?

Which punctuation is correct?

19) The second president of the United States was:

18) Who was the second president of the United States?

If the stem is a complete question, it should end with a question mark. The answer and the distractors would be capitalized, but not have periods unless they were complete sentences.If the stem is part of a statement that will be completed by one of the choices, it should end with a colon. Colons precede lists and the choices constitute a list. If the choices complete the statement, each should begin with a lower case letter and end with a period because the stem plus the choice will make a complete statement.

19) Who was the second president of the United States?A) John AdamsB) Warren HardingC) Thomas JeffersonD) James Madison

What is our opportunity for improvement?

19) Who was the second president of the United States?

Format… Skip a line between the stem and the choices to make it easier for students to visually separate the two components. Also keep both the stem and the choices on the same page, if there is not enough room, move the whole question to a new page.

(True or False)If every teacher were given his or her own computer, most of the difficult problems currently limiting wider use of computer-assisted instruction would be solved.

What is the opportunity for improvement?

(True or False)If every teacher were given his or her own computer, most of the difficult problems currently limiting wider use of computer-assisted instruction would be solved.

“Most of the difficult problems”? How many is that? 99%? 52%?True False items are most appropriate for lower-level cognitive skills. (Keep in mind it is a 50/50 guess) Although True-False items are often misused, it is possible to construct true-false items that cover the analysis level of Bloom’s Taxonomy.Other points to keep in mind when writing True/False items:•Be sure every item is wholly true or wholly false.•Whenever possible, avoid such terms as ‘generally’ or ‘usually’. (Although not as much as a giveaway as “always” or “never”, they are still open to varying interpretations.)•Be sure that items are not dependent upon insignificant facts. Make sure each item asks something of importance and worth remembering.•Be sure that correct items are not consistently longer or shorter than incorrect items.•Avoid the use of double negatives, but if you use any negative at all, emphasize it.

Connect the state with its state bird.

___Connecticut A) Robin

___Idaho C) Bluebird

___Ohio D) Cardinal

___Mississippi F) Mockingbird

___Wisconsin H) Lark Bunting

___Minnesota I) Common Loon

___North Dakota J) Brown Thrasher

___South Carolina K) Great Carolina Wren

___Iowa L) Eastern Goldfinch

What is our opportunity for improvement?

Connect the state with its state bird.

___Connecticut A) Robin

___Idaho C) Bluebird

___Ohio D) Cardinal

___Mississippi F) Mockingbird

___Wisconsin H) Lark Bunting

___Minnesota I) Common Loon

___North Dakota J) Brown Thrasher

___South Carolina K) Great Carolina Wren

___Iowa L) Eastern Goldfinch

Matching Items are used most easily to measure low-level cognitive skills such as recall or comprehension. Link People to dates, capitals to states, labels for charts, etc.In this question, neither list is in alphabetical order so students must continually scan the list if they know the answer. If the list is 1-1 match, student can guess right answers by eliminating some options. So…

Connect the state with its state bird.

___Connecticut A) Bluebird

___Idaho B) Brown Thrasher

___Iowa C) Cardinal

___Mississippi E) Common Loon

___North Dakota G) Great Carolina Wren

___Ohio H) Lark Bunting

___Oregon I) Mockingbird

___South Carolina J) ) Mountain Bluebird

___Texas K) Robin

1. Sort both lists chronologically2. Add 1-many options… in this case, some birds are used multiple times for different

states, and some not at all. Alternatively, have more choices than matches. This prevents guessing by elimination. (Note this in the instructions too!)

3. Ensure all items are around one topic… (seems obvious, but again…)4. Shorten list… usually no more than 10

Fill-in-the-Blank vs. Essay

How is grading different for these two items?

Fill-in-the-Blank vs. Essay

For completion items, write items that can be completed with a single word or short phrase. When students are required to ‘fill in’ more than a few words, the grading of the item is no longer black and white, and it ceases to be a completion item.

The first World War began in ____________.

What is our opportunity for improvement?

The first World War began in ____________.

Be sure that only one word or phrase can correctly complete the sentence.

21) ______________ was the second president of the United States.

22) The second president of the United States was ______________.

Which completion item is easier for the student?

21) ______________ was the second president of the United States.

22) The second president of the United States was ______________.

Put blanks near the end of the sentence so the student is guided toward the correct response.

The letter from the 1776 General Congress to King George that severed all political connections between America and Britain is known as the_________________ ___ ___________________.

What is our opportunity for improvement?

The letter from the 1776 General Congress to King George that severed all political connections between America and Britain is known as the_____________ ______________ _______________.

Make all blanks the same length. Making the length of the lines correspond to the words, gives away hints for students who are guessing.

__________ __________ wrote _________ ___________.

What is our opportunity for improvement?

__________ __________ wrote _________ ___________.

No more than 2 blanks in an item. The more blanks in an item, the greater the chance the student will be unable to determine the question.

1. In-depth sampling of students’ knowledge of a specific topic.

2. Students must interpret, evaluate, and organize data; draw conclusions; make inferences; and express thoughts coherently.

3. Emphasizes higher-level cognitive skills such as synthesis and evaluation.

4. Easier to construct than objective tests.

Benefits and Risks of Essays

1. Teacher fatigue, subconscious biases, and other extraneous variables can affect grade.

2. Essay tests inherently biased in favor of students who can write quickly, neatly, and effectively.

3. Reduced reliability and validity because only a few questions are asked.

4. Longer time to grade.5. Increased possibility of

bluffing.