Welcome to Unit III

Recognition Items

EDS 596 Assessment Techniques in Counseling

Usual DisclaimerAvoid prolonged contact with skin

Not to be taken internallyNo animals were harmed during the

production of this presentation

After viewing this presentation avoid operating mobile or dangerous equipment

This presentation contains graphic images which some viewers may find disturbing.

Definition of Test

“…a task or series of tasks used to collect a sample of human behavior assumed to be representative of a

large domain.”

Typically a test is used in an attempt assess or measure some quality or characteristic of an individual. Tests are used to judge, grade, or evaluation someone.

TestsTests are composed of items;

Each item is designed to elicit a response (simple or complex)

The response is evaluated or compared to a standard and a judgment made (right-wrong/correct-incorrect)

Tests are frequently used in the assessment of Achievement, Aptitude or Intelligence.

Items typically fall into one of two large families of item forms

Recognition and Supply

Recognition Items•Highly structured

•Restricted Response Freedom

•Objectively Scored

This family of items includes:The alternative response item

The Matching Exercise and …

The multiple choice/Interpretive Exercise

Some beginning thoughts….a test item represents a link in a chain and is used for one of two purposes….

1. To assess “mastery” of some body of material or skill….or

2. To measure differences in mastery or levels of achievement among students/clients…..

As a link in the chain each item must have one characteristic. Ideally, a client or student who does not know the material should miss it and a student who knows the material or has mastered the material should be able to get it right! The item should

discriminate.

Recognition Items……

Advantages:A. Permits excellent item sampling

B. Easy, quick, and objective scoring

C. Writing is minimized…Primitive Responses

D. Generally good for measuring simple outcomes…

E. No skill is needed for scoring

F. Quick return of scores and scored tests…

Recognition Items……

Disadvantages:A. Bright students may be penalized…

B. Guessing is always possible

C. Items may encourage rote learning….not always bad…

D. Sometime used to measure “trivial” material.

Recognition Items……General Guidelines

A. Test only for important ideas, information, & skills

B. Write items as simply and directly as possible

C. Include only ONE correct or best answer.

D. Sometime used to measure “trivial” material.

E. Avoid “Chaining” questions

F. Avoid specific determiners

G. Never “pull” questions or material from text.

H. Avoid “trick” questions

I. Always review items prior to printing test

The Alternative Response Item

Advantages:A. Excellent item sampling

B. Easy to write and score

B. Good for measuring “simple” outcomes.

C. Good item for clients with poor reading ability

Form: A simple declarative sentence requiring a dichotomous judgment.

The Alternative Response Item

Disadvantages:A. Ease of successful guessing (50%)

B. Encourages rote learning…not always bad

B. Inability to measure complex outcomes

C. Limited to dichotomous material

D. Susceptible to ambiguity and misinterpretation

E. Least discriminating of all test item forms..

F. May penalize bright and insightful students

Guidelines for Alternative Response Items

a. Insure that items are unequivocally true or false

b. Avoid the use of specific determiners

c. Avoid ambiguous, and indefinite terms of degree

d. Avoid negatively worded items..

e. Avoid “double barreled items”

f. Keep T’s and F’s about the same length

g. Avoid detectable response patterns

h. Avoid trivial material

Consider this Question…

True of False…The sun rose today.

Most would agree that the answer is true….it got light this morning.....

However, the more thoughtful among us may conclude that it is False…the sun only appears to rise and set as a function of the earth’s rotation on its axis, for example…..

Alternative Response Items

Poor: T F Because of dramatically increased levels of CO2 emissions from industrialized countries there has been a reduction in the size of the polar ice caps.

This is a double barreled item!

Better: T F Recently there has been a reduction in the size of the polar ice caps.

Alternative Response Items

True or False

Acceleration of a body due to gravity is not 9,8 m/sec2

This item is worded in the negative...

Alternative Response Items

True or False

Weather and climate always have the same meaning….

This item contains a specific determiner, the word “always”

What to do about guessing

• Guessing is possible anytime the correct answer is available for selection

• A student gets the same point if he/she knows the answer or simply get lucky with a guess

• Some students guess far more often than others• Guessing closes the range and makes differences

between students harder to see.

Any attempt to deal with guessing should be directed toward equalizing the impact of

guessing on all students.

Two choices are available

• Try to discourage guessing by imposing a penalty through the use of a score correction formula

•Try to encourage guessing by telling students not to leave any question blank (Probably the best option)

The Score Correction Formula

Tommy takes a 50 item, Alternative Response test, getting 42 correct. His score, correct for guessing would then be…..42-(8/2-1)….42-(8/1)….42-8=34

• That’s not so difficult, is it?

12

842cgS 34842

1

842

cgS

Confidence WeightingTrue or False?

The Avco-Lycoming O-235-C1C produces 130 bhp at 2950 rpm.

Please write down your answer….true or false…

Now…tell me how confident you are of your answer….write down:

1 if your answer is just a guess

2 if you’re kind of certain and

3 if you’re absolutely sure…..

The Matching Exercise

Form: Two columns of information the first of which is referred to as the list of items…the second column is referred to as the list of options. The task is to match the items with the appropriate options….This is the best type of item to use when trying to assess mastery of a paired-association.

Advantages of the Matching Exercise

• Ease of construction

• Excellent item sampling

• Ease of scoring (if properly constructed)

• Best item for paired-associations tasks

Disadvantages of the Matching Exercise

• Encourages simplistic and rote learning…not always bad…

• Basic inability to assess complex learning outcomes

• Guessing is always possible but this can be dealt with through proper construction.

Writing a Matching Exercise

• Items should be in the left column and options in the right.

• Each exercise should consist of homogeneous material.• Keep each exercise relatively short (8-10 items).• Avoid having an equal number of items and options.• Arrange options in some systematic fashion is possible• Avoid specific determiners• Explain clearly in the directions the basis upon which the

match is to be made.• Maintain grammatical consistency.

Match the famous compositions in Column I with the guys who wrote that kind of stuff in Column II. Each choice in Column II may be used one, or not at all.

__ 1. Les Quaaludes A. Offenbach

__2. Pictures of an Exhibitionist B. Rimsky-Korsakov

__3. Marche Slob C. Chopin

__4. Orpheus in His Underwear D. Orff

__5. Le Coq Au Vin E. Liszt

__6. Carmina Piranha F. Stravinsky

__7. Peter and the Moose G. Debussy

__8. Capriccio Escargot H. Prokofiev

__9. A Midsummer Night on Elm Street I. Beethoven

J. Tchaikovsky

K. Rachmaninoff

Match the famous Operas in Column I with the guys who wrote that kind of stuff in Column II. Each choice in Column II may be used once, more than once, or not at all.

__ 1. Rigatoni A. Offenbach

__2. La Boingggg B. Rimsky-Korsakov

__3. Madam Mothballs C. Chopin

__4. Il Fettuccine D. Orff

__5. The Cadillac de Saville E. Liszt

__6. La Sagna

__7. The Marriage of Orthogro

__8. Das Fahrvergnugun

__9. The Magic Galoot

Directions: On the line to the left of each achievement in column A, write the letter from column B that identifies the name of the man noted for that achievement. Each name from column B may be used once, more than

once, or not at all. This is POOR PRACTICE

Column A Column B

__ 1. Invented the telephone A. Alexander Graham Bell

__2. Discovered America B. John Glenn

__3. First U.S. Astronaut to orbit the earth C. Abraham Lincoln

__4. First President of the U.S. D. Ferdinand Magellan

E. Christopher Columbus

F. George Washington

G. Eli Whitney

The Multiple Guess Item

Form: Typically takes the form of a stem with four or five options only one of which is considered to be correct or the best answer. There are two forms…..

Correct answer type: One correct among a number of incorrect ones…..

Best answer type: Several correct responses which differ in DEGREE of correctness or completeness.

Multiple Guess Items

Advantages:

a. Excellent item sampling

b. Ease of scoring

c. Versatile across a range of difficulty

d. Student writing is minimized

e. Permits writing tests which can re-used

Multiple Guess Items

Disadvantages:a. Time consuming to write….for each stem

that is written a correct response must be created as well as three or four incorrect responses which appear as though they could be correct.

b. Students complain that more than one correct response

Multiple Guess Items

Guidelines/Suggestionsa. Stem should be brief, clear, and conciseb. Stem should completely set the taskc. The options should be brief, clear, and concised. Distractors should be plausiblee. Avoid specific determinersf. Correct responses should be randomly placed.g. Use “all of the above” or “none of the above”

only when appropriate and sparingly.

The two types of M/C Items

The Correct Answer Type:

Which of the following continents lies totally above the equator?

a. Africa c. Asia

b. Austrailia d. South America

The two types of M/C Items

The Best Answer Type:

If you found yourself on a desert island, which of the following would you need first to find?

a. Food

b. Water

c. Shelter

d. A “friendly” companion

Forms of M/C Stems

• Which of the following is the best explanation for why most people wet the bed during earthquakes? Interrogatory

• The principle reason for nocturnal enuresis in geologically active regions is: Incomplete sentence

• Mark the answer which best describes the reason for bed wetting in volcanic areas. Imperative

Question Format Stem

10.Which one of the following expressions used by the airlines is an OXYMORON?

A. Complete stop D. Airline food

B. Direct flight E. Friendly Skies

C. Nonstop flight

A specific determiner…

10. A “BLEAM” is an:

A. lizard with 200 eyes, 100 for distance and 100 for reading.

B. animal with the body of a crab and the head of a certified public accountant.

C. a singles game played with dice and a large tube of ointment.

D. savage beast with the head of a lion and the body of a lion, though not the same lion.

Avoid specific determiners.

10. The largest African antelope is an:

A. Hippo

B. Dik-Dik

C. Eland

D. Warthog

Vary the position of the correct response

10. The largest African antelope is a/an:

A. Hippo A. Dik-Dik

B. Dik-Dik B. Eland

C. Eland C. Hippo

D. Warthog D. Warthog

By arranging the options in alphabetical order the correct response has in now in the “B” position.

Vary the position of the correct response

10. The Magna Carta was signed in what year?

A. 1965 A. 1215

B. 1812 B. 1776

C. 1215 C. 1812

D. 1776 D. 1965

By arranging the options in chronological order the correct response has in now in the “A” position.

Avoid Implausible Distrators

10. About how many calories are recommended daily for a girl, age 14, height 1.57m, weight 50kg, moderately active?

A. 0 A. 1,500

B. 2,000 B. 2,000

C. 2,500 C. 2,500

D. 30,000 D. 3,000

Strive for Parallelism

10. The cell islets of the pancreas:

A. Contain ducts

B. Produce insulin

C. Disappear as one grows older

D. Are located around the margins of the pancreas

This is poor form…..

Strive for Parallelism and let the stem set the task…..

10. The cell islets of the pancreas secrete the substance called:

A. adrenalin

B. insulin

C. secretin

D. trypsin

This is much better!

Strive for Parallelism and let the stem set the task…..

10. South America:

A. Is a flat, arid, country

B. Imports coffee from the U.S.

C. Has a larger population than the U.S.

D. Was settled mainly by colonists from Spain

Strive for Parallelism and let the stem set the task…..

10. Most of South America was settled by colonists from:

A. England

B. France

C. Holland

D. Spain

It’s not wise to “chain” questions

Poor:

1. Scurvy is caused by a lack of vitamin:

a. A b. B1 c. B12 d. C

2. A good source of this vitamin is:a. Orange juiceb. Cod-liver oilc. Whole riced. liver

Avoid “Trick” Questions

Poor:

1. Which of the following substances is most essential for human life?

a. Protein b. Water c. Vitamins d. Minerals

1. Better: A healthy person is marooned on a deserted island. To survive, the person needs almost immediately to find a source of:a. Protein b. Water c. Vitamins d. Minerals

Check the Reading Level!

Poor: The promiscuous use of sprays, oils, and antiseptics in the nose during acute colds is a pernicious practice because it may have a deleterious on:

A. the sinuses.

B. red blood cells.

C. white blood cells.

D. the olfactory nerve.

Check the Reading Level!

Better: Frequent use of sprays, oils, and antiseptics in the nose during bad colds may result in:

A. Sinuses infections.

B. mucous membrane congestion.

C. harm to white blood cells.

D. olfactory nerve damage.

Eschew Verbosity

Poor: Cells of one kind belong to a particular group performing a specialized duty. We call this group of cells a tissue. All of us have different kinds of tissues in our bodies. Which of the following would be classified as epithelial tissue?

A. Tendons

B. Adenoids and tonsils

C. Mucous membranes

D. Cartilage

Eschew Verbosity

Better: Which of the following would be classified as epithelial tissue?

A. Tendons

B. Adenoids and tonsils

C. Mucous membranes

D. Cartilage

Think Positively

Poor: Which of the following structures of the ear is NOT concerned with hearing?

A. Eardrum

B. Oval window

C. Semicircular canals

D. Cochlea

Think Positively

Better: Which of the following structures of the ear helps us to maintain balance?

A. Eardrum

B. Oval window

C. Semicircular canals

D. Cochlea

Test only for important ideas

Poor: In 1998 the death rate from accidents of all types per 100,000 population in the 15-24 age group was?

A. 59.1

B. 59.2

C. 59.3

D. 59.4

Test only for important ideas

Better: In 1998 the leading cause of death in the 15-24 age group was from:

A. respiratory diseases.

B. accidents.

C. cancer.

D. rheumatic heart disease.

Measuring Complex AchievementThe Interpretive Exercise

• Apply a principle• Interpret relationships• Recognize and state inferences• Recognize the relevance of information• Develop/recognize tenable hypotheses• Formulate/recognize valid conclusions• Recognize the limitations of data• Design experimental procedures

Measuring Complex AchievementThe Interpretive Exercise

The form: The IE consists of a series of objective items the answers to which may be conjured from a common set of data. These data may be written material, maps, charts, graphs, tables or pictures.

This pie chart depicts Bill’s weekly allowance distribution

1. What is the ratio of the amount Bill spends for school supplies to the amount he spends of movies.

a. 7:2 b. 1:3 c. 2:7 d. 3:1

2. What would be the best title for this pie chart?

a. Bill’s weekly allowance c. Bill’s weekly expenditures

b. Bill’s money Chart d. Bill’s money planning

Interpretive Exercise

Objective: to recognize the relevance of information

Bill lost his overshoe on the way to school. He wanted to put a notice on the bulletin board so other children could help him find it. Which of the following sentences tells something that would help children find the overshoe?

Directions: Circle “yes” if it would help. Circle “no” if it would not help.

Yes No 1. The overshoe was black

Yes No 2. It was very warm

Yes No 3. It was for his right foot

Yes No 4. It was a Christmas present

Yes No 5. It was nice looking

Yes No 6. It had a zipper

Yes No 7. It had a gray lining