Guidelines in writing items for noncognitive measures

of 35 /35
Guidelines in Writing Items for Noncognitive Measures Carlo Magno, PhD De La Salle University, Manila

Embed Size (px)

description

 

Transcript of Guidelines in writing items for noncognitive measures

Page 1: Guidelines in writing items for noncognitive measures

Guidelines in Writing Items for

Noncognitive Measures

Carlo Magno, PhDDe La Salle University, Manila

Page 2: Guidelines in writing items for noncognitive measures

How do I start writing items?

What variable do you want to measure?You need a conceptual frameworkGood conceptual frameworks are presented in scholarly journalsA scale becomes faulty if the framework is faulty or not well-grounded on a theory or modelThe framework should provide the description, definition, and factors of the variable you want to study.Get a framework from the original author of the construct

Page 3: Guidelines in writing items for noncognitive measures

Examples of Framework

Passion by Vallerance et al. (2003)Obsessive passionHarmonious passion

Epistemological beliefs by Schommer (1993)

Fixed abilitySimple knowledgeQuick learningCertain learning

Page 4: Guidelines in writing items for noncognitive measures

Examples of Framework

Achievement emotions by Pekrun (2006)

Positive activating emotions: enjoyment, hope, pridePositive deactivating emotions: reliefNegative activating emotions: anger, anxiety, shameNegative deactivating emotions: hopelessness, boredom

Page 5: Guidelines in writing items for noncognitive measures

Look for the definition in the frameworkThere are good definitionsThere are also bad definitions

Page 6: Guidelines in writing items for noncognitive measures

Using definitions

• Good definitions– Accurate– Easily understood– Provides behavioral

manifestation of the construct

– No contradictions

• Bad definitions– Overlaps with other

constructs– Highly technical– Difficult to frame

examples– Parsimonious

(exemption to the rule)

Page 7: Guidelines in writing items for noncognitive measures

What is a good item?

Items need to be consistently understoodItems need to be consistently administered or communicated to respondentsUnless measuring knowledge is the goal of the question, all respondents should have access to the information needed to answer the item accuratelyRespondents must be willing to provide the responses called for in the item

Page 8: Guidelines in writing items for noncognitive measures

Gathering factual dataEnsures that respondents have the same understanding of what is to be reported.Faulty: I ate butter in the past week.Better: I ate butter in the past 7 days not counting any margarine.Faulty: My income is 1,000 Php.Better: My total family income including from all sources, before deductions for taxes, for me and my family members living with me for 2008 is 1,000 Php.

Page 9: Guidelines in writing items for noncognitive measures

Knowing and remembering Form of an answer: Faulty: In the past 30 days, were you able to climb

a flight of stairs with no difficulty, with some difficulty, or were you not able to climb the stairs at all?

Better: How many miles are you from the nearest hospital?

Reducing social desirability: Ensure confidentiality Emphasize in the introduction and in other ways

the importance of the accuracy of answers Use self-administration rather than interviewer

administrator, or have respondents enter their answers directly into the computer.

Page 10: Guidelines in writing items for noncognitive measures

Questions to measure subjective states

Defining what is to be rated:Example: In general, do you think government officials care about your interest a lot, some, only a little, or not at all?Example: Do you consider crime to be a big problem, some problem, or no problem at all?Response task:Overall, how would you rate your health?-excellent, very good, good, fair, poor?

Page 11: Guidelines in writing items for noncognitive measures

Questions to measure subjective states

Agree-disagree format:Example: My health is excellentExample: I like my mother

Rank order:Which candidate do you prefer?Here are some factors some people consider when deciding where to live. Which is most important to you?

Proximity to workQuality of schoolParksSafetyAccess to shopping

Page 12: Guidelines in writing items for noncognitive measures

Questions to measure subjective states

Narrative answerWhat do you consider to be the most important problem facing your local city government today?

Measuring responses to ideas:Higher taxes generally hurt the rich and benefit the poor. Do you agree or disagree?In general, would you like to have more money spent on the parks and playgrounds in your neighborhood area or not?

Page 13: Guidelines in writing items for noncognitive measures

General rules for designing good survey instruments

The strength of the survey research is asking people about their firsthand experiences. Ask questions to which most people have informed answer.

Beware of asking about information that is only acquired second-hand.

Beware of hypothetical questions. Beware of asking about solutions to complex

problems. Questions should be asked one at a time. Avoid asking two questions at once. Avoid questions that impose unwarranted

assumptions. Beware of questions that include hidden

contingencies.

Page 14: Guidelines in writing items for noncognitive measures

General rules for designing good survey instruments

A survey question should be worded so that all respondents are answering the same questions.

To the extent possible, choose the words in questions so that all respondents understand their meaning and all respondents have the same sense of what the meaning is.

To the extent that words or terms must be used that have meanings that are likely not to be shared, provide definitions to all respondents.

The time period referred to by a question should be unambiguous.

If what is to be covered is too complex to be included in a single question, ask multiple questions.

Page 15: Guidelines in writing items for noncognitive measures

General rules for designing good survey instruments

If a survey is to be interviewer administered, wording of the questions must constitute a complete and adequate script such that when the interviewer reads the question as worded, the respondent will be fully prepared to answer the question.

If definitions are to be given, give them before the question itself is asked.

A question should end with the question itself. If there are response alternatives, arrange the questions so that they constitute the final part.

All respondents should understand the kind of answer that constitutes an adequate answer to a question.

Page 16: Guidelines in writing items for noncognitive measures

General rules for designing good survey instruments

Avoid questions that begin with adverbs: how when, where, where, why, to what extent. Such questions do not specify the terms of an adequate answer. Specify the number of responses to be given to question for which more than one answer is possible.Survey instruments should be designed so that the task of reading questions, follow instructions, and recording answers are as easy as possible for interviewers and respondents.

Page 17: Guidelines in writing items for noncognitive measures

Presurvey evaluation questions

Before the actual survey is done, testing should be done to find out if respondents can understand it, if they can perform the tasks required, and if interviewers will read it as recorded.Some steps:

Focus group discussionDo the questions appropriately cover what respondents are suppose to describe?Are the response tasks that the questions will pose tasks that respondents are able and willing to perform?Do the words or descriptions proposed in the questions convey consistent meaning, so that respondents have a common understanding of question they are to answer?

Page 18: Guidelines in writing items for noncognitive measures

Presurvey evaluation questions

Intensive individual interviewFind out how respondents understand the questionsRespondents are brought to special settings like the laboratoryReading questions to respondents

Field pretesting

Page 19: Guidelines in writing items for noncognitive measures

Guide to questionnaire construction

Match your objectives with your questionsKeep the language pitched to the level of the respondents.Try to pick words that have the same meaning for everyone.Avoid long questionsDo not assume that that your respondents possess factual information, or first hand opinions.Establish the frame of reference you have in mind.In forming a question, either suggest all possible alternatives to the respondent or don’t suggest any.

Page 20: Guidelines in writing items for noncognitive measures

Presurvey evaluation questions

Protect your respondents ego.If you’re after unpleasant orientations, give your respondent a chance to express his positive feelings first so that he’s not put in an unfavorable light.

Ex. What do you like about X? What don’t you like about X?

Decide whether you need a direct question, an indirect question, or an indirect followed by a direct question.Decide whether the question should be open or closed.Decide whether general or specific questions are needed.Avoid ambiguous words

Page 21: Guidelines in writing items for noncognitive measures

Presurvey evaluation questions

Avoid biased or leading questionsPhrase questions so that they are not unnecessarily objectionable.Decide whether a personal or impersonal question will obtain the better response.Questions should be limited to a single idea or a single reference.

Page 22: Guidelines in writing items for noncognitive measures

22

Examples of Response Formats

Multiple response item:

Single-response item

Page 23: Guidelines in writing items for noncognitive measures

23

Examples of Response Formats

Lickert Scale

Page 24: Guidelines in writing items for noncognitive measures

24

Examples of Response Formats

Verbal frequency scale

Page 25: Guidelines in writing items for noncognitive measures

25

Examples of Response Formats

Ordinal scale

Page 26: Guidelines in writing items for noncognitive measures

26

Examples of Response Formats

Forced Ranking Scale

Paired Comparison Scale

Page 27: Guidelines in writing items for noncognitive measures

27

Examples of Response Formats

Comparative Scale

Page 28: Guidelines in writing items for noncognitive measures

28

Examples of Response Formats

Linear Numeric Scale

Page 29: Guidelines in writing items for noncognitive measures

29

Examples of Response Formats

Semantic differential scale

Page 30: Guidelines in writing items for noncognitive measures

30

Examples of Response Formats

Adjective checklist

Semantic distance scale

Page 31: Guidelines in writing items for noncognitive measures

31

Examples of Response Formats

Fixed sum scale

Page 32: Guidelines in writing items for noncognitive measures

32

Examples of Response Formats

Multiple rating list

Page 33: Guidelines in writing items for noncognitive measures

33

Examples of Response Formats

Multiple rating matrix

Page 34: Guidelines in writing items for noncognitive measures

34

Examples of Response Formats

Diagram scale

Page 35: Guidelines in writing items for noncognitive measures

35

Examples of Response Formats

Graphic scale