Educational Educational Action ResearchAction Research

Todd TwymanTodd Twyman

Week 2Week 2

Gathering Quantitative Gathering Quantitative DataData

NumbersNumbers Attendance records test scores grades Attendance records test scores grades

specific counts of behavior responses to specific counts of behavior responses to selected response survey questions etcselected response survey questions etc

Consider the validity of the source of your Consider the validity of the source of your numbers for the specific use you intendnumbers for the specific use you intend Has this use been empirically validatedHas this use been empirically validated How do you know the numbers are How do you know the numbers are

lsquogoodrsquolsquogoodrsquo

Designing Your Designing Your Dependent MeasureDependent Measure

11 Identify the Identify the constructconstruct

22 Identify the Identify the purposepurpose of the measure of the measure

33 Define the Define the contentcontent to address the to address the purpose and constructpurpose and construct

44 Decide on the Decide on the structurestructure of the test of the test

55 Formalize Formalize administrationadministration protocols protocols

66 Write down the Write down the scoringscoring methods methods

Structure of the MeasureStructure of the Measure

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scores What level of the observed scores What level of knowledge are you hoping to knowledge are you hoping to measuremeasure Format of the MeasureFormat of the Measure

SelectionProduction ResponseSelectionProduction ResponseWrittenOralDemonstrationetcWrittenOralDemonstrationetc

Structure of the MeasureStructure of the Measure

Context of the ProblemContext of the Problem SituatedNot SituatedSituatedNot Situated

Context of AdministrationContext of Administration TimedNot TimedTimedNot Timed GroupIndividualGroupIndividual StandardizedNot StandardizedStandardizedNot Standardized

Scoring the BehaviorScoring the Behavior

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scoresthe observed scores Type of Scoring SystemType of Scoring System

ObjectiveSubjective ObjectiveSubjective (analyticholistic rubrics)(analyticholistic rubrics)

DichotomousPartial-CreditDichotomousPartial-Credit

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Gathering Quantitative Gathering Quantitative DataData

NumbersNumbers Attendance records test scores grades Attendance records test scores grades

specific counts of behavior responses to specific counts of behavior responses to selected response survey questions etcselected response survey questions etc

Consider the validity of the source of your Consider the validity of the source of your numbers for the specific use you intendnumbers for the specific use you intend Has this use been empirically validatedHas this use been empirically validated How do you know the numbers are How do you know the numbers are

lsquogoodrsquolsquogoodrsquo

Designing Your Designing Your Dependent MeasureDependent Measure

11 Identify the Identify the constructconstruct

22 Identify the Identify the purposepurpose of the measure of the measure

33 Define the Define the contentcontent to address the to address the purpose and constructpurpose and construct

44 Decide on the Decide on the structurestructure of the test of the test

55 Formalize Formalize administrationadministration protocols protocols

66 Write down the Write down the scoringscoring methods methods

Structure of the MeasureStructure of the Measure

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scores What level of the observed scores What level of knowledge are you hoping to knowledge are you hoping to measuremeasure Format of the MeasureFormat of the Measure

SelectionProduction ResponseSelectionProduction ResponseWrittenOralDemonstrationetcWrittenOralDemonstrationetc

Structure of the MeasureStructure of the Measure

Context of the ProblemContext of the Problem SituatedNot SituatedSituatedNot Situated

Context of AdministrationContext of Administration TimedNot TimedTimedNot Timed GroupIndividualGroupIndividual StandardizedNot StandardizedStandardizedNot Standardized

Scoring the BehaviorScoring the Behavior

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scoresthe observed scores Type of Scoring SystemType of Scoring System

ObjectiveSubjective ObjectiveSubjective (analyticholistic rubrics)(analyticholistic rubrics)

DichotomousPartial-CreditDichotomousPartial-Credit

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Designing Your Designing Your Dependent MeasureDependent Measure

11 Identify the Identify the constructconstruct

22 Identify the Identify the purposepurpose of the measure of the measure

33 Define the Define the contentcontent to address the to address the purpose and constructpurpose and construct

44 Decide on the Decide on the structurestructure of the test of the test

55 Formalize Formalize administrationadministration protocols protocols

66 Write down the Write down the scoringscoring methods methods

Structure of the MeasureStructure of the Measure

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scores What level of the observed scores What level of knowledge are you hoping to knowledge are you hoping to measuremeasure Format of the MeasureFormat of the Measure

SelectionProduction ResponseSelectionProduction ResponseWrittenOralDemonstrationetcWrittenOralDemonstrationetc

Structure of the MeasureStructure of the Measure

Context of the ProblemContext of the Problem SituatedNot SituatedSituatedNot Situated

Context of AdministrationContext of Administration TimedNot TimedTimedNot Timed GroupIndividualGroupIndividual StandardizedNot StandardizedStandardizedNot Standardized

Scoring the BehaviorScoring the Behavior

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scoresthe observed scores Type of Scoring SystemType of Scoring System

ObjectiveSubjective ObjectiveSubjective (analyticholistic rubrics)(analyticholistic rubrics)

DichotomousPartial-CreditDichotomousPartial-Credit

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Structure of the MeasureStructure of the Measure

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scores What level of the observed scores What level of knowledge are you hoping to knowledge are you hoping to measuremeasure Format of the MeasureFormat of the Measure

SelectionProduction ResponseSelectionProduction ResponseWrittenOralDemonstrationetcWrittenOralDemonstrationetc

Structure of the MeasureStructure of the Measure

Context of the ProblemContext of the Problem SituatedNot SituatedSituatedNot Situated

Context of AdministrationContext of Administration TimedNot TimedTimedNot Timed GroupIndividualGroupIndividual StandardizedNot StandardizedStandardizedNot Standardized

Scoring the BehaviorScoring the Behavior

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scoresthe observed scores Type of Scoring SystemType of Scoring System

ObjectiveSubjective ObjectiveSubjective (analyticholistic rubrics)(analyticholistic rubrics)

DichotomousPartial-CreditDichotomousPartial-Credit

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Structure of the MeasureStructure of the Measure

Context of the ProblemContext of the Problem SituatedNot SituatedSituatedNot Situated

Context of AdministrationContext of Administration TimedNot TimedTimedNot Timed GroupIndividualGroupIndividual StandardizedNot StandardizedStandardizedNot Standardized

Scoring the BehaviorScoring the Behavior

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scoresthe observed scores Type of Scoring SystemType of Scoring System

ObjectiveSubjective ObjectiveSubjective (analyticholistic rubrics)(analyticholistic rubrics)

DichotomousPartial-CreditDichotomousPartial-Credit

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Scoring the BehaviorScoring the Behavior

Consider the construct under Consider the construct under investigation and the intended use of investigation and the intended use of the observed scoresthe observed scores Type of Scoring SystemType of Scoring System

ObjectiveSubjective ObjectiveSubjective (analyticholistic rubrics)(analyticholistic rubrics)

DichotomousPartial-CreditDichotomousPartial-Credit

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Scoring the BehaviorScoring the Behavior

Type of ReportingType of Reporting Letter gradenumber or Letter gradenumber or

percentage correct2 category percentage correct2 category checklistnarrative reportchecklistnarrative report

Basis for Interpreting ScoresBasis for Interpreting Scores Norm-Criterion-Individual-Norm-Criterion-Individual-

referencereference

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Selected Response ItemsSelected Response Items

Make sure each answer choice is a Make sure each answer choice is a viable option (syntactically possible)viable option (syntactically possible)

Make sure each answer choice is Make sure each answer choice is approximately the same lengthapproximately the same length

Randomly order the correct answer Randomly order the correct answer choice locationchoice location

Avoid confusing question stems that Avoid confusing question stems that might detract from peoplersquos might detract from peoplersquos understandingunderstandingExampleExample

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Poor ExamplePoor Example

During the past month have you felt depressed0 = No1 = Yes once in a while2 = Yes some of the time3 = Yes most of the time4 = Yes all of the time

Harder to develop questions and response categories

May force invalid responses1048766Less depth and substance

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Poor Example contPoor Example cont

1048766Do you think it should be possible for a pregnant woman to obtain a legal abortion Circle your response

1 = Yes2 = No

Respondents unable to explain qualify or clarify answer

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Dos and DonrsquotsDos and Donrsquots

Be ConcisePoorHow do you feel about building an ice

arena in downtown Coquitlam where the railroad property has been sitting unused for a number of years

BetterAn ice arena should be built on the railroad property in downtown Coquitlam

1 = Strongly agree2 = Agree3 = Disagree4 = Strongly disagree

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Dos and Donrsquots contDos and Donrsquots cont

Use simple clear languagePoorHow often do you punish your

toddlerBetterHow often do you put your

toddler into timeout Check only one___ Once a day___ Several times a day___ Once a week ___ Several times a week

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Dos and Donrsquots contDos and Donrsquots cont

Make questions concretePoorDid you enjoy the bookBetterHave you recommended the book to anyone

elseUse mutually exclusive and exhaustive

categoriesPoorWhat is your marital status___ Married ___ SingleBetterWhat is your marital status___ Married ___ Divorced ___

Separated___ Widowed ___ Never Married

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Dos and Donrsquots contDos and Donrsquots cont

Limit ldquoskiprdquopatternsDo you participate in sports1 = No (GO TO QUESTION 3)2 = Yes (Check all sports that apply)___ Football ___ Volleyball___ Basketball ___ Soccer___ Swimming ___ Other(Specify_________)

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Dos and Donrsquots contDos and Donrsquots cont

Use caution when asking personal questions

PoorHow much do you earn each year $______________

BetterIn which category does your annual income last year best fit

___ Below $10000 ___ $10001-$20000___ $20001-$30000 ___ $30001-$40000___ $40001-$50000 ___ $50001-$60000___ $60001-$70000 ___ over $70001

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Formatting a SurveyFormatting a Survey

Focus on clarity of questionsFocus on clarity of questions Consider carefully the of answer Consider carefully the of answer

choiceschoices Use appropriate language for your Use appropriate language for your

target audiencetarget audience Have your survey reviewed for Have your survey reviewed for

appropriateness prior to useappropriateness prior to use

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Formatting a SurveyFormatting a Survey

Include a short explanation of what you are Include a short explanation of what you are planning to do with the results why you are planning to do with the results why you are seeking their input Make a plug for why their seeking their input Make a plug for why their input is really important to you (how it will input is really important to you (how it will help improve something if possible)help improve something if possible)

Try to fit all the survey items on 1 piece of Try to fit all the survey items on 1 piece of paper ndash double sided is OKpaper ndash double sided is OK

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Formatting a SurveyFormatting a Survey

If possible phrase the question stems as ldquoI statementsrdquo rather than as questions

Include demographic information that will allow you to group respondents for later analysis ndash be sure to consider how you will analyze the data prior to administering the survey so you have this covered

ExampleExample

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Your TurnYour Turn

Write questions for a survey or a Write questions for a survey or a selected response test appropriate selected response test appropriate for your Action Research study (10 for your Action Research study (10 minutes)minutes)

Wersquoll hold lsquocontent reviewrsquo focus Wersquoll hold lsquocontent reviewrsquo focus groups as a means of providing you groups as a means of providing you with feedback on your instrumentwith feedback on your instrument

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve

Content Review Focus Content Review Focus GroupsGroups

Groups of 5 peopleGroups of 5 people Decide on appropriate questions for the Decide on appropriate questions for the

group to consider (2 minutes)group to consider (2 minutes) Take turns facilitating [for discussion of Take turns facilitating [for discussion of

your own measurement instrument] and your own measurement instrument] and acting as note-taker (5 minutes)acting as note-taker (5 minutes)

Review each personrsquos survey or test and Review each personrsquos survey or test and provide feedback on ways to improveprovide feedback on ways to improve