Hunter Gehlbach

2013 AAPOR Convention, Boston MA

An Overarching Process forEnhancing the Validity of Survey Scales

1

Overview

1. Literature review

2. Interviews & focus groups

3. Compare literature review & interview data

4. Develop items

5. Expert validation

6. Cognitive pretesting

(Gehlbach & Brinkworth, 2011)

2

Background: the scales in question Parent support (social and learning) Fit of the child at the school Efficacy Child behaviors (positive and negative) Engagement School climate Communication

Roles Barriers

3

Step 1:Literature review of parent support

Goals:◦ Determine how parent support is defined Parent home-based support pertains to the specific behaviors that

parents enact to foster their children’s learning and social /emotional development in the home.

◦ Identify items that could be adapted for our scale Examples: provide enrichment, help children manage emotions

4

Step 2:Interviews and focus groups Goal:

◦ Make sure parent support construct reflected whatpotential participants experience Incorporate parent voices in developing parent support

scale

5

Interview Protocol

Interviewed 10 parents of children in preK-12

Each interview/focus group lasted 45-60 minutes

Used semi-structured interview protocolconsisting of 11 questions, including:

◦ What are some of the goals that you have foryour child this year and what have you done tohelp them reach those goals?

6

Literature Review Parent interviews Indicators

Direct homework aid (Patall,Cooper, & Robinson, 2008)

Explain concepts, notassistance withcompleting assignments

Help with school-relatedtasks

Step 3:Compare interview & lit. review Goal: Full conception of the parent support

construct that both academics & participantsagree with◦ Begin using participants’ language/terminology

7

Step 4:Develop items & construct questionnaire

Goal: Representative construct coverage anddevelop items that adhere to best practices:

◦ Should I craft items as statements or questions?

◦ Should I use construct-specific or agree-disagreeresponses?

◦ How many response options should I use?

◦ Does it matter if I have a mid-point?

◦ Should I label them with words? numbers? both?8

Developing parent support items

Learning Support (13 items)◦ How often do you help your child understand

the content s/he is learning in school?

Social Development Support (6 items)◦ How often do you help your child deal

appropriately with his or her emotions?

9

Step 5:Expert validation Goal: Establish content validity

◦ Construct relevance of individual items

◦ Developmental appropriateness

◦ Cultural appropriateness

◦ Check for missing indicators of home-basedparental support

Changes:

1 item added

2 items revised10

Step 6:Cognitive pretesting/interviewing Goal: Make sure respondents are

understanding the items as desired bysurvey developer◦ Have respondents repeat the questions in their

own words

◦ Think-aloud as they answer

11

Changes

• Revised 5 items

• Example: How well do you know your child’s friends?

• Deleted 3 items:

• Example: How much effort do you put into helping yourchild learn to work out disagreements with others?

Cognitive pretesting

12

13

Learningsupport

Talkaboutwhat

class islearning

Helpchildlearn

concepts

Engage inoutsidehome

learningactivities

Helpchild

learn todo thingson own

Socialsupport

Talkwhen

child hasproblem

Knowhow childis doing

socially inschool

Knowchild’sclose

friends

.495(.047)

.739(.047)

.661(.035)

.742(.030)

.699(.038)

.734(.036)

.611(.040)

.818(.036)

Pilot testing

CFA Model Fit:Chi-square =18.56**, RMSEA = .056, RMSEA 90% CI = .028-.084.,CFI = .986

Summary

Multiple check-points & methods forcatching problems Builds in validity on the front end Balances input from respondents and

academics…but time and labor expenses

14

APPENDIX

15

Sample indicators / measures relatedto parent support

16

Step 2: Parent Support ScaleInterview Sample

17

Sample characteristics # of parents (%)

Race / Ethnicity

White 7 (70%)

African American 2 (10%)

Arab 1(10%)

Gender

Mother 9 (90%)

Father 1 (10%)

# of children

1 child 3 (30%)

2+ children 7 (70%)

Child grade level

Pre-K-Grade 5 9

Grades 6-8 6

Grades 9-12 3

Expert validation of parent supportscale Data collected from 16/25 invited experts

Average relevance of items = 4 out of 5

Deleted one item for low relevance toconstruct:◦ How much effort do you put into preparing

your child for the next school year?

Issues identified in clarity and content 18

Pilot Testing: Factor Structure

1-factor2-factor (socialand learning)

Free parameters 35 36Chi-square test ofmodel fit value 59.94*** 18.56**RMSEA 0.092 0.056RMSEA 90% CI 0.069-0.117 0.028-0.084CFI 0.958 0.986TLI 0.936 0.977WRMR 0.818 0.557

19

The two factor model fits the data significantly better than aone-factor model (chi-square test for difference testing =205.253, df = 1, p <.0001)

School Fit1. How well do you feel your child’s school is preparing him/her

for his/her next academic year?

2. Given your child’s cultural background, how good a fit is his/herschool?

3. How well do the teaching styles of your child's teachers matchyour child's learning style?

4. How well do the activities offered at your child’s school matchhis/her interests?

5. How much of a sense of belonging does your child feel at his/herschool?

6. At your child's school, how well does the overall approach todiscipline work for your child?

7. How comfortable is your child in asking for help from schooladults?

20

School Climate How well do administrators at your child’s school create a school

environment that helps children learn?

How motivating are the classroom lessons at your child's school?

How fair or unfair is the school's system of evaluating children?

To what extent do you think that children enjoy going to yourchild's school?

Overall, how much respect do you think the children at your child'sschool have for the staff?

Overall, how much respect do you think the teachers at your child'sschool have for the children?

How much does the school value the diversity of children'sbackgrounds?

21

Roles Make time for doing fun activities that are unrelated to schoolwork

Help children deal with their emotions appropriately

Identify what children are most interested in learning

Ensure good communication between home and school

Call attention to decisions about learning that do not seem to be inthe best interest of the children

Make sure the children’s learning environment is safe Make sure children are supported to do their best in school

Make sure that the children understand what is being taught atschool

Make sure that the children have an adult to talk to at school

Make sure that children have enough time set aside to do all of theirschool-related work

Ensure children have good relationships with their peers22

Parent Support Social Support

◦ How often do you and your child talk when s/he is having a problemwith others?

◦ To what extent do you know how your child is doing socially at school?

◦ How well do you know your child's close friends?

Learning Support

◦ How often do you have conversations with your child about what his/herclass is learning at school?

◦ How often do you help your child understand the content s/he is learningin school?

◦ How often do you help your child engage in activities which areeducational outside the home?

◦ How much effort do you put into helping your child learn to do thingsfor himself/herself?

23

Self-Efficacy How confident are you that you can motivate your child to try hard

in school?

How confident are you in your ability to support your child'slearning at home?

How confident are you that you can help your child develop goodfriendships?

How confident are you in your ability to make sure your child'sschool meets your child's learning needs?

How confident are you in your ability to make choices about yourchild's schooling?

How confident are you in your ability to connect with otherparents?

How confident are you in your ability to help your child deal withhis or her emotions appropriately?

24

Child Behaviors Positive Behaviors

◦ How much effort does your child put into school-related tasks?

◦ How motivated is your child to learn the topics covered in class?

◦ On average, how well does your child work independently onlearning activities at home?

◦ How regularly does your child read for fun?

◦ In general, how well does your child learn from feedback abouthis/her work?

Negative Behaviors

◦ When working on school activities at home, how easily is yourchild distracted?

◦ How often does your child give up on learning activities that s/hefinds hard?

◦ How often does your child struggle to get organized for school?25

Engagement How often do you meet in person with teachers at your child's

school?

In the past year, how often have you visited your child's school?

In the past year, how often have you discussed your child's schoolwith other parents from the school?

How involved have you been in fundraising efforts at your child'sschool?

How involved have you been with a parent group(s) at your child'sschool?

In the past year, how often have you helped out at your child'sschool?

26

Barriers to Engagement How big of a problem are the following issues for becoming involved with

your child's current school?◦ How busy your schedule is

◦ Transportation-related challenges

◦ Childcare needs

◦ Concerns about getting to the school safely

◦ Your child does not want you to contact the school

◦ Negative memories of your own school experience

◦ You feel unsure about how to communicate with the school

◦ The school is not welcoming to parents

◦ The school provides little information about involvement opportunities

◦ The school doesn't communicate well with people from your culture

◦ School staff seem too busy

◦ You worry that adults at the school will treat your child differently if you raise a concern

◦ You do not feel a sense of belonging with your child's school community

27