ScWk 240 Week 7 Measurement Part 2, & Sampling October 8, 2012 1ScWk 240 Week 7.

ScWk 240Week 7

Measurement Part 2, & Sampling

October 8, 2012

1ScWk 240 Week 7

Today’s Agenda

• Finish Validity & Reliability• Survey development• Intro to Sampling• Literature review – writing tips and your

questions

ScWk 240 Sampling 2

Review: Potential for systematic or random error?

1. People are more alert in the morning than after lunch ____________

2. The meaning of “living independently” varies cross-culturally __________

3. Male respondents tend to minimize their aggressive tendencies _________

4. A 16 y.o. taking a test: “I just circle ‘C’ each time and hope for the best”

5. The person entering data is not careful enough and mis-codes some of the responses ________

3ScWk 240 Week 6

Student Report

• How is validity related to cultural issues? --Dawei

ScWk 240 Sampling 4

Reliability • Concerned with whether or not a particular

technique done repeatedly would yield the same result each time– Inter-rater reliability—Two researchers rating the same

behavior: do they rate them the same?– Test-retest reliability—Is the response from the first test

the same as the response from a re-test?– Alternate forms reliability—Is there the same response

from slightly different versions of same measure?– Internal consistency reliability & use of coefficient alpha—

Are similar questions (about the same concept) in survey are answered in the same way?

ScWk 240 Week 6 5

You‘ll see this in your lit review…• Methods section—measures used: the author

will report on reliability testing (if done) using the coefficient alpha– A number that tells you, overall, how well the

questions in an instrument measuring the same concepts correlate with each other (internal consistency reliability)

– A coefficient alpha is considered good if it is .80 or above

– The author will then report “The instrument showed good internal consistency reliability”

ScWk 240 Week 6 6

Validity• Concerned with the accuracy of meaning—are

you measuring what you originally intended?• Types of validity– Face validity – experts’ understanding of concept– Content validity – Does measure cover all relevant

dimensions?– Criterion-related validity – Does measure of concept agree

with (or relate with) another related concept (external criterion)?

– Construct validity – Does measure agree with or correlate with measures of other theoretically related concepts?

ScWk 240 Week 6 7

Validity examples1. Validity studies show that a set of questions about alcohol

dependence correlates well with the observed daily amount of alcohol consumption. Criterion-related validity

2. Most experts would agree that children with Attention Deficit Disorder with Hyperactivity have difficulty staying on task. Face validity

3. Couples who test as having low levels of marital satisfaction communicate less effectively, leading to high levels of conflict and, eventually, divorce. Construct validity

4. The Beck Depression Scale covers those dimensions of depression that are most supported by evidence. Content validity

8ScWk 240 Week 6

Student reports

• What are some strategies for ensuring reliability when developing survey questions? -- Amanda

• What are some strategies for ensuring validity when developing survey questions? -- Sarah

ScWk 240 Sampling 9

Writing survey questions to maximize validity & reliability

• Question written at appropriate level of understanding (age, knowledge level, language spoken)

• Avoid “double-barreled” question– “Are you satisfied with services and why?”

• Questions are sensitively arranged (emotionally-laden ones later)

• Instructions make sense• Scaling of attributes makes sense

– Consistent language – Mutually exclusive choices

ScWk 240 Sampling 10

What’s wrong with this?

How satisfied are you with the services you received?

1. Very satisfied2. Somewhat satisfied3. Don’t know4. Somewhat dissatisfied5. Terrible


Should I use open- or closed-ended questions?

• Student Report: What is the main difference between “open-ended” and “closed-ended” questions? --Selene

• Qualitative vs. quantitative survey purposes


Why sampling?

• The problem of quality control on the assembly line• Usually can’t observe (take measurements from)

100% of a population• Making the decision on who or what to observe

involves understanding– What population do you want to study?– Whether you want breadth or depth of information

***Discussion: So, how do you decide that?

13ScWk 240 Sampling

First, a few terms to know…

• Study population--Your study’s focus– As stated in your problem/research statement, and maybe your

hypothesis– You want to generalize your study’s results to this larger population.

“Generalize” – to extend the findings of your sample to the larger population

• Sampling frame--Those eligible to be selected into your study– Could be your entire study population, or more typically it’s a subset

of the study population meeting criteria for inclusion in your study– Often, literally, a list (such as a list of agency clients)– Not every study population comes with a handy sampling frame (e.g.

homeless people)• Sample--Those ultimately selected to be in your study



SamplingFrame

Sample

Study Population

Example: “The impact of a parenting class on teen parents’ stress levels”


SamplingFrame—list of all teen parents in a school

Sample

All teen parents

From this sample, we want to

generalize to the population of all

teen parents

Two main types of sampling:

• Probability sampling -- everyone in the study population has an equal chance of selection into your study. Keywords: random assignment

• Non-probability sampling -- everyone in the study population does not have an equal chance of selection

***Discussion: what are the pros and cons of both?


The Logic of Probability Sampling• Overall purpose of probability sampling: to allow the researcher to

estimate characteristics (parameters) of the study population from a sample’s characteristics– e.g. “How much does the sample represent the population from

which it was drawn?”• Why do we want to “estimate population characteristics”?

– Because in most cases we don’t really know the real population characteristics (but we’d like to know!)

• So, we estimate a population parameter (e.g. average depression score) of the population, from a sample statistic (average depression score of the sample)

• Since research means “never having to say you’re certain”, we won’t be 100% perfect in our estimates, although we can quantify how confidently we feel our sample matches the population. How? Stay tuned for ScWk 242.


How to do probability sampling

• Simple random sample—assigning numbers to potential participants and selecting numbers (people) randomly. Similar to picking them out of a hat

• Systematic random sample—choosing every kth person--selecting persons at a predetermined interval

• Stratified random sampling--Simple or systematic random sampling with sub-groups

• Cluster sampling—a multi-stage procedure of randomly choosing levels of analysis units (e.g. cities, neighborhoods, schools, classrooms)


Non-probability sampling• Quota sampling—sample chosen based on predefined

characteristics of study so that sample will have same proportion of those characteristics as in the study population

***What’s the difference between this and stratified random sampling?• Snowball sampling—for difficult-to-locate populations: finding more

sample participants based on recommendations from others in the study

• Purposive sampling —selecting sample that is thought to yield the most comprehensive understanding of the study’s topic

• Convenience sampling (a.k.a. “availability sampling”)—selects participants simply based on their immediate availability. (Note—participants may also coincidentally comprise a purposive sample, but not necessarily)


Two families of sampling

Probability sampling

• Simple random sampling• Systematic sampling• Stratified random sampling • Cluster sampling

Non-probability sampling

• Quota sampling• Snowball sampling• Purposive sampling• Convenience sampling


What type of sampling is this…?

• For an experimental study of cognitive behavioral therapy, every 10th adult with anxiety disorders is selected randomly from a sampling frame

• An agency manager walks down the hall and asks staff opinions about the new caseload policy

• The same agency manager makes sure to ask both women and men

• For a study on standardized testing in schools, researchers randomly select cities in California, then randomly select elementary, middle and high schools in those cities

• A social work researcher seeks out Latino women who were victims of domestic violence


Student reports for next week

1. Name some strategies to recruit difficult-to-reach populations, and retain them for follow up Annalisse

2. What is required in order to prove causality? Jenna

3. What is the most important difference between experimental group designs and all others? (Hint—has to do with group assignment) Ashley


Concepts to know• Closed-ended questions• Open-ended questions• Population• Sample• Sampling frame• Population parameter• Statistic• Generalizibilty • Probability sampling (types)• Non-probability sampling (types)


ScWk 240 Week 7 Measurement Part 2, & Sampling October 8, 2012 1ScWk 240 Week 7.

Documents

Transcript of ScWk 240 Week 7 Measurement Part 2, & Sampling October 8, 2012 1ScWk 240 Week 7.