2014 HYS Regional Workshop

Using your HYS Results April/May 2015

Welcome!

• Introduce presenters • General housekeeping • Powerpoint and workbook • Find out who is here

Workshop purpose and objectives

Help people understand and use their HYS results: • Overview of 2014 Results • HYS background & administration • Statistical issues • Accessing your data • Results available on AskHYS • Talking about your results

Overview of 2014 Results

The rates of alcohol use among 8th and 10th grades has dropped by half. Nearly 11,000 fewer students are using alcohol compared to 2010.

Q. Had alcohol during the past 30 days?

0%

10%

20%

30%

40%

50%

60%

1998 2000 2002 2004 2006 2008 2010 2012 2014

6th Grade 8th Grade 10th Grade 12th Grade

Source: WSSAHB 1998 (spring), 2000 (fall), HYS 2002, 2004, 2006, 2008, 2010, 2012, 2014 (fall)

-24%

Alcohol Use The prevalence of alcohol use has declined significantly in all grades.

Marijuana Use Marijuana use did not change significantly from 2012 to 2014

0%

10%

20%

30%

40%

50%

1998 2000 2002 2004 2006 2008 2010 2012 2014

6th Grade 8th Grade 10th Grade 12th Grade

Year 10th Grade Marijuana

Use

1998 26.6%

2000 21.9%

2002 18.3%

2004 17.1%

2006 18.3%

2008 19.1%

2010 20.0%

2012 19.3%

2014 18.1%

Q. Used marijuana/hashish during the past 30 days?

Source: WSSAHB 1998 (spring), 2000 (fall), HYS 2002, 2004, 2006, 2008, 2010, 2012, 2014 (fall)

Perception of Risk from Marijuana Use Increasingly more students think using marijuana regularly is not risky

21%

35%

46%

0%

10%

20%

30%

40%

50%

2002 2004 2006 2008 2010 2012 2014

8th Grade 10th Grade 12th Grade

Q. Using marijuana regularly has no risk/only slight risk

21%

18%

27%

16%

9%

35%

21%

10%

34%

18%

8%

Sad or hopeless? Serious thoughts about suicide?

Attempt suicide?

8th Grade 10th Grade 12th grade

Mental Health Status More than one in four students had depressive feelings in the past year.

Over 100,000 youth (12-17 year olds) seriously considered suicide in the past year, which is about one in every six students.

NOTES: Depressive feeling: felt so sad or hopeless for two weeks in a row that they stopped doing usual activities.

Percent of students answering “Yes”

29%

16%

26%

14% 15%

7%

9%

2%

Had depressive feelings in the past year No depressive feelings in the past year

Substance Use and Mental Health Status Students who had poor mental health are also more likely to report substance use.

NOTES: Tobacco products include cigarettes and chewing tobacco.

Use Alcohol

Use Marijuana

Use Tobacco*

Use Pain Killers

Use Alcohol

Use Marijuana

Use Tobacco* Use Pain

Killers

Substance Use and Poor Academic Performance

37%

47% 52%

47%

20%

Use Alcohol Use Marijuana Use Tobcco Use Pain Killers to Get High

No Substance Use*

Low Grades

Low Grades

Low Grades

NOTES: * Did not use alcohol, marijuana, tobacco products (cigarettes or chewing tobacco), or pain killers in the past 30 days.

Low Grades

Washington State 10th graders who reported substance use are twice as likely to have low grades in school than those who do not use substances

Q.

Low Grades

Putting them all together, what were your grades like last year? Percent of students who report receiving “C”, “D”, or “F” mostly.

Use Alcohol

School Risk and Protective Factors Scale Scores “Protected”, “At Risk”

• School Opportunities for Prosocial Involvement

• Decisions about class activities, talk to

teacher one-on-one, work on special projects, get involved in extracurricular activities, class discussion involvement…

13%

25% 24%

6%

17% 15%

0%

5%

10%

15%

20%

25%

30%

Current cigarette …

Current alcohol …

Current marijuana …

Not Protected Protected

Protected by “School Opportunities” ? Washington 10th graders who are protected by “School Opportunities” have lower substance use rates.

4%

16%

11%

3%

13%

25% 27%

7%

0%

5%

10%

15%

20%

25%

30%

Current cigarette

smoking*

Current alcohol

drinking*

Current marijuana

use*

Current other

drug use*.

Not at Risk At Risk

Washington 10th graders with the risk factor “Academic Failure” have higher rates of substance use than students without that risk factor.

At Risk for “Academic Failure”?

New Questions in 2014

Social/emotional learning

Marijuana questions

Texting and driving

Sexual orientation

Anxiety/worrying

Gangs at school

Stopping/reporting bulling

HYS Background & Administration

Past youth surveys

U Student Alcohol and Drug Use Survey W Washington State Adolescent Health Behaviors Y Youth Risk Behavior Survey H Healthy Youth Survey

U U W W W Y W H H H H H H

88 90 92 95 98 99 00 02 04 06 08 10 12 14

H

Statewide support for HYS

+ Schools, ESDs, Local Health & Community Partners!

2014 HYS participation

Over 223,000 students…

In all 39 counties…

• In 215 school districts…

• In 989 schools took HYS

Small School District Pilot

Could survey extra grades 7, 9 and 11

186 districts were eligible as small (150 or less in a grade)

All schools in the district could participate

Could get combined Middle (6/7/8) & High School (9/10/11/12) results

Goal = increase the number of respondents, smaller confidence intervals, more stable results

Pilot participation

81 districts and 141 schools in those districts participated

101 schools received combined middle and/or high school results

Interleaved survey forms for Survey

forms for 8th, 10th and 12th graders

Also for 9th and 11th

graders in the Pilot

Optional questions A-

enhanced and B-

enhanced

Survey form for 6th

graders

Also for 7th graders in the Pilot

Survey form content

Form A ½ students

Core All

students

Form B ½ students

Form C All 6th graders

Using HYS Data

• How have you used HYS data in your school or community?

• What issues are you seeing in your school or community?

• What would you like to look at in your data or have you looked at to understand these issues?

Why is HYS important?

“Why do we do this survey?

Because students are willing to share what’s going on in their lives.

This isn’t filtered through parents or educators; it’s coming directly from students. And what they tell us is very important.

It gives us insights into what they’re thinking and what concerns them. From that, we are able to provide relevant programs and

services that can help them be more successful.”

- Randy Dorn, Superintendent of Public Instruction

HYS Press Release March 2015

Statistical Issues: Fundamentals & Understanding Your Results

Common questions you may get about HYS

1. Are youth telling the truth? Can we trust our HYS results?

2. Why take student instruction time to do a survey?

• What are common questions/concerns you get about the HYS???

What is reliability?

The extent to which a measure, procedure or instrument yields the same result on repeated trials. A survey item is reliable if it consistently produces the same results under the same circumstances.

Administration Procedures:

• Standardized administration procedures

• Importance of survey

• No names

• Only student sees their answers

• Students resource list

What is validity?

The degree to which the results are likely to be true, believable, and free of bias and can be generalized to a larger population. A survey item is valid if it accurately measures the concept it is intended to measure.

HYS uses questions from established surveys.

Data Cleaning Procedures, that remove respondents with:

• Inconsistent answers

• High level of substance use (all substances, every day)

• Dishonest

• Wrong grade

Generalizability

• Can we “generalize” from the survey to the population?

• Challenges to generalizability:

o Response rates (why 70%?)

o Are different for schools, districts, counties

o Need to consider how these challenges affect interpretation of the results

Do I need to generalize?

• Yes, if you want to apply the results to a larger population. o 8th graders in our district said….

• Yes, if you want to compare to others or results over time

• No, if you want to just describe the students surveyed, in that moment, without confidence intervals: o Students at our school who took the survey said….

What are Confidence Intervals?

• The confidence interval is the ± number after each item estimate

• A confidence interval accounts for the fact that the reported value is probably a little different than the true value for all of the students

• A 95% confidence interval, for example, means that we are 95% confident that the true value is within the ± range

• Confidence intervals are important when you generalize results to a larger population.

What do CIs look like?

Your survey results say 18.0% (± 2.0%) of Grade 10 students used marijuana.

18.0 – 2.0 = 16.0

18.0 + 2.0 = 20.0

Interpret as between 16.0% and 20.0% used marijuana and looks like:

0 5 10 15 20 25 30

CurrentMarijuana Use

Your survey results say 18.0% (± 5.0%) of Grade 10 students used marijuana.

18.0 – 5.0 = 11.0

18.0 + 5.0 = 23.0

Interpret as between 11.0% and 23.0% used marijuana and looks like:

0 5 10 15 20 25 30

CurrentMarijuana Use

What do CIs look like?

Why are CIs different sizes?

The size of a confidence interval is affected by: • Number of students.

o In general, the more students surveyed, the smaller the confidence interval.

• Inherent variability.

o If most students answer a survey question in the same way, then there is less variability. The more variable the answers, the wider the CIs.

Why do we need CIs if data are valid?

• CIs account for variability, NOT the validity of the data.

• CIs allow for the comparison of results to others and

to ourselves over time.

• To protect you---say, from exaggerated claims.

Determining statistical significance

• The probability that differences in results are not due to chance alone.

• When using 95% confidence intervals, a difference between two groups is considered statistically significant if chance could explain it only 5% of the time or less.

Comparing CIs: significant

Local students 25% ±5% (so range is 20% to 30%)

State students 36% ±3% (so range is 33% to 39%)

CI’s don’t overlap, so difference is significant

0 5 10 15 20 25 30 35 40

Local State

20% 30%

33% 39%

Comparing CIs: non-significant

Local students 25% ±5% (so range is 20% to 30%)

State students 28% ±3% (so range is 25% to 31%)

Local CI range overlaps and overlaps the point estimate, so the difference is not significant

0 5 10 15 20 25 30 35 40

Local State

20% 30%

25% 31%

Point estimate, 28%

CI overlap

0 5 10 15 20 25 30 35 40

Local State

Comparing CIs: inconclusive

Local students 25% ±5 (so range is 20% to 30%)

State students 32% ±3 (so range is 29% to 35%)

CIs overlap, but don’t include a point estimate, so you can’t tell if the difference is significant

20% 30%

29% 35%

Point estimate, 32%

CI overlap

Practice using confidence intervals

If you don’t have your own report, use these results

26.9% to 31.9% 35.4% to 43.8%

Tool for testing significance

If you need to know for sure, there is a “Tool” to test for significance at: www.AskHYS.net/Training The spreadsheet tests the difference between two point estimates and their 95% CI to compute a p-value. • If your p-value is less than 0.05, then your difference

is significant. • Only use this test if you have at least 30 students. • Don’t use this test if you have 0% or 100%

Example – Comparing to the state

Since we couldn’t tell if there was a difference by using our CIs – let’s try the “Tool”: • Your students: 25.0% (±5.0) • Students statewide: 36.0% (±3.0)

P-value is less than 0.05, so there is a significant difference.

25.0

36.0

5.0

3.0

Your students

Statewide

0.0002177

Using results with large confidence intervals

If you are a small school, district or county your CIs will be large... • The estimate may be good, but only of the

students who took the survey this year.

CIs don’t take participation rates into account: • Low participation – there may be additional

bias • High participation – you can feel more

confident in your point estimates.

Questions?

Getting Access to AskHYS

Levels of access

County, ESD and State results are available to everyone.

School district and school building HYS results are available only with the permission of the superintendents’ offices.

Log on for school district and building access

Use your EDS login and password

Contact your District Data Security Manager for appropriate permissions to your building or district data

For district and building access

susan@rainiertheory.com

********

Successful login

Frequency Reports

Opening a frequency report www.AskHYS.net/Reports

Frequency report title and sections

• Introduction & Overview • Highlights • Selected Results by Gender • Understanding Your Report • Frequency Results • Risk & Protective Factor

Scales • Risk & Protective Factor

individual questions • Item Index • Core Questions

Intro - Survey participation

Submitted a survey

Submitted a “usable” survey

Students enrolled

Enrolled ÷ usable surveys

Pilot Combined High School Report participation rates

Report highlights

Selected results by gender

Smallest “n” example

A minimum n = 40 is needed if:

50% of the local respondents are

female and 50% are male

50% X 40= 20 females

50% X 40= 20 males

50.0% of females smoke

cigarettes and 50.0% of males

smoke cigarettes

50.0% X 20 = 10 females

50.0% X 20 = 10 males

% in each cell

Female Male

None 50% 50%

1 or more 50% 50%

n in each cell

Female Male

None 10 10

1 or more 10 10

Understanding Your Report

“N” is shorthand for “number” --- the number of students who took this survey or who answered a survey question. There are multiple reasons why different questions have different “N’s”: • Question is only on Form A or B • Question is optional • Question is near the end of the survey • Question is difficult to answer, or too

personal

Samples of n’s from different survey questions for a district with 400 10th graders:

N for a core question (all/most students)

N for question on Form A only (half the students)

N for optional question only on Form B (half the students, and some schools requested surveys without option questions)

N for near the end of the survey

17. Have you ever, even once in your life: Used marijuana? (n=398)

a. Yes 49.0% (±3.0)

b. No 51.0% (±3.0)

18. Have you ever, even once in your life: Used inhalants? (n=200)

a. Yes 11.0% (±4.0)

b. No 89.0% (±4.0)

100. How old were you when you had sexual intercourse

for the first time?† (n=100)

a. I have never had sexual intercourse 70.0% (±8.0)

b. 11 yhears old or younger 2.0% (±2.0)

c. 12 years old 2.0% (±2.0)

d. 13 years old 5.0% (±2.0)

e. 14 years old 8.0% (±6.0)

f. 15 years old 9.0% (±5.0)

g. 16 years old 3.0% (±3.0)

h. 17 years old or older 1.0% (±1.0)

150. Does your school have a counselor? (n=375)

a. Yes 90.0% (±3.0)

b. No 5.0% (±2.0)

c. Not sure 5.0% (±2.0)

Frequency results

18.0% (±2.0) 26.7% (±2.2)

82.0% (±2.0) 73.3% (±2.2)

10.0% (±1.0) 8.5% (±1.0)

6.0% (±1.0) 5.3% (±0.6)

1.0% (±0.5) 2.6% (±0.5)

1.0% (±0.5) 10.3% (±1.3)

(n=400) (n=9,000)

a. None

b. 1-2 days

c. 3-5 days

d. 6-9 days

e. 10 or more days

Any use in past 30 days

29. Use marijuana or hashish (weed, hash, pot?)

Between 16% and 20%:

• 18% - 2% = 16%, low

• 18% + 2% = 20%, high

During the past 30 days, on how many days did you:

Other frequency results

Pilot Multi-grade Middle School results

10.0% (±4.0) 8.0% (±1.0)

90.0% (±4.0) 92.0% (±1.0)

5.0% (±3.0) 3.5% (±1.0)

3.0% (±2.0) 2.5% (±0.5)

1.0% (±0.5) 1.0% (±0.2)

1.0% (±0.5) 1.0% (±0.2) 1

(n=100) (n=300)

a. None

b. 1-2 days

c. 3-5 days

d. 6-9 days

e. 10 or more days

Any use in past 30 days

29. Use marijuana or hashish (grass, hash, pot?)

2.0% (±2.0) 4.0% (±3.0)

98.0%(±2.0) 96.0% (±4.0)

2.0% (±2.0) 2.0% (±1.0)

0.0% (±0.0) 2.0% (±1.0)

0.0% (±0.0) 0.0% (±0.0)

0.0% (±00) 0.0% (±0.0) 1

(n=100) (n=100)

Pilot combined Middle School results

Grade 6 Grade 7 Grade 8 MS Combined

% (±CI) % (±CI) % (±CI) % (±CI)

8.0% (±1.0) 5.2% (±0.9)

92.0% (±1.0) 94.8% (±0.9)

3.5% (±1.0) 2.3% (±0.4)

2.5% (±0.5) 0.9% (±0.2)

1.0% (±0.2) 0.6% (±0.2)

1.0% (±0.2) 1.4% (±0.3) 1

a. None

b. 1-2 days

c. 3-5 days

d. 6-9 days

e. 10 or more days

Any use in past 30 days

29. Use marijuana or hashish (grass, hash, pot?)

Your Students Small Statewide

% (±CI) % (±CI)

Why are there asterisks?

• Results suppressed at school level to protect anonymity

• School chose not to ask optional question • Results by gender require a minimum number of respondents

Risk and protective factors

Each “factor” is a “scale” measured with two or more questions, so that the factor includes multiple dimensions of the risk or protection.

Risk factors that predict youth substance use, violence and delinquent behaviors. Protective factors that can protect youth from the effects of those risks.

Prevention strategies can be designed to lower risk and increase protection.

E.g., Academic Failure What grades do you usually get How far do you think you will get in school

Risk & Protective Factors Over Time

Report extras

Questions by topic

Core questions

PowerPoint Slides

Opening PowerPoint slides

Participation rates provided

Example of a typical slide

Example of a Pilot typical HS slide

Fact Sheets

Opening a fact sheet

Opening Fact Sheets

2014 Fact Sheet chart

Trend chart and table

State comparison chart

*

Pilot state comparison MS chart

*

Statewide relationship charts

All grade Fact Sheet charts

All grade Fact Sheet statewide relationship charts

Comparisons over time

A few things to think about first:

• Did the survey question change?

• Were challenges to generalizability similar? o Response rates o Groups of students missing

• Do you expect a change?

Trends

• Fact sheets do provide data back to 2004 for some questions – but do not provide a test for overall change in trend (only year to year comparisons)

• To measure trends you need statistical software o Free software from the National Cancer

Institute – uses your point estimates and confidence intervals:

http://surveillance.cancer.gov/joinpoint

Questions?

QxQ Online Data Query System

Q x Q = Question by Question

Running frequencies and crosstabs • Run frequencies: responses to individual

questions. Just like the results in your Frequency Reports.

• Run crosstabs: looking at the relationship between two questions - crossing one question by the second question.

Running crosstabs – HYS requirements

Cell size • State level minimum 5 per cell. • Sub-state level minimum 10 per cell. **NOTE** • The squares that have results in them are the

“cells”. • The number of respondents in each cell is

called the “n”, or also the “cell size”.

Crosstabs – cell size error message

“At least one cell in the results table contained a count of less than 10.” Output is suppressed.”

Cell size limitations can be frustrating! If your results are suppressed because of cell

size: • Select the “collapsed” answer option instead of

the “surveyed” option. • Re-run your crosstab at a higher geography – like

county rather than district.

Opening the Q x Q www.AskHYS.net/Analyzer

Opening the Q x Q www.AskHYS.net/Analyzer

Running a frequency - question selection

Frequency - response selection

Frequency - drag and drop

Frequency output

Running a crosstab – question selection

Alcohol and marijuana example Row Variable = Current Alcohol, Column Variable = Current Marijuana

Year=2014, Grade=8 Among 8th graders who didn’t drank: • 3.6% used marijuana (cell 2)

Among 8th graders who drank: • 49.4% used marijuana (cell 4)

Current Alcohol Drinking and Current Marijuana Use

96.4% ±0.7%

9,1850

3.6% ±0.7%

345

50.6% ±3.9%

421

49.4% ±3.9%

411

100.0%

9,530

100.0%

832

2014

Correct way: Race and bullying

Row Variable = Race, Column Variable = Bullying Year=2014, Grade=8

Among White non-Hispanics: • 29.1% were bullied

Among Hispanics: • 26.3% were bullied

Ra

ce

Bullied

Wrong way: Race and bullying

Row Variable = Bullying, Column Variable = Race, Year=2014, Grade=8

Among White non-Hispanics: • 51.3% were bullied

Among Hispanics: • 16.5% were bullied

Bu

llie

d

Race White non-

HispanicHispanic

American

Indian or

Alaska

Native non-

Hispanic

Asian or

Asian

American

non-Hispanic

Black or

African

American

non-Hispanic

Native

Hawaiian or

other Pacific

Islander non-

Hispanic

Other non-

Hispanic

Multiracial

non-HispanicTotal

48.7% 18.1% 3.0% 11.0% 4.1% 1.7% 7.6% 5.8% 100.0%

± 5.8% ± 5.3% ± 0.7% ± 3.5% ± 1.2% ± 0.4% ± 0.8% ± 0.8%

3,672 1,363 226 831 312 130 572 434 7,540

51.3% 16.5% 4.0% 7.0% 7.0% 1.5% 8.8% 7.0% 100.0%

± 5.0% ± 5.1% ± 0.9% ± 2.2% ± 1.0% ± 0.5% ± 1.2% ± 1.4%

1,509 486 117 205 114 45 259 205 2,940

no

days

any

days

Check you Q x Q results

You can compare your Q x Q frequency results with your Frequency Reports (www.AskHYS.net/Reports) • Page 4 in your reports includes crosstabs by gender Ask someone to review your results **NOTE** There may be slight differences because of the way different statistical programs round numbers.

Questions?

Crafting a Communication Message Around HYS

Talking about survey results

Keep it simple Round to a whole number

Survey results are estimates

Say “about”

Acknowledge low participation – “among those who took the survey”

Carefully include your confidence interval, if at all

If 75.3% (±5), between 70 and 80 percent

Plus or minus 5 percent

Different ways to talk with numbers

Painful: 75.3% (±5.1) of 8th graders feel safe at school.

Less painful: About 75 percent

About 3/4ths

About 3 out of 4

Turn it around, if that is the point you need to make: About 25% of 8th graders don’t feel safe

1 in 4

If you have 200 8th graders, turn it into the number of students, about 50 of our 8th graders don’t feel safe

Use visuals

There are charts in your PowerPoint slides and fact sheets

Create other types of charts or visuals:

7 ½ out of 10 students…

Where do you start?

Know your facts about HYS

How, when and to whom it was administered

Details about the questions

What information do you have?

HYS reports, slides, fact sheets

What do your 2014 HYS results say?

What HYS questions address your topic?

How do your results compare to the state results or results from previous years?

Where do you start?

Do the numbers make sense?

What else is needed to tell the whole story? Additional data sources

Information from informed people

Define your audience

Ask yourself: “Are there specific opinions I need to influence?”

Administration

Parents

Reporters

Grant application reviewers

What’s your Single Overriding Communication Objective (SOCO)?

This is the “so what” of your message. Ask yourself:

“If I could only make one point, what would it be?”

Tells your audience what you want them to learn or do

Practice: Using your results

Pick a scenario below and use HYS on page 60 of your Workbook to develop a SOCO or a key message.

The school board has invited your team to talk with them and make your case about the need for more early prevention programs among youth.

The son of a prominent city council member was arrested for driving under the influence of marijuana. He attends the local high school. A radio station calls and asks you to provide a sound bite about marijuana use among youth in your community.

A local agency is offering funds for programs that support youth. Support your grant application for a mentoring program.

Share Your Message

Share your results with the group

Example of SOCOs

“The school board needs to take action to prioritize marijuana prevention programs for our middle school children”

“Every parent in our community should talk to their kids about marijuana and driving”

“Funding this program will reduce the excess danger that children in our community face from marijuana”

What ideas do you have for using HYS data after leaving here today?

Reflection

Questions?

Thank you!

From the Healthy Youth Survey Planning Committee

• DASA-DBHR: Martha Perla, Linda Becker, Steve Smothers,

Grace Hong

• OSPI: Krissy Johnson, Dixie Grunenfelder

• DOH: Kevin Beck, Anar Shah, Lillian Bensley, Cathy Wasserman, Riley Peters

• Liquor Control Board: Mary Segawa

• Looking Glass: Joe Kabel and Susan Richardson

Questions?

About this training:

Krissy Johnson: Krissy.Johnson@k12.wa.us

Susan Richardson: susan@rainiertheory.com

Other HYS questions:

healthy.youth@doh.wa.gov

www.AskHYS.net