Pilot STEM in OST Evaluation

Preliminary Report

March 2012 University of California , Irvine

Study Sample Selected• 17 programs in 9 Regions participating

Pilot Evaluation Study• Regions 1, 2, 3 4, 7, 8, 9, 10, 11

• Total of 65 sites recruited• 52 Elementary• 9 Middle School• 4 K-8

• Criteria for Selecting Study Sample:

Range of STEM curriculum approaches Range of student age groups (grades 3-12) Diverse ethnic and socio-economic backgrounds,

representative of students in the State of California Internet access

Overview of Evaluation Activities:

Selected Study Sites engage in the following activities:

Administer pre (fall 2011) and post (spring 2012) online student surveys

Administer online staff surveys at two time points: pre (fall 2011), and end-of-year (spring 2012)

Document Weekly Stem Activities—to be recorded daily by site implementers on STEM Activity Documentation Forms

Provide UC Irvine with copies of program schedule and lesson plans

Pilot Study Launch Fall 2011ACTIVITIES TO DATE:

Selection of pilot study sample sites

Set up of staff pre-survey and student pre-survey online links

Communication with pilot study sites:

1. Email sent informing Program Liaison of selected sites and overview of evaluation activities to be carried out

2. Individual emails sent to STEM Implementers at each study site including:1) Individual staff IDs assigned & instructions distributed for staff

surveys2) Site IDs assigned & instructions distributed for student

surveys3) STEM Activity Documentation Form (electronic and hard

copies) distributed with instructions & envelopes for returning forms to UC Irvine

3. Packages sent to each study site or program liaison with:1) Hard copies of STEM Activity Documentation Forms and

instructions2) Prepaid and addressed envelopes completed forms to UC

Irvine

Winter 2012 Evaluation Activities

Follow-up Communications & Reminders:1. Program liaisons sent a list of study sites and

corresponding staff IDs and site IDs Nov. 30/Dec. 1, 2011

2. Reminders sent to Program Liaisons & STEM Implementers• Reminders to complete surveys and begin

sending STEM Activity Documentation Forms

• Accounting of data collected from their program sites

• All site and staff codes and survey instructions resent with email

Ongoing contact with program Program Liaisons & STEM Implementers by phone and email

Data Collected to Date—March 15 2012

90 staff pre-surveys have been completed

35 sites have completed 1,277 student pre-surveys

Preliminary results of Staff and Student Pre-Surveys summarized in slides that follow

103 STEM Activity Documentation forms received reporting on 310 Individual Stem Activities Items reported by STEM implementers about each activity include:

Date and Duration of Activity Name of activity STEM content area addressed Number of students and grade level 4 point ratings of

1. Level of student Engagement

2. Level of challenge

3. Overall assessment of success of activity

Data are being entered in the data base for analysis and will be correlated with staff survey data and student survey site level data.

Summary STAFF Pre-Survey Data

Data collected December 2011–March 2012

SUMMARY OF STAFF PRE-SURVEY RESULTS

90 staff surveys collected between November 2011 and March 2012

73% respondents female.

1/3 are between 18 and 25 years old.

72% are 35 or under.

18-2526-3536-4546-55Over 55

AGE OF STAFF

STAFF SURVEY—LEVEL OF EDUCATION ACHIEVED

Nearly half (48%) have attended college or have an AA degree

42% have B.A. Degrees or higher

High School Diploma or GED

Attended classes/training not related to a degree

Attended college

Completed two-year college degree (AA)

Completed four-year college degree (BA)

Attended graduate school

Completed Master’s degree

Completed Doctoral degree

0.00% 5.00% 10.00% 15.00% 20.00% 25.00% 30.00% 35.00%

3%

1 %

29%

19%

29 %

12 %

7%

0.00%

Level of Education n = 90

16%1%

7%

1%

36%3%

34%

2%

Staff Ethnicity (n = 90)

African, African-American

American Indian or Alaskan Native

Asian, Asian-American

Filipino

Hispanic or Latino

Pacific Islander

White, Caucasian

Other

Staff Reflect the Diversity of Students and Communities Served

STAFF POSITION IN AFTERSCHOOL PROGRAM

Less than 6 mo.

6-12 mo.

1-2 years

2-3 years

3-5 years

5-9 years

10 years or more

0% 10% 20% 30%

24%

16%

15%

12%

21%

12%

0%

Time in Current Po-sition

(n =90)

54%42.5%

3.5%

Current Position (n = 87)

Site Coordina-torActivity Leader at one siteProgram Coach work-ing with several sites

40% staff have less than one year in current position33% have more than 5 years in position27% have 1-3 years in position

STAFF EXPERIENCE IN OTHER SCHOOL SETTINGS62% have some experience as a classroom aide or TA

23% 1-5 years 14% more than 5 years

37% reported having some classroom teacher experience

15% 1-5 years 10.5% more than 5 years

15% have some school administrative staff experience

8% student support staff experience

Few staff report any administrative experience (6%)

Professional Experience

n < 6 mo 6-12 mo 1-2 yrs 2-5 yrs 5-10 yrs 10+ yrs

School Administrator

74 3% 0 0 3% 0 0

Student Support Staff

74 1% 0 3% 1% 0 3%

Administrative Staff 76 3% 3% 5% 4% 0 0

Classroom Teacher 79 2.5% 9% 4% 11% 2.5% 8%

Instructional Specialist

75 4% 3% 3% 9% 3% 3%

Classroom Aide, TA 84 14% 11% 11% 12% 6% 8%

Staff Self-Identified Instructional Roles

Science

Engineering

Reading/Language

Arts

Athletics

Other

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

53%

31%

16%

67%

66%

37%

57%

81%

23%

Instructional Roles (n = 90)

Variety of STEM Activities Implemented

18%

15%

9%

11%

16%

9%

22%

Type ImplementedLife Science

Physical Science

Earth Science

Technology

Information & Commu-nication Technologies

Engineering

Mathematics

Prompt: In your current position, do you implement any STEM activities with students? If so, please specify which types of activities (select all that apply):

None

15-30 min.

30-60 min.

1-2 hours

2-3 hours

3-5 hours

5-10 hours

10 hours or more

0% 5% 10% 15% 20% 25% 30%

22%

10%

25%

24%

11%

3%

3%

2% n = 90

Prior Experience & Time per Week Implementing STEM

Prompt: In your current position, how much time per week do you spend implementing STEM activities with students?

22% spend NO time implementing STEM Nearly half (49%) do at least 30 minutes to 2 hours of

STEM per week

30 % have no prior experience implementing STEM at another program

Staff Meetings Around STEM

Never Less than once a month

Once a month

2-3 times a month

Once a week

2-3 times a week

4-5 times a week

0%

5%

10%

15%

20%

25%

30%

0%

5%

18%

28%27%

13%

9%

21%19%

23%

13%

19%

4%

0%

Program IssuesSTEM Issues

21 % report never discussing STEM at staff meetings

55% discuss STEM in staff meetings at least once a month to once a week

n =90

Frequency of Staff Discussion of Program and STEM Issues

Compensation for Staff Meetings

( n = 89)

$ 66% staff report being compensated for all meetings

$ 10% for most meetings

$ 7% for some meetings

$ 15% receive no compensation for meetings

2% staff report having no meetings

Training and Support

0%

10%

20%

30%

40%

50%

60%

70%

11%

62%

16%

7%2% 2%

42%

54%

4%0% 0% 0%

Any Training

A little over half (52%) of staff surveyed have had at least 1-4 sessions of Stem Related Training in the past academic year

42 % have had NO Stem Training

n = 90

STAFF RELATIONS WITH TEACHERS

Curriculum concepts being taught in school

STEM concepts being taught in school

Homework assignments

Needs or progress of individual student

Issues related to classroom space

8%

32%

3%

6%

22.00%

20%

29%

14%

9%

26%

32.00%

14%

11%

16%

17%

11%

11%

17%

18%

12%

7%

4%

26%

26%

9%

> 1x/wk1x/wk

2-3x/mo1x/mo

n = 90

During the past academic year, how often have you discussed…

32% never speak with classroom teachers about STEM concepts taught in classroom

STAFF RELATIONS WITH PARENTS

Talk with parents about STEM activities

Hold STEM-related events for parents

34%

76%

37%

17%

17%

7%

6%

> 1x/wk1x/wk2-3x/mo1x/mo< 1x/moNever

1%

n = 89

76% of staff report never holding STEM-Related events for parents

36% say they never speak with parents about STEM activities

Staff Outcome Measures Beliefs & Sense of Competency

n Range Mean SD Alpha

Staff Beliefs about STEM

90 1.60-4 2.85 0.49 .73

Staff Sense of Competency

89 2-4.17 3.12 0.46 .79

Two staff outcome measures:5 point rating scale

o strongly disagree, disagree, neither agree or disagree, agree, strongly agree [some reversed coded]

Staff beliefs about STEM in the afterschool program. Example Items

I think the students enjoy doing STEM Activities In general, I think these students [in the afterschool program] are very

capable of doing hands-on science activities I don’t think there is enough time at the program for students to learn

much about STEM

Staff sense of competency implementing STEM in the afterschool program Example Items

I have a strong background in at least one area of STEM I do not know enough about Science, Technology, Engineering and/or

Mathematics to teach any of them well I feel confident about teaching Science, Technology, Engineering and/or

Mathematics in the afterschool program

Summary STUDENT Pre-Survey

DataData collected

December 2011–March 2012

Student Survey Sample

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

51% 49% 51%56% 58%

47%49% 51% 49%44% 42%

53%

BoysGirls

1,277 in Full Sample

51% Boys and 49% GirlsGrade and Gender of Student

Respondents

Time to Complete Survey

Full sample Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8

10% 11%8%

11%8%

13%

6%

19%

14% 13%

23%25%

31%

60%

22%

27%

20%17%

26%28%

20%

10%

14%

11%8%

6%

2% 3%

39%

34%

48%

42%

34%

26%

11%

< 5 min 5-9 min 10-15 min > 15 min Don’t know

51% students overall report taking 15 minutes or less to complete survey

55% Middle School & 28% Elementary students report taking 9 minutes or less

Survey taking Experience

6% 4% 4%

12% 12% 11%

19.17%26.% 25%

64%58% 60%

Not at all true

A little true

Mostly true

Really true

Full sample: n = 1200

The majority of students find the surveys Easy to Read, Understand

and Answer• 83-85% state “mostly true” or “really true”

STUDENT OUTCOMESPre-Survey Results

1,277 surveys completed by students grades 3-8

Eight scales:1. Work Habits

2. Misconduct

3. Social Competencies

4. Math Efficacy

5. Science Efficacy

6. Interest and Engagement in STEM

7. Future Outlook

8. Science Career Aspirations

Work Habits

Full Sample Grade 3 (n = 339)

Grade 4 (n = 374)

Grade 5 (n = 356)

Grade 6 (n = 132)

Grade 7 (n =40)

Grade 8 (n = 36)

0

0.5

1

1.5

2

2.5

3

3.5

2.91 2.99 2.92.74 2.64

2.42 2.42

Mean Sc...

Work Habits—mean of 6 items Assessed on a 4-point scale (1 = not at all true, 4 = really true)Sample items include:

“I work well by myself” “I finish my work on time.”

Full Samplen = 1,277SD = .67Alpha = .81

Misconduct

Full sample

Grade 3 (n =

334)

Grade 4 (n =

364)

Grade 5 (n =

350)

Grade 6 (n =

131)

Grade 7 (n =

40)

Grade 8 (n =

36)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.44 0.440.37

0.450.5 0.51

0.66

Full Samplen = 1,255SD = .52Alpha = .85

Misconduct—mean of 9 itemsAssessed on a 4-point scale: (0 = never, 3 = more than once a week)

• Lower score means LESS misconduct Sample items include:

“I have gotten into a fight at school” “I have taken something that belongs to someone

else.”

Full sample

Grade 3 (n =

331)

Grade 4 (n =

358)

Grade 5 (n =

349)

Grade 6 (n =

131)

Grade 7 (n =

39)

Grade 8 (n = 36)

2.75

2.8

2.85

2.9

2.95

3

3.05

3.1

2.95

2.912.93

2.99 3

2.86

3.08

Mean Sc...

Full Samplen = 1,244SD = .66Alpha = .73

Social CompetenciesSocial Competencies—mean of 7 itemsAssessed on a 4-point scale: (1 = not at all true, 4 = really true)Sample items include:

“I work well with other kids” “I can tell other kids what I think, even if they disagree with me.”

Full sample

Grade 3 (n =

327)

Grade 4 (n =357)

Grade 5 (n =

349)

Grade 6 (n =

131)

Grade 7 (n =

39)

Grade 8 (n =

36)

2.5

2.6

2.7

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

3.273.32

3.37

3.28

3.13

2.812.91 Mean Sc...

Full Samplen = 1,239SD = .80Alpha = .84

Math EfficacyMath Efficacy—mean of 4 itemsAssessed on a 4-point scale: (1 = not at all true, 4 = really true)Sample items include:

“I expect to do well in math” “I am interested in math.”

Full sample

Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 82.5

2.6

2.7

2.8

2.9

3

3.1

3.2

3.3

3.4

3.13 3.13

3.27

3.12

2.892.94

2.76

Mean Score

Full Samplen = 1,239SD = .80Alpha = .84

Science EfficacyScience Efficacy—mean of 4 itemsAssessed on a 4-point scale: (1 = not at all true, 4 = really true)Sample items include:

“I expect to do well in science” “I am interested in science.”

Full sample

Grade 3 (n =

312)

Grade 4 (n =

355)

Grade 5 (n =

348)

Grade 6 (n =

131)

Grade 7 (n =

39)

Grade 8 (n = 35)

0

0.5

1

1.5

2

2.5

3

3.5

2.89 2.91 2.99 2.92.74 2.64

2.42

Mean Sc...

Full Samplen = 1,220SD = .56Alpha = .93

Excited, Engaged and Interested Science Learner

[PEAR-Harvard]Excited, Engaged & Interested Science Learner—mean of 24 itemsAssessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)

• Some items reverse codedSample items include:

“Science is something I get excited about.” “I am curious to learn more about science, computers or

technology.” “Science is boring” “I pay attention when people talk about recycling to protect our

environment.” “I enjoy visiting science museums or zoos.”

Future Outlook

Full sample

Grade 3 (n =

307)

Grade 4 (n =

354)

Grade 5 (n =

347)

Grade 6 (n =

130)

Grade 7 (n =

39)

Grade 8 (n =

35)

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.61

3.44

3.63

3.74

3.61

3.54

3.63

Mean Score

Full Samplen = 1,212SD = .56Alpha = .93

Future Outlook—mean of 7 itemsAssessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)Sample items include:

“I will go to college.” “I will have a job that I enjoy doing.”

Science Career Aspirations

Full sample

Grade 3 (n = 302)

Grade 4 (n = 354)

Grade 5 (n = 347)

Grade 6 (n = 130)

Grade 7 (n = 39)

Grade 8 (n = 35)

2.4

2.5

2.6

2.7

2.8

2.9

3

3.1

2.92

3.03 3.04

2.81

2.68

2.79

2.65

Mean Sc...

Full Samplen = 1,207SD = .82Alpha = .85

Science Career Aspirations—mean of 4 itemsAssessed on a 4-point scale: (1 = strongly disagree, 4 = strongly agree)Sample items include:

“I will have a career in Science, Technology, Engineering, or Mathematics.”

“I will graduate with a college degree in a major area needed for a career in science.”

Next StepsInitiate Post-Survey administration: April 15, 2012-June 15, 2012

Ensure staff have received information, instructions and forms for completing Activity Documentation Forms

UC Irvine to Request sample copies of site schedule and weekly lesson plan

Final analysis of pre/post student and staff outcome measures and program implementation data

Refinement of measures for 2012-13 STEM in OST evaluation study