Noyce Program Evaluation Conference

Thursday, December 6, 2007

Frances LawrenzMichelle Fleming

Pey-Yan LiouChristina Madsen

Karen Hofstad-Parkhill

1

Outline of Today’s Presentation

• Overall goals for our project• Data overview

– Scholar survey data– Scholar interview data– PI survey data– Scholar and PI survey comparison data

• Your additional questions and suggestions

2

Overall Goals for Our Project

• To contribute to the knowledge base about effective strategies for attracting and retaining high quality STEM teachers

• To collaboratively develop a plan to evaluate the Noyce Program that will document overall program accomplishments while celebrating the uniqueness of each project

• To conduct the evaluation and disseminate findings in a utility-oriented fashion

3

4

Methodology

Data sources:• Monitoring data collected for NSF • PI online survey (N=65)• Scholar online survey (N=557)• Faculty on line survey (in progress)• Interviews of scholars and districts (in progress)

Analyses: Frequencies, %s and cross tabs

Bias: Voluntary response, low number of “leavers”

5

Overview of Included Data • Selected charts about the influence of the Noyce Funding

• Scholar yes no items by group and year: items about cultural and high needs education, mentoring opportunities, educational experiences, and the influences affecting the decision making process.

• Scholar Likert items by group and year: items about teaching ELL students, choosing the same teacher preparation program again, over 50% of students receive free or reduced lunch in their school, the influence of the Noyce money on completing the certification program and taking a teaching job.

• PI by type of program and year: items about activities the Noyce funding supports, different screening used for selecting Noyce scholars, required field experiences and hours, and how integral the Noyce scholarship is to the teacher preparation program

• PI scholar comparisons on 16 items: items about cultural and high needs education, mentoring opportunities, and educational field experience

6

Scholar Survey• Project overview – provides a broad description of their teacher

certification program

• Program characteristics and organization – provides more details about the opportunities and requirements of their teacher certification program

• Teaching environment and experience – provides details of their teaching setting, and their feeling about that setting

• The decision to become a teacher – explores the timing and factors that led them to pursue a career in education

• Background and experience – provides details about their previous work and education

• Overall experience – open-ended responses that explore some of these issues in more depth.

7

Selected Charts Showing the Influence of the Noyce Funding

• Timing of learning about Noyce Funding

• Effect of funding by timing of learning on decision to be a teacher

• Effect of funding by timing of learning on decision to teach in a high needs school

• Differences on timing and effect by career changer status

• How learn about funding

• Attitudes and school characteristics

• Influence of funding on commitment to Noyce goals

8

Did you first learn about the Noyce Scholarship before or after you decided to become a teacher?

*Scholar survey Section IV:4

Timing of Learning about the Noyce Funding

9

Timing of Learning about the Noyce Funding by Year

*Scholar survey Section IV:4

Did you first learn about the Noyce scholarship before or after you decided to become a teacher?

(percentage of scholars’ response by year)

10

Would you have become a teacher if you had not received the Noyce scholarship? (Scholars said ‘Before’ in the previous question )

*Scholar survey Section IV:6

Effect of Funding on Decision to Become a Teacher (Before Group)

11

Would you have become a teacher if you had not received the Noyce scholarship? (Scholars said ‘After’ in the previous question)

*Scholar survey Section IV:6

Effect of Funding on Decision to Become a Teacher (After Group)

12

Would you have decided to teach in a high needs school if you had not participated in the Noyce scholarship program? (Scholars said ‘Before’ in the previous question)

*Scholar survey Section IV:7

Effect of Funding on Decision to Teach in a High Needs School (Before Group)

13

Would you have decided to teach in a high needs school if you had not participated in the Noyce scholarship program? (Scholars said ‘After’ in the

previous question)

*Scholar survey Section IV:7

Effect of Funding on Decision to Teach in a High Needs School (After Group)

14

Timing of Learning about Funding for Career Changers

Did you first learn about the Noyce Scholarship before or after you decided to become a teacher?

(Analysis for those who said they were career changers.)

*Scholar survey Section V:4

*Scholar survey Section IV:4 15

Timing of Learning about Funding for Non Career Changers

*Scholar survey Section V:4

*Scholar survey Section IV:4

Did you first learn about the Noyce Scholarship before or after you decided to become a teacher? (Analysis for those who said they were

NOT career changers.)

16

Did any of the following help you learn about the Noyce scholarship?

*Scholar survey Section IV:6

How students learned about the Noyce Funding by Year

17

Other ways student learned about funding…..

• students• friends• professors in the

department of education, mathematics…

• directors of the program• instructors• teachers• emails

• radio advertisement• flyers• newspaper advertisement • bulletin notices• program office• college of education

scholarship database• workshops• financial aid offices

18

Scholars’ Attitudes about Their Jobs and Their Certification Programs

*Scholar survey Section II:6 19

Which of the following characterize your school? (percentage for scholars who responded “applies”)

*Scholar survey Section III:3

Characteristics of Schools in which Scholars are Employed by Year

20

Influence of Funding in Scholar Commitment to Noyce Goals by Year

*Scholar survey Section IV:8

The Likert-scale is as follows:

Very influential:4; Somewhat influential:3; Not very influential:2; Not at all influential:1.

The value in this graph is the mean for each item by year. 21

Scholar Yes-No Survey Data• There were 28 Items

• The % yes ranges from 0 to 100

• There were 15 Items showing significant differences by year

• There were 15 Items showing significant differences by group

• Considerations: small numbers, many chi square analyses, and different populations by item.

22

Questions about the Scholar Yes/No Survey Data

• From 2002-2006, the scholars who are in the certification program and/or teaching full time indicated an increase in education about how to work in high needs schools.

Do you feel the Noyce program is related to this increase?

• According to the data, the scholars (who are working as full-time teachers) indicated that, over the 2002-2006 time span, they have received more mentoring experiences from the districts compared to the certification program.

Do you find this to be true? If so, why do you think that is?

23

Scholar Likert Survey Data• There were 28 Items.

• The means range from 1.00 to 4.64.

• There was 1 Item showing a significant difference by year

• There were 5 Items showing significant differences by group

• Considerations: small numbers, many comparisons, and different populations by item

24

Questions about the Scholar Likert Survey Data

• From 2002-2006, scholars total mean scores on the opportunity to “Develop specific strategies for teaching students from diverse racial and ethnic backgrounds.” has slightly increased from 3.12-3.84 on a 5.00 scale.

– Is this sufficient?– What particular strategies do you promote in your program?

• From 2002-2006, the Noyce money had an increasing impact on the scholars’ decision to remain teaching in a high needs school to fulfill the scholarship commitment.

– How influential is the Noyce money on scholars’ initial commitment to teach in high needs schools?

– Past the initial commitment, do you find the scholars continue teaching in high needs schools?

25

Scholar Interview Model

26

Questions about the Scholar Interview Model

• In terms of your work with the scholars, do you see any connections or relationships between the scholars’ SES and their desire to work in high needs schools or choosing to teach as a career?

• Do you see the Noyce money playing a role in scholars’ decisions to work in high needs schools?

• Would you like the interviews to focus on any particular decisions, factors or other issues?

27

PI Survey• Project overview – provides a broad description of the

teacher certification programs offered at this site

• Noyce money – explores what the money provided by the Noyce grant does and does not do, and the roles it plays in the program most scholars participate in

• Program characteristics and organization – provides more details about the opportunities and requirements of the teacher certification program that most scholars participate in

• Contacts and partners – this information will be used for later investigation by the evaluation team, and is not included here

28

PI Survey Data

• There were 42 yes/no items and 23 Likert items

• The % of yes ranges from 0 to 100

• The Likert means range from 1.28 to 4.00

• No differences by year

29

Questions about the PI Survey• The question “How Integral is the Noyce scholarship funding to your

teacher preparation program,” has changed every year from ,”The program would continue mostly unchanged without the funding” to “The program would be substantially different without the funding.” – Do you think that the projects funded more recently are more affected by the

Noyce funding than those funded initially?

• According to the data, student teaching experiences in a high needs district are offered but not necessarily required by all programs Noyce scholars participate in. – Should these types of experiences be required?

30

Scholar and PI Comparison Data on 16 Survey Items

• The scholars felt the following experiences were more available than the PIs did – Interaction with adults from other cultures– Educational field experiences (not student teaching)– Research field experiences – Student teaching in NON high needs schools

• The PIs felt the remaining 12 experiences were more available than the scholars did

• Limitation was that the PI and scholar response options were slightly different

31

Questions about the Scholar and PI Comparison Survey Data

• According to the data, the Principal Investigators (85 % required 9% offered) and Scholars (82 %) agreed that the scholars receive education about different cultures.– Are these percentages high enough? – If culture education is not provided, how else are these issues dealt with?

• According to the data, 67% of the scholars indicated they received supervised classroom teaching experiences in high needs schools (i.e. student teaching, internship). However, 87% of the PIs indicated that their institutions require supervised classroom teaching in high needs schools.– Why do you think there is a such difference?– Do you feel this is important? Why or why not?

32

Do you have any questions or suggestions you would like to share?

33

Thank you!

34