Project Director – Dr. Mark LungDept of Natural & Environmental SciencesWestern State College of Colorado

Project Evaluator – Dr. Dave ShannonEducational Research MethodologyAuburn University

PartnersWestern State College of Colorado (IHE)

Hindsdale SDGunnison Watershed SD

Ouray SDMoffat County SD

Project DescriptionProject Description

Professional development Professional development program for middle school math program for middle school math and science teachers in rural and science teachers in rural Colorado schools.Colorado schools.

Includes: Includes: Authentic experience with the Authentic experience with the

process of science and its tools process of science and its tools (including math)(including math)

Training and equipment for Training and equipment for implementing project-based (i.e. implementing project-based (i.e. inquiry) lessons in the classroom. inquiry) lessons in the classroom.

1. Field research experience 1. Field research experience (2 weeks) (2 weeks)

- Data collection/analysis- Data collection/analysis- GPS/GIS training- GPS/GIS training- Creating inquiry lessons- Creating inquiry lessons- Teachers leave with - Teachers leave with experience, photos, data to experience, photos, data to build lessons build lessons

Project ActivitiesProject Activities

2. Implementation Workshop 2. Implementation Workshop (2 days)(2 days)

- Skill/content reinforcement- Skill/content reinforcement - Implementing lessons in - Implementing lessons in

classrooms classrooms

3. Job-embedded coaching3. Job-embedded coaching - Observation of inquiry - Observation of inquiry

lessons lessons - Immediate - Immediate

discussion/feedbackdiscussion/feedback - Repeat observations and- Repeat observations and

feedbackfeedback

Project OutcomesProject Outcomes

1. Increase teacher knowledge, 1. Increase teacher knowledge, experience, and confidence with experience, and confidence with science concepts and applications science concepts and applications (e.g. hypotheses, variables, designing studies, (e.g. hypotheses, variables, designing studies, scientific interpretation, etc.)scientific interpretation, etc.)

2. Increase the use of inquiry-2. Increase the use of inquiry-based instruction in classroomsbased instruction in classrooms

3. Increase in student attitude 3. Increase in student attitude towards and achievement in towards and achievement in science and mathscience and math

Year 1 Year 2 Year 3

Activity/Evaluation TimelineActivity/Evaluation Timeline

C1

C2

Field experience

Implementation workshop

Job-embedded coaching

Activities

1 2 3 4

1 2 3

Evaluation

Outcome 2

Outcome 1

June 06 June 07 June 08

Evaluation DesignEvaluation Design

Design - Cross-sectional and longitudinalDesign - Cross-sectional and longitudinal Sample – All participating teachers and Sample – All participating teachers and

students. Comparison groups of students students. Comparison groups of students sampled when possible.sampled when possible.

Data Collection – surveys, content Data Collection – surveys, content knowledge assessments, class knowledge assessments, class observations, lesson plan analysisobservations, lesson plan analysis

EvaluationEvaluation Outcome 1: Increase teacher knowledge, Outcome 1: Increase teacher knowledge,

confidence and experience in science processconfidence and experience in science process Instruments – Instruments –

• PPIPPI – s – self-perceptions of knowledge, confidence, and elf-perceptions of knowledge, confidence, and experience. (administered as pre- and post for both cohorts experience. (administered as pre- and post for both cohorts of teachers)of teachers)

• Science Teacher SurveyScience Teacher Survey – attitudes toward teaching – attitudes toward teaching and learning science, motivating students, teacher efficacy, and learning science, motivating students, teacher efficacy, and the use of constructivist and inquiry-based teaching and the use of constructivist and inquiry-based teaching methods. (administered annually to all teachers)methods. (administered annually to all teachers)

• Content knowledge assessmentContent knowledge assessment– – tailored to the tailored to the project’s objectives regarding science concepts and project’s objectives regarding science concepts and applications.applications. (administered as pre in June 2007, post (administered as pre in June 2007, post planned for later this year)`planned for later this year)`

Evaluation Evaluation

Outcome 2: Increase use of inquiry-based Outcome 2: Increase use of inquiry-based instruction in the classroominstruction in the classroom Instruments – Instruments –

• **Science Teacher Inquiry RubricScience Teacher Inquiry Rubric (STIR) - has (STIR) - has been used as an observation tool as well as a been used as an observation tool as well as a measure of self-reflection by teachers. measure of self-reflection by teachers. (administered in fall and spring) (administered in fall and spring)

• Teacher and Student Surveys Teacher and Student Surveys

*Beerer and Bodzin 2003, 2004

EvaluationEvaluation

Outcome 3: Increase student attitude and Outcome 3: Increase student attitude and achievementachievement Instruments –Instruments –

• AttitudesAttitudes - - Survey of Science Classrooms – Survey of Science Classrooms – student attitudes toward science, academic efficacy, goal student attitudes toward science, academic efficacy, goal orientation and motivation, and inquiry-based teaching orientation and motivation, and inquiry-based teaching practices in the classroom. (administered in February 2007)practices in the classroom. (administered in February 2007)

• KnowledgeKnowledge – The examination of state-mandated – The examination of state-mandated achievement tests (CSAP) and the MOSART tests are achievement tests (CSAP) and the MOSART tests are planned for the upcoming year.planned for the upcoming year.

ResultsResultsOutcome 1: Teacher Knowledge, Outcome 1: Teacher Knowledge,

Experience, ConfidenceExperience, Confidence PPI –PPI –

Reliabilities estimates exceed .97Reliabilities estimates exceed .97

Cohort 1 teachers (n=7) reported higher post perceptions Cohort 1 teachers (n=7) reported higher post perceptions with increases of approximately 1 point (on a 5 point with increases of approximately 1 point (on a 5 point scale) occurred on each of the three PPI scales, with scale) occurred on each of the three PPI scales, with

slightly larger gains for confidenceslightly larger gains for confidence.. Cohort 1 (experienced) teachers also reported more Cohort 1 (experienced) teachers also reported more

positive perceptions that the new teachers from Cohort positive perceptions that the new teachers from Cohort 2.2.

Outcome 1 (continued)Outcome 1 (continued) Science Teacher SurveyScience Teacher SurveyTeacher Beliefs and AttitudesTeacher Beliefs and Attitudes

• While all teachers generally agreed that While all teachers generally agreed that they can overcome student difficulties and they can overcome student difficulties and attribute student success to their efforts, attribute student success to their efforts, these beliefs were more positive for these beliefs were more positive for teachers from cohort 1 (2006)teachers from cohort 1 (2006)

• Teachers from cohort 2 (2007) were more Teachers from cohort 2 (2007) were more skeptical about student learning potentialskeptical about student learning potential

Outcome 1 (continued) Teacher Content Knowledge

• In general, cohort 1 teachers self-reported having more knowledge on the PPI, especially in terms of:

• Describing inquiry-based learning• Designing inquiry-based lessons and assessments

• Cohort 1 teachers also performed better than Cohort 2 teachers on the Content Knowledge Assessment in terms of:

• Describing the principles of inquiry-based learning• Comparing research designs and designing studies to

investigate the effects of inquiry

ResultsResultsOutcome 2: use of inquiryOutcome 2: use of inquiry

Science Teacher SurveyScience Teacher Survey• Both cohorts of teachers cohorts generally believed in their ability to Both cohorts of teachers cohorts generally believed in their ability to

use constructivist, student-centered, inquiry-based approachesuse constructivist, student-centered, inquiry-based approaches• Experienced teachers (cohort 1) were more likely to indicate that Experienced teachers (cohort 1) were more likely to indicate that

they made special efforts to recognize individual progress and they they made special efforts to recognize individual progress and they encouraged student collaboration.encouraged student collaboration.

While the overall mean for the Inquiry-based While the overall mean for the Inquiry-based teaching measurement scale was just at the teaching measurement scale was just at the midpoint, several items resulted in higher midpoint, several items resulted in higher average responses by participating students. average responses by participating students. Specifically, students reported:Specifically, students reported:

Outcome 2 (continued)Outcome 2 (continued)

Survey of Science ClassroomsSurvey of Science ClassroomsStudents taught by participating teachers reported that they:Students taught by participating teachers reported that they:• were involved in small group discussions to make sense of science,were involved in small group discussions to make sense of science,• used math skills to help with scienceused math skills to help with science• gave written explanations about how they solved science problemsgave written explanations about how they solved science problems• discussed alternative explanations regarding science questions and discussed alternative explanations regarding science questions and

problemsproblems• did science projects that lasted several daysdid science projects that lasted several days

Outcome 2 (continued)Outcome 2 (continued)

STIRSTIR 9 classrooms observed with inquiry-based lesson 9 classrooms observed with inquiry-based lesson

plansplans RubricRubric

• 6 items relating to different components of inquiry6 items relating to different components of inquiry• scoring: 1 (low) – 5 (high)scoring: 1 (low) – 5 (high)

ResultsResults Mean (Mean (++SD)SD)• Cohort 1: Cohort 1: 3.1 ( 3.1 (++1.4)1.4)• Cohort 2:Cohort 2: 3.6 ( 3.6 (++0.8)0.8)• Considerable variation among teachersConsiderable variation among teachers

Low scores primarily due to leading students to contentLow scores primarily due to leading students to content

ResultsResultsOutcome 3: Student Attitude/AchievementOutcome 3: Student Attitude/Achievement

Student AttitudesStudent Attitudes• Eleven measurement scales yielded reliabilities ranging Eleven measurement scales yielded reliabilities ranging

from .63 to .86 (Median = .82)from .63 to .86 (Median = .82)• Students averaged above the mid-point (3) on 8 of the 11 scalesStudents averaged above the mid-point (3) on 8 of the 11 scales• Students:Students:

• Have a positive attitude toward science and that science is usefulHave a positive attitude toward science and that science is useful• Exhibit a Mastery Goal Orientation (task-oriented) toward learning Exhibit a Mastery Goal Orientation (task-oriented) toward learning

science as opposed Performance Approach (competition) or Avoidance science as opposed Performance Approach (competition) or Avoidance (embarrassment) (embarrassment)

• Believe in their ability to learn (Academic Efficacy)Believe in their ability to learn (Academic Efficacy)• Perceive that their teachers believe in their ability to learn sciencePerceive that their teachers believe in their ability to learn science

Outcome 3 (continued)Outcome 3 (continued)

Student AchievementStudent Achievement• The use of state-mandated tests (CSAP) is being The use of state-mandated tests (CSAP) is being

explored, but the Science tests are only administered in explored, but the Science tests are only administered in grades 5 and 8 – providing limited usefulness for the grades 5 and 8 – providing limited usefulness for the projectproject

• The use of commercially-prepared tests (e.g., the The use of commercially-prepared tests (e.g., the MOSART tests) and teacher-made tests are being MOSART tests) and teacher-made tests are being explored for potential us in the project. explored for potential us in the project.

Teacher provides an opportunity for learners to engage with a scientifically oriented question.

Learner is prompted to formulate own questions or hypothesis to be tested.

Teacher suggests topic areas or provides samples to help learners formulate own questions or hypothesis.

Teacher offers learners lists of questions or hypotheses from which to select.

Teacher provides learners with specific stated (or implied) questions or hypotheses to be investigated.

No evidence observed.

Teacher engages learners in planning investigations to gather evidence in response to questions.

Learners develop procedures and protocols to independently plan and conduct a full investigation.

Teacher encourages learners to plan and conduct a full investigation, providing support and scaffolding with making decisions.

Teacher provides guidelines for learners to plan and conduct part of an investigation. Some choices are made by the learners.

Teacher provides the procedures and protocols for the students to conduct the investigation.

No evidence observed.

Teacher helps learners give priority to evidence which allows them to draw conclusions and/or develop & evaluate explanations that address scientifically oriented questions.

Learners determine what constitutes evidence and develop procedures and protocols for gathering and analyzing relevant data (as appropriate).

Teacher directs learners to collect certain data, or only provides portion of needed data. Often provides protocols for data collection.

Teacher provides data and asks learners to analyze.

Teacher provides data and gives specific direction on how data is to be analyzed.

No evidence observed.

5 4 3 2 1Attribute

Learners formulate conclusions and/or explanations from evidence to address scientifically oriented questions.

Learner is prompted to analyze evidence (often in the form of data) and formulate own conclusions/explanations.

Teacher prompts learners to think about how analyzed evidence leads to conclusions/explanations, but does not cite specific evidence.

Teacher directs learners' attention (often through questions) to specific pieces of analyzed evidence (often in the form of data) to draw conclusions and/or formulate explanations.

Teacher directs learners' attention (often through questions) to specific pieces of analyzed evidence (often in the form of data) to lead learners to predetermined correct conclusion/explanation (verification).

No evidence observed.

Learners evaluate their conclusions and/or explanations in light of alternative conclusions/ explanations, particularly those reflecting scientific understanding.

Learner is prompted to examine other resources and make connections and/or explanations independently.

Teacher provides resources to relevant scientific knowledge that may help identify alternative conclusions and/or explanations. Teacher may or may not direct learners to examine these resources, however.

Teacher does not provide resources to relevant scientific knowledge to help learners formulate alternative conclusions and/or explanations. Instead, the teacher identifies related scientific knowledge that could lead to such alternatives, or suggests possible connections to such alternatives.

Teacher explicitly states specific connections to alternative conclusions and/or explanations, but does not provide resources.

No evidence observed.

Learners communicate and justify their proposed conclusions and/or explanations.

Learners specify content and layout to be used to communicate and justify their conclusions and explanations.

Teacher talks about how to improve communication, but does not suggest content or layout.

Teacher provides possible content to include and/or layout that might be used.

Teacher specifies content and/or layout to be used.

No evidence observed.

5 4 3 2 1Attribute

Disclaimer The instructional practices and

assessments discussed or shown in these presentations are not intended as an endorsement by the U.S. Department of Education".