Collecting and Analyzing Student Engagement Data as A Key Component for School Improvement,...

Collecting and Analyzing Collecting and Analyzing Student Engagement Data as Student Engagement Data as A Key Component for School A Key Component for School Improvement, Organizational Improvement, Organizational

Learning, and Increased Learning, and Increased AchievementAchievement

West Virginia School Improvement SpecialistsAugust 18, 2010Jerry Valentine

Professor EmeritusUniversity of Missouri

[email protected]

Copy, print, or use only with written permission (via email) from Dr. Valentine ([email protected]).

Criteria for these slides…Criteria for these slides…

Sound, research based knowledgeSound, research based knowledge Practical use in schoolsPractical use in schools Practical use for you as a SI Practical use for you as a SI

specialist…you can use all, most, specialist…you can use all, most, some, a few of the slides with your some, a few of the slides with your schools.schools.

Perspectives on Student Perspectives on Student EngagementEngagement

Addressing Engagement is Addressing Engagement is critical to learning…here are critical to learning…here are

some basics:some basics:

Three Broad Forms of Student Three Broad Forms of Student Engagement in the ClassroomEngagement in the Classroom

Cognitive

Physical Emotional

Physical and Emotional Support CognitivePhysical and Emotional Support Cognitive

OUR FOCUS TODAY IS COGNITIVE OUR FOCUS TODAY IS COGNITIVE ENGAGEMENTENGAGEMENT

Logical Perspectives on Physical Logical Perspectives on Physical Presence, Mental Attention, and Presence, Mental Attention, and

Cognitive EngagementCognitive Engagement Attendance is a precursor to attention in class… Attention is a precursor to cognitive engagement… Cognitive engagement is a precursor to new learning for most

students (building knowledge, understanding and ability). A few students are physically present in class but consistently

cognitively absent. All students are, occasionally, physically present in class but

cognitively absent. For most students…attendance, attention, and cognitive

engagement are linked to learning through student motivation

For most students…motivation to cognitively engage is a function of:

Teacher-Student Relationships Emotional Security Content Relevance Challenging and Realistic Learning Experiences

Students are motivated to engage Students are motivated to engage when…when…

MotivationMotivation

See Value inSee Value in

the Learningthe Learning

Can make an Emotional/PersonalCan make an Emotional/Personal

Link to the Content/ProcessLink to the Content/Process

(Relate to Prior Knowledge/Experiences)(Relate to Prior Knowledge/Experiences)

Believe TheyBelieve They

Can Learn ItCan Learn It

Components of Meaningful Components of Meaningful Cognitive EngagementCognitive Engagement

Necessary ComponentsNecessary Components AttendanceAttendance in School in School AttendanceAttendance in Class in Class AttentionAttention during Learning during Learning

ExperiencesExperiences Cognitive Engagement Cognitive Engagement during during

Learning ExperiencesLearning Experiences Relevant and Challenging Relevant and Challenging

Learning ExperiencesLearning Experiences Student Self-Reflection/ Student Self-Reflection/

Assessment/Goal SettingAssessment/Goal Setting Student MotivationStudent Motivation to Attend and to Attend and

Cognitively EngageCognitively Engage

Supportive Processes Teacher builds Teacher-

Student Relationships School Protection of

Classroom Learning Time Teacher Use of Allocated

Classroom Learning Time Teacher Design of

Relevant Challenging Lesson

Teacher is Enthusiastic Supportive, Facilitative toward Student Effort

ATTENDANCE: SCHOOL-CLASSROOMINTEREST IN CONTENT AND

ACTIVITIES

Emotional Security to Engage

Physical Presence Necessary to Engage

Emotional Commitment to Engage with Effort

ENGAGEMENT AND ACHIEVEMENT: THE BIG ENGAGEMENT AND ACHIEVEMENT: THE BIG PICTUREPICTURE

R E L A T I O N S H I P S

CHALLENGING RELEVANT LEARNINGKNOWLEDGE

DEVELOPMENT“SURFACE”

EMBED BASIC FACTS/SKILLSCOMPREHENSIONAPPLICATION

THINKING DEVELOPMENT“H-O/DEEPER”ANALYSIS

PROBLEM-SOLVINGCRITICAL THINKINGCREATIVITY/INNOV.SYNTHESIS

Cognitive Engagement

Terminology and Basic Numbers… Academic School Day

Beginning to ending time for school--e.g. 8:00-3:25 (445 minutes) Allocated Learning Time

Scheduled in-class time What % of the Academic School Day is Allocated Learning Time? (85%-90%) (375-400 minutes)

Engaged Learning Time Students are truly cognitively engaged in the learning experience What % of the Allocated Learning Time is Engaged Learning Time? (80-90%) (300-360 minutes) Surface Learning Time

Simple comprehension, fact-finding, recall, and skill development/practice

What % of Engaged Learning Time is Surface Learning? (80-85%) (250-300 minutes)

Deeper Learning Time Analytical, critical, creative, innovative, and synthesis types of

thinking What % of Engaged Learning Time is Higher-Order Deeper Learning? (15-20%) (30-60 minutes)

Table Reflective Discussion The basics of engagement are neither

complex nor new… What information was affirming of

what you already knew? Did you hear or read something that

expanded your perspective about, or understanding of, student engagement? If so, what?

What stands out in your mind right now about student engagement??????

What stands out the most in What stands out the most in your mind right now about your mind right now about

engagement?engagement?

Engagement and Engagement and Achievement: Achievement:

The Classic StudyThe Classic Study(Yair, 2000)(Yair, 2000)

Yair Study of Cognitive Yair Study of Cognitive EngagementEngagement

Individual Student Engagement in Grades 6-8-10-12 Grade 6 in K-6 Schools and Grades 6 and 8 in K-8

Schools Grades 6 and 8 in Middle Schools Grades 10 and 12 in High Schools Students selected randomly and stratified by gender,

race, and ability level Data Collection Design

Digital wristbands Randomly buzzed 8 times daily for a week >3500 self-reports during in-class (allocated) learning

time Students Described:

Where are you? What are you doing? Who are you interacting with? What is on your mind?

How much are you concentrating? How challenging is it? How difficult is it? How interesting is it? How important is it to you? What else are you doing?Source: Yair, Educational Administration Quarterly, Vol. 36, #4 (October 2000)

Cognitive Engagement andCognitive Engagement andStudent CharacteristicsStudent Characteristics

Boys were 21% more likely to be engaged than girls.

Asian and white students were about 25% more likely to be engaged than African American and Hispanic students

Sixth graders are 29% more likely to be engaged than 12th graders; 23% more than 10th graders, and 6% more than eighth graders

The more students are engaged, the higher are their grade point averages

Source: Yair, Educational Administration Quarterly, Vol. 36, #4 (October 2000)

In which subject is cognitive In which subject is cognitive engagement the highest?engagement the highest?

Subjects: Mathematics English Reading Science Social Science

Rank order:

14325

Compared to Math and Science and Reading, students in English and Social Studies are more than 40% less likely to be engaged cognitively with the content


In which instructional method In which instructional method is cognitive engagement the is cognitive engagement the

highest?highest?Instructional Method

Teacher lecture Class discussions

(whole group T-led)

Laboratory (Hands-on work)

Group work Individualized

(seat) work Presentation

(creating/making)

EngagementRank

Passive for all

Active for all

Is the Learning Active/Passive?

Sources: Yair, Educational Administration Quarterly, Vol. 36, #4 (October 2000); Valentine (NSDC Conference (December, 2010)

6

5

1

2

4

3

Passive for most

Active for all

Active for all

Passive for all

Engagement and Instructional Engagement and Instructional MethodsMethods

When compared to teacher lecture learning experiences, the odds that students will be cognitively engaged are: 125% higher during group learning

experiences 115% higher during a laboratory learning

experiences 90% higher during class presentation

learning experiences 70% higher during individualized learning

experiencesSource: Yair, Educational Administration Quarterly, Vol. 36, #4 (October 2000)

Engagement in Relevant and Engagement in Relevant and Challenging Learning Challenging Learning

ExperiencesExperiences The odds of students being engaged during

the “most relevant” lessons were 108% higher than the lessons the students described as “least relevant.”

The odds of students being engaged during the “most challenging” lessons were 90% greater than when students identified the lessons as “least challenging.”

Relevance and challenge were the two most important lesson design strategies linked to student engagement in the study.


Engagement and Student Engagement and Student Recall…Recall…

The following slides are interesting, but the exact information reported is unique to the setting about which they are reporting. Do not take the findings literally across the board. Realize that they are being used herein as a way to make a point about engagement and recall, rather than as an empirical finding. You will see what I mean when I explain them…

And, as you look at the slides, realize that we all know this, both intuitively as well as through our own knowledge of how students learn and through our knowledge of general research about instructional methods.

Learning Pyramid: Recall in Two Learning Pyramid: Recall in Two Weeks from Weeks from Audio-VisualAudio-Visual Instruction Instruction

(Edgar Dale, Audio-Visual Methods in Teaching)(Edgar Dale, Audio-Visual Methods in Teaching)

Active

Passive

Read

HearLook at Pics

View movie, exhibit, demonstration

Participate in discussion; give talk

Do dramatic presentation; simulation, do real thing

10%

50%

70%

90%

20%

30%

Caveat: pyramid is considered more theoretical than empirical…use only to stimulate discussion about Passive and Active forms of learning.

How much we remember when…

1. We Say It 2. We Hear It and See

It 3. We See It 4. We Say and Do It 5. We Read It 6. We Hear It

Robert Pike, 1989; public speaker and adult trainer (Very similar to the Edgar Dale report)

412365 ---PASSIVE--- -ACTIVE-

Retention Rates and Instructional Methods

1. Practice by Doing 2. Reading 3. Lecture 4. Demonstration 5. Discussion Group 6. Audio-Visual 7. Teach Others and/or

Immediate Use of Learning

Warren (1989) “New Movement Seeks to Replace Rivalry in Class with Team Spirit,” Education.

Instructional Method

71542 63 --PASSIVE-- -ACTIVE-

Retention Rates and Instructional Methods

Instructional MethodInstructional Method Lecture Reading Audio-Visual Demonstration Discussion Group Practice by Doing Teach

Others/Immediate Use of Learning

Retention RateRetention Rate 5%5% 10%10% 20%20% 30%30% 50%50% 75%75% 90%90%

Warren (1989) “New Movement Seeks to Replace Rivalry in Class with Team Spirit,” Education.

Facts, Facts, Facts…Facts, Facts, Facts…

The previous slides were loaded with facts about engagement, most of which are research-based…

Which information surprised you the most? Why?

What surprised you the What surprised you the most?most?

The IPI Process for The IPI Process for Collecting and Collecting and

Collaboratively Studying Collaboratively Studying School-Wide Engagement School-Wide Engagement

Data Data

What is The IPI Process?What is The IPI Process? The Instructional Practices Inventory Process is

a set of faculty-led strategies for collecting valid/reliable student engagement data and for collaboratively studying the data with the goal of increasing and enriching learning experiences throughout the school.

The process serves a school best when teacher-leaders are the data collectors and the facilitators of the faculty collaborative study of the data.

When implemented with integrity, data analyses document that the process fosters instructional change and organizational learning.

Rationale for Collecting and Rationale for Collecting and Studying Engagement As A Studying Engagement As A

FacultyFaculty Student Engagement is clearly linked

to student achievement Collaborative Conversations are a

cornerstone to establishing a learning organization

Organizational Learning is a key to maintaining currency of knowledge and application of best practices

Development of the Instructional Development of the Instructional Practices Inventory ProcessPractices Inventory Processfor Profiling and Studyingfor Profiling and Studying

Student EngagementStudent Engagement When: 1995-96 Who: Bryan Painter co-developer Why: Originally to understand change in

instruction and engagement during a two-year comprehensive, systemic school improvement project of 30 schools

Data today available from thousands of schools representing hundreds of thousands of classroom observations

Data Collection Mental Image:Data Collection Mental Image:Collect Large Volume of “Snap Shots” Collect Large Volume of “Snap Shots”

of Student Engagement…of Student Engagement…

Collaborative Study Mental Image: Collaborative Study Mental Image: Faculty Analyze the Data, Problem Faculty Analyze the Data, Problem

Solve, and Design for ChangeSolve, and Design for Change

The IPI process…The IPI process…

provides the opportunity to provides the opportunity to create an OPTIMUM profile of create an OPTIMUM profile of student engagement in learning…student engagement in learning…

that teachers will view as fair and that teachers will view as fair and accurate, and thus…accurate, and thus…

use as a basis for periodic use as a basis for periodic reflective, problem-solving, reflective, problem-solving, collaborative conversations.collaborative conversations.

The IPI does not The IPI does not profile the types of profile the types of instructional instructional activities in which activities in which students are students are engaged.engaged.

The IPI profiles how The IPI profiles how students are students are engaging in engaging in learning during the learning during the instructional instructional activities.activities.

The Six IPI Student The Six IPI Student Engagement CategoriesEngagement Categories

H-0/Deeper Verbal Learning Conversations

Synthesis, Creativity/InnovationCritical AnalysisProblem SolvingAnalysis

All Other H-O/D Engagement

Pre-2004:18-20% Post-2004:

13-15%

APA 14 Learner-Centered Principles; ES: Construct Meaning; HS: Socratic

3-5%

18-20%

Students are attentive to teacher-led instruction

Independent or group work designed to build basic understanding, new knowledge or pertinent skills

3: Teacher is attentive to, engaged with, or supportive of students

2: Teacher is not attentive to, engaged with, or supportive of students

25-30%

5-10%

30-40%

30-40%

Students are not engaged in learning directly related to the curriculum

3-5% High Achiev: 0-1%Low Achiev: 8+; 10+; 20+

RecallMemorizationSkill PracticeFact-Finding

Disengagement

6 5 4 3 2 1

What does this visual imply about the six IPI What does this visual imply about the six IPI Categories ? Categories ?

NOT A NOT A HIERARCHYHIERARCHY

Six distinct categories…ways of classifying how students are engaged. A 6 is not better than a 5; a 5 is not better than a 4; etc. A 5 is not better than a 3, it is different than a 3 and there are times when the 3 is the most appropriate learning experience for the students.In a lesson or a unit, categories 6, 5, 4, 3 are all valuable learning experiences for the students.

Influences on Achievement?Influences on Achievement?

6 5 4 3 2 6 5 4 3 2 1 1Which category has the single-most

influence on student achievement? 1

Which two categories when combined have the most negative impact on student achievement?

1 2

Which two categories when combined have the most positive impact on student achievement?

5 6

The relative impact of 1-2 vs 5-6 is devastating… Categories 1-2 have slightly more than 2 times the influence of categories 5-6 on achievement in most grades.

Typical Engagement Typical Engagement Percentages from the IPI Percentages from the IPI Data before IPI Process Data before IPI Process

ImplementationImplementation(Data from 2004-2009(Data from 2004-2009

…the NCLB era)…the NCLB era)

What schools (grade-levels) have What schools (grade-levels) have the highest percentages of Higher-the highest percentages of Higher-

Order/Deeper Engagement? Order/Deeper Engagement?

Early Early ChildhoodChildhood

ElementaryElementary MiddleMiddle HighHigh Voc-TechVoc-Tech Alternative HSAlternative HS

Typical Percentages:

18.4%

17.4%

16.5%

15.5%

35.1%

21.8%

What schools (grade-levels) have What schools (grade-levels) have the lowest percentages of Student the lowest percentages of Student

Disengagement? Disengagement?

Early Early ChildhoodChildhood

ElementaryElementary MiddleMiddle HighHigh Voc-TechVoc-Tech Alternative HSAlternative HS

Typical Percentages:

1.3%

3.1%

3.9%

6.2%

2.2%

2.3%

Typical Percentage--2004-2009 (> 46,000 Class Typical Percentage--2004-2009 (> 46,000 Class Obs.)Obs.)

IPICategory EC ES ML HS V-T-C Alt.

6 13.37 13.29 11.45 11.30 29.99 15.39

5 5.06 4.09 5.06 4.24 5.14 6.43

4 46.88 44.12 39.71 37.60 25.51 32.88

3 28.53 28.62 30.71 29.08 31.58 35.54

2 4.86 6.80 9.13 11.62 5.62 7.49

1 1.30 3.08 3.93 6.16 2.16 2.27

5-6 18.43 17.38 16.51 15.54 35.13 21.82

2-3 30.69 35.42 39.85 40.70 37.20 43.03

4-5-6 65.31 61.50 56.22 53.14 60.64 54.70

1-2-3 34.69 38.50 43.78 46.86 39.36 45.30

Practice Coding Examples Practice Coding Examples to Develop an to Develop an

Understanding of the Understanding of the Categories in the Categories in the

ClassroomClassroom

Classroom Coding Classroom Coding Examples…Examples…

Read the example and chat briefly Read the example and chat briefly with your neighbor…what do you with your neighbor…what do you think is the correct code?think is the correct code?

1—2—3—4—5—61—2—3—4—5—6

IPI Example AAs you enter the 8th grade science classroom,

the students are listening attentively to the teacher give them simple directions for the litmus experiment they will begin in a few minutes. The teacher explains the process step by step. You learn from two students that the class also did litmus tests last month. As you begin to leave the room the teacher is re-explaining the main points of the process.

Which of the IPI categories should be coded for this example?

IPI Example AAs you enter the 8th grade science classroom, the

students are listening attentively to the teacher give them simple directions for the litmus experiment they will begin in a few minutes. The teacher explains the process step by step. You learn from two students that the class also did litmus tests last month. As you begin to leave the room the teacher is re-emphasizing the main points the students are to recognize and the steps they are to take when they do the litmus test.

IPI Code: Category 4—Students are attentive to Teacher-Led Instruction

IPI Example BAs you enter the sophomore English

classroom, the students are creating (original) poems. The teacher is moving among the students encouraging them as they work. They have a rubric on their desks that clarifies expectations about rhyme, meter, imagery, content, emotion, and length. The students are defining a topic and creatively explaining it through poem. As you read their work, you are impressed with their depth of thought and emotion.


IPI Example BAs you enter the sophomore English

classroom, the students are creating (original) poems. The teacher is moving among the students encouraging them as they work. They have a rubric on their desks that clarifies expectations about rhyme, meter, imagery, content, emotion, and length. The students are defining a topic and creatively explaining it through poem. As you read their work, you are impressed with their depth of thought and emotion.

IPI Code: Category 6—Students are engaged in Higher Order/Deeper, Creative thought

IPI Example CAs you enter the 4th grade math class

the students are seated at their tables completing a workbook assignment. When you look at their work you see they are independently computing practice division problems. They have been doing these types of problems off and on for months. The teacher is working at her computer creating a test and has her back to the students.


IPI Example CAs you enter the 4th grade math class

the students are seated at their tables completing a workbook assignment. When you look at their work you see they are independently computing practice division problems. They have been doing these types of problems off and on for months. The teacher is working at her computer creating a test and has her back to the students.

IPI Code: Category 2—Students are engaged in practice seatwork and the teacher is not attentive to, engaged with, or supportive of their work.

IPI Example D

As the you enter the freshman honors history class, the students are watching selected segments of the movies Pearl Harbor and Midway. The students are not taking notes, just watching. The teacher is standing by the DVD/VCR player watching the segments with the students. You can tell from the books on the students’ desks that the class is studying WWII. You are in the room about two minutes.


IPI Example D

As the you enter the freshman honors you enter the freshman honors history class, the students are history class, the students are watching selected segments of the watching selected segments of the movies movies Pearl HarborPearl Harbor and and MidwayMidway. The . The students are not taking notes, just students are not taking notes, just watching. The teacher is standing by watching. The teacher is standing by the DVD/VCR player watching the the DVD/VCR player watching the segments with the students. You can segments with the students. You can tell from the books on the students’ tell from the books on the students’ desks that the class is studying WWII. desks that the class is studying WWII. You are in the room about two minutes.You are in the room about two minutes.

IPI Code: Category 3—Students attentive to video resource and teacher is attentive to students

IPI Example EAs the you enter the high school art class,

the students are in small groups of four or five. Each group has a print of a classic painting and the students are discussing their analyses of the paintings. The discussions are stimulated by two “why” and “what if” questions written on the board that require collective analysis of the artwork. One student in each group is taking notes for the group. As you begin to leave the room two minutes after entering, you hear the teacher say that it is time to explain their group analyses and defend them to the whole class.


IPI Example EAs the you enter the high school art class, you enter the high school art class,

the students are in small groups of four the students are in small groups of four or five. Each group has a print of a classic or five. Each group has a print of a classic painting and the students are discussing painting and the students are discussing their analyses of the paintings. The their analyses of the paintings. The discussions are stimulated by two “why” discussions are stimulated by two “why” and “what if” questions written on the and “what if” questions written on the board that require collective analysis of board that require collective analysis of the artwork. One student in each group is the artwork. One student in each group is taking notes for the group. As you begin taking notes for the group. As you begin to leave the room two minutes after to leave the room two minutes after entering, you hear the teacher say that it entering, you hear the teacher say that it is time to explain their group analyses is time to explain their group analyses and defend them to the whole classand defend them to the whole class

IPI Code: Category 5—Students engaged in higher-order verbal learning conversations

IPI Example FAs the you enter the first grade class, 12 of

the 17 students are organized in two reading groups of 6 each on one side of the room. Each student is reading silently. A parent volunteer is circulating among the two groups to help as needed with difficult words. When you talk quietly with a few of those students and the parent, you learn that the students have challenging stories to read and they are working on developing reading skill and story comprehension. The teacher is in the opposite side of the room reading softly to the other five students.


IPI Example FAs the you enter the first grade class, 12 of

the 17 students are organized in two reading groups of 6 each on one side of the room. Each student is reading silently. A parent volunteer is circulating among the two groups to help as needed with difficult words. When you talk quietly with a few of those students and the parent, you learn that the students have challenging stories to read and they are working on developing reading skill and story comprehension. The teacher is in the opposite side of the room reading softly to the other five students.

IPI Code: Category 6—Non readers learning to read and comprehend requires at least analysis to obtain meaning from the sentences.

IPI Example GAs you enter the fourth grade language As you enter the fourth grade language

arts classroom it is obvious that the 24 arts classroom it is obvious that the 24 students are taking a paper and pencil students are taking a paper and pencil test. The teacher is seated at a student test. The teacher is seated at a student desk near the back of the room observing desk near the back of the room observing the students as they take the test. You the students as they take the test. You walk among the students for a few walk among the students for a few moments to read some of the test items. moments to read some of the test items. The test is a “recall” test, of multiple The test is a “recall” test, of multiple choice and fill in the blank items. The choice and fill in the blank items. The test is still in progress when you leave test is still in progress when you leave the classroom two minutes after entering the classroom two minutes after entering the room. the room.

Which of the IPI categories should be coded for

this example?

IPI Example GAs you enter the fourth grade language As you enter the fourth grade language

arts classroom it is obvious that the 24 arts classroom it is obvious that the 24 students are taking a paper and pencil students are taking a paper and pencil test. The teacher is seated at a student test. The teacher is seated at a student desk near the back of the room desk near the back of the room observing the students as they take the observing the students as they take the test. You walk among the students for a test. You walk among the students for a few moments to read some of the test few moments to read some of the test items. The test is a “recall” test, of items. The test is a “recall” test, of multiple choice and fill in the blank multiple choice and fill in the blank items. The test is still in progress when items. The test is still in progress when you leave the classroom two minutes you leave the classroom two minutes after entering the room. after entering the room.

IPI Code: Category 3—Students are engaged in recall-level seatwork with the teacher attentive to the students.

Collecting Engagement Collecting Engagement Data with Validity and Data with Validity and

ReliabilityReliability

Procedures for Collecting Engagement Data Procedures for Collecting Engagement Data with Validity and Reliability so Teachers with Validity and Reliability so Teachers

View the Data as Fair and AccurateView the Data as Fair and Accurate

Systematically move from classroom to classroom based upon the floor plan of the school.

Observe all learning settings proportionately across the school.

Code student learning during the first moments of initial entry into classroom as if you took a snapshot upon entry.

Focus on students, not the teacher. Code the predominant engagement pattern Collect a large volume of data points

throughout the school day

Procedures for Collecting Engagement Data Procedures for Collecting Engagement Data with Validity and Reliability so Teachers with Validity and Reliability so Teachers

View the Data as Fair and AccurateView the Data as Fair and Accurate Conduct data observations on “typical”

school days. Have candid faculty discussions about

“jazzing-it up” on data collection days. Select higher-numbered code when a clear

picture between two codes is not evident. Code during regular learning time, not

during transitions between content areas. Protect anonymity of all observations; never

link an observation with a teacher.

Who Should Collect the Data?Who Should Collect the Data? Teacher-Leaders Should Collect the DataTeacher-Leaders Should Collect the Data

Observations provide teachers with broader perspective Observations provide teachers with broader perspective about learningabout learning

Teachers are not evaluatorsTeachers are not evaluators Faculty embrace data more quickly when teachers collect Faculty embrace data more quickly when teachers collect

the data and facilitate conversations about the datathe data and facilitate conversations about the data Each school should develop an IPI Team of 3-5 teacher Each school should develop an IPI Team of 3-5 teacher

leadersleaders Regional Office and State Support Staff Can HelpRegional Office and State Support Staff Can Help

Short-term strategy to build interest/comfortShort-term strategy to build interest/comfort Most impact occurs when a school builds internal capacity Most impact occurs when a school builds internal capacity

to collect and study the datato collect and study the data Long-term support in design of faculty work sessions Long-term support in design of faculty work sessions

Principals are capable of Collecting Data for Profiles, Principals are capable of Collecting Data for Profiles, however, the process usually fails because…however, the process usually fails because… Teachers link principal’s observations with evaluationTeachers link principal’s observations with evaluation Teachers perceive the IPI process as an “administrator-Teachers perceive the IPI process as an “administrator-

driven” processdriven” process Teacher ownership is critical to consistent day-by-day Teacher ownership is critical to consistent day-by-day

changes in instructional design…and teacher ownership changes in instructional design…and teacher ownership does not occur if staff view the process as administrator-does not occur if staff view the process as administrator-drivendriven

Collaborative Faculty Collaborative Faculty Study of the IPI DataStudy of the IPI Data

Collaborative Faculty Study of the IPI Collaborative Faculty Study of the IPI Data…Core, Non-Core and Total Pie Data…Core, Non-Core and Total Pie

ChartsChartsCore ClassesCore Classes Non-Core ClassesNon-Core Classes

Collaborative Faculty Study of the IPI Collaborative Faculty Study of the IPI Data…Core, Non-Core and Total Pie Data…Core, Non-Core and Total Pie

ChartsChartsTotal for All ClassesTotal for All Classes

Keys to Studying the Data Keys to Studying the Data Use multiple short (45-50 min) study sessions

Study in a timely manner…compared to waiting weeks for a professional development day

Control the study environment… arrange setting with tables for small groups permit faculty to be seated with friends after beginning the session, number-off the teachers

to mix them randomly create small groups of 5-6 teachers per group

Use Whole Group and Small Group Strategies Think of the faculty meeting as an opportunity to

model good “category 5” engagement in learning

Common Elements for Each IPI Common Elements for Each IPI Faculty Collaborative Faculty Collaborative

ConversationConversation Discuss Typicality Of School day Of Instructional classroom practices (Jazz-it-up effect?)

Compare current and prior profiles…define issues to celebrate and issues of concern that need to be addressed

Build new knowledge about engagement strategies

Conclude session with discussion of value/worth of session…reflection/meta-cognition

Facilitate the collaborative study in whole group setting and small group settings with whole group share-out (make the learning a “5”)

Can You Interpret this Image of a Highly Can You Interpret this Image of a Highly Collaborative Faculty Study Session???Collaborative Faculty Study Session???

4>5 4>55

5 5

55 5

55

5

5

55

555

55

5 5

5 5 5<4 5<4

Higher-Order Faculty Collaborative Higher-Order Faculty Collaborative Learning Conversations?Learning Conversations?

Teacher

Teacher

Teacher

TeacherTeacher

Teacher Teacher

Teacher

Powerful learning yet under-utilized in faculty study

Collaborative Conversations We learn to study and problem-solve together as a faculty

Collective Collective Commitment Grows Commitment Grows We are getting on the same We are getting on the same page and learning from each page and learning from each otherother

Faculty CollectiveEfficacy GrowsWe believe we can and we believe we are “making a difference”

Student Academic Success is more Prevalent An expectation that students will succeed becomes the norm throughout all classrooms

Students Realize They Are More Successful Learners I can see my successes and I like it

Student Self-Efficacy Increases I believe I can learn this and I will learn it

Student Student AcademiAcademi

c c SuccessSuccess

ORGANIZATIONAL LEARNING

INSTRUCTIONAL BEST PRACTICES

PSYCHOLOGY OF LEARNING

Trust Grows

Collaborative Conversations Impact Student Learning (Valentine, 7-2010)

Instructional Activities that Engage Students more Effectively Increase Disengagement declines; Higher- Order/Deeper Engagement Increases

Thinking and Talking about Engagement Strategies become more Common Consciously design units and lessons with engagement in mind

Heartland Heartland SchoolSchool

Learning Communities can be like Silos…where

knowledge, like grain, is isolated, stored.

knowledge is not transferred to other silos. Collaboration/sharing/cohesion are missing.

RtI

CoopLrng

DiffInst

Etc.

PLC Study Group

IPI Longitudinal StudyIPI Longitudinal Study(1996-2008)(1996-2008)

Overview of 2009 StudyOverview of 2009 Study Data from 1996 to 2008 were compiled in 2008-2009

and analyzed using descriptive statistics, correlations and regressions, hierarchical linear modeling, and structural equation modeling. 243 schools from 105 school districts More than 550 days of data collection produced more than

70,000 classroom observations 125 of the 243 Schools completed a detailed “process” survey

allowing us to study “integrity to the recommended procedures”

Factors studied included Variables beyond the control of the school (e.g. FRL, %

minority) Variables within the control of the school (e.g. teacher

educational level, teacher experience, instructional practices) Dependent Variables

Student Achievement (Math and Communication Arts) Instructional Practices (higher-level and non-higher-level

engagement)

Higher-Order/Deeper Higher-Order/Deeper Engagement and AchievementEngagement and Achievement

A 20% increase in Higher-order/Deeper levels of engagement (categories 5-6) is projected to produce, on the state’s high-stakes achievement test, a: 6% increase in Communication Arts scores

i.e. a school with 50% of students passing the CA portion of the test would have 56% passing

7.3% increase in Mathematics scores i.e. a school with 50% of students passing the

Math portion of the test would have 57.3% passing

Disengagement and Disengagement and Seatwork with Achievement Seatwork with Achievement A 20% increase in categories 1-2-3 is

projected to produce, on the state’s high-stakes achievement test, a:7% decrease in Mathematics

i.e. a school with 50% of the students passing the Math portion of the state test would go down to 43% pass rate

Interesting Middle School DataInteresting Middle School Data

Reducing student and teacher disengagement in middle schools is twice as impactful on achievement as is increasing higher-order thinking. In middle schools, an increase in student

disengagement and teacher disengagement during seatwork (Categories 1-2) by 20 percent reduces the percent of students passing the state test by approximately 10%...

Changing HO/D (Categories 5-6) engagement enough to make a 10% upward swing in achievement pass rates would require an increase of approximately 41 percent

Interesting High School DataInteresting High School Data

For high schools, reducing student and teacher disengagement is even more impacting on achievement than in middle schools, while increasing HO/D is comparable on state achievement tests. In high schools, an increase in student/teacher

disengagement and seatwork (Categories 1-2) to 20 percent reduces the percent of student passing the state test by approximately 14%...

Changing HO/D (Categories 5-6) engagement enough to make a 14% upward swing in Communication Arts achievement pass rates would require an increase of approximately 45 percent

High Implementation Integrity, High Implementation Integrity, FRL & Achievement FRL & Achievement

Schools that implemented the IPI with integrity did not see the same degree of negative influence of FRL on state achievement scores compared to schools that implemented the IPI with low integrity. In short, achievement in schools with

high levels of IPI integrity are not as negatively impacted by poverty as would otherwise be the case.

Key Components of High Key Components of High Implementation of the IPI Implementation of the IPI

ProcessProcess Multiple data collections per year Collaborative faculty study of the

data following each data collection Data collection by teachers Faculty collaborative study of the

data led by teachers Level of faculty receptivity

(openness) to the IPI process during initial stages and subsequent stages

Faculty Receptivity to the IPI Process Faculty Receptivity to the IPI Process before Beginning Data Collectionsbefore Beginning Data Collections

Using a five-point scale to measure faculty receptivity to the IPI process: Low L-M Moderate M-H High 1 2 3 4 5

A school with the highest receptivity to the IPI data collection and collaborative conversations process before the first data were collected is projected to have 6% higher levels of HO/D engagement over schools with low-moderate receptivity

Faculty Receptivity to the IPI Process Faculty Receptivity to the IPI Process after the First Data after the First Data

Collection/ConversationCollection/Conversation Using a five-point scale to measure faculty

receptivity to the IPI process: Low L-M Moderate M-H High 1 2 3 4 5

A school with the highest receptivity to the IPI data collection and collaborative conversations process after their first data collection and collaborative study is projected to have 12% higher levels of HO/D engagement and teacher directed learning compared to schools with low-moderate receptivity

Keys to Early Faculty Keys to Early Faculty ReceptivityReceptivity

Develop a team of teacher leaders to collect data

Teacher leaders explain, and engage the faculty in discussing, the process before beginning the IPI process

Faculty are aware of the dates when IPI data will be collected

Teacher leaders maintain integrity during and after data collection

Teacher leaders facilitate the faculty study of the data

Principal facilitates and actively supports and encourages the process

Frequency of Data Collection and Frequency of Data Collection and Collaborative Study Make a Collaborative Study Make a

DifferenceDifference10 data collections

followed by collaborative conversations

over 3 years equated to…

14% increase in higher-order/deeper engagement (Categories 5-6)

AND 13% decrease in disengagement and

seatwork (Categories 1-2-3)

Keys to Data CollectionsKeys to Data Collections Collect data quarterly Build the 4 dates/times for faculty

study into school calendar Work backward from the dates when

data can be studied by the faculty as the basis for selecting the days data will be collected

Collect data approx. one week prior to time when faculty will study the data

The True Value of the IPI The True Value of the IPI Process Resides in the…Process Resides in the…

faculty collaborative conversations following each data profiling in which

teachers constructively study the data, problem solve, strategize, and learn together

as they collectively work to raise the bar of instruction across the whole school.

Let’s Sum it up…Let’s Sum it up…ENGAGEMENTENGAGEMENT

Document engagement using a valid and reliable observation process

Categories 3-4-5-6 are each important for Learning throughout a unit and/or lesson

Eliminate Disengagement (1) Reduce Teacher Disengagement (2) Reduce Seatwork (2-3) Increase HO/D Engagement (5-6) Increase HO/D Verbal Learning

Conversations (highly powerful process for learning) (5)

Let’s Sum it up…Let’s Sum it up…Collaborative Learning Collaborative Learning

ConversationsConversations Teacher leaders are the appropriate data collectors and should collect the engagement data in their own schools

Data collectors must be certificated to ensure validity and reliability.

Teacher leaders, with support from principals, should “facilitate” the study of the data

Multiple data collections and collaborations per year are necessary for meaningful, long-term change

Each faculty collaborative study session should: Discuss typicality during the data collection process Study data profiles and define issues to address Build new knowledge per defined issues Reflect on the value of the collaborative learning experience Develop commitment and learning by engaging staff

meaningfully in small/whole group learning conversations!

Recommended Annual Recommended Annual EventsEvents Based upon our most recent research about

successful use of the IPI, collect and study data three or four times a year.

The following is a recommended timeline: Early Fall—review process and categories; prep new

faculty; revisit “jazzing-it-up” Between school start and holiday break—collect

profiles twice and have collaborative conversations as soon as possible after each collection

Between holiday break and spring break, collect profile data and have collaborative conversation

Near the end of school year, collect profile data and have collaborative conversation.

Key IPI Reflective Questions After Year Key IPI Reflective Questions After Year One of ImplementationOne of Implementation

How many times have IPI data been collected? How many times have IPI data profiles been studied/discussed by the

faculty? Before the first IPI data collection, did the faculty discuss the IPI

process? If so, who led the discussion? How receptive to the use of the IPI was the faculty before the first data

collection? How receptive to the use of the IPI was the faculty after the first data

collection? How receptive to the use of the IPI is the faculty today? Did the faculty study/discuss the IPI profiles after each data collection? Who has been leading the study/discussion of the profiles? When the faculty discussed the data, how were the discussions

organized? Generally, who collects the data for the IPI profiles? Overall, who has provided the leadership for the faculty

study/discussions of the data? Overall, what have been the outcomes from the faculty’s

study/discussion of the data? Overall, what has been the faculty’s attitude/receptivity toward the IPI

process?

Longitudinal Effect…Longitudinal Effect…

.05

.35

.30

.25

.20

.15

.10

.45

.40

Sept

MayFebOctApril

Dec

Categories 2-3

Categories 5-6

Collaborative Higher- Collaborative Higher- Order Learning Order Learning Conversations Conversations

prepare students for prepare students for life! life!

If time…discuss the following about school

change.

[email protected]

Jerry ValentineProfessor Emeritus

University of Missouri

ReferencesReferences Collins, J. (2009). Collins, J. (2009). Higher-order thinking in the high-stakes accountability era: Linking student Higher-order thinking in the high-stakes accountability era: Linking student

engagement and test performanceengagement and test performance. Unpublished doctoral dissertation, University of Missouri. . Unpublished doctoral dissertation, University of Missouri. Gauen, K. (2009). Gauen, K. (2009). The Impact of the Instructional Practices Inventory on an Illinois Middle The Impact of the Instructional Practices Inventory on an Illinois Middle

SchoolSchool. Unpublished doctoral dissertation, Lindenwood University.. Unpublished doctoral dissertation, Lindenwood University. Painter, B. (1998). Painter, B. (1998). The Impact of Student Engagement on Student Achievement and The Impact of Student Engagement on Student Achievement and

Perceptions of Student-Teacher Relationships.Perceptions of Student-Teacher Relationships. Unpublished doctoral dissertation, University of Unpublished doctoral dissertation, University of Missouri.Missouri.

Valentine, J., Clark, D., Hackmann, D., and Petzko, V. (2004Valentine, J., Clark, D., Hackmann, D., and Petzko, V. (2004 ). A National Study of Leadership in ). A National Study of Leadership in Middle Level Schools, Volume II: Leadership for Highly Successful Middle Level SchoolsMiddle Level Schools, Volume II: Leadership for Highly Successful Middle Level Schools . Reston, . Reston, VA: National Association of Secondary School Principals. VA: National Association of Secondary School Principals.

Valentine, J. (2005Valentine, J. (2005). Statistical Differences for the Percentages of Student Engagement as ). Statistical Differences for the Percentages of Student Engagement as Measured by IPI Categories between Very Successful and Very Unsuccessful Middle SchoolsMeasured by IPI Categories between Very Successful and Very Unsuccessful Middle Schools . . University of Missouri, Columbia, MO: Middle Level Leadership Center.University of Missouri, Columbia, MO: Middle Level Leadership Center.

Valentine, J. and Collins, J. (2009a). Valentine, J. and Collins, J. (2009a). Improving Instruction by Profiling Student Engaged Learning Improving Instruction by Profiling Student Engaged Learning and Creating Collaborative Teacher Learning Conversations.and Creating Collaborative Teacher Learning Conversations. National Association of Secondary National Association of Secondary School Principals, Annual Conference, March 1, 2009. School Principals, Annual Conference, March 1, 2009.

Valentine, J. and Collins, J. (2009b). Valentine, J. and Collins, J. (2009b). Analyzing the Relationships among the Instructional Analyzing the Relationships among the Instructional Practices Inventory, School Climate and Culture, and Organizational LeadershipPractices Inventory, School Climate and Culture, and Organizational Leadership . American . American Educational Research Association, Annual Meeting, April 14, 2009, San Diego, CA.Educational Research Association, Annual Meeting, April 14, 2009, San Diego, CA.

Valentine, J. (December, 2009). Valentine, J. (December, 2009). The Instructional Practices Inventory: Using a Student Learning The Instructional Practices Inventory: Using a Student Learning Assessment to Foster Organizational LearningAssessment to Foster Organizational Learning. National Staff Development Council, Annual . National Staff Development Council, Annual Convention, December 8, 2009, St. Louis, MO.Convention, December 8, 2009, St. Louis, MO.

Valentine, J. (2010). ) Valentine, J. (2010). ) Establishing a Faculty-wide Collaborative Study of Student EngagementEstablishing a Faculty-wide Collaborative Study of Student Engagement , , National Association of Secondary School Principals, Annual Conference, San Diego, CA March National Association of Secondary School Principals, Annual Conference, San Diego, CA March 14, 2010.14, 2010.

Yair, G. (2000). Not just about time: Instructional practices and productive time in school. Yair, G. (2000). Not just about time: Instructional practices and productive time in school. Educational Administration Quarterly, 36(4), 485-512. Educational Administration Quarterly, 36(4), 485-512.

Suggested Resources…Suggested Resources… For more detailed information about the IPI see For more detailed information about the IPI see

Instructional Practices Inventory: Profiling Student Instructional Practices Inventory: Profiling Student Engagement for School ImprovementEngagement for School Improvement (Valentine, 2005) (Valentine, 2005) available from available from [email protected]@missouri.edu..

For more detailed information about Project ASSIST see For more detailed information about Project ASSIST see Frameworks for Continuous School Improvement: A Frameworks for Continuous School Improvement: A Synthesis of Essential ConceptsSynthesis of Essential Concepts (Valentine, 2001) and (Valentine, 2001) and Project ASSIT Research paper presented at available at AERA, Project ASSIT Research paper presented at available at AERA, April 2006 in San FranciscoApril 2006 in San Francisco) available from ) available from [email protected]@missouri.edu..

Also see The Instructional Practices Inventory: Using a Also see The Instructional Practices Inventory: Using a Student Learning Assessment to Foster Organizational Student Learning Assessment to Foster Organizational Learning Learning Valentine, 2007, also 2009 NSDC Annual Convention Valentine, 2007, also 2009 NSDC Annual Convention paper) available by request from paper) available by request from [email protected]@missouri.edu.

For information about IPI Level I (Coder Training) and Level II For information about IPI Level I (Coder Training) and Level II (Advanced Faculty Work Session) workshops(Advanced Faculty Work Session) workshops email email [email protected]@missouri.edu..

IPI Coder Reliability is IPI Coder Reliability is Developed through IPI Level I Developed through IPI Level I

WorkshopWorkshop

Minimum Reliability for user endorsement:Minimum Reliability for user endorsement: .80 for site-based school improvement data.80 for site-based school improvement data .90 for research.90 for research

Coder Reliability StudyCoder Reliability Study w/o Workshop .05-.20 Reliability avg: .17w/o Workshop .05-.20 Reliability avg: .17 With Workshop .80-1.0 Reliability avg: .93With Workshop .80-1.0 Reliability avg: .93

If you would like a copy of If you would like a copy of this PowerPoint this PowerPoint presentation…presentation…

Email me at Email me at [email protected]@Missouri.edu Request the PPT by date and location Request the PPT by date and location

of the presentationof the presentation I will reply and attach a copy of the I will reply and attach a copy of the

PPT plus the handouts and a couple of PPT plus the handouts and a couple of manuscripts about the IPI.manuscripts about the IPI.

For information about IPI Workshops For information about IPI Workshops in your area, contact me by email.in your area, contact me by email.

Website: www.MLLC.org

Email: [email protected]

Phone: (573) 882-0944

Questions, contact the IPI developers at:

http://www.mllc.org/

mailto:[email protected]

Additional Slides of Additional Slides of Interest…Lack of time Interest…Lack of time prevented these slides prevented these slides from being used in the from being used in the

presentationpresentation The following slides may have some The following slides may have some

value for understanding the IPI value for understanding the IPI process. They have been used in process. They have been used in different presentations over time different presentations over time about the IPI and/or student about the IPI and/or student engagement in learning.engagement in learning.

Recommended ResourcesRecommended Resources www.APA.org “Learner-Centered www.APA.org “Learner-Centered

Psychological Principles (1997)Psychological Principles (1997) ““Powerful Learning (ASCD and Powerful Learning (ASCD and

www.ascd.org; (Brandt 1998)www.ascd.org; (Brandt 1998) Marzano, et al. (ASCD):Marzano, et al. (ASCD):

The Art and Science of Teaching (2007)The Art and Science of Teaching (2007) Classroom Instruction that Works (2001)Classroom Instruction that Works (2001) Designing and Assessing Educational Designing and Assessing Educational

Objectives (2008)Objectives (2008)

What does it take to change What does it take to change engagement?engagement?

Sequence of Payoff

What it Takes

6 (3) Combined Categories 5-6:

All HO/D

New KnowledgeImplementation

SkillCommitment

5

43

2(2) Categories

One Plus Two: All Disengagement

AwarenessCommitment

1(1) Most Direct:

Stu. Disengagement

AwarenessCommitment

Overall Implementation of the IPI Process Overall Implementation of the IPI Process Recommended Practices for Data Recommended Practices for Data

Collection and Collaborative Collection and Collaborative Conversations Conversations

Using a five-point scale to measure school Using a five-point scale to measure school implementation of the recommended practices:implementation of the recommended practices: Low L-M Moderate M-H HighLow L-M Moderate M-H High 1 2 3 4 51 2 3 4 5

A school implementing the IPI A school implementing the IPI recommended practices with high integrity recommended practices with high integrity is projected to have 8.4% higher levels of is projected to have 8.4% higher levels of HO/D engagement over schools with low-HO/D engagement over schools with low-moderate implementationmoderate implementation

Higher Order/Deeper EngagementHigher Order/Deeper Engagementvs. vs.

Not Higher Order/Surface Not Higher Order/Surface EngagementEngagement

Analysis, Critical Thinking, Problem Analysis, Critical Thinking, Problem Solving, Decision Making or Solving, Decision Making or Application from Analysis, Application from Analysis, Creativity, Innovation SynthesisCreativity, Innovation Synthesis

------------------------------------------------------------------------------------------

Recall, Memorization, Fact Finding, Recall, Memorization, Fact Finding, Simple Understanding, Practice to Simple Understanding, Practice to Internalize Skills or ProcessesInternalize Skills or Processes

6 – Student Active Engaged 6 – Student Active Engaged Learning (HO/D)Learning (HO/D)

Higher-Order/Deeper thinking through Higher-Order/Deeper thinking through analysis, problem solving, critical analysis, problem solving, critical thinking, creativity, innovation, and thinking, creativity, innovation, and synthesis.synthesis.

Common Examples (if HO/D): Common Examples (if HO/D): Inquiry-based approaches such as project Inquiry-based approaches such as project

and problem-based learning, research and and problem-based learning, research and discovery learningdiscovery learning

Authentic demonstrationsAuthentic demonstrations Independent metacognition, reflective Independent metacognition, reflective

journaling, and self-assessmentjournaling, and self-assessment Higher-order responses to higher-order Higher-order responses to higher-order

questions.questions.

5—Student Learning 5—Student Learning Conversations (HO/D)Conversations (HO/D)

Higher-Order Student-Student Verbal Higher-Order Student-Student Verbal Learning Conversations constructing Learning Conversations constructing deeper meaning and understanding deeper meaning and understanding through the conversationsthrough the conversations

Common examples (if HO/D):Common examples (if HO/D): collaborative or cooperative learningcollaborative or cooperative learning Peer tutoring, debate, and questioningPeer tutoring, debate, and questioning Partner research and discovery/exploratory Partner research and discovery/exploratory

learninglearning Socratic learningSocratic learning Small group or whole class analysis and Small group or whole class analysis and

problem solving, metacognition, reflective problem solving, metacognition, reflective discussions or writing, and self assessmentdiscussions or writing, and self assessment

4—Teacher-Led Instruction 4—Teacher-Led Instruction (Not HO/D)(Not HO/D)

Students are attentive to teacher-led Students are attentive to teacher-led instruction as the teacher leads the instruction as the teacher leads the learning experience by disseminating learning experience by disseminating content knowledge and/or directions for content knowledge and/or directions for learninglearning

Common Examples:Common Examples: Teacher-directed Q/A, lecture, explanationsTeacher-directed Q/A, lecture, explanations Teacher direction givingTeacher direction giving Teacher demonstrationsTeacher demonstrations

3—Student Work with 3—Student Work with Teacher Engaged (Not Teacher Engaged (Not

HO/D)HO/D) Students engaged in independent or Students engaged in independent or

group work designed to build basic group work designed to build basic understanding, new knowledge, pertinent understanding, new knowledge, pertinent skills. Teacher is attentive to, engaged skills. Teacher is attentive to, engaged with, or supportive of the students’ work.with, or supportive of the students’ work.

Common Examples: (Teacher Common Examples: (Teacher Engaged)Engaged)

Fact findingFact finding Building skill or understanding through Building skill or understanding through

practice, seatwork, worksheets, chapter practice, seatwork, worksheets, chapter review questionsreview questions

Multi-mediaMulti-media

2—Student Work with 2—Student Work with Teacher Not Engaged (Not Teacher Not Engaged (Not

HO/D)HO/D) Same as Category Three except the Same as Category Three except the

teacher is not attentive to, engaged with, teacher is not attentive to, engaged with, or supportive of the students’ work.or supportive of the students’ work.

Common Examples: (Teacher Not Common Examples: (Teacher Not Engaged)Engaged)

While students are working, teacher is:While students are working, teacher is:Out of the roomOut of the roomWorking at computerWorking at computerGrading papersGrading papers

1—Students Not Engaged 1—Students Not Engaged in Learningin Learning

Students are not engaged in learning Students are not engaged in learning directly related to the curriculumdirectly related to the curriculum

Common Examples:Common Examples: Students talking, daydreaming, or otherwise Students talking, daydreaming, or otherwise

inattentiveinattentive Students misbehavingStudents misbehaving Students not doing their assigned workStudents not doing their assigned work

Activities that are typicallyActivities that are typicallyHigher-Level Learning Higher-Level Learning

Common Examples:Common Examples: Project-based learningProject-based learning Research/Hypothesizing/Testing/Concluding/Research/Hypothesizing/Testing/Concluding/

DefendingDefending Questioning…Why, What if, Compare/ContrastQuestioning…Why, What if, Compare/Contrast Socratic SeminarsSocratic Seminars Thinking about thinking...metacognitionThinking about thinking...metacognition Cooperative LearningCooperative Learning Problem-based learningProblem-based learning Reflective Discussions and Writing TasksReflective Discussions and Writing Tasks

Less common examples…how do you make Less common examples…how do you make these higher-order?these higher-order? Watching and analyzing a movieWatching and analyzing a movie Paper/pencil testsPaper/pencil tests Dodge ball! Dodge ball!

Why are Higher-Order Learning Why are Higher-Order Learning ConversationsConversations so Significant? so Significant?

Learning is enhanced through social Learning is enhanced through social interaction. When students engage in H-interaction. When students engage in H-O learning conversations the benefits O learning conversations the benefits are: are: Motivation to learn (social nature)Motivation to learn (social nature) Depth of knowledge and understandingDepth of knowledge and understanding Breadth of knowledge and understandingBreadth of knowledge and understanding Recall of knowledge and conceptsRecall of knowledge and concepts Transfer of knowledge and conceptsTransfer of knowledge and concepts

Recommended Annual Recommended Annual EventsEvents Based upon our most recent research about Based upon our most recent research about

successful use of the IPI, collect and study data successful use of the IPI, collect and study data three or four times a year.three or four times a year.

The following is a recommended timeline:The following is a recommended timeline: Early Fall—review process and categories; prep new Early Fall—review process and categories; prep new

faculty; revisit “jazzing-it-up” effectfaculty; revisit “jazzing-it-up” effect Between school start and holiday break—collect Between school start and holiday break—collect

profiles twice and have collaborative conversations profiles twice and have collaborative conversations as soon as possible after each collectionas soon as possible after each collection

Between holiday break and spring break, collect Between holiday break and spring break, collect profile data and have collaborative conversationprofile data and have collaborative conversation

Near the end of school year, collect profile data and Near the end of school year, collect profile data and have collaborative conversation.have collaborative conversation.

Recommended Year One Recommended Year One Timeline/TasksTimeline/Tasks

Aug/SeptAug/Sept Sept/OctSept/Oct Nov/DecNov/Dec Jan/FebJan/Feb Apr/MayApr/May

Orient new Orient new faculty; faculty; remind oldremind old

Data Data collection; collection; collaborative collaborative conversationconversation




Discuss:Discuss:CategoriesCategoriesProcessProcessJazz-it-upJazz-it-upNot Not EvaluationEvaluation

Discuss:Discuss:Typicality Typicality CelebrationCelebrationIssuesIssuesValueValue

Discuss:Discuss:Typicality Typicality CompareCompareCategories Categories 5-6 5-6 ExamplesExamplesHomeworkHomeworkValueValue

Discuss:Discuss:TypicalityTypicalityCompare Compare longitudinallylongitudinallyCategories Categories 5-6 5-6 ExamplesExamplesValueValue

Discuss:Discuss:TypicalityTypicalityCompare Compare longitudinallylongitudinallySet goals for Set goals for next yearnext yearValueValue

Commonalities to Each Faculty Commonalities to Each Faculty Collaborative IPI ConversationCollaborative IPI Conversation

TypicalityTypicality School daySchool day Instructional classroom practicesInstructional classroom practices

Comparison of profilesComparison of profiles Substantive learning experiencesSubstantive learning experiences Value or worth…reflection, meta-cognitionValue or worth…reflection, meta-cognition Facilitation of issues from whole group to Facilitation of issues from whole group to

small group to whole group share-outsmall group to whole group share-out

Higher-Order Student Classroom Higher-Order Student Classroom Learning Conversations?Learning Conversations?

Teacher

Student

Student

StudentStudent

Student Student

Student

Powerful learning yet under-utilized

Faculty Work Session: Analysis and Faculty Work Session: Analysis and Discussion of the Profile DataDiscussion of the Profile Data

Small and Whole Group Analyses and Small and Whole Group Analyses and DiscussionDiscussion What do we see in the profiles that we can What do we see in the profiles that we can

feel good about or celebrate?feel good about or celebrate? What do we see in the data profiles that we What do we see in the data profiles that we

should be concerned about and thus study should be concerned about and thus study and discuss more deeply?and discuss more deeply?

How do we build a cache of good ideas on How do we build a cache of good ideas on engagement, especially good HO/D engagement, especially good HO/D engagement?engagement?

Faculty Discussion: Are these types of Faculty Discussion: Are these types of data valuable to us?data valuable to us?

Faculty Work Session: Post-Faculty Work Session: Post-session Requestssession Requests

Request for sub-group analyses…Request for sub-group analyses… Can we have a profile for the math Can we have a profile for the math

program? program? Individual teacher self-Individual teacher self-

assessment…assessment… Can I build a profile of my students’ Can I build a profile of my students’

engagement using this process?engagement using this process? Value/benefit of self-ratings vs. Value/benefit of self-ratings vs.

accuracy/reliability of self-ratings?accuracy/reliability of self-ratings?

Faculty Work Session: Deeper Faculty Work Session: Deeper Analyses with Longitudinal Analyses with Longitudinal

Perspective and Goal SettingPerspective and Goal Setting How do we begin to share knowledge How do we begin to share knowledge

about effective strategies that will about effective strategies that will change the percentages?change the percentages? Small groups collaboratively brainstorm Small groups collaboratively brainstorm

good examples of categories 5-6 used in good examples of categories 5-6 used in past week (create examples from within…)past week (create examples from within…)

Move conversation to whole faculty sharingMove conversation to whole faculty sharing Move conversations after faculty meeting to Move conversations after faculty meeting to

sub-groups such as content areas, teams, or sub-groups such as content areas, teams, or grade levelsgrade levels

Type and share all examples with all facultyType and share all examples with all faculty

Faculty Work Session: Goal Faculty Work Session: Goal Setting after three or four Setting after three or four

data collections…data collections… For each category percentage, should For each category percentage, should

we:we: Increase?Increase? Maintain?Maintain? Decrease?Decrease?

If change is appropriate…If change is appropriate… How much?How much? By when?By when?

What do we address first that will have What do we address first that will have the most direct impact on student the most direct impact on student learning?learning?

““Typical” Profiles…Typical” Profiles…notnot normsnorms

Are there differences between Are there differences between typical profiles by grade levels, typical profiles by grade levels, (elem., middle, and high school)?(elem., middle, and high school)?

Are there differences between Are there differences between typical profiles for core and non-typical profiles for core and non-core classes? core classes?

Are there differences between Are there differences between profiles for more effective and less profiles for more effective and less effective schools?effective schools?

IPI Protocol Examples for Data IPI Protocol Examples for Data CollectionCollection

Data observations on “typical” daysData observations on “typical” days Systematically move from classroom to Systematically move from classroom to

classroom based upon the floor plan of the classroom based upon the floor plan of the schoolschool

Focus on students, not the teacherFocus on students, not the teacher Code student learning during first moments of Code student learning during first moments of

observationobservation When observation is borderline between two When observation is borderline between two

codes, select higher codecodes, select higher code Code during regular learning time, not Code during regular learning time, not

transitions between content areastransitions between content areas Classrooms of special education and student Classrooms of special education and student

teachers are observed and codedteachers are observed and coded Classrooms of substitute teachers are coded Classrooms of substitute teachers are coded

and included in profile if higher-order learningand included in profile if higher-order learning All observations are anonymousAll observations are anonymous

Active-Passive EngagementActive-Passive Engagement

What are some examples of What are some examples of ACTIVEACTIVE STUDENT ENGAGEMENT in learning?STUDENT ENGAGEMENT in learning?

What are some examples of What are some examples of PASSISVEPASSISVE STUDENT ENGAGEMENT in learning?STUDENT ENGAGEMENT in learning?

Being actively engaged does not Being actively engaged does not necessarily mean good learning!necessarily mean good learning!

IPI ProcessIPI Process

Fits Concept of Learning Fits Concept of Learning OrganizationsOrganizations Faculty Discussions/Analysis of Data Faculty Discussions/Analysis of Data

about Teaching/Learningabout Teaching/Learning ““Teachers engaged in the data collection”Teachers engaged in the data collection” ““Teachers engage in regular, reflective collaborative Teachers engage in regular, reflective collaborative

conversations about the data profiles.”conversations about the data profiles.” ““On-going collection and collaborative problem-On-going collection and collaborative problem-

solving conversations over time.”solving conversations over time.”

should…should…

Be associated with Be associated with staff evaluationstaff evaluation

Identify individual Identify individual teachers or classesteachers or classes

Be collected by Be collected by administratorsadministrators

Reflect the activity in Reflect the activity in which the students are which the students are engagedengaged

Create an optimum profile of Create an optimum profile of engagement teachers will view as fair engagement teachers will view as fair and accurateand accurate

Provides the basis for reflective, Provides the basis for reflective, collaborative conversationscollaborative conversations

Provide baseline data and insight Provide baseline data and insight about subsequent changes in about subsequent changes in engagementengagement

Support school improvement and Support school improvement and professional development plansprofessional development plans

Serve as a basis for action researchServe as a basis for action research

Be used in context with multiple Be used in context with multiple measures of student success measures of student success

Reflect how students are engaging in Reflect how students are engaging in learninglearning

should not…An effective student engagement profiling process:An effective student engagement profiling process:

The following four slides are The following four slides are data from the Justin Collins data from the Justin Collins

study, 2009study, 2009

Number of Collections and Number of Collections and Collaborative Conversations with Collaborative Conversations with

Engagement (Q5)Engagement (Q5) If a typical school in this study If a typical school in this study

engaged in 10 data collections and engaged in 10 data collections and collaborative conversations over the collaborative conversations over the course of three years, the school course of three years, the school would have a 13 point decrease in would have a 13 point decrease in categories 1-2-3 from current categories 1-2-3 from current average of 38% and a 14 point average of 38% and a 14 point increase in higher level thinking as increase in higher level thinking as measured by categories 5-6 from measured by categories 5-6 from current average of 22%.current average of 22%.

Engagement and Student Engagement and Student

AchievementAchievement If a school in the study had a 20% increase in If a school in the study had a 20% increase in

categories 1-2-3 it would have had a decrease categories 1-2-3 it would have had a decrease in achievement of 7% on Math performancein achievement of 7% on Math performance In our study that meant a school at the 42% In our study that meant a school at the 42%

pass rate would go down to 35% pass ratepass rate would go down to 35% pass rate If a school in the study had a 20% increase in If a school in the study had a 20% increase in

categories 5-6 it would have had a 7.3% categories 5-6 it would have had a 7.3% increase on Mathematics performanceincrease on Mathematics performance In our study that meant a school at the 42% In our study that meant a school at the 42%

pass rate would go to 49.3% pass rate pass rate would go to 49.3% pass rate N=135

3-level analysis

Faculty Receptivity andFaculty Receptivity andEngagement (Q6b)Engagement (Q6b)

If a typical school in this study If a typical school in this study introduced the use of the IPI in a introduced the use of the IPI in a manner that resulted in a level of manner that resulted in a level of receptivity (openness on a 5-unit receptivity (openness on a 5-unit scale) to the IPI process that was scale) to the IPI process that was high (5) compared to low-moderate high (5) compared to low-moderate (2), the school would see an increase (2), the school would see an increase of 6 points of categories 5-6. of 6 points of categories 5-6.

Faculty Receptivity after Initial Faculty Receptivity after Initial Analysis and Student Engagement Analysis and Student Engagement

(Q11)(Q11) If a faculty’s level of receptivity If a faculty’s level of receptivity

on a 5-unit scale to the IPI on a 5-unit scale to the IPI process after the faculty’s first process after the faculty’s first profile analysis was high (5) profile analysis was high (5) compared to low-moderate (2) compared to low-moderate (2) the school would see a 12 point the school would see a 12 point gain in the engagement for gain in the engagement for categories 4-5-6.categories 4-5-6.

Overall Benefits of IPI Process with Overall Benefits of IPI Process with Engagement (Q14)Engagement (Q14)

If a typical school in this study engaged in If a typical school in this study engaged in the use of the IPI to the degree that the the use of the IPI to the degree that the overall benefits of collaborative overall benefits of collaborative conversations and deeper understanding conversations and deeper understanding about instructional design increased 3 units about instructional design increased 3 units on a 5-unit scale assessing overall impact of on a 5-unit scale assessing overall impact of the IPI on the school culture and instruction, the IPI on the school culture and instruction, the resultant increase in Core Higher-level the resultant increase in Core Higher-level engagement would go up 8.4 points from 23 engagement would go up 8.4 points from 23 to 31.4%.to 31.4%.

Note that the following data Note that the following data are Pre-2005 Data…review are Pre-2005 Data…review

and use with cautionand use with caution

Typical Percentages by Grade Typical Percentages by Grade LevelsLevels

E. S.E. S. M. S.M. S. H. S.H. S.

Student Active Engaged Student Active Engaged LearningLearning 15-2515-25 15-2015-20 15-2015-20

Student Learning Student Learning ConversationsConversations 3-53-5 3-53-5 3-53-5

Teacher-Led InstructionTeacher-Led Instruction 35-4035-40 35-4535-45 30-4030-40

Student Work with Student Work with Teacher EngagedTeacher Engaged 20-3020-30 20-3020-30 15-2015-20

Student Work with Student Work with Teacher not EngagedTeacher not Engaged 5-105-10 10-2010-20 15-2015-20

Complete Complete DisengagementDisengagement 3-83-8 5-105-10 5-155-15

Jerry Valentine 2004

Typical Percentages by Core/Non-Typical Percentages by Core/Non-CoreCore

All LevelsAll Levels

All All ContentContent

All All LevelsLevels

CoreCore


Non-CoreNon-Core

Student Active Engaged Student Active Engaged LearningLearning 15-2015-20 <15<15 <25<25

Student Learning Student Learning ConversationsConversations 3-53-5 5-105-10 <5<5

Teacher-Led InstructionTeacher-Led Instruction 30-4530-45 >40>40 <40<40

Student Work with Student Work with Teacher EngagedTeacher Engaged 20-3020-30 >25>25 <25<25

Student Work with Student Work with Teacher not EngagedTeacher not Engaged 10-2010-20 >20>20 <20<20

Complete Complete DisengagementDisengagement 5-105-10 >5>5 <5<5


Typical Percentages by School Typical Percentages by School EffectivenessEffectiveness


All All ContentContent

MoreMore

EffectiveEffectiveLessLess

EffectiveEffective

Student Active Engaged Student Active Engaged LearningLearning 15-2015-20 >25>25 15-2015-20

Student Learning Student Learning ConversationsConversations 3-53-5 5-105-10 <5<5

Teacher-Led InstructionTeacher-Led Instruction 30-4530-45 35-4535-45 30-4030-40

Student Work with Student Work with Teacher EngagedTeacher Engaged 20-3020-30 15-2515-25 >25>25

Student Work with Student Work with Teacher not EngagedTeacher not Engaged 10-2010-20 5-105-10 10-2010-20

Complete Complete DisengagementDisengagement 5-105-10 <3<3 >5>5


Percentages for High Achieving and Low Achieving Middle SchoolsPercentages for High Achieving and Low Achieving Middle Schools

HighlyHighly

AchievingAchievingLowLow

AchievingAchieving

***.001***.001

**.05**.05

*.10*.10

Student Active Engaged Student Active Engaged LearningLearning 29.329.3 16.016.0 **Student Learning Student Learning ConversationsConversations 3.33.3 0.20.2 ******Teacher-Led InstructionTeacher-Led Instruction 40.540.5 33.233.2

Student Work with Student Work with Teacher EngagedTeacher Engaged 17.317.3 28.428.4 ******Student Work with Student Work with Teacher not EngagedTeacher not Engaged 8.58.5 13.613.6

Complete Complete DisengagementDisengagement 1.01.0 8.48.4 ******Jerry Valentine 2004


HighlyHighly


AchievingAchieving

***.001***.001

**.05**.05

*.10 *.10

Student Active Engaged Student Active Engaged LearningLearning

32.632.6 16.216.2 ****Student Learning Student Learning ConversationsConversations

Teacher-Led InstructionTeacher-Led Instruction57.857.8 61.661.6 **Student Work with Student Work with

Teacher EngagedTeacher Engaged

Student Work with Student Work with Teacher not EngagedTeacher not Engaged

9.59.5 22.022.0 ****Complete Complete DisengagementDisengagement



HighlyHighly


AchievingAchieving

***.001***.001

**.05**.05

*.10*.10

Student Active Engaged Student Active Engaged LearningLearning

73.173.1 49.449.4 ******Student Learning Student Learning ConversationsConversations

Teacher-Led InstructionTeacher-Led Instruction

Student Work with Student Work with Teacher EngagedTeacher Engaged

26.826.8 50.450.4 ******Student Work with Student Work with Teacher not EngagedTeacher not Engaged

Complete Complete DisengagementDisengagement


Activities that Activities that TypicallyTypically produce produce Higher-Order/Deeper Engagement Higher-Order/Deeper Engagement

Common Examples:Common Examples: Project-based learningProject-based learning Research/Hypothesizing/Testing/Concluding/Research/Hypothesizing/Testing/Concluding/

DefendingDefending Questioning…Why, What if, Compare/ContrastQuestioning…Why, What if, Compare/Contrast Socratic SeminarsSocratic Seminars Thinking about thinking...metacognitionThinking about thinking...metacognition Cooperative LearningCooperative Learning Problem-based learningProblem-based learning Reflective Discussions and Writing TasksReflective Discussions and Writing Tasks

Less common examples…how do you make Less common examples…how do you make these higher-order?these higher-order? Watching and analyzing a movieWatching and analyzing a movie Paper/pencil testsPaper/pencil tests Dodge ball! Dodge ball!

Organizational Learning Organizational Learning Reduces the Knowledge--Reduces the Knowledge--Implementation GapImplementation Gap

Society and Students We Serve Change Continuously

Expert Knowledge of Best Practices

Our Knowledge of Best Practices

Our Implementation if We Maintain Knowledge

Our Implementation w/ Moderate Knowledge

Our Implementation with No New Knowledge No Knowledge, Low Attitude, Low Effort

Collecting and Analyzing Student Engagement Data as A Key Component for School Improvement,...

Documents

Transcript of Collecting and Analyzing Student Engagement Data as A Key Component for School Improvement,...