Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern...
-
Upload
derick-stone -
Category
Documents
-
view
217 -
download
1
Transcript of Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern...
![Page 1: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/1.jpg)
Inside the ‘Black Box’ Of School Improvement: Measuring Change
James P. Spillane
Northwestern University
Chicago
March 11, 2008
Institute of Education Sciences,Spencer Foundation, National Science Foundation, Carnegie Foundation
www.distributedleadership.org
![Page 2: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/2.jpg)
Organizing for Teaching and Learning
• Formal Structure
• Formal positions• Organizational routines
• Relational Structure: A Social Network Approach
• Norms and Beliefs• Network Structure• Substance of Interactions
![Page 3: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/3.jpg)
Managing and Leading Instruction
• Heroics of Leadership• Principal Plus• Styles• Behaviors
• The missing piece - the practice of leading & managing
• A Distributed Perspective
• Attention to both formal and informal leaders• Practice is about interactions, not just actions
• The Subject Matters
![Page 4: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/4.jpg)
Research Instruments
• Logs• Experience Sampling Method (ESM) log• Leadership Daily Practice (LDP) Log• End of Day Log (EOD)
• Social Network Survey
• Mixed Method Studies
![Page 5: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/5.jpg)
Today’s Talk
• Motivating the work: Why Look Inside Schools?
• Social Network Survey Design & Validation
• Organizational ‘measures’• Norms & beliefs• Structure• Substance
![Page 6: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/6.jpg)
Motivating the Work
Why Look Inside Schools?
![Page 7: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/7.jpg)
Motivating the Work
• ‘Mediator Variables’
• “factors that must be changed or modified first by the treatment before main outcomes can be affected” (Petrosino, 2000, p. 50).
• “generative mechanism through which the focal independent variable is able to influence the dependent variable of interest” (Baron & Kenny, 1986, p. 1173).
• Build and test program/intervention theory (Weiss,1997).
• Hypotheses generating/theory development
![Page 8: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/8.jpg)
Anchoring the Work: What to Measure
• First Dimension: Normative Structure• Social Trust• Collective Responsibility• Innovation
• Second Dimension: Relational Structure• Tie strength• Tie span
• Third Dimension: Substance of Interactions
![Page 9: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/9.jpg)
Social Network Instrument
Design and Validation Work
![Page 10: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/10.jpg)
Social Network Survey
Screen shot from SSSNQ Version 1 - Math Advice Questions
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
![Page 11: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/11.jpg)
Social Network Instrument
Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 1
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
![Page 12: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/12.jpg)
Social Network Instrument
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 3
![Page 13: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/13.jpg)
Social Network Instrument
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Screen Screen Shot from SSSNQ Version 2 – Math Advice Questions Page 2
![Page 14: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/14.jpg)
School staff network survey
M/R R/MRandomization
math name generator
math name interpreter
RWLA name generator
RLWA name interpreter
RWLA name generator
RWLA name interpreter
math name generator
math name interpreter
![Page 15: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/15.jpg)
Three ways in which network data is susceptible to bias
• How many alters are named– fatigue or satisficing
• Who is named– censoring or priming
• Why those names were chosen/alter attributes– question scope redefinition
![Page 16: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/16.jpg)
Average number of alters listedBy subject area and treatment group
Subject areaR/M
(n = 126)M/R
(n = 138) Difference
RWLA 3.2 1.9 1.3*
Math 1.5 1.7 -0.2
Total 4.7 3.6 1.1*
![Page 17: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/17.jpg)
Some Results
The Instrument in Use
![Page 18: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/18.jpg)
The First Dimension: Norms and Beliefs
• Why should we care:– Trust moderates sense of vulnerability and uncertainty that
accompanies change.– Trust enables joint problem solving.– Trust among school staff is linked to gains in student
performance.
• The Evidence from the field– Teacher-teacher trust– Teacher-principal trust
![Page 19: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/19.jpg)
F A D C E G J I B H Total CCSRSchools
CCSRTeachers
strongly agree
strongly disagree
agree
disagree
Trust among teachers
![Page 20: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/20.jpg)
M O L D A R K N E S B F H I C J G P Q Total CCSRSchools
CCSRTeachers
Trust among teachers
strongly agree
strongly disagree
agree
disagree
![Page 21: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/21.jpg)
The Second Dimension: Relational Structure
• Frequent and influential interactions enable transfer of tacit, complex knowledge and joint problem solving.
• Relationships that span the formal organization allow for new ideas and minimize group think.
• Relationships that encompass all school staff members affects the degree to which staff members are engaged in joint work, and the consistency of advice provided.
![Page 22: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/22.jpg)
Measuring Relational Structure
• Tie Strength
– Average number of advisors– Average number of advisors with at least weekly contact – Average influence
• Tie Span
– Average number of advisors in other grade level teams/departments– Average number of advisors external to the school
• Network Concentration
– Percent of ties to formal leaders– Distribution of advisors by number of advisees
![Page 23: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/23.jpg)
Color corresponds to staff role
Reading/Writing/Language Arts
Math
Science
Social Studies
Special Education
6th Grade
Special Subjects
Administration
A
BC
Relational Structure
![Page 24: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/24.jpg)
Color corresponds to staff role
Reading/Writing/Language Arts
Math
Science
Social Studies
Special Education
6th Grade
Special Subjects
Administration
Relational Structure
M2
MathCoordinator
M2
M2
![Page 25: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/25.jpg)
Average in-degree: M2 associates vs. Non-M2 associates
8.4
2.1
7.0
0.3
5.3
0
1
2
3
4
5
6
7
8
9
6th Grade Math Other
M2
Non-M2
N=6 N=95 N=16 N=35 N=541
![Page 26: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/26.jpg)
• In math network, 8 out of 10 schools have two 6th grade team members connected to math department
Spanning Sub-units
Color corresponds to staff role
Reading/Writing/Language Arts
Math
Science
Social Studies
Special Education
6th Grade
Special Subjects
Administration
![Page 27: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/27.jpg)
• In RWLA network, 7 out of 10 schools have zero or one 6th grade team member connected to RWLA department
Spanning Sub-units
Color corresponds to staff role
Reading/Writing/Language Arts
Math
Science
Social Studies
Special Education
6th Grade
Special Subjects
Administration
![Page 28: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/28.jpg)
• In about half of the schools, an M2 associate provides the bridge between the 6th grade team members and the math department
Spanning Sub-units
Color corresponds to staff role
Reading/Writing/Language Arts
Math
Science
Social Studies
Special Education
6th Grade
Special Subjects
Administration
M2M2
M2
M2
coordinatorM2 Math
![Page 29: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/29.jpg)
Average external out-degree: M2 associates vs. Non-M2 associates
2.10
0.510.69
0.08
1.00
0
0.5
1
1.5
2
2.5
6th Grade Math Other
M2
Non-M2
N=5 N=86 N=16 N=29 N=445
M2 participants seek advice from more sources outside of their schools compared to their colleagues, suggesting that the M2 participants may be key brokers between their schools and external sources.
![Page 30: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/30.jpg)
How is the red group different from the blue group?
![Page 31: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/31.jpg)
• In some schools, one or two key players provided practically all of the advice (a high network concentration).
• In other schools, many teachers were involved (a lower network concentration).
• Generally, math networks were more concentrated than RWLA networks.
Structure: The Distribution of Influence
0
2
4
6
8
10
12
14
0
2
4
6
8
10
12
14
more concentrated less concentrated
![Page 32: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/32.jpg)
• In some schools, one or two key players provided practically all of the advice (a high network concentration).
• In other schools, many teachers were involved (a lower network concentration).
• Generally, math networks were more concentrated than RWLA networks.
Concentrated Influence?
0
2
4
6
8
10
12
14
0
2
4
6
8
10
12
14
more concentrated less concentrated
M2 Math teachersM2 Math teachers
Self-contained teacherMath teacher
M2 Math teacher/coordinator
![Page 33: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/33.jpg)
The Third Dimension
– The Substance of interactions among staff -- what teachers talk about!
– Instruction is a multi-faceted practice including • Deepening content knowledge • Course content • teaching strategies, • Materials• Strategies for low performing students• Assessing students understanding
![Page 34: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/34.jpg)
0
1
2
3
4
5
6
O B H I M L C G F S D R P N A Q K J E Total
Less than Weekly Contact
Daily or Weekly Contact
RWLA NetworkAverage out-degree by school
![Page 35: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/35.jpg)
Math NetworkAverage out-degree by school
0
1
2
3
4
5
6
M O C L J D B R S G H I Q P N A K F E Total
Less than Weekly Contact
Daily or Weekly Contact
![Page 36: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/36.jpg)
Math NetworkPercent of ties that deal with each content area
0%
10%
20%
30%
40%
50%
60%
70%
80%
Deepening yourcontent knowledge
Planning orselecting course
content andmaterials
Approaches forteaching content to
students
Strategiesspecifically to assist
low-performingstudents
Assessing students’understanding of the
subject
Other
District Average
Individual School
![Page 37: Inside the ‘Black Box’ Of School Improvement: Measuring Change James P. Spillane Northwestern University Chicago March 11, 2008 Institute of Education.](https://reader036.fdocuments.us/reader036/viewer/2022062421/56649e115503460f94afd2d6/html5/thumbnails/37.jpg)
www.distributedleadership.org
The instructional practices and assessments discussed or shown in these presentations are not intended as an endorsement by the U.S. Department of Education.