Data Coaching Network
Transcript of Data Coaching Network
Data Coaching Network A Collaboration Between The Office of Superintendent of Public Instruction & The Association of Education Service Districts Year Two Report 2013 Submitted by Sue Feldman, Education Service District 112 Additional edits by Sue Furth and Tara Richerson, OSPI
1
Report Summary
Data coaching is an example of how well the network of nine Education Service Districts
(ESD) can work collaboratively with the Office of Superintendent of Public Instruction
(OSPI) to provide training and new program implementation equitably throughout the
state. Each ESD participated in the data coaching institutes in year one and year two, which
resulted in interesting and unique approaches to data coaching in each ESD region. Given
the same training, each ESD team developed a unique and regionally relevant approach to
data coaching. Coming together to learn with and from each other was valued by each ESD
and provided necessary support data coaching implementation.
Data coaching is a good step toward improving data practices throughout the state. Each
ESD is providing some form of data coaching, primarily by integrating data coaching into
other ESD-funded services. This is a slow approach to meeting the actual needs of school
districts. If education is going to take advantage of the potential in data use, there continues
to be a serious need for data practice supports in most school districts. ESDs may be the
most efficient, effective, and economical way to provide these supports. Education is lagging
behind in the big data movement. While education generates local, state, and national data,
the system of education in Washington State has established few resources to use these
data. It is a rare school district with the software infrastructure to display and analyze data
and an even rarer school district with the analytic expertise to provide accurate, clear,
timely, and relevant analysis of data for decision making at the classroom, school, or school
district level. ESDs continue to be the best location for data coaches because ESDs can and
do provide economical and equitable access to education services. ESDs will need two key
resources to increase their data use support to school districts. First, ESDs and school
districts will need to establish data-sharing agreements. Second, ESDs will need funding to
hire quantitative expertise to work alongside data coaches who are currently prepared to
facilitate data practices, but are generally not prepared to conduct sophisticated analyses.
2
Current State of Data Use in Washington State’s System of Education Data use in education has become an established expectation while it continues to be an
unclear set of practices (Goren, 2012). In Washington State, teachers are expected to use
data for instructional purposes and school principals are expected to observe teachers’ uses
of data as a part of the new teacher and principal evaluation system. Student growth data is
also to be used as a part of the teacher and principal evaluation process. While these
expectations are in policy, educators are awaiting the implementation of a reliable and valid
assessment system that generates growth data, aligned to grade level expectations (Ingram,
Louis, & Schroeder, 2004; Supovitz & Klein, 2003).
After more than ten years of investments in testing apparatus and data infrastructure, it
remains a rare educator in Washington State who can develop a reliable or valid classroom
assessment, conduct an item analysis on test questions, or run an inferential analysis to find
a Z-score to compare test scores across unlike tests. While the lack of data tools and
technical expertise present a challenging situation for professional development, educators
in general are still drawn toward the promise of data; unfortunately, the Legislature has not
yet allocated resources to develop the data expertise necessary to realize this promise
(e.g.,Coburn & Talbert, 2006; Datnow & Park, 2009; Knapp, Copland, Swinnerton, &
Monpas-Huber, 2006).
In an effort to improve data use practices of teachers, school and school district leaders,
OSPI and the Washington School Information Processing Cooperative (WSIPC) garnered a
commitment from each of the nine Education Service Districts’ Assistant Superintendents
for Teaching and Learning to send an inter-program team to 12 days of data coaching
institutes. The institutes took place over a two-year period. ESD teams met for two days at a
time with trainers from Public Consulting Group (PCG) to support a network of data
coaching initiative and a common set of data coaching protocols for the network of nine
ESDs. A research initiative was implemented, in conjunction with the data coaching
institute, to document and analyze the process of the data coaching initiative. As a new
program intending to change practices, the data coaching initiative provided a good
opportunity to study program implementation and changes in statewide practice, in real
time. This is the report of the year two study of that effort.
3
Study Design Data coaching, sponsored originally by OSPI and WSIPC, in partnership with the network of
nine ESDs, moved into its second year of implementation in 2012–13. Data coaching
continued to expand its collaborative reach to include multiple departments at OSPI, school
districts, and other education-focused organizations including special education
cooperatives, early childhood education programs, and GATE: Graduation: A Team Effort.
Data coaching began with the development of a set of protocols to support school district
leadership teams to increase and improve their data practices. There is strong theoretical
support for the concept of data coaching, even if there is not a dedicated revenue stream to
support it (e.g, Halverson, Grigg, Prichett, Thomas, & Wisconsin-Madison, 2006; Kowalski,
Lasley, & Mahoney, 2008; Marsh, Pane, & Hamilton, 2006; Means, Padilla, DeBarger, &
Bakia, 2009).
Year Two Research Questions:
1. How, if at all, does data coaching support collaboration within and between OSPI,
ESDs, and school districts?
2. How, if at all, does studying the implementation of data coaching support the
implementation of data coaching?
Year one research questions will continue to be a focus of the study:
1. How, if at all, did the data coaching institute shape the work of the ESDs in year two
of implementation?
2. What, if anything, was the tangible outcome of ESDs participating in three data
coaching institutes during the 2011–2012 school year?
3. To what extent are ESDs ready to pursue data coaching as a service to their regions?
4. What more do ESD participants need in order to provide data coaching services in
their regions?
5. What are the supports and constraints on ESD resources for data coaching?
4
Study Purpose. This study was primarily designed to inform the implementation process. It
documented the second year of data coaching implementation and explored the emergent
data coaching activity within and between ESDs, school districts, departments at OSPI, and
WSIPC to understand how people participated in and developed capacity for data coaching
in Washington State during the 2012–2013 school year.
Study Method. This mixed methods (Cohen, Manion, & Morrison, 2007; Lodico, Spaulding,
& Voegtle, 2006) study used survey data collected after the final data coaching institute in
the spring of 2013, field observation data, interview data collected in semi-structured brief
interviews with data institute participants and project leaders, and documentary evidence
related to the project formation and implementation.
Data Collection. Interview data and documentary data were collected throughout the year.
Survey monkey was used to collect survey data. The first year survey produced better than
70 percent response rates. The response rate of actual data coaches was close to 100
percent. The second year survey had better than an 87 percent return. Additional data will
be collected throughout year two including data coaching institute field notes and
observations, document analysis and interviews with ESD data coaching team members,
school district data coaching participants, and OSPI leaders involved with data coaching.
Data Analysis. Field-notes and interview data and the qualitative survey responses were
analyzed (Miles, Huberman, & Saldana, 2013) using Nvivo software. A coding schema was
developed, based on the research questions, and free coding was used to capture themes
that emerged through the analysis process. Survey data were analyzed using descriptive
statistics to examine emerging trends in data coaching activity, influences of data coaching
on school district-based practices, and influences on data practices at OSPI and WSIPC.
5
Findings and Discussion Following the practices from the first year of the data coaching institutes, monthly phone
calls between OSPI leaders, interested ESD data coaches, and PCG facilitators were used to
monitor implementation and plan the data coaching institutes. The emerging interest of
data coaches seemed to be improving their data analysis skills. Therefore, the data coaching
institutes invited a variety of data experts, who work with different administrative data
sets, theoretically available to data coaches, to present to the data coaching teams. The
presentations were designed to introduce data coaches to the data sets and to analytic
practices that might be useful for making sense of the data. Here are three examples:
1) During the second data coaching institute, Paul McCold, the data analyst with the
state Office of Migrant and Bilingual Education, introduced data coaches to the data
sets he used to analyze the progress bilingual students were making in the
Washington State school system. He also taught data coaches how to conduct an
elaboration using inferential statistics.
2) The team working on the new longitudinal student data system for OSPI introduced
their data system and showed data coaches the types of questions that could be
answered by the system, which included teacher salaries and school district
revenues, making it easy to access two data sets that had previously been difficult to
locate.
3) At the school and district level, a reading specialist from one of the ESDs presented a
process she had invented to determine how each student in a school district was
meeting reading standards. Using formative and summative assessments and
aligning each assessment to a small set of standards and curriculum materials, she
demonstrated how to generate formative assessments for instructional decision-
making.
These three presentations, and multiple other presentations, were designed to improve
coaches’ data analysis skills; yet there is a long way still to go to prepare the data coaches in
the use of inferential statistics. The data analysis conducted through most of the data
coaching continues to be basic descriptive statistics. Survey results indicate ESDs are not
yet ready to offer data analysis using inferential statistics. Using inferential statistics was
the lowest rated service among all the data coaching work ESDs might do. Interviews
corroborated that all ESD data coaches continue to want more sophistication in the data
6
analysis skills. Data coaches are interested and willing to participate in training on inferential
statistics if it is done in the context of their districts’ data and the questions their districts are
trying to answer.
More coaching than data. The majority of the data coaches came into data coaching
confident and competent in the facilitation of socio-cultural practices. The data coaching
tool kit, developed during year one of the data coaching initiative, focused primarily on the
socio-cultural features of transforming district leadership teams into data leadership teams.
For example, the tool kit includes protocols to guide a conversation about what data the
team has and uses and for what purposes; protocol for facilitating a conversation about how
to increase the data sources used for decision making; facilitating data reviews; and
facilitating data informed decision making. Data coaches are comfortable leading these
processes that focus primarily on facilitation of productive interactions between people on
leadership teams. Data coaches are less confident in their ability to access and analyze
district data. While the data coaching institute did not expect competence in data analysis as
an outcome, by the end of the second year, the data coaches reported wanting to be more
competent in data analysis.
ESD data coaches want to respond to their districts’ needs for actual data analysis support.
Except in rare districts, schools and school districts do not have data analyst positions or
people with those skills and time to do data analysis. While there is a growing expectation
that teachers and administrators analyze data, there are actually few districts with the
staffing capacity to do this with rigor or consistency. ESD data coaches may be a good
solution to increasing districts’ data analysis capacity, but it will require hiring actual data
analysts at ESDs to work with the data coaches or training the data coaches as data analysts,
which was not the focus of the data coaching institutes.
Direct-coaching & certification. Data coaching certification was introduced during the
first year of data coaching as the focus of year two and the culminating event of the
institutes. Certification was a part of the original work plan between OSPI, WSIPC, and PCG,
but the data coaches questioned what the certification meant and the overall purpose or
value of the certification. A common perspective among data coaches was that certification
was superfluous. By virtue of being chosen to represent their ESDs at the data coaching
7
institute, they were already recognized as competent in the work and further certification
was not necessary. Given this questioning of the value and purpose of certification, PCG and
the lead OSPI facilitator of the data coaching institutes refocused the certification
framework not as a pass or fail process but instead as a tool to focus the direct-coaching
conversations.
Also in the original scope of PCG work, for year two, were two site visits made to each ESD.
These site visits were preceded by a pre-visit planning phone call. These site visits all took
place, although the original direct coaching was altered because no ESD data coaching team
implemented data coaching with a district leadership team using the data coaching tool kit as
designed. That is not to say that there was no data coaching conducted with district
leadership teams, but the majority of data coaching was conducted at the school level rather
than the district level. The majority of data coaching, in year two, did not focus on
developing a data leadership team, or transforming a leadership team into a data
management team; instead, data coaching was one activity that a leadership team may have
facilitated along with many other activities. This being the case, it was challenging for the
PCG coaches to provide direct coaching and it was challenging for the ESD data coaching
teams to use the certification framework. What the direct coaching did reveal was the
unique implementation of data coaching in each ESD. Each ESD team had a clear and
coherent plan and data projects they were leading. Each was uniquely constructed to utilize
the strengths and interests of the team members and leverage funding opportunities to
include data coaching, which was unfunded.
Data, data systems and data displays. Data coaching began as an initiative between OSPI
and the Washington Student Information Processing Cooperative (WSIPC). As WSIPC serves
as the most prevalent student data management system in the state, they were facing the
challenge of transforming the data system from a transactional system—designed primarily
for reporting data upstream—into a data system that could be easily used for decision
making at all levels of the system. Additionally, OSPI’s focus was the implementation and
sustainability of several large statewide initiatives including, but not limited to, Teacher and
Principal Evaluation (TPEP), Common Core State Standards (CCSS), the State Longitudinal
Data System (SLDS), and the Student Growth Model. It may be counter-intuitive that
teachers and principals have a hard time accessing student data, since student data
8
generally originates in the classroom and at the school level. The structure of the data
system, concerns about student privacy, and the lack of time allocated in the school day for
teachers, principals, and even district office directors to manage, sort, analyze, and use the
data put into the data system meant that the data were underutilized. Data sets improve
with use, and without constant use, the data were not high quality. Data were missing, and
of the data that were in the system, some of it was inaccurate. Inaccuracy is normal in data
sets that are not critically analyzed.
In a transactional data system the emphasis is on data input for upstream reporting. The
system in Washington State was fairly tight in terms of keeping the data moving up stream.
For example, many of the data coaches, who worked down the hall from the student
coordinator, were unfamiliar with the Skyward data system’s reporting capacity. It was not
unusual for data coaches to remark that they could not get access to the data for the school
they were supposed to be coaching. This issue of data access was not only a problem for
data coaches, it was a problem for teachers, principals, and district office directors
concerned with using data to make decisions. District-based data warehouse systems were
not the only data management systems that educators found difficult to access. The state
report card data is easy to view within a school or district view but limited for comparing
across schools and districts. State report card data was restricted to the school level so it
was impossible to analyze across third graders, for example. There were Excel spreadsheets
available for quick down loads of state report card data if an educator was inclined to work
in Excel or SPSS and conduct his/her own analyses. Even the data coaches were not
confident in their ability to conduct their own inferential statistics. The Query System was
another data system that most data coaches found difficult to access. It requires permission
from the district data coordinator, and while some data coaches were granted permission,
others did not get as far as identifying the right person in the district office to grant
permission. The data that were the most protected and often the most controversial were
the free and reduced-price lunch data. Some districts do not tell their teachers who qualifies
for free or reduced-price lunch and other districts did share those data if requested.
Complicating the data access and data use situation further, whether educators were
granted permission to have free and reduced-price lunch data or not, they tended to make
inferences about who did or did not qualify. In some cases, they made decisions based on
their impressions of students. The extent to which a district opens or closes their data
9
access to educators was a feature of the data environment that data coaches continuously
navigated with greater and less success. It is possible to make data requests to OSPI and at
least one data coach did request research data. The first request was fulfilled after six
months and the second request was fulfilled after three months and constant follow-up
requests urging the data office to prioritize the request. Transactional data systems were a
big step in the right direction in the early 2000s, but like all student information systems, by
the time they are debugged and ready to launch, they are outdated. By the time people learn
to use the systems they are even further behind. This is a problem not just in Washington
State but across the country (J. Wayman, 2003; J. C. Wayman, Brewer, & Stringfield, 2009; J.
Wayman & Chu, 2009; J. Wayman & Stringfield, 2006). For example, HOMEROOM is a data
display system designed primarily for teachers’ use as a response to the need to make a
commonly used transactional data system interactional. It was intended to replace the
conditionally formatted Excel spreadsheets that were emerging all over the state, developed
by teachers lucky to be trained in Excel. These teachers seemed to be few and far between
and rather than train the entire teaching cohort to use Excel, an entrepreneurial software
engineer developed a software option for teachers to view their student data at their
desktop. School Data Solutions (SDS), like most software developers, worked with a user
group from across the state to inform the development of the tool. There were no data
coaches in the development group, but data coaches took an active interest in the
development and potential implementation of HOMEROOM, keeping track of its
development and the implementation plan throughout 2012–13 school year.
As the SDS system HOMEROOM implementation began, new users immediately identified
new needs for the software, and new glitches in the use of the software. Some school
districts were using a variety of software for data display. For example, even within OSPI,
the School Success program was purchasing Data Director for the school success schools;
the Student Longitudinal Data System was being developed. The Education Research Center
was developing a web-based data display for education data, as was the Center for Research
on Education Data at UW Bothel. In some school districts, one school might be using
HOMEROOM, while the School Success program required them to use Data Director and
Indistar and they may have also been using SWISS for behavior data. None of these systems
work together. None of these data systems worked seamlessly and all of them required
more training than had first been expected, leading to these products being underutilized.
10
Additionally, as has been the case with much education software, about the time it is ready
for implementation, it has already been made obsolete by the early adopters who want it to
do more and better (Feldman & Tung, 2001; Goertz, Olah, & Riggan, 2010; Kerr, Marsh,
Ikemoto, Darilek, & Barney, 2006; Talbert, Scharff, & Lin, 2008). The promise of data
systems may be realized in the coming years as the developers continue to respond to the
users’ needs and the data coaches continue to support teachers learning to analyze student
data for instructional decision making.
Survey Results. The survey was conducted after the close of the final data coaching
institute. It was sent to all the ESD-based data coaches who participated in the year two
data coaching institutes, which made it a survey of the whole population of data coaches.
There was a 90 percent response rate on the survey. The full year two survey results are
included as an appendix of this report. Below is a summary and discussion of some of the
results.
Demographics. The majority of data coaches worked as directors or assistant
superintendents at ESDs. Each data coaching team included staff from a variety of programs
within their ESD, making these data coaching teams uniquely inter-program work groups.
Fifty-five (55) percent of the data coaches participated in at least eight days of training. And
only 10 percent of the data coaches participated in less than three days of training. The only
distinct difference in survey responses between the data coaches with more than eight days
of training and the data coaches with three or fewer days of training was that the data
coaches with less training were more sure, than their peers with more training, that their
ESDs were ready to provide all the data coaching services.
Discussion of team make-up. Due to program funding structures and tight funding
expectations, it is unusual for program staff to have the opportunity to work with staff from
outside of their area. This initiative was an opportunity for Educational Service District
(ESD) staff to work on an inter-program project. For example, the ESD112 data coaching
team had members who were student data coordinators from the Communication &
Information Management (CIM), special education, early childhood, prevention
intervention, research and assessment, and STEM. These inter-program teams were
constructed by design, although in some ESDs they also emerged by default. Given how busy
11
ESD staff are, team members may have been chosen because they had the flexibility to
participate, more than the inclination.
Data coaches rated the data coaching institutes as average or above average learning
experiences. The year one survey indicated that the majority of data coaches believed that
the data coaching institute did not provide new information or tools, but affirmed what they
already knew and already did. There also seemed to be further evidence of this tendency to
absorb data coaching into what the data coach already knew and did in year two. For
example, the most common data-coaching venue across ESDs, in year two, was with School
Success schools. The actual data practices reported tended to be the required/expected data
practices of the School Success program, which many of the data coaches had been
facilitating for multiple years before they were trained as data coaches. While there are
some similarities between the data practices in the School Success program and the data
practices in the data coaching tool kit including an inquiry cycle, problem definition, root
cause analysis, and action planning, the data coaching tool kit was intended to be used for
the establishment of data leadership/management teams which is a different focus than
school success. The research literature on implementing new practices in schools concurs
suggesting it is more likely for new practices to be absorbed into current practice, than to
change practice or replace current practice (J. P. Spillane, 2002; J. P. Spillane & L.K., 2002; J.
Spillane & Stein, 2005).
Data practices in schools and school districts. Just over half of the data coaches indicated
that it was uncommon for schools and school districts to ignore student achievement data.
All data coaches reported seeing some evidence of data coaching spreading in their regions,
but the majority of the data coaches (79 percent) also indicated schools and school district
data use focused primarily on students’ annual achievement test scores, which offer limited
use. Forty (40) percent of the data coaches reported that, while some school districts are
searching for data sources beyond student achievement data, only 21 percent of data
coaches reported that it was common across most school districts in their regions to search
for additional data sources for decision making.
Given the challenges generating, collecting, accessing, displaying, and analyzing data, it is
surprising that schools and school districts have not hired data expertise. Only 18 percent of
12
the data coaches reported schools or school districts hiring data expertise. The majority of
data coaches reported they had no knowledge of schools or school districts creating and
hiring a data expert position. It does seem reasonable that ESDs might be able to provide
data expertise for districts and schools given the challenges and the constraints that schools
and school districts face improving their data use.
ESD readiness to provide services. The majority of ESD team members reported, from the
start of this initiative, that they were ready to provide data coaching services through their
ESDs. Ninety (90) percent of the data coaches reported being ready to offer or already
providing analysis of student achievement data for a school and 76 percent were ready to
provide it for a whole school district. Over 83percent of the coaches reported being ready to
facilitate data analysis with teachers, principals, or central office administrators. That said,
73 percent of the data coaches reported that they were not yet ready to offer data analysis
using inferential statistics. The student achievement data analysis that ESDs are prepared to
provide falls short of inferential statistics. Descriptive statistics may be the norm and
considered adequate for students achievement test score analysis, but without knowing
how to do inferential statistics, it may be hard for the data coaches to assess what they are
missing. Seventy-six (76) percent of the data coaches reported being ready or already
offering training on the data coaching tool kit. Sixty six (66) percent of the data coaches
reported being ready or already offering planning for data coaching. Fifty-nine (59) percent
of the data coaches reported being ready or already offering support for developing
classroom assessment that show student growth. Overall, at least one data coach in each
ESD was ready or already offering all the data practices including generating data, collecting
data, displaying data, analyzing data, and facilitating decision making using data with
teachers, principals, or central office administrators. The data coaches are ready, and 47
percent of the data coaches reported that school districts were requesting some form of
data support from the ESDs, creating at least some possibility for data coaching to continue
to spread, even if it is not funded.
Data coaches indicated that the protocols in the data coaching tool kit were the same or
similar to protocols they already used. For example, the inquiry cycle at the center of data
coaching is similar to the cycle for school improvement planning, school success, and the
new teacher and principal evaluation process. The only service ESDs are already offering or
13
ready to provide is data analysis using inferential statistics. This service was not expected to
be an outcome of the data coaching institutes. The data coaching institute trained coaches to
facilitate meetings in which the district leadership teams were the experts on their own
data. The coaches were not expected to bring data, find data, or use the district data. Instead
the coaches were expected to facilitate the transition of district leadership teams to data-
focused leadership or data management teams. This situates data coaches as facilitators of
the social-cultural practices of leadership teams rather than data analysts or data
specialists.
Collaboration between ESD data coaching teams. The survey asked, to pursue data
coaching as a service to your districts, what supports would you want. The most commonly
reported support was to continue to meet with the other ESD teams to share emerging
practices. Of the data coaches who indicated further training, the training they want is in
data analysis skills.
14
Data Coaching Highlights From Each ESD During the data coaching institutes, each ESD presented on the data coaching work they had
accomplished between sessions. Unlike year one, when it was common for ESDs to report
no work on data coaching between institutes, in year two all the ESDs conducted some work
related to data coaching. It is worth highlighting at least one accomplishment of each ESD to
illustrate how each ESD took the data coaching training and tool kit and produced
something uniquely appropriate to their locally situated context. The work below highlights
only a small fraction of the varied data coaching work that emerged in each ESD during the
2012–13 school year1.
ESD 101, Walking our Talk. ESD 101 has a strong commitment to practicing internally
what they expect their districts to practice. To this end, the ESD developed an extensive
summer training for ESD staff and for a school district leadership team to learn the data
coaching cycle together. The district agreed to share their data and to model the cycle in a
fish bowl session. Each ESD department brought their staff to session and, after the district
modeled a stage of the cycle, the ESD teams practiced the stage internally. This process was
designed to span the full school year, and to replace some of the time spent in ESD
leadership team meetings.
ESD 105, Data Coaching Coop. ESD 105 has a strong culture of cooperative work with
their districts. They developed a cooperative of five districts all interested in increasing and
enhancing their data practices through data coaching. The original design included whole
co-op meetings for all the districts to the data coaching cycle together in one training. The
ESD differentiated training for each of the districts in the cooperative, and rather than
having the districts come to the ESD, the ESD went to the districts to provide personalized
training for each district. The flexibility and adaptability of ESD staff to the needs of the
school districts is a common and expected attribute of ESDs. Responding to the request for
differentiation was therefore viewed as a logical progression for learning.
1 North Central ESD is not included only because there was no opportunity for a timely interview.
15
ESD 112, Special Education. ESD 112 runs a large special education co-op. In 2012–13 the
longtime leader of the department retired and the new leader came in from a school district
where the district’s data were readily available for teachers and school and district leaders.
This was not the case for ESD 112 co-op teachers or directors. The data coaching team
focused on training the special education co-op directors to use data coaching as a tool to
increase and enhance their work with the general education teachers and school and
district leaders.
ESD 113, Early Childhood. ESD 113 leads a multi-county, long-standing, successful Head
Start program. The early childhood leadership team and the data coaching team worked
together to develop a training process to meet the growing demands for data practices in
Head Start programs. Drawing on inquiry cycle and some of the protocols in the data
coaching tool kit, the data coaching team designed a focused tool kit to train Head Start
leadership teams to meet the new data-use expectations in their programs. ESD 113 began
by training the local Head Start leaders working at ESD 113 and quickly found themselves
responding to requests for training throughout Washington State and far beyond
Washington State.
OESD 114, Inquiry Cycles. OESD 114 joined the data coaching institutes with confidence in
the data coaching cycle. They had been successfully leading cycles of inquiry in their math
and science partnership (MSP) work for many years. They were ready and able to take the
data coaching cycle into the majority of funded work including school success, MSP, and the
new teacher and principal evaluation project, which also relies on an inquiry cycle.
ESD 123 School Success. ESD 123, focused much of their data coaching work on their focus
and priority schools in the school success program, working to meet the goals for school
improvement. Data coaching made a nice match with the expectations within the school
success program helping districts collect, analyze, and use data for strategic planning and to
monitor changes toward improvement.
NWESD 189, HOMEROOM. NWESD 189 sits in the shadow of WSIPC. At the start of year
two, the lead data coach from the ESD became the leader of the ESD’s data center, working
closely with WSIPC. Given this proximity to WSIPC, NWESD 189 was the first ESD to support
16
the implementation of HOMEROOM, WSIPC’s data display system designed primarily for
teachers. The training success and challenges were worked out at NWESD 189 and the rest
of the ESDs benefited from their implementation trials. The ESD used the implementation of
HOMEROOM as an invitation for data coaching. Many of the opportunities to conduct
training on the tool kit were associated with HOMEROOM implementation. This was
perhaps the implementation example that was closest to what was originally conceived by
WSIPC.
Puget Sound ESD 121, Using data to determine cultural competence throughout the
early childhood program. The data coaching team at Puget Sound ESD 121 was led by the
early childhood department. This team focused on learning to use data to assess and adjust
their own cultural competence. This was work that was happening across all the programs
at the ESD, and the early childhood team had the advantage of time together to discuss their
data needs and uses for this assessment process.
17
Research Questions Answered in Brief
1. How, if at all, did the data coaching institute shape the work of the ESDs in year two
of implementation?
The assistant superintendents for teaching and learning at the ESDs have worked over the
past ten years to become a collaborative network. The data coaching institutes provided
another opportunity to further this interest. The data coaches from each ESD used the data
coaching institutes to share useful practices, and develop and share plans for new services.
Data coaches reported that they would like to continue meeting and learning from and with
each other. This collaborative approach of ESDs ensures that best practices spread across
ESDs. This means districts throughout the state can have access to the best practices that
emerge in a small district hundreds of miles away, equalizing access and opportunity across
all the districts in the state.
2. What, if anything, are tangible outcomes of ESDs participating in three data coaching
institutes during the 2011–2012 school year?
ESDs are all providing some form of data coaching services. Each ESD produced a creative
and locally-appropriate approach to data coaching.
3. To what extent are ESDs ready to pursue data coaching as a service to their regions?
All ESDs are ready to pursue all aspects of data coaching as designed. This means data
coaches are ready to facilitate district or school leadership teams becoming data-focused
leadership teams or data management teams. Across all the ESDs data coaches went beyond
the PCG approach to data coaching, and designed their own locally appropriate approach to
increasing and enhancing data practices in their region.
4. What more do ESD participants need in order to provide data coaching services in
their regions?
Data coaches want to learn basic inferential statistics to help their schools and school
districts. Data coaches indicated strong interest in continuing to meet to learn with and
18
from each other. They want to learn statistics in the context of the problems of practice
their schools and districts face using available data. The data coaching toolkit does not
require that data coaches know or use inferential statistics, but the majority of data coaches
indicated that statistics would be useful for data coaching. Data coaches were expressly
trained not to bring data, analyze data, or comment on the districts’ data, but instead to
facilitate the district leaders’ bringing data, analyzing data, and discussing data. Given that
most of the data coaches were not comfortable accessing or analyzing their districts’ data,
this helped them to be more comfortable in their role at the start. By the second year, the
data coaches wanted to know more about accessing data and analyzing data with and for
their districts. There is clearly an emerging need for data analysts throughout the states.
5. What are the supports and constraints on ESD resources for data coaching?
From one perspective, there seem to be no constraints on data coaching. Each is ready to
pursue data coaching and there is already evidence that each ESD developed a creative and
locally appropriate way to bring data coaching into the current work in their region. From
another perspective, without funding, data coaching will not be implemented as intended.
No ESD implemented data coaching as intended by PCG. Data coaching was largely absorbed
into the current practices of each ESD. This initiative showed that ESDs can and will find
creative and locally invited ways to bring new practices into their work.
19
Conclusion and Recommendations
What would a fully mature and productive data use practice look like for a state system of
education?
Integrated data systems with built in analytics. It is a rare school district and only one ESD
that has a full time researcher on staff, with responsibility for analyzing data sets to answer
pressing problems of practice in the field. For now, data-informed decision making is largely
rhetoric due to poor access to data and lack of personnel with data analysis skills. One
trained researcher, with a well-trained support person, could increase data use
substantially. ESDs now have data coaches ready to facilitate data conversations. What they
are not as ready to do is access and analyze the data.
Data access and analysis continues to be a problem. There are multiple sites (e.g., ERDC,
EDRC, OSPI Report Card, SLDS) where some education data and some data analytics can be
found and used. These are far from comprehensive and do not provide the flexibility
necessary for sophisticated analysis. It is possible to request administrative data from OSPI
and, with a data sharing agreement, a researcher can have student level data identified with
research numbers rather than names. These requests tend to take three to five months to
process. In context this is not as long as some school systems. Currently, Chicago tells
researchers to expect two years for data requests. All of this is much too slow for developing
robust data use practices in education.
There are additional obstacles to robust data practices. The data are strongly protected
from the student data coordinators. This is, of course, necessary for sensitive student data,
but it is also a challenge for ESD data coaches who need to access the data to facilitate data
analysis for the districts. This challenge can be overcome with data sharing agreements
between school districts and ESDs, but it can take up to a year for an ESD researcher to put
in place a data sharing agreement with each school district in his/her region. Then the data
needs to be accessed district-by-district, and comparisons between districts take much
more time than necessary, given the data could be rolled up into regional data sets directly
by WSIPC and OSPI. To be fair WSIPC has developed an administrative review that rolls up
data at the ESD level, but it is limited to the data that is put into the system at the school
20
level, which may not include state assessment data. Not only would ESDs benefit from a
definition of data access, small districts could benefit from sharing data to increase their
analysis population.
Working with sensitive data. Data use in education suffers from concerns about how data
might be misused. Some of these concerns are warranted. For example, individual students’
grades are private and must be protected. But from whom?
If the ESDs are going to support data-informed decision making, data analysis to help solve
pressing problems of practice, or data analysis to produce better understanding of the social
and political dynamics of their districts, data will have to be shared. As an example of just
how challenging the situation is currently, when HOMEROOM was being implemented in
pilot districts, it was set up for teachers to see only the students who were currently
enrolled in their classes. In middle school and high school, this made it impossible to
explore how their students were doing in other classes or to see if attendance, for example,
was a problem at any other time of day than their classes. Teachers, who are in a good
position to identifying pressing problems of practice, were extremely limited in their access
to their students’ administrative data. Even more difficult for teachers and administrators in
many districts is the restriction on analyzing free and reduced-price lunch data. When
schools are told they are failing their students who receive free and reduced-price lunches,
but they are not allowed to know who these students are, there is a serious limitation to
useful data analysis. It is perhaps time to re-evaluate what “businesses need to know”
stipulation in FERPA to include at least data analysis professionals who work in the interest
of the school district.
If data analysis is an important new approach to making decisions there are currently few
positions in schools, school districts, or at ESDs for data analysis. This clearly takes
specialized skills, generally developed in graduate programs with a research focus. It might
be necessary to allocate resources for these positions, as many school districts have across
the country.
21
The future of data use in education. While educators are both absorbing data into their
current practices and pressing for better tools (assessments, data displays, data analytics)
to increase the effectiveness of their data use, it is still unclear who (teachers, principals,
central office directors) needs to learn what in order to turn data into the powerful tool it
purports to be. It remains an empirical question what the actual benefits are of data use in
education (Goren, 2012). The promise of big data, as used by businesses like NIKE, and
Albertsons, to predict who will shop for what, in which store, on which day has little chance
of helping algebra teachers determine how to help their students learn functions. The
promise of geo-coded data and special analytics can provide the legislature with the exact
cost of the transportation routes for each students in the state; however, it cannot provide
the exact cost of educating each of those transported students.
The new Smarter Balanced assessments, nicely aligned to the new national
standards, will provide student achievement data; however, its promise is already
constrained by the challenges of forging data sharing agreements between school districts
and states. In other words, for all the promise of data, education has a long way to go to
predict, or accurately map, or describe and explain leading and learning in schools. As
wonderful as it would be to have an analytic data system to predict, describe and explain
the strengths and needs of each student, each teacher and each education leader remains far
from that vision.
Recommendations and Promising Possibilities. 1. Think big data. The promise of data-use reaches far beyond the narrow constraints
of current data practice in most school districts. The STARS geo-spatial data system,
developed and used by OSPI’s transportation group, is an example of the where
education data use might aim. Using geo-spatial data analytics and cartography tools
allows education analysts to map multiple layers of large data sets, across the state
landscape to display complex and under considered relationships important to
understanding complex and pressing problems of practices.
2. Think beyond student achievement data. These test score data are limited and often
useless in helping teachers and school leaders making consequential decisions. At
the same time, as big data analysis capacity is being developed, teachers and leaders
need data tools close at hand to support quick decisions including how to improve
instruction tomorrow. For these decisions video cameras are powerful. Collecting
classroom observation data with phone and tablet cameras provides a new source
22
of empirical data for teachers and principals to analyze actual classroom interaction.
No longer reliant on student achievement data as a proxy for learning, teachers and
principals can review and critically analyze learning in real time providing clear,
accurate, and meaningful evidence based feedback for teachers to use to perfect
their practice. Lack of trust constrains the use of video in many schools. The lack of
trust may be addressed as teachers and principals learn to collaboratively, analyze
classroom data from a growth prospective
Finally, this report is a product of a promising data practice. Data coaching began with a
research plan. Beyond evaluating the data coaching institutes, the leaders had wanted to
bring qualitative tools to document and study what happens when you bring all nine ESDs
together to collaborate on a new initiative. Using some of the tools from evaluation
including providing feedback on program implementation as it is emerging, the research
focused not on whether the data institute accomplished preset expectations but instead,
recognized that there were no preset expectations and it was therefore important to follow
what was emergent and new. Partnering with researchers to study the implementation of
new policy and programs is becoming more common, AIR has partnered with the Teacher
and Principals Evaluation Project (TPEP) to study the policy implementation and provide
feedback to the state steering committee throughout the implementation and providing
insight into the process, which is emergent and new. This form of researcher-practitioner
partnership holds significant promise for empowering education practitioners to learn from
their practice to continuously invent their practice in the local and highly complex,
education environments they lead.
23
References
Coburn, C., & Talbert, J.E. (2006). Conceptions of Evidence Use in School Districts: Mapping the Terrain. American Journal of Education, 112.
Cohen, L., Manion, L., & Morrison, K. . (2007). Research Methods in Education (Sixth ed.). New York: Routledge.
Datnow, A., & Park, V. (2009). School system strategies for supporting data use. Routledge, New York.
Feldman, Jay, & Tung, Rosann. (2001). Using Data-Based Inquiry and Decision Making To Improve Instruction. ERS Spectrum, 19(3), 10-19.
Goertz, M., Olah, L., & Riggan, M. (2010). From Testing to Teaching:The use of interim assessment in classroom teaching: Univeristy of Pensylvania.
Goren, P. (2012). Data, Data, and More Data- What's an Educator to Do? American Journal of Education, 118(2), 233-237.
Halverson, R., Grigg, J., Prichett, R., Thomas, C., & Wisconsin-Madison, University of. (2006). The New Instructional Leadership: Creating Data-Driven Instructional Systems in Schools. Journal of School Leadership.
Ingram, D., Louis, K., & Schroeder, R. (2004). Accountability policies and teacher decision making: Barriers to the use of data to improve practice. Teachers College Record, 106, 1258-1287.
Kerr, K.A., Marsh, Julie A., Ikemoto, Gina Schuyler, Darilek, Hilary, & Barney, Heather. (2006). Strategies to Promote Data Use for Instructional Improvement: Actions, Outcomes, and Lessons from Three Urban Districts. American Journal of Education, 112(4), 496-520.
Knapp, Michael S., Copland, Michael A., Swinnerton, Juli A., & Monpas-Huber, Jack. (2006). Data-Informed Leadership in Education: Center for the Study of Teaching and Policy.
Kowalski, T., Lasley, T, & Mahoney, J. (2008). Data-Driven decisions and school leadership:Best practices for school improvement Boston: Pearson.
Lodico, M.G., Spaulding, D.T., & Voegtle, K.H. (2006). Methods in educational research: From theory to practice. San Francisco: Jossey-Bass.
Marsh, J. A., Pane, J. F. , & Hamilton, L. S. . (2006). Making Sense of Data-Driven Decision Making Sense of Data-Driven Decision
Making in Education: Evidence from Recent RAND Research. . Santa Monica, CA: RAND.
24
Means, B., Padilla, C., DeBarger, A., & Bakia, M. (2009). Implementing data-informed decision making in schools—teacher access, supports and use. In C. o. M. a. S. U.S. Department of Education (Ed.). Washington, DC: Evaluation, and Policy Development.
Miles, M.B., Huberman, A.M., & Saldana, J. (2013). Qualitative Data Analysis: An Expanded Source Book (Third ed.). Thousand Oaks, CA: Sage.
Spillane, J.P. (2002). Local theories of teacher change: The pedagogy of district policies and programs. Teachers College Record, 104(3), 377-420.
Spillane, J.P., & L.K., Seashore. (2002). School improvement process and practices: Professional learning for building instructional capacity. The LSS Review, 1(2), 12-13.
Spillane, James, & Stein, Mary Kay. (2005). What can researchers on educational leadership learn from research on teaching? Building a bridge. In W. Firestone & C. Riehl (Eds.), A New Agenda for Research in Education Leadership (pp. 28-45). New York: Teachers College Press.
Supovitz, J.A., & Klein, V. (2003). Mapping a course for improved student learning: How innovative schools use student performance data to guide improvement. Philadelphia, PA- Consortium for Policy Research in Education.
Talbert, Joan, Scharff, Nell, & Lin, Wendy. (2008). Leading School Improvement with Data:A Theory of Action to Extend the Sphere of Student Success. Palo Alto.
Wayman, J. (2003). Multiple imputation for missing data: What Is It and how can I use It? (C. F. S. O. O. Schools, Trans.). Johns Hopkins University.
Wayman, J. C., Brewer, C., & Stringfield, S. (2009). Leadership for effective data use. Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA.
Wayman, J., & Chu, V. (2009). Preparing educators to effectively use student data systems: Routledge, New York.
Wayman, J., & Stringfield, S. (2006). Technology-Supported Involvement of Entire Faculties in Examination of Student Data for Instructional Improvement. American Journal of Education, 112.
25