Northwestern DBIR brown bag

25
Design-Based Implementation Research: Working in Partnership(s) to Transform the Relationship of Research and Practice Bill Penuel (@bpenuel) University of Colorado Boulder Barry Fishman (@barryfishman) University of Michigan May 23, 2013

description

Brown Bag presentation by Barry Fishman and Bill Penuel at Northwestern University School of Education and Social Policy on Design-Based Implementation Research (DBIR), presented on Thursday, May 23rd, 2013

Transcript of Northwestern DBIR brown bag

Page 1: Northwestern DBIR brown bag

Design-Based Implementation Research:

Working in Partnership(s) to Transform the Relationship of Research and Practice

Bill Penuel (@bpenuel)University of Colorado Boulder

Barry Fishman (@barryfishman)University of Michigan

May 23, 2013

Page 2: Northwestern DBIR brown bag

Acknowledgments• National Science Foundation 1054086

• Annie Allen, University of ColoradoBritte Haugan Cheng, SRI InternationalNora Sabelli, SRI InternationalAndy Krumm, SRI International

Page 3: Northwestern DBIR brown bag

Where We Begin• Many promising educational interventions have been

developed, validated, tested... Then what?

• The majority fade away as funding ends or attention turns elsewhere

• A few are sustained in (or near) the contexts where they were developed

• A very few are brought to scale and are used across settings/contexts

• Why is the rate of “success” so low?

Page 4: Northwestern DBIR brown bag

A Validity Problem• Interventions are usually developed in “hothouse” environments

• Researchers seek to reduce sources of variation in evaluations to increase internal validity

• Funded research focuses more on developing and validating interventions from basic research (Type I translation) than on understanding or closing gaps between research and practice (Type II translation)

Page 5: Northwestern DBIR brown bag

Type I Translation

Type II Translation

What is being translated?

Translating principles from basic learning

research into interventions

Translating interventions

developed for one or a few settings into

interventions that are scalable to many

settings

What kind of research is involved?

Design-based researchEfficacy and

effectiveness trials

Implementation research

(for example, DBIR)

Page 6: Northwestern DBIR brown bag

Type I Translation

Type II Translation

What kinds of questions does translational

research answer?

What/how do people learn from this design?

What do problems in learning or implementation

suggest about redesign?

What kinds of capacities are required to implement

this design?What supports are needed for people implementing the design to adapt it in ways congruent with the design’s core principles?

Who is involved?

Learning scientists, classroom teachers,

subject matter experts, often also software

developers

Learning scientists, organizational researchers, teacher leaders, school and

district administrators, often also publishers and

enterprise software engineers

Page 7: Northwestern DBIR brown bag

Design-Based Research• The objective of DBR is to develop theory, not

just empirically tune “what works”• “Design experiments create the conditions for

developing theories yet must place these theories in harm’s way.”

• The theory is expected to do “real work in practical educational contexts”

(Cobb, Confrey, diSessa, Lehrer, Schauble, 2003)

Page 8: Northwestern DBIR brown bag

Design-Based Implementation Research• Blends learning sciences and policy

implementation research traditions and methods

• Learning sciences: iterative, collaborative, guided by and informing theories of learning/teaching

• Policy implementation: focus on conditions for implementation effectiveness, guided by and informing theories of institutional change and organizational learning. Focus on the design of systems and infrastructure

Page 9: Northwestern DBIR brown bag

DBIR Principles1. A focus on jointly-defined problems of teaching and

learning practice

2. A commitment to iterative, collaborative design

3. A concern with developing knowledge and theory through disciplined inquiry

4. A goal of developing capacity for sustaining change in systems

Source: Penuel, W. R., Fishman, B., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337.

Page 10: Northwestern DBIR brown bag

Example: The Center for Learning Technologies in Urban Schools

Page 11: Northwestern DBIR brown bag

Commitment to Iteration & Collaboration

• All work unfolded over multiple phases of pilot testing and revision, starting with a small core of teachers and then expanding

• Design teams formed “work circles” that included university and school personnel

• Project leadership worked to forge new relationships within and across institutions

– E.g., Bringing curriculum and technology leadership together in Detroit Public Schools

– E.g., Convening technology leadership across districts

Page 12: Northwestern DBIR brown bag

Concern for Developing Knowledge and Theory Through Inquiry• The goal throughout was not only to create better

materials

• Goal was to develop new knowledge by developing the theory of educative curriculum materials, with a range of subgoals, e.g.:

– Study of teacher learning through professional development that linked PD to student outcomes

– Research on cognition supported by technology

– Evolving theory of system capacity for supporting reform, and how to design for systemic change

Page 13: Northwestern DBIR brown bag

A Concern with Developing Capacity to Sustain Change

Page 14: Northwestern DBIR brown bag

Evidence Standards for DBIR

Page 15: Northwestern DBIR brown bag

Design and Development

EfficacyTrials

EffectivenessStudies

Involvement of R&D Team

Involvement of Evaluators & Practitioners

Page 16: Northwestern DBIR brown bag

Design and Development

EfficacyTrials

EffectivenessStudies

Involvement of R&D Team

Involvement of Evaluators & Practitioners

Sustainability?

+0.50

+0.00

Page 17: Northwestern DBIR brown bag

Replication… Really?

• Organizational replication is often thought of as a process that yields reliable results at the expense of local and professional control

• Peurach and Glazer (2011) argue for a “knowledge-based logic” in which program developers collaborate with schools to produce, use, improve, and retain practical knowledge

Page 18: Northwestern DBIR brown bag

Key Differences from (old) IES Model

• Teams plan for scaling and sustainability from the start

• Designers maintain their involvement throughout, because design is ongoing and iterative

• Powerful innovations are not expected to survive in the wild without changes at the system level

Page 19: Northwestern DBIR brown bag

Generating Forms of Evidence in an Era of “Fast Science”

• Get it mostly right, fast.– How do you know you “got it right”?– What’s a “minimum viable” innovation to test?

• Fail early, fail often.– How can you put your theory in harm’s way from the

start?– What documentation can benefit your own and other

teams?– When do you do the “big test”?

Page 20: Northwestern DBIR brown bag

Matching Phase of Development to Phase of Research

Phase of Development

Driving Questions Sources of Evidence

Problem Negotiation What problem of practice should be the focus of our joint work?

Available data from multiple sectorsResearch evidencePerspectives and values of stakeholders (including nonschool actors)

Co-design What should be the focus of our work?To what extent do teams leverage the diverse expertise of stakeholders?

Design RationalesEthnographic accounts of design processes

Page 21: Northwestern DBIR brown bag

Matching Phase of Development to Phase of Research

Phase of Development

Driving Questions Sources of Evidence

Early implementation How do implementers adapt the innovation to their local contexts?How do implementers use the innovation to reconstruct their practice?What are the appropriate measures of impact?

Observations of implementationInterviewsAssessment design

Efficacy What is the potential impact of the innovation on teaching and learning?What mediates impacts on learning?

Randomized Controlled TrialsInterrupted Time Series DesignsExplanatory Case Studies

Page 22: Northwestern DBIR brown bag

Matching Phase of Development to Phase of Research

Phase of Development

Driving Questions Sources of Evidence

“Translation” (Type II) What supports are needed to implement the program effectively?What are the conditions for sustainability?

Experimental comparisons of different means of supportExplanatory comparative case analysis

Page 23: Northwestern DBIR brown bag

Challenges in DBIR

• Establishing shared relevance• Balancing rewards across partners• Collaborating across disciplines• Synchronizing time tables and cycles• Earning trust

Page 24: Northwestern DBIR brown bag

Funders Interested in DBIR

• National Science Foundation• US Department of Education– RELs– Research-practice partnerships– Continuous Improvement Research in

Education

• William T. Grant Foundation– Use of research evidence RFP– Funding of work in research-practice

partnerships (white paper, Coburn-Penuel study, Carnegie study)

Page 25: Northwestern DBIR brown bag

Coming Soon

Forthcoming volume on DBIR:

Fishman, B., Penuel, W. R., Allen, A., & Cheng, B. H. (Eds.). (2013). Design-based implementation research: Theories, methods, and exemplars. National Society for the Study of Education Yearbook, Vol. 112(2). New York: Teachers College Record.

Also: http://tinyurl.com/dbir-aera