How ed-tech evaluations provide the evidence of success that ...Introduction Study design vs. needs...

How ed-tech evaluationsprovide the evidence of success that school and district buyers require

Curiosity for better learning

Introduction Study design vs. needs

ESSA + evidence claims Final thoughts

Appendix

Case studies

Finding a research partner

Things to consider

01 07

02 09

10

12

04

05

TABLE OF CONTENTS

You’re an ed-tech creator who’s spent months—maybe years—developing a new app or software product that you think will improve and transform teaching and learning. It’s been a long journey as you went from a great idea to a persuasive demo to development and production. Now you’re excited to roll the product out to schools so teachers can use it with their students.

But during initial conversations and product demonstrations with school staff and administrators, you’re getting questions about your product’s evidence base, impact, and something called ESSA tiers—questions that you rarely, if ever, got a few years ago when you launched other ed-tech products. What does all of this mean? What’s driving this focus on your product’s evidence of success?

With this brief, we’ll give you the background on:

• Why schools and districts are asking, more than ever before, about evidence of your product’s success before they commit to using it.

• How to responsibly address these questions through an evaluation of your product, including key considerations before engaging in evaluation and what to expect during the evaluation process.

• Best practices for ensuring the evaluation of your product is insightful and actionable.

• The types of evaluation necessary to use federal funds to purchase.


INTRODUCTION

1

ESSA lists four progressively rigorous “tiers of evidence” that would qualify a product to be purchased using Title I and other federal dollars (more detail about the ESSA tiers can be found at the end of this brief).

Education decision-makers are being pressed to be good stewards of federal dollars; with this comes the expectation that vendors will provide evidence that their product meets the ESSA requirements.

What type of evidence claims do you want to make about your product?When considering the types of claims you’d like to be able to make about your product and how you can meet the ESSA evidence requirement, you’ll have to weigh the pros and cons, in time and cost, of conducting

a study that conforms to each of the different tiers and what types of product evaluations can produce the necessary level of rigor.

As an example, a correlational study, which can show a relationship (or association) between your product and improved student outcomes, can produce promising evidence more cheaply than a randomized controlled trial (or experiment). The drawback is that it will not allow you to make causal inferences such as, “The use of our product resulted in (or caused) better student outcomes,” which would be more compelling to a prospective user.

The 2015 Every Student Succeeds Act (ESSA) requires that states, districts, and schools wishing to leverage federal funds to purchase ed-tech products and other school interventions ensure purchases have an evidence base of performance.

ESSA Tiers of Evidence

TIER 1—Strong Evidence

TIER 2—Moderate Evidence

TIER 3—Promising Evidence

TIER 4—Demonstrates a Rationale

ESSA + EVIDENCE CLAIMS

Curiosity for better learning 2

if you want to claim . . . then, you’ll need . . .

“All else held constant, our program has a statistically significant impact on student/school achievement.” Tier 1—Strong Evidence

“Schools/students that used our program demonstrated significantly higher achievement than similar students/schools that did not use our program.”

Tier 2—Moderate Evidence

“There is a positive, statistically significant relationship between using our program and student achievement.” Tier 3—Promising Evidence

“Our program is grounded in rigorous research and a theory of change that leads us to believe that, if implemented, it would impact student achievement.”

Tier 4—Demonstrates a Rationale

So, what do you want to be able to say about your product? Following are examples of claims you might want to make, and the level of evidence that would be required. The claim you want to make will guide the design, implementation, and cost of the evaluation.


How should I select a research partner?principal investigators to ensure they have doctoral-level training in research methodologies.

Finding a critical friendThe “independent perspective” is key—and why it’s best to work with a reputable organization that can provide unbiased results that remain true to the data (even if they’re not your desired results). A good research partner will be able to provide insights to help you make improvements—be it with product design, user experience, training, or implementation support. So, you may find it helpful to work with a team that has broad expertise in education—including curriculum, instruction, assessment, and implementation—so they can help you make sense of research results in light of the practical realities of school improvement as well as help you identify meaningful ways to improve your product.

How can your product reach the K–12 market with top-quality evidence of success backing it up?

Contact McREL at 800.858.6830 or [email protected] to get started.

If your ed-tech company has an in-house research team with the capacity, skills, and experience needed to plan and conduct a rigorous evaluation, you could evaluate your own product—but this runs the risk of being viewed as a conflict of interest, especially if you’re asked if the product meets the ESSA evidence requirement. It would be better to work with independent researchers from a reputable organization who can provide expertise, integrity, and valuable insights to improve your product.

How do you find the right research partner? An online search for program evaluation, education research, or ed-tech efficacy studies can give you a starting list of companies. Check out their websites and look for case studies, example reports, and other documents that demonstrate their capacity and experience in studying products similar to yours. Contact them and ask about the background, experience, and credentials of their

FINDING A RESEARCH PARTNER


What things should I consider prior to launching a study?Once you’ve identified a research partner, here are a few things to consider as you work with them to develop your plan.

Timelines and costsThe type of claim you want to be able to make will affect both the timeline and cost of your study. Studies producing more rigorous results (for ESSA tiers 1 or 2) generally require more time and personnel than those with lower levels of rigor (ESSA tiers 3 or 4). Producing “strong evidence” (tier 1), for example, requires a randomized controlled trial (experiment), which typically takes a year or more to complete. You’ll need to recruit a large pool of participants, randomly assign units (schools, teachers, students) to be in a control group, support members of the experimental group in using your product, ensure the product is implemented with fidelity for the required duration to generate measurable effects, and ultimately collect, analyze, and report data from all sites. All of these activities, of course, increase the cost

of the study; hence most rigorous research studies cost $100,000 or more to complete—not including your in-kind contribution of the product and, if part of your intervention, training for participants in the experimental group.

A study at the “promising evidence” (tier 3) level, on the other hand, can usually be completed in a matter of months, especially if you’ve already collected data from existing users that you can provide to your research partners to analyze and summarize in a written report. And a “demonstration of rationale” (tier 4) report can typically be completed in weeks, especially if you’re already clear about the research base for your product and want your research partner to help you create a well-documented and cogent rationale for your product.

Institutional Review Board (IRB) requirementsBefore data collection can begin, researchers—and, in some cases, participating schools or districts—

THINGS TO CONSIDER


must secure approval from an IRB to ensure ethical compliance in the proposed research. Your IRB may consist of members of your team, your research partners, and clients who are committed to ensuring the integrity of your research design and the rights of study participants. IRB requirements include ensuring all participants understand and agree to the data collection methods, confidentiality protections, participants’ rights, and the researchers’ responsibilities. Your research partners should demonstrate they have well-established protocols for data integrity, security, and privacy before the study begins.

Signing a Memorandum of Understanding (MOU)Before you launch your study, be sure to draft an MOU with your research partner to establish study expectations among participating schools, districts, your company, and researchers. This MOU, which representatives from all participating organizations should sign, should include statements about how data will be shared and kept secure, and how participant confidentiality will be protected.

Recruiting participantsThe types of claims you want to make about your product will guide how you recruit participants. If you’re seeking to make claims about how the product impacts users compared to non-users, you’ll need data on not only students (or teachers) using the product (the treatment group) but also those not using it (the control group). You will need to recruit schools that are willing to participate in your study, and then randomly assign half of them to the control group. Also, bear in mind that for random control experiments you cannot include current users of your product because this would “contaminate” the study with sites that have already benefited from the product. Thus, you and your research partner will need to recruit schools and districts that do not currently use the product.

With all this in mind, it’s no surprise that recruitment is often one of the most challenging and time-intensive elements of a study. Many schools and districts are reluctant to divert teacher and student attention to anything that doesn’t closely align with their existing improvement plans and strategic priorities even if it’s offered at no cost—a common perk of study participation. Be prepared to cast a wide net to find enough participants.


Remember, it’s your study. Be sure it meets your needs.

product design. Later, when your product is ready, you can commit to a larger (and more costly) rigorous study. In short, don’t let anyone talk you into putting the cart before the horse.

Are users implementing with fidelity?Effective implementation tends to be the exception, not the norm, with education interventions. Ed-tech products are no different. Districts use only a small fraction of the licenses they purchase from software companies. So, it’s likely that actual use of your product falls short of what’s needed for it to demonstrate positive effects. Before committing to an efficacy study, you may want to learn how well your product is being used—how long, how frequently, and whether that use would be considered “enough” or “appropriate.” For example, imagine you’ve created a formative assessment app that’s designed to be used 10 minutes a day to measure student learning and guide instruction.

You, no doubt, believe in your product. You’re certain it has tremendous benefits for those using it. So, you may be confident it’s ready for an evaluation. And you’re likely to find researchers who are willing to oblige—after all, big, rigorous studies are exciting and, frankly, lucrative. But before you go “all in” on a major study, it’s a good idea to consider how well the study design matches your needs.

Is your product fully developed?For starters, consider your product’s stage of development. If you’re still in beta, there’s no need to conduct a rigorous study. Not only does your product likely still have some kinks to work out, but also by the time the study is released to the world, in a year or two, both the product and the state of the market may have changed considerably. So, if you’re still in early-stage development, a better option may be an implementation study—collecting preliminary data on your product’s effects while documenting users’ experiences with it to help you improve product-related training, address implementation challenges, and loop user feedback into

STUDY DESIGN VS. NEEDS


You’d want to know how many teachers actually use it a) daily, b) for 10 minutes, and c) to adjust instruction. Ideally, you’d collect these implementation data alongside outcome data to show the benefits of full versus partial implementation so you can refine recommendations for usage.

Is your professional development effective?Something else to consider before committing to a major study is whether your training program—if that is part of your product—effectively helps educators use your product as designed. Bear in mind that studies

show most professional development opportunities do little to change educators’ practices—especially if they’re one-shot trainings without follow-up guidance or coaching to help teachers embed new tools or practices in their classrooms. So you may need to review your professional development program to see if it provides participants with the knowledge and skills they need to implement your product effectively—and whether they need follow-up guidance and coaching to be able to sustain consistent, high-quality use of your product over time. Here again, before diving into an evidence-of-impact (tiers 1 or 2) study, you may wish to engage in an implementation study while collecting data to demonstrate promising (tier 3) evidence.


Remember the real purpose of your researchBeyond compliance with federal regulations or improving marketing and sales pitches, there’s a deeper purpose, of course, to researching the impact of your product. Having worked with many ed-tech companies, we know their leaders, employees, and investors are driven by a desire to create something that improves student outcomes. That’s why, as a nonprofit dedicated to improving school systems and enhancing student learning experiences worldwide, we support ed-tech

companies with our research services—to ensure that the multibillion-dollar investment school systems continue to make in education technology has an impact where it matters most, in the lives of educators and the students they serve.

We hope you, too, will view research as we view it—as on ongoing quest to transform our education system so that it provides rich learning experiences and opportunities for all children.

To find out how to build an evidence base for your product, contact McREL at 800.858.6830

or [email protected].

FINAL THOUGHTS


Tier 1—Strong Evidence: supported by one or more well-designed and well-implemented randomized controlled trials (RCTs). RCTs are viewed as the gold standard of studies, involving rigorous research methods and random assignment of participants to treatment and control groups.

Driving the questions: ESSA “tiers of evidence”The 2015 federal education act known as ESSA (Every Student Succeeds Act) prioritizes the use of evidence-based interventions in schools across the country. The U.S. Department of Education developed standards describing four levels (“tiers”) of scientific rigor used to evaluate education interventions (See Table A on p. 11). These four tiers of evidence are:

APPENDIX

Tier 2—Moderate Evidence: supported by one or more well-designed and well-implemented quasi-experimental studies. Still a rigorous research design, but these studies don’t use random assignment to create treatment or control groups. Instead, the researchers identify commensurable comparison groups.

Tier 3—Promising Evidence: supported by one or more well-designed and well-implemented correlational studies which will determine if there is a relationship (correlation) between the intervention and proposed outcome.

Tier 4—Demonstrates a Rationale: practices that have a well-defined logic model or theory of action, are supported by research, and have some effort underway by a state department of education, school district, or research organization to determine effectiveness.

Interventions applied under Title I, Section 1003 (School Improvement) are required to have strong, moderate, or promising evidence (tiers 1–3) to support them. All other programs under Titles I–IV can rely on tiers 1–4. (Source: https://www.cde.ca.gov/re/es/evidence.asp).


type of claim you’d like to make

ESSA tier study design timeline typical

cost

“Use of our program led to higher student achievement in math.”

Tier 1 Strong

Evidence

At least one well-designed and implemented randomized controlled trial/experimental study.

6–18 months $$$$

“Schools/students that used our program demonstrated significantly higher achievement than similar students/schools that did not use our program.”

Tier 2 Moderate Evidence

At least one well-designed and implemented quasi-experimental study. This is a rigorous method that uses a comparison group.

4–12 months $$$

“There is a positive, statistically significant relationship between using our program and student achievement.”

Tier 3 Promising Evidence

At least one well-designed and implemented correlational study which will determine if there is a relationship between the intervention and proposed outcomes.

4–12 months $$

“We believe that, if implemented, our program will impact student achievement.”

Tier 4 Demonstrates

a Rationale

Practices that have a well-defined logic model or theory of action are supported by research, and have some effort underway by a state department of education, school district, or research organization to determine effectiveness.

2–3 months $

Table A. ESSA Tiers of Evidence and Study Design Requirements


A logic model for tier 4 evidence

An ed-tech firm says, “We believe that our program will improve student test scores”—a claim that would require tier 4 evidence.

Company A creates an online lesson planning tool that helps teachers align lessons to standards and deliver them with evidenced-based teaching strategies. As it brings the

product to market, potential customers ask about its efficacy: Will it improve student achievement? The company is still developing and refining the tool’s features and content, so it’s not sure how to collect or report efficacy evidence. When it

Case study

contacts a research organization to see if it can provide an impact analysis, the researchers explain that, because the product is still in development, it’s more appropriate to develop a demonstration of rationale, which requires tier 4 evidence. Working together, they develop a logic model that illustrates the research principles underlying the product’s design and shows how helping teachers design and deliver better lessons is likely to positively impact student achievement. The company’s sales team now has a user-friendly report to share with prospective customers that shows how the tool is designed to support teachers, develop their capacity, and engage students in rigorous learning. The logic model now serves as a testable model to guide product refinement.


The “gold standard” isn’t always necessary

A product developer wants to say, “Schools/students that used our program demonstrated significantly higher achievement than similar students/schools that did not use our program”—a claim that would require tier 2 evidence.

After Company B releases a virtual textbook to multiple districts, anecdotal evidence suggests the product is having a positive effect on student achievement. So, Company B contacts a research organization

13Curiosity for better learning

Case study

to see if it will conduct a “gold standard” evaluation—a randomized controlled trial.

The researchers explain that this will require starting with new users of the product and randomly assigning users into treatment or control groups. Timeline? At least one year.

Company B’s sales cycle requires a quicker turnaround for sharing results with potential

customers, so the researchers suggest a quasi-experimental tier 2 study. That way, schools in the same district or state that have not yet adopted the program can serve as the comparison group.

This McREL issue brief was prepared by Faith Connolly,

Mathias Smrekar, and Hsiang-Yeh Ho.

© McREL 2020

[email protected]

Learn more atmcreleval.tech


How ed-tech evaluations provide the evidence of success that ...Introduction Study design vs. needs...

Documents

Transcript of How ed-tech evaluations provide the evidence of success that ...Introduction Study design vs. needs...