Efficacyhow to prove that your edtech product works

@joeatmarsImpact8 2015

Source: GSV Advisors

Tradition: focus on inputs Efficacy: focus on outputs

…focused on teaching …focused on learning

…focused on inputs …focused on outputs

…tradition-driven …data-driven

…teacher as content authority …teacher as content guide

…teacher responsible for cause > effect …teacher iterates towards effect

…teacher is the sole proprietor of a private practice

…teacher is a member of a Professional Learning Community

…”best practices” …context sensitive

Educational researchEducational tradition

New methods of measuring impact (for teachers and researchers)

1. Action-based research2. Developmental evaluation3. Design-based research4. Control groups, A/B tests5. SAMR

In medical terms, “efficacy” refers to the ability of a product to provide a beneficial effect.

Impact Frameworks

1. IRIS 2. Pearson3. Learning Canvas

• The learning benefit of your product is notself-evident

• Social metrics are often proxies for learning outcomes

• Social metrics should be used to make hypotheses about learning outcomes

From social metrics to learning metrics

Example: you have a product that teaches math online with a parent-teacher bridge component. If you can increase parent-teacher communication by 20 min/week you can hypothesize that student achievement will increase based on “Parental Involvement and Student Achievement: A Meta-Analysis” (Jeynes, 2005)

Your social metric + existing learning research = learning hypothesis

From social metrics to learning metrics

Impact Frameworks

1. IRIS 2. Pearson3. Learning Canvas

From Good Intentions to Real Impact

5 Levels of Efficacy Method

Level 1: Why might the product have an impact onlearning?

Hypothesis

Level 2: Data shows some change amongst users Anecdotal evidence; surveys

Level 3: Data shows the change is because of the product

Control Groups

Level 4: Independent evaluation of impact/explanation of learning impact

Independent researchers

Level 5: Ability to replicate/scale the learningimpact

Replication studies

Impact Frameworks

1. IRIS 2. Pearson3. Learning Canvas

LEARNING GAINS

LEARNING PROPOSITION

END USER

TECHNOLOGY

THEORY

METRICS

LEARNING OBSTACLES

Who are the students who use the product?

What learning gains do you expect to see?

What hi-tech/low-tech do the students use?

What learning theory predicts these learning gains?

What metrics will you collect to prove the learning gain?

14 2

3

8

6

5

7

9

SCHEDULEHow often do the students interact with the product?

PARTNERS

What partners do you need to execute?

- Age- Grade- Gender- Special needs- Socioeconomic- Culture- Hand-outs

- iPad- Lab equipment- Sports equipment

- Once/week- 30 min/day- On demand- Beginning of

each course- Home use

- Increase in marks- 21st C skills- Complex thinking- EQAO scores

- Scientific studies- Theory search- Existing products- Benchmarks

- Marks- Self-assessment- Engagement- Attendance

- Iterate on hypothesis, unanticipated metrics

- Teachers- Researchers- Lead students- Principals- Superintendents

(After implementation) What learning gains resulted from the product’s use?

- Final metrics, confirmation of hypothesis

(After implementation) Did students get stuck anywhere? Were there any unpredictable negative side effects to the product’s use?

LEARNING GAINS

LEARNING PROPOSITION

END USER

TECHNOLOGY

THEORY

METRICS

LEARNING OBSTACLES

SCHEDULEPARTNERS