HoTEL OEP ELIG Pearson Learnshop - part 2

How to Guide Innovation in a Changing Education Ecosystem

An ELIG-Pearson Interactive LearnshopFriday 6th December 2013: 11:45 - 13:30

Facilitators

Kelwyn LooiVaithegi VasanthakumarFadi KhalekDr. Adam Black

Dr. Andreas Meiszner Elmar Husmann

Online Educa Berlin l December 20132

Our team

Kelwyn Looi

Office of the Chief Education Advisor, Pearson

Vaithegi Vasanthakumar

Office of the Chief Education Advisor, Pearson

Dr. Andreas Meiszner

European Learning Industry Group (ELIG), The Netherlands

Elmar Husmann

European Learning Industry Group (ELIG), Germany

Fadi Khalek

VP-Higher Ed & Voc. Learning Solutions, Pearson Education EMA

Dr. Adam Black

Chief Learning Technologies Office, Pearson ELT;

SVP Efficacy and Global Scale of English Products, Pearson English, UK


Agenda

1. Introduction: Key Objectives

2. Background information: ELIG, HoTEL Project and Pearson

3. Introduction to Efficacy

4. The Efficacy Framework in Practice

5. Summary: Reflections and Q&A

1. An Introduction to the Learnshop

This interactive Learnshop aims critically to reflect how to innovate in a profoundly changing education ecosystem.

Pearson’s Efficacy Framework will be tested as a [e.g. stand-alone] means to support the different stakeholders to innovate in TEL / education

Key objectives

For attendees the learnshop would provide the opportunity:

1. For all attendees: to acquaint yourself with the efficacy framework as a tool to engender learning-focused discussions when assessing and evaluating prospective innovations

2. For a case owner: to examine your own ‘case’ through the lens of a rigorous and structured framework, providing key takeaways at both the transversal and individual case level

3. For a participant: to be involved at the input phase of the project, allowing for feedback provided on evaluating the efficacy framework as a support model to be incorporated into future iterative decisions of the holistic model


2.1 ELIG: We change the way Europe learns


2.2 The HoTEL Project

• The assumption based on which a project was conceived is that the innovation cycle in the field of TEL is so long that:

1. It seldom leads to exploitable innovations

2. Often traps potentially disruptive innovations into the chronological limits of EU-funded projects in the field

• HOTEL tries to address this challenge by designing and testing an innovation support model (specific for TEL) in three labs (the HOTEL Learning Exploratoria)


2.3 Pearson aims to use its position in education to improve the lives of learners

• Our vision is to help millions of people make progress in their lives through learning

• We want to be able to prove that our products and services have a measurable impact

• Efficacy is how we’ll do this.


• More than ever before, there is a shared understanding that high quality education drives personal, economic and societal growth

• Governments, individuals, employers and institutions recognise the need to deliver high quality learning

• New technology makes it increasingly possible to see what works and what doesn’t in education to drive our ability to help learners achieve their goals


3.1 The moment is now to make a lasting impact on learner lives

• We are reviewing all of our products and services to ensure they deliver on learner outcomes

• We introduced a framework that is systematically being applied across the company to help us do this

• For some products it will take time to demonstrate efficacy but we will work hard to ensure that they are set on a path to efficacy


3.2 Measuring and improving

3.3 Embedding and leading

• We adapted core business processes driving the company to focus on delivering learner outcomes

• We will measure our success not only by financial returns but also by learner outcomes

• We hope to demonstrate the knowledge gained from this approach and encourage others to adopt it


3.4 Collaborating and exchanging

• We cannot do this alone - we are just one part of the education system

• We want to work with partners, leading educators and organisations to co-create solutions

• We will share our approach with everyone, invite feedback on it and work together to tackle the great unmet education needs worldwide


3.5 Our commitment

• We will review all our acquisitions, investments and existing products and services to ensure they improve learner outcomes

• We will build and share a global bank of evidence on best practices in education

• We will publicly report our impact on learning outcomes, and set targets for improvement, across our business by 2018


3.6 There is greater demand for, and opportunities to prove efficacy

Greater demand for measurable learning

Greater opportunity to prove outcomes


4. Our efficacy framework aims to measure likelihood of impact

• What outcomes are we trying to achieve?

• What evidence do we have to believe it is possible to achieve these outcomes?

• What plans are in place to implement these outcomes?

• What capacity is in place to deliver on outcomes?


Efficacy Framework: Likelihood of impactCriteria area Rating Rationale summary

• Action plan

• Governance

• Monitoring and reporting

• Internal capacity and culture

• User capacity and culture

• Stakeholder relationships

Outcomes

• Intended outcomes

• Overall design

• Value for money

• Comprehensiveness of evidence

• Quality of evidence

• Application of evidence

Evidence

Planning and implementation

Capacity to deliver

Efficacy

KeyGreen: Requires small number of minor actions. Amber/green: Requires some actions (some urgent and some-non urgent).Amber/red: Requires large number of urgent actions.Red: Highly problematic requiring substantial number of urgent actions.

Purpose:

• To better understand the use of the Efficacy Framework as a tool to support TEL innovation

Instructions:

• 1 case owner per group – run through the efficacy framework

• Participants to join a case group and ask questions to the case owner aligned to the framework

• Based on additional knowledge and context, group to rate the case using the efficacy framework

• Pearson & ELIG team to facilitate the discussion and note down next steps

Timing:

• ~ 45 minutes total


4.1 The Efficacy Framework in Practice

Efficacy Framework: Outcomes

Overall design

• Is the product designed in a way that will most effectively help your target group reach their goals?

• Does the design allow you to automatically collect evidence of your progress?

• Have you adapted the design based on feedback from users?

• Could the design by used by others?

Value for money

• Do you understand the benefits of your product or service to your target group, relative to other options?

• Is the cost of the product/service competitive, considering the benefits it would deliver?

Intended outcomes

• Have you identified specific outcomes for your target group?

• Do you have a way to measure the intended outcomes?

• Do you have ambitious and measurable targets in place, and deadlines for achieving them?

• Are your intended outcomes clearly documented and understood by the relevant people within and outside your organisation?

Example of green rating Example of red rating

• All outcomes are specific and clearly documented.

• People within and outside my organisation understand the intended outcomes and are able to communicate them clearly.

• Future targets are ambitious and achievable.

• Outcomes can be regularly measured against set targets.

• Design is superior to other options/competitors with features focused on delivering outcomes.

• Real-time evidence is generated.

• The design can be adapted and developed.

• Others could use this design, and it has been shared with them.

• Feedback/research has allowed me to identify what benefits the product/service needs to deliver to users.

• Feedback and return-on-investment research shows that the cost of the product/service reflects the benefits delivered.

• Outcomes are not documented or specific.

• People within and outside my organisation do not understand the intended outcomes or communicate them in the same way.

• Targets do not exist to measure outcomes against.

• Outcomes are only defined at a high level.

• No feedback from users exists (either formal or informal), and the benefits of using this product/service are unclear to our team and our users.

• Perceptions of value for money and user experience are poor.

• The design does not meet target group expectations and is difficult to use.

• The design does not reflect intended outcomes.

• The design does not allow for the collection of feedback.

• The design is specific to a local situation and cannot be replicated.

Quality of evidence

• Does the evidence you have collected link directly to what you are trying to achieve?

• Is the evidence you have collected unbiased; applicable to your product/service; recent; and does it measure success over a period of time?

• Is the evidence you have collected relevant, representative and where possible at an individual level?

Application of evidence

• Is the evidence stored and accessible to relevant people? Is it available in an electronic and searchable format?

• Has the evidence you have collected been analysed to help inform the design of your product/service?

• Has the evidence you have collected been analysed to help inform other decisions about your product/service?

Comprehensiveness of evidence

• Do you collect evidence using a range of methods (quantitative, qualitative, internal and external for example)?

• Do you collect evidence for all stages of your product/service (from early conception to design and then to implementation)?

• Do you have evidence from all users of your product/service?


• A wide range of evidence has been collected via internal/external, and quantitative/qualitative methods.

• Evidence relates to all stages of my product/service.

• Evidence exists from all users.

• Evidence collected effectively proves how well we are meeting our objectives.

• Rigorous research methods have been used.

• Evidence relates to the specific and relevant use of the product/service.

• Evidence was gathered over a period of time.

• Of the evidence that does exist it is not linked directly to what I am trying to achieve.

• The evidence that exists is: biased; not from a relevant use of the product/service; out of date.

• The evidence is not representative of how a learner would use this product/service.

• All evidence is readily accessible and searchable.

• The evidence is used regularly to inform the design of my product/service.

• Collected evidence is also used to inform non-design decisions.

• The evidence that exists cannot be accessed quickly via electronic means.

• The design of my product/service has not been changed as the result of evidence.

• Major decisions about my product/service are not underpinned by evidence.

• Evidence is collected via a limited range of methods and does not balance qualitative and quantitative sources.

• Evidence is mainly anecdotal and patchy, and does not take into account the product/service’s lifecycle, features, or users.

Efficacy Framework: Evidence

• Our action plan has not been updated and adapted.

• Where feedback exists, it is delayed.

• Our team is are unaware of issues or fails to act on them.

• Team routines are informal and not focused on monitoring progress.

• Team-members do not know who makes key decisions.

• Roles for people outside the core team are poorly defined.

• New team members are unclear of key processes and do not have documentation to refer to.

Monitoring and reporting• Do you update your plan based on progress, adapt it

where necessary and communicate this with your stakeholders?

• Do you get/have access to real-time feedback from your users?

• Do you identify issues early, discuss these honestly and find solutions?

• Do you have tools and routines in place to monitor progress (such as emails, calls, document-sharing)?


Governance• Do people within and outside your organisation

understand who is responsible for decision-making regarding your product/service?

• Have you documented who is responsible for the work, and who should be consulted and informed? Do the relevant people understand this?

• Have you identified the key processes required to implement your product/service and are these clearly documented?

Action plan• Do you have a plan in place to achieve your

outcomes, including milestones, actions, responsibilities and timelines?

• Does your plan include short- and long-term priorities.

• Have you identified any potential risks and included actions to mitigate these in your plan?

• Do you regularly update your plan and communicate changes to relevant people/institutions?

• Electronic plan exists with clearly identified steps, responsibilities and deadlines.

• The plan includes short and long-term priorities.

• The plan is regularly updated and all relevant parties are aware of the changes.

• Team-members know who makes decisions, and each member of the team (within and outside my organisation) is clear about their role.

• The processes we have in place are documented and well understood, and new members are fully briefed.

• Data is collected in real-time and analysed to provide feedback.

• Monitoring of the product/service alerts me to issues in real time.

• Tools and routines are in place to identify and solve problems.

• No electronic plan exists.• Plan is informal with actions,

responsibilities and timelines unclear.

• Milestones lack clarity and are either too ambitious or not stretching enough.

• Potential risks have not been formally identified or planned for.

Efficacy Framework: Planning & Implementation

• Our team lacks the appropriate skills and resources to deliver the desired outcomes.

• Our culture feels negative, traditional and not focused on outcomes.

Stakeholder relationships

• Have you identified who your key stakeholders are and do you understand their needs and concerns?

• Do you regularly communicate with your stakeholders?

• Is there a culture of partnership and collaboration between your organisation and your stakeholders?

User capacity and culture

• Do the target group understand the objectives and their roles in achieving them?

• Does the product/service reflect the user’s skillset and available resources?

• Do users have the people, skills, time, or resources to achieve their goals?

• Have you put measures in place to build users’ skills?

Internal capacity and culture• Does your organisation have the right number of

people, and people with the right skillsets to enable you to deliver your desired outcomes?

• Does your organisation have a culture focused on delivering outcomes, and is it collaborative and innovative?

• Do leaders within your organisation support your work and are there opportunities to work with others across the organisation?


• Team has right number of people with appropriate skillset and experience.

• Culture is focused on delivering outcomes and is collaborative and innovative.

• Team has appropriate budget.

• The target group understand the objectives and their roles.

• The product/service takes the user’s skillset into account and there are mechanisms in place to build skills.

• Users have the appropriate resources to achieve their goals.

• We meet with stakeholders frequently, and have formal and informal conversations.

• Conversations with stakeholders have led to a culture of trust and partnership over a sustained period of time.

• The target group and existing users are not aware of what the product/service should help them to achieve and what they need to do to get there.

• The product/service is ill-suited to the user and attempts to build users’ skills are ineffective.

• Users do not have the resources and skills to meet their goals.

• The team and stakeholders have uncertain relationships.

• Miscommunication occurs frequently and solving problems in a joint fashion is difficult.

Efficacy Framework: Capacity to deliver

On November 15th, Pearson launched a dedicated website:

http://efficacy.pearson.com

outlining the company’s focus on efficacy and commitment to put the learner at the heart of the global strategy.

An interactive version of the efficacy framework also features on the website.


Reference material: Efficacy website

http://efficacy.pearson.com/


Identifying dialogue and collaboration with the wider education community as crucial to accelerate

progress, Pearson has also published two reports:

• The first, Asking More: The Path to Efficacy, sets out the imperative for measuring and improving

learning outcomes worldwide.

• The second, The Incomplete Guide to Delivering Learning Outcomes, shares in detail the

company’s new approach to contributing to that goal and the progress it has made so far


Reference material: Efficacy publications

“The future will belong not

to those who focus on the

technology alone but to

those who place it in this

wider context and see it as

one element of a wider

system transformation.”


Reference material: Alive in the Swamp

Quote is from Michael Barber, Chief Academic Advisor, Pearson

https://neo.pearson.com/docs/DOC-263271

You can visit http://hotel-project.eu/, for:

• Project overview: Work programme and Innovation model

• More on the Learning Exploratorium Labs

• Details of future events

• Find us on LinkedIn: (TEL Innovation laboratory)

• Contact: Andreas Mesizner ([email protected]) if you are interested in further opportunities for collaboration


How can I find out more?

http://hotel-project.eu/

mailto:[email protected]



You can visit efficacy.pearson.com, for:

• More information about our approach

• To use our online interactive efficacy tool

• Keep up to date on our events

• Read up on the role of efficacy in education in two publications: Asking More, and The Incomplete Guide

• Find us on LinkedIn (Open for Learning) and Twitter (@PearsonPLC)

• Contact: [email protected] if you are interested in further opportunities for collaboration


How can I find out more?




HoTEL OEP ELIG Pearson Learnshop - part 2

Education

Transcript of HoTEL OEP ELIG Pearson Learnshop - part 2