Assessing OER impact across varied organisations and learners: experiences from the Bridge to

Success initiative

Tim Coughlan

Horizon Digital Economy Research & School of Computer Science,

The University of Nottingham (UK)

Patrick McAndrew & Beck Pitt

Institute of Educational Technology, The Open University (UK)

Nassim Ebrahimi

Anne Arundel Community College, MD, USA.

Overview and Aims

• Introducing the Topic

• Introducing Bridge to Success

• Research Methods Used

• Overview of Users and Pilot Contexts

• Definitions of “Impact”

• Issues in Assessing “Impact”…

• Towards an Effective General Model for Research

Introducing the Topic

• How to understand the impact of OER and of initiatives?

• What is meant by impact?

• What do different stakeholders want to know?

• What are the particular complexities of research in this space?

• How could we overcome them?

What is Bridge to Success?

The project aimed to offer free, open educational resources to prepare adults to successfully and confidently transition to a college environment in the US, to pursue advanced qualifications, or to be successful in their chosen careers…

Next Generation Learning Challenges

Research Methods“What is the impact of Bridge

to Success on student success and educator practice?”

Product Design/Adaptation Process

Contexts of Use

Student and

Institutional Experience

Student Outcomes

Project Impact

Evaluation Research

Surveys

Quizzes

Interviews & Focus Groups

Pre- and Post-Surveys

Analysis of interviews, meetings, emails etc.

Courses underwent accessibility, usability and design analysis

Action Research

Periodic Interviews with keystakeholders and the project team

Quantitative

Analytics of website use

Institutional student data

Who piloted the content?

• A total of 16 US college and non-college institutions (October 2011 – December 2012).

• As at August 2012 a total of 17 out of 26 completed pilots had been conducted with cohorts of low-income students. A further 16 pilots were ongoing at this date (total = 42 distinct pilots).

• As at February 2013 9 US college and non-college institutions were continuing to utilise Bridge to Success materials, in a variety of contexts.

Overview of Users and Pilots

Defining “Impact”

Differences across Stakeholders• Funders

• Policy Makers

• Educational Organisations

• Academic Research

Independent Learning

Problem solving

Collaboration

Communication

Mastery of Subject

Defining “Impact”

Measures of Impact:• Retention / Completion / Persistence.

• Whether a student has “mastered” material.

• “Deeper Learning.”

• Changes to educator practice.

• Increased learner confidence.

Independent Learning

Problem solving

Collaboration

Communication

Mastery of Subject

Assessing “Impact” in Community Colleges• Nine colleges piloted Bridge to Success materials during the

project’s pilot phase (October 2011 – August 2012).

As at August 2012

• Of the completed pilots that have submitted 100% of their data, and as reported by instructors, 88% of students who completed B2S materials persisted to next semester.

• 98% of these students mastered the subject matter and 96% mastered deeper learning.

63% of these college students were low income.

Of these low income students, 99% persisted to the next

semester, 96% mastered the subject matter and 94%

mastered deeper learning.

Assessing “impact” in non-college institutions

“I’m not a computer literate person, I always dealt with

things hands-on, it’s paper and pencil and I deal with things

hands-on … I mean I came out of school in ’88…For me to go

online and learn online, it’s a challenge.” (Bridge to Success

pilot participant, March 2012)

“The National Assessment of Educational

Progress found that 41% of eighth graders in

free and reduced lunch programs have home

Internet access compared to 72% of their

financially better off peers.” (K

aminski, Seel

& Cullen, 2003)

EXAMPLE ONE

International Workforce Development Agency: supported people across Maryland for over 90 years.

This pilot utilised specific units of the B2S maths course until end of August 2012 to enable students to complete a new requirement math pre-test for a Weatherization program.

Students who originally failed the math entrance exam, 80% passed after working with B2S resources for a 1-3 week period.

Assessing “impact” in non-college institutions EXAMPLE TWO Family Support Centre in residential area where:

“85% of families are headed by a single female parent;90% have not completed high school and do not have a GED;95% are unemployed, underemployed or receive welfare or other social services.”(Source: Case Study Two Family Support Centre leaflet 2012)

Piloting both Bridge to Success courses to support preparation for General Education Development (GED) and pre-GED examinations.

Outcome of Spring 2012 pilot with 8 participants: • 2 obtained their GED certificate,

• 2 less advanced learners referred to another programme,

• 2 students sat their GED examinations but need to retake,

• 1 participants is now employed following a job training programme and will take their GED during March 2013,

• 1 dropped out of Waverly programme.

“Challenges that are unique to

our setting and population”

Use of B2S materials to date

Between 1 October 2011 – 17 March 2013:

147,183 Page Views

18,591 Unique Visitors

Bridge to Success (B2S)

exceeded its unique user and

pilot participant ta

rgets: 1822

current/completed participants

and, by September 2012, over

6,000 unique users of B2S

materials.

How should we really measure impact?

• It is inherent in the Open concept that resources are used in different contexts and in varied ways.

• This is part of what makes for great impact

• But it makes it complex to holistically measure or compare the value of an OER or an OER initiative

Research Methods and Outcomes

• Usage Statistics

• Show broad evidence of amount of use, trends, locations of users

• Can show paths of individual users / identified groups

• No data on why many events happen (e.g. did a teacher tell them to stop at that point or did they get stuck?)

• In-built pre / post unit assessments

• Show evidence of improvement in scores

• But only in a sample who took both quizzes

Research Methods and Outcomes

• Observations and ethnographic work

• Provide evidence of impact outside the expected

• Begin to understand why it works or where it doesn’t

• Time consuming, reliant on opportunities, doesn’t scale well

• Institutional Data

• Longer-term, detailed understanding of impact on individuals

• Heterogeneous

• Requires long-term involvement and relationship building

• Can support institutional comparison, so may be wary of sharing

• Hard to find suitable comparator if targeting at (e.g. underachievers)

Research Methods and Outcomes

• Pre / post course surveys

• Can be used to identify learner and their context in the system and link with other data (e.g. institution ID)

• End point of use is hard to know, and unlikely to be when a learner wants to fill in a survey

• Creates an initial barrier to getting on with using the resources

• Surveys / Interviews with Instructors

• Understand context of learning and use of OER

• Elicit instructors judgement on types of impact (e.g. ‘Deeper Learning’ definitions)

• Can be combined with support and advice

• Scope for definitions to be interpreted differently

What is required?

• Understanding the different contexts of learners

• Identifying individuals and learning contexts across data sources

• Operationalising varied impact definitions to be studied at scale

• Being able to adapt and expand to opportunistic assessments

An Effective General Model

Conclusions and Next Steps

• Broadness of use should be acknowledged in OER research

• Methodological and technical innovation needed

• Mixed methods approaches

• Relationship building and long term planning

• More research on use of Bridge to Success content via the Hewlett funded OER Research Hub project

• Reversioning of the courses to take into account user feedback and analysis

THANK YOU!

Tim Coughlan tim.coughlan@nottingham.ac.uk

@t1mc

Beck Pitt Beck.Pitt@open.ac.uk

@BeckPitt

Patrick McAndrew p.mcandrew@open.ac.uk

@openpad

Nassim Ebrahimi nebrahimi@aacc.edu

Resources

• Watch a review of Bridge to Success functionality and the courses: https://www.youtube.com/watch?v=9pHyYO5d01I

• Visit the Bridge to Success website: http://bridge2success.aacc.edu/

• Visit the OER Research Hub website: http://oerresearchhub.org/

Bibliography• Atkins, D. E., Seely Brown, J., Hammond, A. L., A Review of the Open

Educational Resources Movement: Achievements, Challenges and New Opportunities (2007) (http://www.hewlett.org/uploads/files/Hewlett_OER_report.pdf)

• Bailey & Cho Issue Brief: Developmental Education in Community Colleges prepared for The White House Summit on Community Colleges (October 2010)

• Bridge to Success Report One: Adaptation, Integration and Engagement

• Case Study Two Family Support Centre Leaflet 2012

• College Board: Advocacy & Policy Centre Education Pays 2010 (http://trends.collegeboard.org/sites/default/files/education-pays-2010-full-report.pdf)

• College Board: Advocacy & Policy Centre Trends in Student Aid 2012 (http://trends.collegeboard.org/sites/default/files/student-aid-2012-full-report-130201.pdf)

• Kaminski, K. Seel, P. & Cullen K. Technology Literate Students? Results from a Survey (2003, EDUCAUSE Quarterly, No 3)