Understanding patterns in OER Impact: the role of mapping and curation

Dr. Rob FarrowThe Open University

StructureOER Research Hub Collaborative Research Evidence ModelMapping and CurationPreview of Project ResultsOER Impact Map

#oerrhub

@philosopher1978

• Research project at The Open University (UK)

• Funded by William & Flora Hewlett Foundation for two years

• Tasked with building the most comprehensive picture of OER impact

• Organised by eleven research hypotheses

• Collaboration model works across different educational sectors

• Global reach but with a USA focus

• Openness in practice: methods, data, dissemination

OER Research Hub

oerresearchhub.org#oerrhub

Rationale

• Consolidatory phase for OER movement after years of investment and piloting

• The need for an evidence base: advocacy, strategy, policymaking

• Lack of robust evidence for OER impact

• The evidence ‘problem’ in OER impact research

• Isolated ‘pockets of innovation’

• No agreement over best methods for impact comparison

• Compare diverse experiences to assess the influence of ‘open’

• Mixed methods data from experimental + non-experimental research

CollaborationDeveloping an ecology of sharing

• Research instruments applied consistently across collaborations:surveys, interview questions, focus groups, etc.

• Supplemented by integration of secondary research

• ‘Agile’ research, sprinting

• Thematic and methodological cohesion provided by research hypotheses

Research Process

InclusiveHolisticValidation of experiences at the ‘cutting edge’Open to diverse perspectives

Incommensurability?Granularity?Synthesis?

Practical Advantages / Methodological Disadvantages

How will comparison of experiences be made rigorous?

HypothesesKey claims about open education

Keyword Research Hypothesis

Performance OER improve student performance/satisfaction

Openness People use OER differently from other online materials

Access OER widen participation in education

Retention OER can help at-risk learners to finish their studies

Reflection OER use leads educators to reflect on their practice

Finance OER adoption brings financial benefits for students/institutions

Indicators Informal learners use a variety of indicators when selecting OER

Support Informal learners develop their own forms of study support

Transition OER support informal learners in moving to formal study

Policy OER use encourages institutions to change their policies

Assessment Informal assessments motivate learners using OER

‘Evidence’ is only evidence in relation to a claim or hypothesis: the project hypotheses form the core of the metadata model.

Evidence ModelDesign for information architecture

Key Questions for Evidence Model

What evidence exists to support key claims about OER?

How do patterns of OER impact differ across countries?

How do patterns of OER impact differ according to sector?

How can the veracity of evidence be checked?

How can diverse evidence forms be meaningfully collected together?

• TextTitle

• Text / HTML - Supports embedding of multimediaCopy

• Association of evidence with hypothesisHypothesis

• Evidence is either +ve/-ve in relation to a hypothesis Polarity

• GeolocationLocation

• School (K12) / College / Higher Education / InformalSector

• Academic citation or Hyperlink / URLCitation

Evidence: Information Architecture

Multimedia embedding

Sample Records

• Popular OS platform

• Easy to customize• Over 27,000

existing plugins

Wordpress

• CSV importer• Location plugin• JSON

Customization• Project code

available on GitHub• Easy to link, share

and engage

Openness

Technical Development

SynthesisMapping and curation

Synthesis

Synthesis Methods• Isolating data by hypothesis, sector, country, or any combination• Collaborative curation of research data• Data visualization, reporting• Editorial quality control exercised centrally

Validation• Iteration through current and future patterns of evidence• Open citation trails allow public auditing of evidence• Community voting

in service of The Open University

OER Impact Mapoermap.org

OER Impact Map: Mapsoermap.org

• Country Summary Map - evidence nodes organized by country

• OER Project Map - OER initiatives and projects around the world

• OER Evidence Map - all impact evidence is categorized according to the OER Research Hub hypotheses

• OER Policy Map is the single largest curated collection of OER policies

• OER Impact Map aggregates the other maps

• Tweetmaps show a geographical summary of tweets for a particular Twitter hashtag. E.g. #oermap for outreach; #oerrhub for tracking project activity.

• Collection of maps by others

Global Impact Summary

Select country to zoom

in service of The Open University

Flowmap changes according to country selected…

Impact Summary (USA)

Node colour = sector

Node border = polarity

Positive Evidence

Negative Evidence

“Uncategorized Evidence”

OER Evidence Map

OER Policy MapL

R

N

I

Local/Institutional policies

Regional policies

National policies

International policies

CC Open Policy NetworkPOERUPSPARC

in service of The Open University

OER ProjectsOER Project Map

in service of The Open University

OER Impact Map

in service of The Open University

OER Impact Map

Combines other maps

Supports filtering

tweetmap tiny.cc/oermap

#oermap

in service of The Open University

Hypothesis Reporting

Hypothesis Reporting

in service of The Open University

in service of The Open University

Mapping & Curation: Summary

• A collaborative approach to OER impact research is recommended

• Mapping and curation are effective tools for building a collective understanding of a problem and building an evidence base

• Open methods for researching openness: openly licensed instruments; remix and reuse data; working openly; sharing

• Takeaway message: the whole is greater than the sum of the parts

OER Research HubPreview of consolidated survey results

OER impact on Student Performance:

• More than 1/3 of educators agree (or strongly agree) that OER increase student satisfaction with learning

• a higher percentage of formal students (55.7%) agree (or strongly agree)

• OER benefits for formal learners: increased interest in subjects taught (60.1%, n=398); increased experimentation with new ways of learning’ (49.4%, n=327); gaining confidence (48.6%, n=322).

The importance of open licensing:

• Comparatively high level of adaptation amongst all types of users

• Two thirds of educators (67.5%) consider open licensing important and are familiar with Creative Commons logo (55.7%)

• But only 12.4% of educators create resources and publish them on a Creative Commons license

• This is, however, consistent with the fact that only 26.8% (n=215) of educators are concerned with knowing whether they have permission to use or change a resource.

OER and access to education:

• A majority of informal OER users already educated to degree level

• However, some learners are using OER as a replacement for formal education to which they might not otherwise have access.

• 88.4% of all learner respondents (n= 3761) indicated that the opportunity to study at no cost was significant, and many formal students use OER as replacement for higher education

• Data from OpenLearn suggests that that OER increase access to education for informal learners with disabilities

OER impact on educators & reflective practice

• Broader range of teaching practices

• More reflection on method

• More comparison with other teachers

• More collaboration

• Use OER for inspiration

• Broadening coverage of curriculum

• Bringing in multiple perspectives on topics

Financial savings:

• Strong evidence that open textbooks save money

• Almost 75% of educators believe that OER saves students money

• Just under 80% of students believe they are saving money

Indirect benefits for institutions:

• Making college affordable encourages students to continue to study

• Savings made through digitization and reuse

OER Impact Map: Next Steps

• Adding qualitative data analysis to evidence base

• Increasing the evidence base through desk research

• Using the map as a storytelling tool

• Refinement of user interface

• A badging system to reward contributors and commenters

• New hypotheses, new contributors, (hopefully) community uptake

• School of Open course on open research; Open Data; CC-BY research tools

• Survey data ‘exploration tool’

in service of The Open University

Survey Data Explorer (prototype)

in service of The Open University

Survey Data Explorer (prototype)

in service of The Open University

Survey Data Explorer (prototype)

Mapping & Curation: Reflection

• Would mapping & curation be useful in your own practice?

• What would you map & why?

• How would you categorize your data to make it meaningful for others?

• What would you omit from the map to make it more effective?

• What would your map look like? Even a very basic attempt to produce a map can help identify the most relevant data

• Dissemination: mapping also helps focus on visual communication and conveying message

• Cartographic pedagogies: we use this for research, but it could also be used for teaching as an artifact shared between learners

Join us in building understanding of open education

School of Open course on #openresearch (Sep 2014)

Take our general OER impact survey

Contribute to OER Impact Map

Thanks for listening!oerresearchhub.orgoermap.orgrob.farrow@open.ac.uk@philosopher1978