Reflections on the evolving landscape of OER use

Reflections on the evolving landscape of OER use

Liz Masterman & Joanna Wild OER13: 26th March 2013

Lond

on s

kylin

e: C

C B

Y-N

C-S

A E

lizab

eth

Mat

thew

s

research

into!

V!

About this presentation

  This is the PowerPoint presentation (in PDF format) from our paper at the OER13 Conference. To make the slides more meaningful, we have appended the Notes pages.

  Please note that the presentation as a whole has the licence CC BY-NC-SA, even though individual images may have different licences.

The Projects

OER Impact Study

  Captured process of search and evaluation

  Benefits were speculative rather than actual

  Exercise caution in quantitative measures of use

  Relevance and benefits to learning are contingent on task at hand

(Left) http://commons.wikimedia.org/wiki/File%3AImpact_Lettering.jpg: Public domain; (Right) CC BY David White

OER Engagement Study

Joanna Wild

  Novice expert progression: ‘ladder’

  Little evidence of remixing and releasing as new OER

  OER use is a means to an end, not a goal in itself

  What constitutes ‘optimal’ engagement?

World War I Centenary: Continuations and Beginnings

  Relationship to other resources

  OER complement or supplement, not replace other materials

  Awareness of (or adherence to) licensing terms remains low

University of Oxford

UK OER Synthesis & Evaluation Project

Digest of Findings

Attitudinal factors

  ‘I’d never as an academic – when writing something – incorporate something from someone else’s book without acknowledging them, so how do I think I can just take things off the internet?’

(WW1C)

Brian Talbot http://www.flickr.com/photos/b-tal/278749604

Logistical factors   ‘They had 3,000 so you end

up checking on them randomly.’ (OER Impact)

  ‘A general scarcity of decent resources … seemed to be the main obstacle’ (OER Impact)

  In comparison with a Google search, WW1C resources ‘are very focused’ (WWIC)

http://commons.wikimedia.org/wiki/File%3AATS_Search_Light.jpg: Public domain

Logistical factors   ‘…[coming] away with

something that you hadn’t expected to find’

  ‘Give yourself a morning or an afternoon to look [at] it, and by chance you some across a great deal more than you would by searching using a search engine’

  ‘Interesting things sometimes take you on a path to something else interesting’

(WWIC) All images accessed via http://ww1centenary.oucs.ox.ac.uk

In Flanders Fields Museum

Paul Reed

University of Oxford

Pedagogic factors   ‘It just covers your subject from

a slightly different angle’1

  ‘A student wouldn’t know to do a lot with the resource’ 2

  ‘…explain things in a visual or in an interactive way’ 1

  ‘…more comfortable with small pieces so that I can control the context’ 1

  ‘The university brings it that stamp of authority and quality’ 1

  ‘Not a perfect image, … but adequate for purpose’ 1

http://commons.wikimedia.org/wiki/File:3D_Jigsaw_Puzzle.jpg: Public domain 1 = OER Impact; 2 = WW1C

Pedagogic factors   ‘Whole packages … kind of

assume that we are kind of neutral deliverers of objective content to those passive recipients, and that’s not what happens’

  ‘We haven’t made it and used our thought processes to make it … And then you spend a couple of hours re-jigging it to sound like you; using the concepts but put it in your style ’

(OER Impact)

http://pixabay.com/get/d82ade6eaf2788458a6e/1363368359/states-40679.png: Public domain

Challenges to the Researcher

1. What are we referring to? The heterogeneity of OER

University of Oxford All images accessed via http://ww1centenary.oucs.ox.ac.uk

2. OER in relation to other resources

  ‘OER are just another resource that one would have to pull together for one’s own purposes, much as one would use other types of resource’ (OER Impact)

  ‘OER are just regarded as any other form of teaching material, they are not separated out’ (OER Engagement)

Elizabeth Matthews Original image of sweetshop in Battery Point, Hobart, Tasmania

What is distinctive about OER? Three propositions

1.  OER make it possible to use learning materials created by others and distribute them legitimately

2.  Wide-scale production of OER is increasing the stock of resources available for teachers to use where:

a)  they can’t create their own, or b)  they need to teach outside their field, or c)  a third-party self-study module is acceptable

3.  Engagement with OER opens up opportunities for transformation in personal practice ‘open’ culture of teaching, learning and scholarship. However, this transformation is not a given

3. Evidence and measurement

‘Measuring results and postulating what led to them is difficult for any multidimensional, longitudinal behavior, and establishing that a meaningful change in such behavior really did occur requires real evidence that really is hard to obtain. Furthermore, in a new field such as open education, practitioners are inclined to justify assumptions and demonstrate differences from existing practice, rather than make comparisons that are based on measures and metrics which are themselves evolving’ (Walker, 2008: p. 78–9)

Walker, E. (2008). Evaluating the Results of Open Education, In T. Iiyoshi and M.S.V. Kumar (eds.), Opening up education : the collective advancement of education through open technology, open content, and open knowledge (pp. 77–88). Cambridge, MA: The MIT Press.

Quantitative measures   ‘Targets … put pressure in

the wrong way’   ‘Once you get above half

the course being OER, you have to massively reshape what you do to match what’s out there’

  ‘They might [have] looked at a whole bunch of stuff, and the 5% might be the only quality that they are happy with’

(OER Engagement)

Elizabeth Matthews Basketball scoreboard, Olympic Games, 2012

Qualitative measures   ‘I found something great,

this has saved me some time, this is helpful for me’

  ‘Does OER lead to a better practice? I think it does, just because of the dialogue around it’

  ‘it gives you different ideas, but it also makes you question your own practice … you can compare yourself to others’

(OER Engagement)

Crossett Library, Bennington College http://www.flickr.com/photos/crossettlibrary/4384982036/

‘Optimal’ engagement with OER: differing agendas?

Dave Pape Original image: http://resumbrae.com/ub/dms423/23/

Learning

(versus)

Branding, reputation

Conclusion   Users and resources evolving more rapidly than practices?   General climate of openness   ‘Mixed economy’ of OER and resources with more

restrictive licences   Methodological issues a key area of work   OER ‘sometimes as ‘signs that [open educational

practices] are going on, sometimes as drivers to make them happen, sometimes just in the background’ (Beetham, 2011)

  Balance to be struck between ‘learning’ and ‘branding’ in defining and measuring ‘optimal engagement’ with OER

Having researched into OER use for over two years in three projects, Joanna and I thought that we would take the opportunity of OER13 to reflect on what we’ve learned.

We have modified the Ftle of this paper since it was originally submiHed.

The geographical metaphor is no longer quite apposite to what I’m going to talk about, but nevertheless I’ll have to pay lip service to it.

>>CLICK

And in wriFng the paper that accompanies this presentaFon, we found ourselves reflecFng on our research and the challenges confronFng it as much as on the results.

The presentaFon falls into three secFons: 1.  Brief run-‐through of the projects. 2.  A short ‘digest’ of the main findings 3.  An overview of the methodological issues in researching this area

1

March 26, 2013 OER13 Paper 36 -‐ Masterman & Wild

In this paper we focus on data collected that tell us something about the individual’s relaFonship to OER use.

We are not reporFng data on departmental and insFtuFonal strategies for promoFng use of OER.

2


OER Impact Study – funded by the JISC OER Programme; explored the landscape as it existed in early 2011.

It was one of the earliest UK studies funded specifically to look at the enabling factors and barriers to the uptake and use of OER.

Its iconic representaFon is the iceberg image – depicFng the relaFonship between OER and what we colloquially referred to as ‘stuff on the web’.

>>CLICK One aspect that disFnguished OER Impact from other studies of reuse were our workshops with 16 pracFsing lecturers, through which we captured the actual process of searching for, locaFng and evaluaFng OER.

This gave us both quanFtaFve data on success rates and the relaFve role of different selecFon criteria, and qualitaFve data ‘from the coal face’.

Moreover, most of these lecturers were encountering OER for the first Fme, so this gave us a sense of the difficulFes of engaging with OER outside a supporFve insFtuFonal context.

Some of the key findings of the study: The benefits were speculaFve: i.e. what lecturers expected them to be rather than what they had actually found.

The study acknowledged insFtuFonal strategies as essenFal for the propagaFon of OER use, but it advised cauFon in the adopFon of quanFtaFve approaches to measuring that use: OER should be used only where they are relevant and genuinely beneficial to students’ learning. And ‘relevant’ and ‘beneficial’ are conFngent on the task at hand.

3


The next project was Joanna’s SCORE fellowship.

Building on embryonic data about pracFce captured by OER-‐Impact, her OER Engagement Study addressed the evoluFon of individual lecturers’ pracFce within the context of insFtuFonal iniFaFves.

She used interviews to explore 1.  How OER use is currently promoted within OER-‐acFve insFtuFons, and 2.  Lecturers’ progression from first-‐Fme to experienced users of OER, using the

metaphor of a ladder. To find out more about the ladder, look at Paper 27, ‘Leveraging passion for open pracFce’, tomorrow morning.

>>CLICK

Selected findings from Joanna’s study:

Lack of remixing/repurposing OER and releasing them as new resources – even though this is one of the vaunted benefits of OER.

The study concluded that OER use should be treated not ‘as a goal in itself, but as a means to an end – a beHer and more engaging course’…

…And it also posed an important quesFon:

The opFmal level of engagement with OER was defined in terms of reusing resources created by others, re-‐sharing resources one has reused, and sharing one’s own materials under open licences. But is this level a desirable goal? Is it what we should be aiming for and anything less falls short?

4


In summer 2012 we we were invited to evaluate this new collecFon of OER which our colleagues had created with funding from the JISC World War I Centenary programme.

We decided to ‘piggy-‐back’ a small piece of research on the evaluaFon, and so we included appropriate quesFons into individual walkthroughs of the tool with Oxford academics and a workshop of contributors and stakeholders from the wider WWI centenary ‘community’.

So, like OER Impact, we looked at the experiences of individual pracFFoners searching for specific resources.

But this Fme we were concentraFng on a single collecFon of resources designed to support interdisciplinary study of a parFcular topic.

And we didn’t try to capture the process.

Another similarity with OER Impact was that a number of our interviewees hadn’t come across OER before.

>> CLICK

The project provided insights into the relaFonship of OER to other resources used by lecturers: namely, complemenFng and/or supplemenFng them, but not replacing them.

We also found a worryingly low awareness of, and adherence to, licensing terms, which I’ll return to shortly.

5


So, in different ways the three projects explored individual lecturers’ aftudes and behaviours in relaFon to adopFng OER developed by others.

But to clarify their relaFonship to each other and to OER studies in general, I thought it would be helpful to locate them within the UK OER Impact Model created by Lou McGill and her colleagues in the OER Synthesis and EvaluaFon project.

All three lie firmly within the ‘individual’ focus, with…

• OER Impact and WWI Centenary in the ‘individual/resource’ quadrant and • OER Engagement in the ‘individual/pracFce’ quadrant.

6


In this part of the presentaFon I’m going to draw together findings from our work regarding the aftudinal, logisFcal and pedagogic factors influencing lecturers’ incorporaFon of OER into their teaching.

Although the fourth factor –insFtuFonal strategies – is equally important, our interest in this paper lies in the individual’s relaFonship to OER, as we saw on the previous slide.

[Looking for conFnuity and change over the three projects. However, this is a bit problemaFc as we didn’t set out to ask the same quesFons in all three – so it was a bit hit and mess whether we collected enough relevant data.]

7


As already noted, the incorporaFon of third-‐party OER builds on longstanding habits of reuse.

An appreciaFon of the specific role that OER can play in one’s teaching hinges on an understanding of the legiFmate use of all third-‐party materials.

But data from all three projects suggest that much groundwork has sFll to be done in this respect.

Survey and interview data from WW1C suggest a somewhat relaxed approach to appropriaFng materials from the Web for one’s teaching: ‘I need that: I’ll use that’.

>> CLICK

This contrasts markedly with lecturers’ rigorous aHribuFon of printed materials.

But copyright does become a concern to these lecturers where they distribute resources to students in digital format and thereby lose control over their further use.

8


The OER Impact study suggested that searching for potenFally suitable resources is the most Fme-‐consuming aspect of OER us.

So by the Fme Joanna did her interviews for the OER Engagement Study, it wasn’t surprising that librarians and learning technologists were coming to play an important role in locaFng OER on lecturers’ behalf.

>> CLICK

Even so, the quanFty of items returned by a search for OER remains problemaFc.

On one hand, a search via a general-‐purpose engine such as Google risks retrieving an unmanageable quanFty of hits.

Moreover, the list might omit potenFally useful resources in OER collecFons which are concealed behind registraFon pages.

On the other hand, the smaller quanFty and scope of specialist OER collecFons mean that searches risk being less successful.

The searches in the OER Impact study had success rates of 46% on OER sites, but 62% on non-‐OER sites.

But, as the third quote points out, what you get from a subject-‐specific OER collecFon in parFcular is a much more focused set of resources.

9


And there’s something called the ‘serendipity factor’: finding something when you didn’t expect to.

Here are 3 items that were unexpected, but happy discoveries for people who looked at WW1C.

>> CLICK

So, what may also be important when lecturers visit a specialist OER collecFon is the promise of what lies within.

If a lecturer’s first search yields an intriguing item, even though it may be irrelevant to the immediate need, their curiosity has been aroused and they will add the OER collecFon to their list of ‘favourites’ to revisit in the future.

• Eton LeHers – found completely by chance; complement course on History and the Novel

• Chinese labourers leave the ruined village of VlamerFnghe on their way to work. – InternaFonal dimension

• ‘ArFllery on the march’ – surprise to discover that it wasn’t just another picture of BriFsh soldiers on the road.

10


Data from all three projects agreed about lecturers’ key consideraFons when evaluaFng an individual OER for a specific purpose.

>> CLICK

These quotaFons directly illustrate the six points listed in the paper:

• fit to the topic, learning outcomes and level of study;

• pedagogic intent: whether the resource has been explicitly developed for an educaFonal purpose or can readily be co-‐opted for such a purpose;

• medium (e.g. textual, visual, audio), which may influence its fit to students’ needs and preferences;

• granularity, on a scale of ‘liHle’ to ‘big’ (cf. Weller, 2010);

• provenance: in parFcular, whether this is trusted

• quality, including scienFfic accuracy and standard of producFon.

11


If data from OER-‐Impact are typical, up to half the OER retrieved in successful searches do not fit lecturers’ requirements exactly.

>> CLICK

Although they can be willing to sacrifice certain aspects of quality and fitness to purpose in return for a CreaFve Commons (or equivalent) licence, the individuality of their teaching voice remains sacrosanct.

In such cases, the lecturer either provides explanatory guidance to students on how to approach the resource or modifies it (if the licence permits).

However, lack of Fme, skills and/or tools means that most modificaFons are limited to ediFng text, and cropping and labelling images.

12


Both in reviewing the literature on OER use and in conducFng our own research, we have encountered some methodological challenges, which I’d like to share with you in the final part of this presentaFon.

OER13 Paper 36 -‐ Masterman & Wild March 26, 2013

13

OER are extremely heterogeneous. To take the WWIC collecFon as an example, it contains:

>>CLICK LiHle OER that were not created for an educaFonal purpose, such as images… >>CLICK: … and Ebooks

It also contains liHle OER that have been created for an educaFonal purpose, such as >>CLICK: Blog posts by academics >>CLICK: InteracFve learning objects made by combining Google maps with open data… >>CLICK: … and by creaFng simulaFons from open data >>CLICK: Podcasts by academics

… as well as: >>CLICK: Links to big OER.

This heterogeneity means that the landscape of use may vary from study to study.

Therefore the conclusions drawn by researchers may differ.

This is important as it means that, say, specific claims about OER that are simply images with CreaFve Commons licences but no embedded pedagogic intent may be mistakenly assumed to apply also to complex stretches of learning with specific intended learning outcomes.

So, as researchers, we need to be careful when we synthesise findings across mulFple projects and make generalisaFons about the benefits of using OER.

OER13 Paper 36 -‐ Masterman & Wild March 26, 2013

14

Comparing results from OER-‐Impact and our previous invesFgaFons into reuse, it becomes clear that many of the benefits that are claimed for OER use hold equally true for the reuse of resources with more restricFve licences.

>>CLICK

Moreover, one of the characterisFcs of ‘expert’ OER pracFce is a ‘holisFc’ approach, in which lecturers view OER as part of an ecology of resources that includes materials with more restricFve copyright terms.

This invites the quesFon: what, exactly, differenFates OER from these other resources?

15


From our work, and from our reading, we suggest that the ‘USP’* of OER lie in three aHributes:

>> CLICK

1.  OER make it possible to use learning materials created by others and distribute them to students and other third parFes legiFmately.

This is not to say that lecturers actually will distribute them legiBmately – but OER make it easier for them to do so – if more Bme-‐consuming.

>>CLICK

2.  The current wide-‐scale producFon of OER is increasing the stock of resources available where a) teachers are unable to create materials of their own, or b) they need to cover an area of the curriculum which lies outside their current field or for which a self-‐study module is acceptable.

>>CLICK

3.  The OER Engagement study yielded evidence that OER use can result in reflecFon on, and transformaFon of, both one’s behaviour and one’s thinking.

To be truly disBncBve, though, the change must go beyond simply using OER. It must extend to a different way of thinking and acBng in relaBon to teaching and learning (open educaBonal pracBce) and, even, to research (open scholarship).

This change is not inevitable – it would be possible simply to appropriate OER into one’s teaching with no change in consciousness.

* Unique selling point (a markeFng term)

16


The OER Engagement Study noted a conFnuing lack of empirically derived evidence, not just of the benefits of OER but also of the factors conducive to their successful incorporaFon into students’ learning.

In our own work, we have also been strongly aware of our reliance on teachers’ self-‐reports, and on qualitaFve data from a relaFvely small number of parFcipants – and insufficient other research to corroborate our findings.

But these methodological concerns are not new…

>> CLICK

…even in 2008 Walker was wriFng, in relaFon to open educaFon as a whole:

[Quote is on the slide.]

…where we can interpret ‘measures and metrics’ as applying to both quanFtaFve and qualitaFve research methods.

17


Shortcomings in quanFtaFve measures include:

>> CLICK

• Imposing targets for the proporFon of OER to be included in a course may compromise the very pedagogy that they are intended to help improve (the first 2 bullet points)

• Actual use of OER is not an accurate reflecFon of intended use, as success rates lie outside the lecturer’s control (the 3rd bullet point).

Also, even where improvements in student performance can be measured, it is not possible to aHribute the improvement solely to the inclusion of OER, as Greaves, Roller and Bradley, point out in a paper wriHen in 2010.

18


QualitaFve measures also have their disadvantages:

>> CLICK

A reliance on self-‐reports, ouen of enthusiasts, may lack the empirical rigour demanded of other fields in educaFonal research.

19


The challenges to measuring the benefits of OER use also go to the heart of a basic tension in the purpose of an insFtuFon’s engagement with OER:

•  Between the creaFon and deployment of OER within the curriculum with the aim to enhance students’ learning,

•  And the creaFon and publishing of OER with a view to enhancing the insFtuFon’s external reputaFon.

Browne et al., in a paper from 2010, refer to ‘two subtly different agendas, learning and branding,* which are not necessarily always in harmony.’

Harley (2008) suggests that the paucity of ‘common terms, metrics, methods, or values for defining use, users, or value’ results, at least in part, from this diversity of stakeholders and perspecJves

So, going back to the quesBon that Joanna posed in her Engagement study… There may be a difference between ‘opFmal’ engagement with OER from the pedagogic perspecFve and…

…‘opFmal’ engagement from the perspecFve of an insFtuFon that wishes to invest in support for the creaFon and use of OER.

An insFtuFon that produces as well as uses OER may well wish to see maximal use of its home-‐grown resources. BUT, as we have already noted, the use of an OER is conFngent on context. AND in some cases the pedagogic purpose may be beHer served by ‘imported’ OER.

And, as Harley asks: ‘How can self-‐reports from a pool of faculty users … be jibed with an ideal vision of sustainable use by those who provide resources and tools?’

* Branding = recruitment of overseas students and enhancement of academic reputaFon.

20


>> CLICK

In reflecFng on our geographical metaphor, we suggest that the landscape may be expanding (in terms of the numbers of insFtuFons and resources) more rapidly than pracFces within it are evolving – at least, judging by the conFnuing low profile of copyright and licensing.

>> CLICK

But overall, we see a favourable disposiFon to the concept of openness. Our current work in the Open Access transiFon project at Oxford suggests that this is reflected in generally posiFve aftudes to open access publicaFon and, to a more limited extent, in open data.

>> CLICK

The role of OER within the overall ecology of third-‐party resources is to complement and supplement resources with more restricFve licences.

But the actual choice of a resource is conFngent on the needs of the moment as much as moFvated by a personal ‘world view’ of open educaFonal pracFce. So there will always be a mixed economy of open and non-‐open resources, and this must be seen as a healthy (and necessary) state of affairs.

SEE NEXT PAGE FOR REST OF NOTES

21


>> CLICK

There are also conFnuing challenges to measuring the benefits of OER by both quanFtaFve and qualitaFve means.

>> CLICK

The data that we have collected and reviewed point to the role of OER not as magical agents of changed pedagogic pracFce and instantly enhanced learning. Rather, they serve as mediaFng artefacts in a process where they move from catalysts of change to signs that change has taken place – as Beetham (2011) says, ‘someFmes as signs that they are going on, someFmes as drivers to make them happen, someFmes just in the background’

>> CLICK

The definiFon and measurement of ‘opFmal’ engagement with OER in terms of an insFtuFon’s teaching and learning strategy occupies a different frame of reference from the definiFon and measurement of ‘opFmal’ engagement in terms of its business model for OER producFon. For open educaFonal pracFce to flourish in such insFtuFons, these perspecFves need to be held in balance.

22


Reflections on the evolving landscape of OER use

Education

Transcript of Reflections on the evolving landscape of OER use