Measuring our Impact: The Open.Michigan Initiative

Emily Puckett RodgersOpen Education Coordinator, Open.Michigan

OpenCourseWare Consortium Global Meeting 2011

Except where otherwise noted, this work is available under a Creative Commons Attribution 3.0 License.Copyright 2011 The Regents of the University of Michigan

CC

: BY-

NC

-SA

, C

hoco

nan

cy1

Flic

kr

Evaluation team

• Emily Puckett Rodgers

• Ted McCarthy• Pieter Kleymeer

Introduction

Evaluation: started September 2010

http://open.umich.edu

“Through a combination of existing and new qualitative (including surveys, interviews and user feedback) and quantitative (including web analytics,

published resources) data, we aim to measure Open.Michigan’s impact on the University of Michigan campus and on

the broader open education landscape, as well as its progress toward overall objectives and mission.”

Evaluation Design

Implementation

Mission and Objectives

Environmental Scans

Document Study and Review

Organizational analyses

Research development and deployment

Analysis

Review

Environmental Scan

General evaluations, activities in Open Education

Specific evaluation projects and documentation: • Tufts OpenCourseWare (2010)• Free to Learn Guide (2010)• MIT OpenCourseWare (2006)• Oxford’s Technology-Assisted Lifelong Learning (2010)

Document Study

• dScribe data • Wiki• Shared internal documents

and spreadsheets

Environments

Open.Michigan’s Strategic Vision

Open.Michigan enables University of Michigan faculty, students, staff and others to share their educational resources and research

with the world.

Open.Michigan’s efforts contribute to two primary goals:

1. to sustain a thriving culture of sharing knowledge at U-M2. to provide comprehensive public access to all of U-M’s scholarly output

Strategic Vision

http://tinyurl.com/openmichiganvision

Measurable Activities

• Building communities of OER producers and users

• Consulting and Outreach Services to facilitate OER production

• Development of Processes and Software to support OER production and publishing

Next Steps: Goals and Objectives document, Communications Plan

Strategic Plan

Guiding Objectives • How and Why is our OER being

used? • Who is using our OER? • What value does OER bring to U-

M? • In what contexts are people

using our OER?

From the Vision document

From the Evaluation plan

Impact: Growth

Open.Michigan platform migrations

eduCommons OER

OERbit /education OER

static Open.Michigan pages

4/1/2008 9/16/20108/27/2009 4/2011

Google Analytics

Impact: Growth

Impact: Contributions

Resources available• ~70 courses published • 2 campuses • 12 schools• ~9 OER modules published • ~48 resources deposited in Deep

Blue

http://open.umich.edu/education

Resource facilitation• Anishinaabe language and

literature• African Health OER Network• “Textbook of the Future”

discussions• Center for Global Engagement

Handbook

• OERca, released 2008 (copyright and content clearance tool)

• OERbit, released 2011 (publishing platform)

• dScribe, process for using volunteers and community members to clear content and publish OER

• Code of Best Practices in Fair Use for OpenCourseWare (2009)

• African Health OER Network

Contributions to Open Education

Impact: Contributions

Institutional Support (U-M)

Dean support: • Dean Woolliscroft (UMMS)• Dean MacKie-Mason (SI)• Dean Potempa (Nursing,

retired)• Paul Courant (MLibrary)

Financial support: • UMMS• Hewlett Foundation• FAIMER• Open Society Institute• Local grants

Institutional support and partnerships: • Medical School• School of Information• MLibrary• MERLOT

African Health OER Network: • Global Reach• African Studies Center• Ghana-Michigan Charter• Ghana Emergency Medicine

Collaborative

Impact: Investment

“Having the big block M on a lot of high quality produced teaching modules that are made available to the world… is mission consistent, so what’s the business that we’re in here? We create and distribute knowledge.” -Paul Courant, “Why Open is Important” interview

Impact: Surveys

40.7% instructors never heard of OCW75.6% students never heard of OCW

CTools survey

http://tinyurl.com/ctools2010survey

Impact: Surveys

Population Size Response Rate Completion Rate

39,958 students 27.5%71%

6,149 faculty 30.4%68%

2,268 staff 16.5%55%

Open.Michigan and Open Educational Resource survey to U-M

Themes: • Sharing habits • Awareness of

Open.Michigan and OER• Use of OER• Impact of OER • Support of OER efforts and

Open.Michigan

Sharing

Impact: Surveys

"I share learning materials with my colleagues to..." ? (faculty responses)

Use

Impact: Surveys

“What type of OER have you used?” (student responses)

Next Steps: participating faculty surveys

(Some) ResultsThemes from survey comments

• Establish Open.Michigan in main channels of U-M communication

• Clarify copyright in OER and provide resources

• Consistently define Open.Michigan and OER

• Describe why/how to use/create OER• Make short-term improvements to OER• Encourage cultural shifts in learning

CC: BY Emily Puckett Rodgers

Impact: Surveys

Next Steps

In the next three years, Open.Michigan will:

① Produce more and richer content as OER with the various campus units, improve modularity, instructional design, and accessibility of U-M OER

② Increase the visibility and discoverability of U-M resources through a combination of marketing and metadata

③ Draw participants from more parts of campus to expand its disciplinary coverage

④ Ensure OER production is an embedded part of the academic life on campus

Evaluation • Finish survey

analysis• Focus Groups

Strategy• Goals and

Objectives• Communications

plan • Advisory

committee

Resources

Contact

Emily Puckett Rodgers

epuckett@umich.eduopen.michigan@umich.edu

Use

• Evaluation wiki page • Evaluation plan• Survey questions• Strategic planning• Aggregate data

• Our Google Analytics data • Aggregate survey results

http://open.umich.edu/wiki

http://tinyurl.com/omevaluation