How to Evaluate Online Teaching & Improve Faculty DevelopmentThomas J. Tobin, B. Jean Mandernach, & Ann H. TaylorOLC International Conference, Orlando - October 16, 2015 

Our Live WebcastTweet your questions and comments to

@OLCTodayusing hashtags

#olc15#OnlineQuality

Our session evaluation:

goo.gl/v6kiaPImage © 2008 Anthony Quintano. Used under Creative Commons BY license from Flickr.com

The goal

Support Faculty

Improve Teaching

Enhance Student Learning

The challenge

Evaluation of Teaching

Faculty Development

1.Identify Desired Results

• How do successful online/blended educators look/act?• What are the skills you want them to have?• What competencies should they master?

2.Determine Acceptable Evidence of Learning

• How will you and your faculty know if they’ve mastered those skills and competencies?

3.Plan Learning Experiences

• How will you help prepare them for success?

Start at the End: Backward Design

6

Integration of services and supportTraining

Mentoring

EVALUATION

Coaching

7

self

students

peers

Admini-strators

Holistic decision-making

training

mentoring

evaluationdevelopment

coaching

ask look

Six Strategies for Evaluating Online Teaching

1. Know your context.2. Study the foundations of evaluation & analytics.3. Implement formative data & evaluation first.4. Automate summative methods next.5. Create a holistic campus plan for

implementation.6. Implement and cycle the analytics plan.

How do faculty know if they

are effective in the online classroom?

• Are students learning?

Student Learning

Indicators of Effectiveness

Online Learning Experience Impact Factors

Institutional Resources

Program & Curriculum

Course Content & Design

Teaching BehaviorsInstitutional Context

Student Characteristics

Technology

Faculty Impact

Teaching Behaviors

Course Content & Design

Six Strategies for Evaluating Online Teaching

1. Know your context.2. Study the foundations of evaluation & analytics.3. Implement formative data & evaluation first.4. Automate summative methods next.5. Create a holistic campus plan for

implementation.6. Implement and cycle the analytics plan.

1. Encourage student-faculty contact

2. Develop reciprocity and cooperation among students

3. Use active learning techniques4. Give prompt feedback5. Emphasize time-on-task6. Communicate high expectations7. Respect diverse talents and

ways of learning

Principles of Good Practice: Technology

Chickering & Gamson (1987)Chickering & Ehrman (1996)

1. Attend to unique challenges of distance learning2. Be familiar with unique learning needs3. Master course content, structure, organization4. Respond to student inquiries5. Provide detailed feedback6. Communicate effectively7. Promote a safe learning environment8. Monitor student progress9. Communicate course goals10.Provide evidence of teaching presence

Faculty Competencies for Online Teaching

Pedagogical

Technical

Administrative

Penn State Competencies for Online Teaching Success

Measurable Online Teaching Behaviors

• Do you log in at least every other day?• Have you posted a biography of at least a paragraph, in addition to contact info?

• Do you post announcements at least weekly?

• Do you answer students in two days or less?

• Do you participate in discussion forums?• Do you give detailed assignment feedback?

(Piña and Bohn, 2014)Image ©2010 Steve Harris. Used under Creative Commons BY license from Flickr.com

• Who has access to this information?• How is the information used to

improve faculty support and programming?

• Who is responsible for evaluation of online teaching? Who is responsible for faculty development?

What standards, competencies, and expectations exist for “good” online teaching at your institution?

Six Strategies for Evaluating Online Teaching

1. Know your context.2. Study the foundations of evaluation & analytics.3. Implement formative data & evaluation first.4. Automate summative methods next.5. Create a holistic campus plan for

implementation.6. Implement and cycle the analytics plan.

Formative Evaluation:The SCARF Loop

Image ©2014 Thomas J. Tobin. Used under Creative Commons BY-SA license.

• improve the educational experiences you provide for your students,

• identify the professional education you need to further develop your capacity to teach well,

• prepare for your performance review with your supervisor, and

• assess your readiness to apply for promotion and tenure.

Image ©2011 F Delventhal. Used under Creative Commons BY license from Flickr.com

Self-evaluation

Formative: Finding the Right Tool

Techniques for Students• Surveys• One-minute papers• Muddiest point• “Suggestion box”• Anonymous tip line

Techniques for Instructors• Self reviews

• Peer reviewsTools• Qualtrics • SurveyMonkey• Survey Gold• Google Template• Email• Your LMS• Administrative

assistant or TA

Six Strategies for Evaluating Online Teaching

1. Know your context.2. Study the foundations of evaluation & analytics.3. Implement formative data & evaluation first.4. Automate summative methods next.5. Create a holistic campus plan for

implementation.6. Implement and cycle the analytics plan.

Student Ratings of Teaching Effectiveness• SEEQ• SALG• SRTE

Data Analytics• LMSs and other technology-based teaching tools

Administrative Evaluations• Checklist for Online Interactive Learning (COIL)

• Quality Online Course Initiative (QOCI)

• Quality Matters (QM) Rubric

• Online Instructor Evaluation System (OIES): Park University

• Peer Review of Online Teaching: Penn State

Summative: Finding the Right Tool

• What can you do to promote formative evaluation?

• How can you utilize summative evaluation to offer more effective and/or targeted faculty development initiatives?

• What other campus groups, organizations or departments can you work with to integrate evaluation and faculty support services?

How can you use evaluation information (formative or summative) to enhance faculty support?

Six Strategies for Evaluating Online Teaching

1. Know your context.2. Study the foundations of evaluation & analytics.3. Implement formative data & evaluation first.4. Automate summative methods next.5. Create a holistic campus plan for

implementation.6. Implement and cycle the analytics plan.

Campus Policy

Create campus policies and procedures for using online data for formative and summative teaching evaluation.

Annual Review, Merit PayStudent ratingsSelf-ratingsTeaching scholarshipAdministrator ratings

 Promotion and Tenure

Administrator ratingsTeaching portfolio

Program Decisions

Student ratingsExit and alumni ratingsEmployer ratings

Six Strategies for Evaluating Online Teaching

1. Know your context.2. Study the foundations of evaluation & analytics.3. Implement formative data & evaluation first.4. Automate summative methods next.5. Create a holistic campus plan for

implementation.6. Implement and cycle the analytics plan.

The Teaching-Learning Lifecycle

Integrate teaching evaluations and analytics to continuously gauge and foster effective online instruction (minus the wires).

Image ©2006 Glogger~commonswiki. Used under Creative Commons BY-SA license from Wikimedia.com

Faculty members want informal learning opportunities, flexible scheduling, short sessions, and one-on-one support.

There isn’t a “best time.” Faculty members want anytime, anywhere professional development.

Rely on your assessment of your own faculty needs when designing the “right” program and resources for your institution.

• What evaluations are currently done at your institution that can inform faculty development initiatives?

• What evaluations can you integrate to more effectively support faculty?

How can you integrate evaluation and faculty development to create more holistic support for faculty at your institution?

Questions and Comments

Thank You!

Let’s Continue the Discussion

Thomas J. Tobint-tobin@neiu.edu

B. Jean Mandernachjean.mandernach@gcu.edu

Ann H. Tayloranntaylor@psu.edu

@OLCToday #olc15 #OnlineQuality

Our Live WebcastTweet your questions and comments to

@OLCTodayusing hashtags

#olc15#OnlineQuality

Our session evaluation:

goo.gl/v6kiaPImage © 2008 Anthony Quintano. Used under Creative Commons BY license from Flickr.com