Effective SLO Assessment in the Online Environment · 2014-08-14 · Effective SLO Assessment in...

Effective SLO Assessment in the Online Environment

Kevin Kelly, EdD

Wiley Learning Institute

San Francisco State University

Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License

Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute


Before we get started…

What is your role at your institution?

Faculty

Professional Development/Instructional Design

Academic Technology/eLearning

Student support staff

Administration – Academic Affairs

Administration – Student Affairs

Administration – Other

3 Effective SLO Assessment in the Online Environment - Module 1

Are you experienced?

HOW LONG HAVE YOU BEEN TEACHING

HOW LONG HAVE YOU BEEN USING ONLINE TOOLS FOR TEACHING

0-1 years

2-5 years

6-10 years

11+ years

0-1 years

2-5 years

6-10 years

11+ years


What are your SLO responsibilities?

SLO and/or assessment committee member

Department coordinator for SLO/assessment

Course author/creator

Instructor – conduct assessment of SLOs

Staff – support the systems where SLO data lives

SLO lead for department

Assessment officer


Overall agenda

Create multiple assessment pathways

Choose appropriate assessment strategies

Choose appropriate assessment tools

Managing consequences of online SLO

assessment



Module 1:

Create multiple

assessment pathways


Kevin Kelly, Wiley Learning Institute

Goal for this module

Redesign an assignment to provide students with multiple pathways to show what they know (or at least think and talk about it)

Review Universal Design for Learning (UDL) principles

Dive further into UDL for assessment

Explore significance for online learning


Apply Universal Design for Learning principles

Universal Design for Learning (UDL)

The core goal is to “Teach every student”

Principles guide instructors to provide multiple pathways for students to succeed

Representation: Share content in more than one format

Engagement: Use multiple ways to make learning meaningful

Expression: Provide more than one way for students to show what they know


UDL Basics – Another way to look at them

Representation Engagement Expression

Provide content or materials in different formats

Motivate learners in different ways

Give learners different ways to show what they know

e.g., for a presentation, post your slides, an audio recording, lecture notes or outline, key media (charts, graphs), etc.

• tie concepts to real-world events or relevant topics

• create collaboration opportunities

• create self-assessment tools for learners to use

• provide different levels of challenge

• let students check their progress

• use different assessment methods


NOTE: These lists are not comprehensive

Thought prompts

Researchers use triangulation to verify their results. UDL proposes something similar for helping students succeed.

What outcome or assignment do you want to assess in the online environment?

Does one assessment provide enough information to show that a student has learned something or reached the outcome?


Meaning for online learning

In some ways, practicing UDL for assessment may be easier in the online environment.

Pros

There are lots of tools – many are free

There are plenty of good examples

Cons

May require some time to set up

Students may not be used to choice at first


Providing assessment choices and levels

Provide alternative questions

Provide different assignments

Let students submit in different formats

Use two or three of these strategies together


Activity

Select assessment strategies that work for example outcomes or for your own outcomes

Effective SLO Assessment in the Online Environment - Module 1 14


Module 2:

Choose appropriate

assessment strategies



Goals for this Module

(Re)imagine assessing your learning outcomes

Consider students’ level of thinking

Consider students’ use of online environments

Select appropriate assessment strategies

Consider reasons to assess in different ways

Consider Universal Design for Learning principles


(Re)imagine Assessing Learning Outcomes

(Re)imagine Assessing Learning Outcomes

Key factors to keep in mind:

Consider the level(s) of learning you want students to achieve or demonstrate

Make the objectives measurable

Avoid common misconceptions

TRUE or FALSE: “online” means all learning or assessment must take place behind a computer screen.

TRUE or FALSE: “distance” education means students are located far away


(Re)imagine Assessing Learning Outcomes Consider Level(s) of Learning

Cognitive (knowledge) - Bloom’s Taxonomy (revised)

Creating: Generate new ideas, products and ways of viewing things

Evaluating: Make a judgment about materials or methods

Analyzing: Break information into parts and describe the relationship

Applying: Implement (use) information

Understanding: Lowest level of understanding

Remembering: Recall/recognize specific information


See http://itc.utk.edu/~bobannon/classifications.html#cognitive_domain

Image source:

http://www.iftf.org

http://itc.utk.edu/~bobannon/classifications.html


Psychomotor (skills)

Adapting: Fine tuning, make minor adjustments in a physical activity

Practicing: Try a specific physical activity over and over

Imitating: Attempt to copy a physical behavior

Observing: Active mental attending of a physical event


See http://ets.tlt.psu.edu/learningdesign/objectives/psychomotor

Image source:

http://www.sharjah.ac.ae

http://ets.tlt.psu.edu/learningdesign/objectives/psychomotor


Affective (attitudes or feelings)

Characterizing by value: Behavior reflects new values.

Organizing: Integrates and prioritizes new values.

Valuing: Demonstrates commitment.

Responding: Shows attention and motivation.

Receiving: Shows awareness.


http://www.uwsp.edu/education/lwilson/curric/affectiv.htm

Image source:

Karin Kirk, SERC

http://serc.carleton.edu

http://www.uwsp.edu/education/lwilson/curric/affectiv.htm

Thought prompt

To what extent is it possible for students to achieve higher level outcomes in online/distance environments?

Harder

Roughly the same

Easier


Creative

Commons

Attribution:

billsoPHOTO

http://www.flickr.com/photos/billsophoto/5243121852





(Re)imagine Assessing Learning Outcomes Consider Environment

Tech-enabled F2F classes

Assessing outcomes might reflect enhanced capabilities with tech

Flipped classes

Assessing outcomes might reflect action or performance focus

Online classes

Assessing outcomes might include student interaction with the world


Activity

Consider example outcomes. What types of assessment work via online/distance learning?

Rethink assessing your outcomes in an online/ distance context.


Select Appropriate Assessment Strategies

Thought prompt

Which activities are supported in online/ distance learning?

Which activities are hindered in online/ distance learning?


Assessing performance

Different strategies used for class assessments

• Written: tests, quizzes, essays, written assignments, case studies

• Oral: recitations, student presentations, oral exams

• Manual: lab practicum, individual and team projects, skills observation

What other types of assessments do you use?


Reasons to assess learning differently

Traditional F2F Methods

• Standardized testing

(multiple choice, etc.)

• Written work (essays,

research reports, etc.)

• Student presentations

• Lab work

• Team projects

Possible F2F Issues

• Does not support all

learners

• Hard to require student

feedback in classroom

• Not enough time in class;

usually not recorded

• Hard to make up if you

miss the lab exam

• Hard to determine who

did what, and when



Module 3:

Choose appropriate

assessment tools



Goals for this Module

Select tools for online class activities

Consider best tool for students to achieve outcomes

Consider tools for different parts of the assessment process

Consider level of support for using the tools

Put it all together with ePortfolios


Select tools for online activities

Using technology for assessment

Low-stakes quizzes / High-stakes exams

Quizzes through a Learning Mgmt System (e.g., Moodle, Blackboard, Desire2Learn)

Written assignments / projects - Grading and feedback process

Screencast (e.g., Screenr, Echo360)

Tablet Apps (e.g., iAnnotate)



Written assignments – iterative process (self-assessment, peer-review, teacher feedback)

Calibrated Peer Review

Online submission tools (e.g., Turnitin)

Wiki or blog tools

Electronic portfolios


cpr.molsci.ucla.edu/‎


Individual or small group projects

Google docs or wiki pages

team organization / coordination / workflow

Blogs or ePortfolios

share and assess final products

Google Picasa + phone pics

math or chemistry equations



Individual or small group presentations

Online presentation tools (e.g., VoiceThread)

Performance-based assessment

Upload observation logs by trusted expert(s)

Audio/video recording of presentations, monologues, etc.

SF State examples: Acting – video files; Business German – audio files



Labs

Commercial lab kits sent to students’ homes

“Kitchen science” labs

Virtual labs / simulations

Intelligent tutors & Adaptive Learning tools

CIRCSIM – first year med students

KnowRe – personalized, real-time math


http://www.cs.iit.edu/~circsim/

https://www.knowre.com/

Model for choosing technology for assessment


synchronous

asynchronous

physical digital

collaborative

individual

x: where

y: when

z: who

Choosing appropriate technology

Questions to consider

• How will using this technology help students learn?

• Do students have access to this technology?

• Do students have access to training and support to use this technology?

• Is this technology accessible to students with disabilities?


Discussion

• Which technologies have you used?

• To assess performance

• Which technologies are you considering for assessment?

• Web 2.0 tools and other online environments

• Mobile technologies

• ePortfolios


Choosing appropriate technology

Resources to investigate

• Virtual resource site - teaching with technology

http://www.umuc.edu/virtualteaching/

• Discussion-based online teaching

http://cgi.stanford.edu/~dept-ctl/tomprof/postings.php (#561)

• How technology influences student success

• MERLOT http://www.merlot.org


http://www.umuc.edu/virtualteaching/

http://cgi.stanford.edu/~dept-ctl/tomprof/postings.php



http://caret.iste.org/index.cfm?fuseaction=answers&QuestionID=1

http://www.merlot.org/

Examples: Integrating Technology

English - ENG 214 – Kory Ching

Writing in a Digital Age

Instructional Tech - ITEC 299 – Kevin Kelly

How 2 Lrn w ur iPod


http://scrivel.files.wordpress.com/2009/07/214syllabus3.pdf

Activity

1. Make a teaching-with-technology plan

2. Share elements of your plan with the group

Objectives Resources Activities Assessments


The role of rubrics

Tie rubric criteria to specific outcomes

Repeatedly assess across multiple assignments


Consider support for assessment tools

Campus apps (e.g., LMS features—quizzes or tests, assignment submissions)

3rd party apps (e.g., Turnitin, Google Picasa)

Publisher tools (e.g., WileyPLUS)

Digital feedback tools (e.g., digital rubric within ePortfolio, audio/video/screencast feedback)


Put it together with ePortfolios

Gathering evidence over time

Electronic portfolios allow different stakeholders to view progress of

ePortfolio resource site: http://eportfolio.sfsu.edu


http://eportfolio.sfsu.edu/







Align with Outcomes at Multiple Levels


ePortfolios enable…


Assessment via ePortfolios

ePortfolios to document:

Self-assessment

Peer review

Instructor feedback (rubric)


Assessment examples

Rubric comments can be shared in addition to automated systems


provide

feedback

& rubrics

encourage

& share

exemplars

cultivate

critical

thinking

elevate

self

evaluation

reward

peer

review

use game

& UDL

elements

provide

guidelines

& grades

foster

learning

improvement

require

attention

to purpose

make

collaboration

a priority

Synthesis


Added benefits of ePortfolios

Demonstration of student achievement

Accreditation

Program Assessment

Student Bridge to Transfer/Workforce

Student Recruitment



Module 4:

Managing consequences

of online SLO assessment


Jim Julius, MiraCosta College, & Kevin Kelly, Wiley Learning Institute

Potential Concerns

Institutions may be out of compliance with regulations re: student authentication

Faculty may not be employing online course design practices designed to encourage academic integrity

Institutions may need to update policies to reflect regulations and/or good practice

Faculty workload issues?


Potential Benefits

Students may develop “showcase”

resources beneficial to their next steps

Thinking about improving online SLO

assessment may result in improvements to

on-ground courses

Other?


Potential actions

Design course and activities in ways to lessen likelihood of students productively cheating


Discuss: Authentication & Academic Integrity

See WCET best practices re: academic integrity & statements from regulators & institutions

Do your institutions have policies that “enforce” these ideas?

What practices do your institutions follow (or not) that might help implement these ideas, regardless of policy?


http://wcet.wiche.edu/wcet/docs/cigs/studentauthentication/BestPractices.pdf


Discuss: Authentication & Academic Integrity

Do you (should you) require use of an institutional learning management system for online assessment?

What do you do about third party systems and technologies?

Under what circumstances should proctoring be encouraged? Required? And what should “proctoring” include?


Points to Ponder

Do tools and processes support meaningful

collection & analysis of course SLO data by

course modality? Is there faculty buy-in for this?

Are Distance Ed (DE) courses defined, tracked,

and reported in a consistent manner?

Is it possible/reasonable to make comparisons

across departmental, college-wide and

statewide DE data? What are the limitations?


Points to Ponder

Do you have a way of tracking DE

program outcomes? (Do you have a way

of tracking DE programs?)

What will the ultimate outcome be when

SLO mania is combined with ratcheting up

DE standards?


Additional Resources

● Universal Design for Learning (UDL) and Online Assessment (Prezi)

● Language on DE Authentication (Google doc)

● WCET Best Practice Strategies for Academic Integrity in Online Ed (PDF)

● Making Online Assessment More Secure (1-pager) (.doc)

61

http://prezi.com/1rm9g73wkic_/udl-and-online-assessment/

https://docs.google.com/document/d/1BLHvrBr1l6QNc4nUXzICMgxtsrRMAA46eSfVSQyubzE/edit?usp=sharing




https://www.dropbox.com/s/gq4i50j3348vjwj/Securing online assessment.doc



Contact info

Email: [email protected], [email protected]

Twitter: @KevinKelly0 (“at Kevin Kelly zero”)

If you use this prezo: Creative Commons BY-NC-SA

To learn more about Wiley Learning Institute:

www.WileyLearningInstitute.com, @LearnApplyShare

62

mailto:[email protected]

mailto:[email protected]

http://www.wileylearninginstitute.com/

Effective SLO Assessment in the Online Environment · 2014-08-14 · Effective SLO Assessment in...

Documents

Transcript of Effective SLO Assessment in the Online Environment · 2014-08-14 · Effective SLO Assessment in...