Program Outcomes Assessment Techniques: Overview and Examples

Steve Steele, Consultant Faculty Online Technology Training Consortium (FOTTC)Applied Sociologist Anne Arundel Community College

sfsteele@mail.aacc.cc.md.us 410-541-2369

“I have never let my schooling interfere with my education.”- Mark Twain

Common Conceptualizations of Evaluation/assessment

Is it viewed a “a reign of terror?”

Is it “stuck on like a barnacle”; an afterthought

Is it an ongoing process in itself?

An organized entity in itself

Part of a management information system

Where are We Going Today?

Evaluation? Assessment? Definitions

Reasons For Doing This

Placement of Evaluation/assessment

The Dynamic Fit Between Planning, Evaluation/assessment and Doing In the Improvement Process

Examples

Summary: Strengths and Weaknesses

Evaluation? Assessment? Definitions

The systematic assessment of the operation and/or outcomes of a program or policy, compared to explicit or implicit standards, in order to contribute to the improvement of the program or policy. (Weiss, 1998)

“The key sense of the term... refers to the process of determining the merit, worth, or value of something...” (Scriven, 1991, pp. 139-141)

“evaluation/assessment research... refers to research purpose rather than research method. Babbie,1989, 326)

Critical Components of Definitions

A systematic process

A judgment will likely be made

Standards for judgment, for success

Intervention

Orientation toward Improvement?

Reasons For Doing This Internal

Clients/Stakeholders

The President

The Dean

Your Chairman

Your students

You!

External Clients/Stakeholders

Societal

Middle States

Professional

Granting agencies

Reasons For Doing This

Have you considered the client’s “corporate culture” when constructing this project/design?

Are you sure that you have focused on the client’s needs for ALL relevant clients and stakeholders?

Placement of Evaluation/assessment

Think of evaluation/assessment before the program, not after.

Integrate evaluation/assessment into the grant proposal, plan or program.

Use evaluative data “for all it’s worth!”

Consider evaluation/assessment as constructive criticism, rather than “doing time!”

The Dynamic Fit Between Planning, Evaluation/assessment and Doing In

the Improvement ProcessIs it part of a larger process?

Plan

DoCheck

Act

HERE!!

Based on E. Deming’s work

Plan

DoCheck

Act

“Plan” and Define it.

What is it? Intervention DefinitionsIntervention DefinitionPlan

Strategic Plan

"...the process of determining what (something) intends tobe in the future and how it will get there..."1

"...the process by which guiding members of anorganization envision its future and develop the necessaryprocedures and operations to achieve that future..."2

Program "(An) effort that marshals staff and projects toward some...defined... goals"3

Project "...time-bounded efforts within a program"4

Process "...what happens between input and output..."5

Course, Training orWorkshop

"...an organized effort in which people or persons learnsomething..."

Something OtherPhenomenon (SOP)

Whatever it is, it doesn't fit one or more of the above!

"I Don't Know" This intervention is ill-defined or undefined.

Example: A Distance Learning Business Plan

http://www.aacc.cc.md.us/ola/

http://www.aacc.cc.md.us/ola/Assessment/BusPlanNov98.html

Goals and Objectives...

http://www.aacc.cc.md.us/ola/Assessment/BusPlanNov98.html#To implement

Plan

DoCheck

Act

“Check”Some Approaches to “Checking”

Keeping track... is it operating (not necessarily successfully!)

Forming something, improving something

Sum up, impact… did it work?

Based on Michael Scriven’s Formative & Summative Evaluation

Some Tools for Tracking, Forming and Summing

Frequency of something

Rates or percentages of something

Level of Need

Opinions, behaviors, feelings

Indicators, Estimates and Forecasts

Survey

Use patterns

Focus groups

A Plan Level Example #1

Wells, Kepner, Barnes, SteeleAACC

Toward an Evaluation/assessment Plan

http://www.aacc.cc.md.us/ola/Assessment/Baldrige.html

http://www.aacc.cc.md.us/ola/Assessment/assessmentStudentStakeholder.htm

Scroll

Measures….

http://www.aacc.cc.md.us/ola/Assessment/studentsurveyresults.htm

A Plan Level Example #2

Barnes, Warner, SteeleAACC

Some Tools for Tracking, Forming and Summing

Frequency of something

Rates or percentages of something

Level of Need

Opinions, behaviors, feelings

Indicators, Estimates and ForecastsSurvey

Use patterns

Focus groups

On Line Enrollment

500

600

700

800

900

1000

1100

Fall 2000 1019 1031

Predicted Actual (unofficial)

A Course Level Example #3

SteeleAACC

Some Formative Checks on Soc111 Online

Plan

DoCheck

Act

How Many Links on the Module Did You

Visit (all modules, N=188 response)? "Site_visit"

16.5%

38.3%22.3%

17.6%

5.3%"Very few of the sit

"Some of the sites""None of the sites"

"Most of the sites"

"All of the sites"

Rate this Module's Learning Value (all modules, N=188 response)?

"Learning_value"

41.0%

50.5%

6.4%

2.1%

"3. Excellent"

"2. Good"

"1. Fair"

"0 Poor"

Rate this Module's Interest Value (all modules, N=188 responses)?

"Interest"

48.9%

44.1%

5.3%

1.6%

"3. Excellent"

"2. Good"

"1. Fair"

"0 Poor"

Summary: Strengths and Weaknesses

StrengthsOpportunity for continuos growth Possibility for continuous improvementMakes planning usefulImproves program and course design

WeaknessesResource and infrastructure commitmentRequires a variety of skillsDifficult to stay “tangent”Demands attention