Hawaii Strategy InstituteApril 16-17, 2010

Terri M. Manning, Ed.D.Center for Applied Research

Central Piedmont Community College

Disbelief & DenialParalysis - Passive resistance

Anger and antagonismResistant & Reactive

Stage 1

Stage 2

Stage 4

DepressionCompliance - Passive reactive

Acceptance & adaptationChallenge & competitionCatalyst - Proactive

Stage 5

Stage 3Bargaining - no time/no moneySeek outside sources

In Education…We’ve learned that things come and they goMost of these trends are purely academic

exercisesThe national educational climate is skeptical

about accountability!They want us to prove that students are

learning, that their lives are improving and that we are good stewards of funds!

The Outcome Assessment Dilemma

What Do The Accrediting Agencies Want?Mandate for Identifying, Measuring and Using Outcome Data

  Identify Assess Analyze Use For

Accrediting Body Outcomes Outcomes Results Improvement

Middle States7, 11, 12, 13, 14 X X   X

New England4.18, 4.28, 4.44, 4.45 X X   X

North Central2c, 3a, 4b, 4c X X   X

Northwest2b, policy 2.2 X X   X

Southern2.5, 3.3.1, 3.5.1 X X X X

Western4.6, 4.7, 4.8 X X X X

Source: Gita Wijesinghe, Florida A&M

What Western Wants - The Critical Pieces

The institution 1. demonstrates a conscious effort to

produce and support student learning (needs to appear “planful”)

2. measures that learning3. assesses how well learning is

occurring4. and makes changes to improve

student learning.

What Western Wants - The Critical ProcessesThe institution

1. organizes its key processes and 2. allocates its resources to effectively support

student learning3. demonstrates its effectiveness by providing

a) evidence of the achievement of student learning outcomes and

b) evidence of institution and program performance.

The institution 1. Uses ongoing and systematic evaluation and

planning to refine its key processes and improve student learning.

But Why?We should periodically take a serious look at

our students? Are they learning? Who is learning best? Are they achieving the outcomes we expected? Should we make changes in programs and

services? Do we need more in-depth services? Do we need a new curriculum or a change in

methodology?Are we placing barriers in the way of student

success?

Myths and Misconceptions About Learning Outcomes Assessment

If three is good then sixteen must be betterWe must assess everything at every level every

semester in every courseIt’s a performance measure that you pass or fail.We have to follow a certain process or pattern to do it

rightEach measure or outcomes must have multiple types of

assessmentAssessment is only the faculty’s jobSpecific assessment is prescribed by the accrediting

agency – certain expectations

Source: Adapted from Lynn Priddy, Higher Learning Commission of NCA, Best Practices in IE Conference, 2008

Great Fallacy #1Grades

Not specific about skills learned or quality of learning

In this day of social promotion, grade inflation and different teaching/learning philosophies, grades tell you virtually nothing.

They are not a measure of outcome achievement.

Two teachers will grade a student differently for the exact same work.

Great Fallacy #2By evaluating teaching, we are evaluating

learning.Only tells you who is happy…”happiness

coefficient”Measures more about the business and science of

teaching – not learningLearning outcomes are not measures of teacher

effectiveness or students satisfaction with the teaching/learning process

Learning outcomes are skills based

Great Fallacy #3I have 65 learning objectives

on my syllabusThe faculty has established

that when students complete ENG 111, they will have met these 65 objectives – so we are establishing and measuring outcomes

What is the problem with this?

Great Fallacy #4Students are able to “self evaluate” how

well they have learned, so to measure outcomes, – we will ask them.

We ask questions like:Since attending the college, my

communication skills, both written and oral, have improved.

Since attending the college, my mathematical and logical reasoning skills have improved.

Since attending the college, I have gained an understanding of how cultural and ethical issues influence the development of society.

Types of AssessmentFaculty are good at assessment.

It is what they do!But there are different types of assessment based on the purpose and its intended use.

Classroom Assessment (also called grading)Goes on throughout the term.Gives students an indication of their

progress, abilities and standing in a course.

Is used to determine how well a given student completes the requirements for a course.

Often predicts how well they will do in the next course.

Assessment“The assessment of student learning can be defined

asthe systematic collection of information about student

learning, using the time, knowledge, expertise, and resources

available, in order to inform decisions about how to improve

learning. (p.2)”

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Assessment CharacteristicsIt is intended not to generate broad

theories but to inform action.Educational situations contain too many

variables to make “proof” possible. Assessment gathers indicators that will be

useful for decision making.It is not limited to learning that can be

objectively tested. … can use the best available indicators.

It does not require standardized test or objective measures.

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Assessment CharacteristicsFaculty regularly assess complex work in their

fields and make judgments about its quality.Faculty can make informed professional

judgments about critical thinking, scientific reasoning, or other qualities in student work, and use those judgments to inform departmental and institutional decisions (p. 2)

Assessment means basing decisions about curriculum, pedagogy, staffing, advising and student support on the best possible data about student learning and the factors that affect it.

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Assessment CharacteristicsA lot of assessment is already going on in

responsible classrooms, departments, and institutions, though we have not always called it that.

Assessment can move beyond the classroom to become program assessment:Classroom assessment – faculty evaluates her own students’

assignments in the capstone course and uses the information to improve her own teaching the next semester

Program assessment – faculty evaluated her own students’ assignments in the capstone course, outlining the strengths and weaknesses of the students’ work in relationship to departmental learning goals. The department uses the data to inform decisions about curriculum and other factors that affect student learning. (page 2-3)

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

DifferencesClassroom assessment informs the studentLearning outcomes assessment informs the

program faculty.It is like formative and summative evaluation.

One is the final judgment – did they pass? The other informs us along the way as to what is and is not working. We adjust – students learn better.

Methods of AssessmentCourse imbedded assessments

Written worksStudent journalsSpeeches

Skills-based assessment (demonstrated skills)Observation checklistsTeamwork assessmentsSurveys that ask about specific behaviors

indicative of changes in values and attitudesNot self-evaluation of actual learning

> > >

Benefits for People

*New knowledge *Increased skills

*Changes in values *Modified behavior *Improved condition *Altered status *New opportunities

INPUTSINPUTS ACTIVITIES OUTPUTS OUTCOMES

(Outcomes answer the “so what” question)

Outcomes are ……... Driven by the mission. Related to overall program goals. Specific to the teachings/activities of your

program/course. Determined by faculty and front-line staff. Measured carefully and specifically.

Inputs

Activities

Long-range

Outputs

Intermediate

Initial OUTCOMES

Different Types of Outcomes

Learning Outcomes (can be at course, program or institutional level)

Program OutcomesAdministrative Outcomes

Definitions and ExamplesLearning Outcomes:

What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students.

Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

Definitions and ExamplesProgram Outcomes:

The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the entire degree versus those who take a few courses? If so what are they?

Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools or graduate programs, lifelong learning issues, contributions to society, the profession, etc.)

Definitions and ExamplesAdministrative Outcomes

Units/programs want to improve services or approach an old problem in a new way.

They want to become more efficient and effective.Typical examples are:

All faculty will attend one professional meeting annually so they can stay up-to-date in their field, or:

Counseling wants to recruit a new counselor with expertise in working with first-generation students, or:

Facilities services wants students, faculty and staff to feel that they are safe on campus.

Five Aspects of Good AssessmentGood Assessment is: Used Cost Effective Accurate and Truthful Valued Derived from Clear and Important Goals

Linda Suskie, Middle States Commission on Higher Education

Good Outcome Assessment is UsedThe results are used to inform important

decisions on important goals.What is the department most concerned with?Are there goals for which the department/unit

needs data (proof)?Don’t create data and a document that sits on

the shelf – it should be used again and again.

Used, cont.Good assessment is planned and purposeful.

Who has been/will be involved in the decision about what to assess?

When will assessment occur (best time, best course)?

What will you do with result or what results do you want to see?

What should be measured first? If you add more variables each year, what should come next?

Used, cont.Good assessment should focus on clear and

important goalsStay away from the vagueIt is clear that English should measure writing

skillsIt is clear that Speech should measure oral

communicationWhat is it clear that you should measure?

Used, cont.Active participation of stakeholders

Get input from the right people State agencies, accrediting agencies, other colleges

with programs like yours Your faculty, staff and students Advisory committees and employers of graduates Anyone else?

Used, cont.Assess teaching-learning process as well as

outcomesWhat is the process and why is it important?

Methods, classroom strategies, online and hybrid –Be concerned with more than the content –

through which processes do student learn best?

Used, cont.Results communicated widely and

transparentlyWhat do you do if they are not learning?If your results are not good? Is that bad?Who needs to know the results?What does it mean to be transparent?

Used, cont.Results used fairly, ethically and responsibly

Faculty should not penalizedOutcome data makes people territorialWhat does it mean to use outcome results

ethically and responsibly

Good Assessment is Cost EffectiveCost effective – efficient and economically

(especially time)Don’t kill your facultyWhat are they hired to do?Do as little assessment as possible and still

obtain meaningful, good dataWhat can you afford (both time and money)?

Cost Effective, cont.Focus on clear and important goals

Pick 3-6 goals at the maximum for which to create outcome assessment

All the courses in your department/program contain hundreds of objectives.

Which ones are most important?

Cost Effective, cont.Start with what you have

What do you have?Assignments and tests?

Can you work with those? (examples)Group projectsClinical hoursCo-op and internship experiences

Cost Effective, cont.Simple

Don’t create an elaborate assessment process unless you have enough staff and resources to carry it out

If you create surveys or assessment tools, keep them simple

If you conduct focus groups, get as much out of them as possible with as few questions as possible

Cost Effective, cont.Realistic expectations

What should your benchmarks be?Will it all “go down” like you want it to?What will happen if faculty have trouble with

the process?Should you start with training?

Accurate and TruthfulRepresent a balanced sample of key goals

including thinking skillsMeasures what is importantMeasures what are considered “critical” to

students in the fieldMeasures more than what students can

memorize – but rather their processing and thinking skills

Accurate and Truthful, cont.Use a variety of approaches, including direct

evidence of student learningTests and classroom assessmentsSurveys of graduates and employersFocus groupsPortfolio assessmentsDirect observations and skill-based

assessments

Accurate and Truthful, cont.Recognize diverse approaches to teaching

and learning and is developed thoughtfullyFaculty who lecture, use small groups,

interactive approaches – will it work for all?Take the time to make sure this assessment

will give you the results you wanted.

Accurate and Truthful, cont.Perpetual works in progress

No one does this well the first timeHealth programs have been doing it the

longest and can help othersIt takes more programs a couple of times

before they create good assessment dataDo you have that kind of time?Our attitude should be “we are always growing

and improving.”

Good Assessment is ValuedAssessment results/efforts are

recognized/honoredWho should value the resultsIf they are valued, what should our actions beWhat about senior administration (part of a

process)If we uncover needs, what should happen?

Valued, cont.Innovation, risk-taking and efforts to improve

teaching and learning are recognized and honoredIf faculty must do assessment and are willing

to try new things to improve results, we should honor and value their efforts.

Assessment will have direct effect of a center for teaching and learning.

Valued, cont.Supported with appropriate resources: time,

guidance, support and feedbackWhat do we need to do this well?What should happen to the results?

Derived From Clear and Important GoalsHave clear and important standards for

acceptable and exemplary student performance Do we test for minimal standards?What is considered good performance?Are there clear standards?Should we compare ourselves with other

schools?

What is an Outcome Objective?A short-term, measurable, specific activity

having a time limit or timeline for completion around a specific outcome

They measure outcomes and are used to show progress toward goals

They specify who, will do what, under what condition, by what standard and within what time period

How to Set Outcome ObjectivesThere’s no magic number

e.g. 80% or 90%What is reasonable?What can you afford?What realistically can your staff accomplish?What percent shows you’re not committed and

what percent shows you’re naïve?

How to Set Outcome ObjectivesExamples:

Fifty percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 60 [D] or better)

By the end of the spring term, 95% of faculty and staff will have completed 20 contact hours of professional development (workshops, college courses, conferences, onsite trainings, etc.)

More RealisticSeventy percent of students will be able to

communicate effectively in writing (complete the writing exam with a grade of 75 [C+] or better)

By the end of the spring term, the professional development office will increase their offerings for faculty and staff by 10% over what was offered last year (workshops, college courses, conferences, onsite trainings, etc.)

How to Set Outcome ObjectivesThe first time you set objectives, be

conservative Allow yourself a pilot semester or year to

determine the appropriate levels of change that can be expected

Don’t pull a rabbit out of your hat (e.g. let’s grow enrollment by 10%)

May need to benchmark (what does it mean)

Why is This Hard?

Because it is education

Because the best results may not happen for years

Because we are so busy doing what we are doing…. we forget why we are doing it

Let’s Look at Healthcare When you have strep throat and go to the

doctor for your antibiotics (your intervention) What are your intended outcomes? Would the doctor ever tell you:

We believe curing disease is a developmental process

We believe there is value in the activity of taking pills and receiving shots

We’re not sure if you’ll get better, it is how all doctors have treated the disease since we learned about it

What would you say to that?

How to Measure Program/Student

Outcomes

* program documents* program faculty and staff* national associations/credentialing boards* key volunteers* former students* parents of students* records of complaints* programs/agencies/employers that are the next

step for your students* other colleges with similar programs, services

and students as yours* outside observers of your program in action

How Often Should we measure objectives or student learning outcomes every year?

When does measurement become too time consuming?

Units need time to put into effect the changes made as a result of outcome assessment before they are thrown back into another cycle. They need time to reflect on changes and results.

Internal Factors:

* Sudden faculty/staff turnover* New teaching philosophy/strategy* Curricular change (campus move)* Unrealistic outcome targets* Measurement problems (lack of follow- through, no effective tracking)

External Factors:

* Community unemployment increases* State funding changes* Related programs (BS or MS programs) close* Public transportation increases fares or shuts

down some routes serving your campus or time slot* Employment trends change

Use Your Findings

Internal Uses for Outcome Findings

Provide direction for curricular changes Improve educational and support programs Identify training needs for staff and students Support annual and long-range planning Guide budgets and justify resource

allocations Suggest outcome targets (expected change) Focus board members’ attention on

programmatic issues Help the college expand its most effective

services Facilitates an atmosphere of change within

the institution

An Example from One College ProgramWorkplace Literacy Program

This program is a literacy initiative that goes directly into the worksite and teaches ESL classes, GED prep and GED classes. Serves mostly immigrants.

During their first attempt at outcome assessment, they surveyed both employers and students.

This was the first time they had ever done this.

What They LearnedEmployers said:

43.8% of employers reported increases in employee performance as a result of participation in the program.

31.3% reported a reduction in absenteeism by participants.

87.5% said classes improved the morale of their employees

37.5% said participants received raises50% said communication had improved.

What Students Said70.2% reported being able to fill out job

forms better35.5% said they could now help their

children with their homework91.1% said they felt better about

themselves44.4% said they had received a raise,

promotion or opportunity as a result of the courses

86.3% said their ability to communicate in the workplace had improved

What Has Happened SinceTheir assessment data has shown up in their

marketing brochures to employers.Their enrollment has grown dramatically.They have received funding and marketing

support from their local Chamber and are considered a model adult literacy program.

External Uses of Outcome Findings

Recruit talented faculty and staff

Promote college programs to potential students

Identify partners for collaboration (hospitals, businesses, etc....)

Enhance the college’s public image

Retain and increase funding

Garner support for innovative efforts

Win designation as a model or demonstration site

Top Ten Skills for the FutureWork ethic, including self-motivation and time management.Physical skills, e.g., maintaining one's health and good

appearance.Verbal (oral) communication, including one-on-one and in a

groupWritten communication, including editing and proofing one's

work.Working directly with people, relationship building, and team

work.Influencing people, including effective salesmanship and

leadership.Gathering information through various media and keeping it

organized.Using quantitative tools, e.g., statistics, graphs, or

spreadsheets.Asking and answering the right questions, evaluating

information, and applying knowledge.Solving problems, including identifying problems, developing

possible solutions, and launching solutions.The Futurist Update (Vol. 5, No. 2), an e-newsletter from the World Future Society, quotes Bill Coplin on the “ten things employers want [young people] to learn in college”

Let’s work through a few?How does the early childhood program

improve the work ethic of childcare workers or the children in childcare centers?

How do IT programs improve teamwork skills?

How does Engineering improve students’ ability to use spreadsheets and read tables/graphs?

How do you improve a student’s ability to:Influence peopleHave time management skillsAsk the right questionsSolve problems in the work environment

Learning Outcomes for the 21st Century Students in the 21st Century will need to be proficient in:Reading, writing, speaking and listeningApplying concepts and reasoningAnalyzing and using numerical dataCitizenship, diversity/pluralism Local, community, global, environmental awareness

Analysis, synthesis, evaluation, decision-making, creative thinkingCollecting, analyzing and organizing informationTeamwork, relationship management, conflict resolution and

workplace skillsLearning to learn, understand and manage self, management of

change, personal responsibility, aesthetic responsiveness and wellness

Computer literacy, internet skills, information retrieval and information management

(The League for Innovation’s 21st Century Learning Outcomes Project.)

ChallengesIdentifying and defining outcomes is the easy

part.The devil is in the details.How do we track it, where does it all go, how

do we score it, compile it, turn it into a comprehensive report.

How do we “demonstrate improvement in institutional quality.”

Some things are hard to measure such as analytical reasoning – we know it in its absence

Things to RememberOutcome measurement must be initiated from the

unit/department level (promotes ownership of process).

Measure only what you are teaching or facilitating.

Measure what is “important” to you or your program.

Be selective (2-3 outcomes only for a course, a select list for programs and institutional outcomes).

Put as much time in to “thinking through” the tracking process as you do into the definition of outcomes.

Spend the time up front in planning and the process will flow smoothly.

It will prove to be energy well spent.

RememberWe do not do outcome evaluation so we can

say we did it.

We do it only for one reason:

To Improve Programs and Services

Where Colleges Get In TroubleOverkill – they evaluate everything that walks

and breaths every semester in every area.No time to “reflect” before they enter back

into another assessment cycle.No focus on “use of results.”No ability to track results and tally them

across the College.

The Insanity Principle

Doing the same thing we have always done but expecting different results.