Introduction toAssessment

Jennifer LowmanCoordinator, Student Persistence Research

Credit for the content of this presentation should go to Marilee Bresciani and her many co-authors; & Gavin Henning & others with Academic Impressions

Outline

• Define Assessment• Identify different Types of Assessment• Review Cycle of Assessment• Define Outcomes– Writing “Learning Outcomes”

• Practical Assessment Techniques• Ethics• Concluding Remarks

What is Assessment?

• Assessment, Evaluation, & Research• Assessment– An effort to gather, analyze, and interpret evidence to

understand effectiveness and improve programming• Evaluation– Using assessment information to make an informed

judgment of program quality or worth• Research– Collection and analysis of data to build or confirm theory

or conceptual foundations

Institutional Assessment

• Mission• Goals or Objectives• Program, Product, Service Inputs• Outcomes• Improvement or Accountability• Assessment Planning part of Programming– Embed Assessment

• Design, Measurement, & Analysis• Reporting & Learning

Embedded Assessment

• Make assessment part of your program, event, service, or activity

• Decades of research shows that students respond to intentional activities that are linked to positive outcomes (Bresciani, Zelna, & Anderson, 2004).– If you tell students what you expect them to learn

and how it will benefit them, they are more likely to learn

Embedded Assessment is…

• Continual assessment for continual improvement

• Track changes in program impact over time, as well as…– Improve measurement over time– Test new measures, while still using old

• Make it a habit – a state of mind!

Culture of Assessment

• Build shared trust: Begin by lowering social and interpersonal barriers to change.

• Build shared motivation: Collectively identify goals worth working toward and problems worth solving.

• Build a shared language: Develop a collective understanding of new concepts needed for transformation.

• Build shared guidelines: Develop a short list of research-based guidelines for using assessment to promote learning

Types of Assessment

Types of Assessment

Tracking UsageNeeds AssessmentSatisfaction StudyCulture/Climate AssessmentOutcomes AssessmentResource Effectiveness StudyBenchmarkingProgram ReviewStrategic PlanningIn

crea

sing

Com

plex

ity &

Inte

grati

on

Forms of AssessmentOperational or Descriptive Assessment• Tracking Usage• Needs Assessment• Satisfaction Study• Culture/Climate Assessment

Formative Assessment• Outcomes Assessment• Resource Effectiveness Study• Benchmarking

Political Assessment• Program Review• Strategic Planning

Types of Assessment?

(activity)

Cycle of Assessment

1. Define issue/problem2. Identify Goals3. Align mission & goals4. Identify stakeholders5. Identify

theoretical/conceptual framework

6. Develop measurable outcomes from goals

7. Identify and measure inputs

8. Classify strategies from framework to reach goals

9. Isolate action steps to implement strategies

Cycle of Assessment: External Perspective

10. Develop formative assessment plan to understand process

11. Measure Outcomes12. Make sense of results13. Report / Present

findings14. Review, Reallocate,

Advocate for Resources15. Retool Program16. Plan for Iteration #2

Cycle of Assessment: Internal Perspective

Mission Purpose

Identify Outcomes

Create Opportunity

Gather, Interpret

Data

Improve Program

Outcomes-Based Assessment

Assessment & Planning

• Assessment and Planning are linked, intertwined, and interrelated

• “Backwards Design” or “Start with the end”• Programs are designed this way too!

Desired Outcome

Program Inputs

From Goals to Outcomes

• Goals/Objectives – end result in broad terms (optimal outcomes)

• Outcomes – more specific statements derived from goals that are measurable– Operational Outcomes – Learning Outcomes– Program Outcomes

Learning v. Program Outcomes

• Learning Outcomes (aka Individual Outcomes) – a learning outcome is participant-centered and describes the desired effect of a program on a student’s cognitive or intellectual state.

• Statements of what students will value, do, or know as a result of participation. – ASK: attitude, skills, & knowledge

• Program Outcomes (aka Aggregate Outcomes) – the desired aggregate effect of a program.

Program or Learning Outcome?

(worksheet)

Learning Outcomes

• Participant - centered– Focus on what students can do– Not on what you did!

• Purpose to reveal the impact of ‘Action Steps’– Action Steps are the point of contact– May or may not be perceived by students– Outcomes may need to take variation in contact into

account– Variation is key for identifying effectiveness!

Learning Outcomes Assessment

• What are students learning and why?• How do we know?• Are students learning what we intend for

them to learn?• How do we use assessment information to

improve learning?• Do the improvements we make work?

SMART Outcomes

• SMART Specific Measurable Aggressive/Attainable Results-oriented Time-bound

Writing Learning Outcomes

• ABCD formula– Audience (Who)• Who is the target?

– Behavior (What)• What should the target be able to know, do, value (ASK)?

– Condition (How)• What Action Steps will facilitate the learning?

– Degree (How much)• How much learning will be demonstrated?

Writing Learning Outcomes

Condition + Audience + Behavior + Degree

As a result of participating in the event +students +

will demonstrate + 3 of 5 skills.

ABC v. ABCD…or why Degree is important!

• The degree of impact you expect to have on your students is important to… – Set criterion for success AHEAD of program

implementation– Demonstrate Minimal Benefit

• Transparency & Accountability– Signal intention to students and stakeholders– Establish standard for data-driven decision-making

Degree of Impact• Independent Indicators (Index)

– Inter-changeable, equally important information– Differences between students at start/finish are not import.

• Ex. 3 of 5 study habits• Ex. 5 of 7 coping strategies

• Continuous Scale– Cumulative or increasing attitude, skill, or knowledge– Expect differences between students at start/finish

• Increased leadership skills from baseline eval to next level (basic to intermed.; intermed. to adv.)

• Process– Student must master step 1 to move onto step 2…etc. – Describe steps in process within LO– Break process in to multiple LOs, but avoid yes/no outcomes

Degree of Impact ≠

Depth of Learning

Degree of Impact & Depth of Learning

• Influenced by the opportunity provided– Exposure – Relevance– Immediacy– Salience– Feedback– …

• L.O. reflects depth of learning in the “Action Verb”

Depth of LearningBloom’s Taxonomy (revised)

• Remembering: recognizing, recalling, labeling• Understanding: interpreting, summarizing,

contrasting• Applying: executing, implementing, classifying• Analyzing: differentiating, organizing, attributing• Evaluating: critiquing, integrating, generalizing• Creating: generating, planning, producing

(teaching)

List five ways identified in the readings that can help you reduce stress? (Remembering)

Based on the floor meeting, discuss at least three reasons why we don’t allow alcohol in the residence halls?

(Understanding) How can you use what you learned in these activities in your student organization?

(Application) After participating in the retreat, compare the effectiveness of leadership tactics?

(Analyzing) As you reflect on this past year as I have served as your organization’s advisor, in

what areas have I been most effective and in what areas can I improve (Evaluating)

Based on our conversation regarding career action steps, recommend four action steps to each of your group members?

(Creating)

The Depth of Learning expected is evident in the Action Verb!

Action Verbs (Observable Behaviors)

Lower-LevelList, Label, NameDefine, DescribeIdentify, InterpretContrast, DiscussClassify, CompleteRelate, ModifyDemonstrate

Higher-LevelSolve, ApplyAnalyze, ExplainInfer, IntegratePlan, CreateDesign, GeneralizeAssess, RankRecommendCompare, Teach

• Don’t Use– Appreciate– Become

aware– Be familiar

with– Know– Learn– Understand

Practice Writing Learning Outcomes

(Activity)