Navigating Navigating AssessmentAssessmentNavigating Navigating AssessmentAssessment

Malissa Martin, Ed.D., ATC, CSCSMalissa Martin, Ed.D., ATC, CSCSDirector, Athletic Training EducationDirector, Athletic Training Education

Department of Health SciencesDepartment of Health SciencesCollege of Mount St JosephCollege of Mount St Joseph

Cincinnati, OHCincinnati, OH

Outcomes• Review components of a Program

Assessment Plan• Develop and analyze components

of an assessment plan• Describe ways evidence is used to

enhance student learning……… ”Close the Loop”

Agenda

• 2:00-2:15 Review Components of Program• Assessment Plan

• 2:15-2:30 Learning Outcomes Indicators• 2:30-2:45 Performance Indicators and Measures • 2:45-3:00 Benchmarks• 3:00-3:15 Closing the Loop• 3:15-3:30 Summary

Assessment is Learning…….

• Students• Curriculum (didactic and clinical)• Institutional fit• Assessment process

Accreditation General Program

Assessment Questions

• How are your LOs appropriate to your mission, program, degree?

• What evidence do you have that your students are achieving your LOs?

• In what ways to do you analyze student learning?

• How do you evaluate and improve the effectiveness of your assessment of student learning?

• Where do you assess your LOs?

Assessment

• “Systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development”

• Marchese, 1987

Programmatic Assessment

• Provides information about groups of students

• Determines student’s ability to integrate individual courses/clinical experiences into a coherent whole

• “Academic Introspection” Denney, 1996

Characteristics of an Assessment Plan

• All LOs are addressed and assessed• Multiple assessment measures• Efficient procedures• Describe people, committees, process

involved• Plan “closes the loop”

Ball State College of BusinessBanta TW & Palomba CA (1999) Assessment Essentials Josey-Bass

Meaning of Assessment

• What should your graduates know, be able to do, and value?

• Have graduates acquired learning?• What are the contributions of the

program to student growth?• How can student learning be improved?

• Hutchings & Marchese, 1990

Common Language• See handout…..

Guided Outcome Questions….

• Where do students start?• What do they encounter along the

way?• Where do they end up?

“The Ideal Graduate”

Components of a Program Assessment

Plan

1. Expected Outcomes (learning outcomes)

SummativeWhat will students be able to do upon

completion of the program?

FormativeWhat will students be able to do along the way

Nichols Model

• 5 Column Assessment Report Model

• SACS

Bloom’s and Assessment Plan

• Varying degrees of learning

• Produce learning vs providing instruction (Barr and Tagg, 1995)

Group Exercise

• At your table, develop 2 program learning outcomes (LOs).

• Excluding– Students will successfully pass the

BOC examination and gain ATC credential

Components…

Where are outcomes addressed?

Courses (list)Clinical programService learning

Components…2. Criteria to assess outcomes

Performance Indicators

How will students meet LO-type of product

When will students meet LO-formative vs summative

Where will students meet LOdidacticclinical extra-curricular experience (service learning)

Components…

3. Measures (how will data be collected)

Portfolio SimulationsPerformance Case studiesWriting samples Poster presentationsOral presentations InterviewsTests Clinical evaluationsSimulationsFocus groupsSurveys

Qualitative and quantitative measuresDirect vs Indirect measures

Group Exercise

• Using your 2 Los, develop– Performance indicators for each

• How will students meet LO– Product

• When will students meet LO– Formative vs summative

• Where in curriculum will students meet LO– Course -Clinical Exp -Post-Grad

– Identify measures used to assess the LO

Performance Assessment

• Authentic assessment• Linked to curriculum/courses• Real examples of student work• Critical and creative thinking• Holistic feedback• Student motivation• Labor intensive

Key PointsPerformance Assessment

• Articulate skills to be examined• Design appropriate tasks to

demonstrate skills• Specify criteria for evaluating

performance• Develop reliable process for rating

performance• Training raters• Evaluating results

Current Trend

• Course-embedded assessment

• Ewell, 1991 and Wright, 1997

Components…

4. Program’s expected level of performance on each outcome (benchmarks)

Numerical score on national examMastery level on writing sampleMasterly level on oral/poster presentationMasterly level a behavioral performance Masterly level on a clinical evaluationHolistic score on ability to solve a case study

At what level is student expected to achieve the outcome?

Group Exercise

• Using your PIs:– Develop criteria or benchmarks for

each PI

Assessment Schedule

Entry into the programEnd of a specific semesterCompletion of a set of coursesCompletion on a certain number of creditsUpon graduationUpon employmentNumber of years after graduation

Should each outcome be assessed every year?

Who collects/analysis/interpret

s results?

Inside Evaluators Outside EvaluatorsStudent affairs AlumniWriting center Employer

Assessment committee BOCATEP committee GRE

Components…

5. Results of data for each outcome

What percentage (number) of students met the LO/PI at the criteria/benchmark established

May want to include sample size

Sharing Results

Who will see the results?

AdministrationFacultyStudentsOutside accrediting agencies

Components…

6. Closing the LoopDoes the assessment plan lead to improvements in academic and co-curricular programs?

Revise curricula, sequence of courses….

Design more effective student orientation

Revise handbook

Increase connection between academic and clinical program

Purchase additional therapeutic exercise equipment

Closing the Loop

• Question

• “What changes to your program have you made based upon results of your program assessment data?”

Closing the Loop

• Curricular changes– Course information (competencies)– Instructional delivery– Course sequencing– Instructional resources (textbooks,

anatomical models, library holdings)– Technology

Closing the Loop

• Clinical program– Clinical instruction– Clinical sites– Clinical rotations– Clinical opportunities– Clinical performances

Closing the Loop

• Space• Budget• Faculty development• Human resources

Group ExerciseClosing the Loop

• With results provided:

Develop a “closing the loop” plan

Assessing the Program Plan

• Follow-up on implemented changes

• How often?• Who?

Trending Data

• Why?

• See changes overtime……• Overtime learning evidence• Charts student’s chronological

achievement

• Lots of work on front-end!

Trending Data

• Semester to semester• Year to year• Each learning outcome (program vs

student)• Clinical evaluations (student)• Clinical evaluations (instructors)• Clinical site evaluations• Teacher evaluations

Key Points to Faculty Involvement

Responsibility

Resources

RewardsPalomba and Banta (1999)

Characteristics of a Good Assessment Plan

• Asks important questions• Reflects the institutional mission• Reflects program goals and Los• Linked to decision making about

– Curriculum -budget -resources

• Involvement of multiple individuals• Contains relevant assessment techniques

Characteristics of a Good Assessment Plan

• Indirect and direct evidence of learning• Reflects what is known about student

learning• Shares information with multiple

audiences• Leads to reflection of faculty, staff, and

students• Allows for continuity, flexibility, and

improvement in the assessment process

Navigating Assessment

Thank You……

Questions ?????????????