3/17/14  

1  

Using Outcomes to Drive Change in your Program!

March| 2014!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Outcomes….!

•  Student Learning Outcomes!

•  Program Outcomes!

•  Clinical Outcomes!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Student Learning Outcome    One row for each SLO  

 

Objectives   (Expected Learning Outcome)  

 List all Objectives of the given SLO in same row (1.1,1.2,1.3).  

Assessment Strategy and Benchmark Note:    The  same  instrument/tool  (e.g.,  an  external  exam,  a  capstone  project,  or  other  cumula>ve  experience)  may  be  used  to  assess  mul>ple  objec>ves.  

Assessment Results    Summarize  the  results  found  for  each  objec>ve,  related  to  the  benchmark.  

Use of Results        Summarize  the  planned  program  changes  intended  to  improve  student  learning.      

   

     

   

           

3/17/14  

2  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Outcomes: Develop a Plan (4): !

•  There must be a comprehensive assessment plan to evaluate all aspects of the educational program. !

•  Assessments used for this purpose must include those defined in Standards 6 and 7. Additional assessments may include, but are not limited to, clinical site evaluations, clinical instructor evaluations, completed clinical proficiency evaluations, academic course performance, retention and graduation rates, graduating student exit evaluations, and alumni placement rates one year post graduation.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Annual Report Section V: Outcomes!

34. Provide one outcome for your program:!35. Provide the objectives that you measured for this

outcome: (2-7)!36. What assessment strategies did you use to

measure each objective: !37. Provide the benchmark(s) you set to determine if

your students are competent based on your assessment strategy: !

38. Which benchmarks were met/not met: !39. What changes did you make in the program as a

result of the outcomes measurement: !

3/17/14  

3  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Section V: Outcomes – WHAT NOT TO DO!!

34. Outcome: Pass the BOC …!35. Objectives: Learn all competencies and proficiencies!36. Assessment strategies: Class Grades! !37. Provide the benchmark(s): Each Student will … !38. Which benchmarks were met/not met: All. !39. What changes did you make in the program as a

result of the outcomes measurement: None!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Outcomes: Develop a Plan (4): !

•  There must be a comprehensive assessment plan to evaluate all aspects of the educational program. !

•  Assessments used for this purpose must include those defined in Standards 6 and 7. Additional assessments may include, but are not limited to, clinical site evaluations, clinical instructor evaluations, completed clinical proficiency evaluations, academic course performance, retention and graduation rates, graduating student exit evaluations, and alumni placement rates one year post graduation.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Developing an assessment plan:!

The Basics!1.  Institutional Mission Statement!2.  Department / Program Mission Statement!3.  Institutional Goals and Outcomes!4.  Department / Program Outcomes!5.  Measureable Objectives!6.  Assessment Strategies / Benchmarks!7.  Administrative Assessment of Results!

3/17/14  

4  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Mission Statement !

•  This is the broad statement of purpose and aspiration of the department or program. It should be closely aligned to the University’s mission statement and should be semi-permanent. !

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Mission Examples!

•  The University of Wisconsin Oshkosh provides a wide array of quality educational opportunities to the people of northeastern Wisconsin and beyond through the discovery, synthesis, preservation and dissemination of knowledge. The interaction of our dedicated faculty, staff and students fosters an inclusive learning environment that prepares our graduates to meet the challenges of an increasingly global society.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Mission Examples!

•  The University of Wisconsin Oshkosh Athletic Training Program exists to provide an effective and interactive learning environment for individuals pursuing a career in athletic training. The program is committed to preparing high caliber graduates equipped to provide quality health care to physically active populations in a changing world.!

3/17/14  

5  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Developing an assessment plan:!

The Basics!1.  Institutional Mission Statement!2.  Department / Program Mission Statement!3.  Institutional Goals and Outcomes!4.  Department / Program Outcomes!5.  Measureable Objectives!6.  Assessment Strategies / Benchmarks!7.  Administrative Assessment of Results!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Institutional Outcomes!

•  How many institutional Outcomes/Goals does your program meet?!

•  Your Assessment Committee’s want to know!!

•  Your institution’s accreditors will look for alignment!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Eastern U. Institutional Learning Outcomes !1. Students develop intellectual curiosity, passion, and agility, valuing the life of the

mind and life-long learning.!2. Students develop their critical thinking, reflection, analysis and communication

skills.!3. Students develop knowledge and competencies in the arts, sciences and

professions.!4. Students develop and expand their Christian worldview, grounded in the

Scriptures.!5. Students discern the ethical consequences of decisions and actions.!6. Students are motivated to assume responsibility for justice and to show a

transformative influence—especially regarding social, political and economic justice.!

7. Students are prepared to live in an interdependent world, aware of societal and global problems and committed to engage in solving them.!

8. Students increase in self-awareness and in their sensitivity towards others and others’ needs and situations.!

9. Students contribute to fostering an environment where diversity is appreciated and reconciliation is practiced.!

3/17/14  

6  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Eastern U. Institutional Learning Outcomes !1. Students develop intellectual curiosity, passion, and agility, valuing the life of the

mind and life-long learning.!2. Students develop their critical thinking, reflection, analysis and communication

skills.!3. Students develop knowledge and competencies in the arts, sciences and

professions.!4. Students develop and expand their Christian worldview, grounded in the

Scriptures.!5. Students discern the ethical consequences of decisions and actions.!6. Students are motivated to assume responsibility for justice and to show a

transformative influence—especially regarding social, political and economic justice.!

7. Students are prepared to live in an interdependent world, aware of societal and global problems and committed to engage in solving them.!

8. Students increase in self-awareness and in their sensitivity towards others and others’ needs and situations.!

9. Students contribute to fostering an environment where diversity is appreciated and reconciliation is practiced.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Developing an assessment plan:!

The Basics!1.  Institutional Mission Statement!2.  Department / Program Mission Statement!3.  Institutional Goals and Outcomes!4.  Department / Program Outcomes!5.  Measureable Objectives!6.  Assessment Strategies / Benchmarks!7.  Administrative Assessment of Results!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Characteristics of outcomes!The outcome is our destination!

!

•  Primary purpose is student learning!•  Should reflect the culture, mission and needs of

an individual institution/program!•  Why we do what we do!•  What does a graduate from your program

“look like”? !

3/17/14  

7  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

What are we telling them?!•  Do our students meet our standards?!•  Do we comply with external standards?!•  How do we compare with peer standards?!•  Value-added – are our students improving?!•  Historical trends – is teaching and curricula

improving?!

= What we expect our graduate’s will look like!–  Knowledge, skills, values and habits!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Not an outcome…

!Entry-level professional knowledge, comprehension, and proficient skills of an orthopedic clinical examination and diagnosis of the knee!

3/17/14  

8  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Not an outcome !Prepare graduates to apply a wide variety of specific health care skills and knowledge!!within each of the following 8 content areas (NATA, 2011): evidence based practice, prevention and health promotion, clinical examination and diagnosis, acute care of injury and illness, therapeutic interventions, psychosocial strategies and referral, healthcare administration, professional development and responsibility. !

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

ATS winning National Championship during clinical rotation with Villanova University!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

UW Oshkosh Students Winners of the 2013 National Quiz Bowl!

3/17/14  

9  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Example Outcome#1:!!Students will use knowledge of basic science and research methodology to interpret evidence-based research related to athletic training to answer clinical questions and to guide clinical practice.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Eastern U. Institutional Learning Outcomes !1. Students develop intellectual curiosity, passion, and agility, valuing the life of the

mind and life-long learning.!2. Students develop their critical thinking, reflection, analysis and communication

skills.!3. Students develop knowledge and competencies in the arts, sciences and

professions.!4. Students develop and expand their Christian worldview, grounded in the

Scriptures.!5. Students discern the ethical consequences of decisions and actions.!6. Students are motivated to assume responsibility for justice and to show a

transformative influence—especially regarding social, political and economic justice.!

7. Students are prepared to live in an interdependent world, aware of societal and global problems and committed to engage in solving them.!

8. Students increase in self-awareness and in their sensitivity towards others and others’ needs and situations.!

9. Students contribute to fostering an environment where diversity is appreciated and reconciliation is practiced.!

3/17/14  

10  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Developing an assessment plan:!

The Basics!1.  Institutional Mission Statement!2.  Department / Program Mission Statement!3.  Institutional Goals and Outcomes!4.  Department / Program Outcomes!5.  Measureable Objectives!6.  Assessment Strategies / Benchmarks!7.  Administrative Assessment of Results!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Writing Objectives!

•  Measurable, demonstrable components of a Student Learning Outcome!–  Specific knowledge, skills, and behaviors!–  Clinical and/or Didactic!

•  Taught developmentally in the curriculum!–  introduced, reinforced/practiced and ultimately applied/

integrated into the practice of the discipline in higher level courses!

•  Foundation for assessments at the course and department/program level.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Writing Objectives!

•  Bloom’s Taxonomy!–  Need varied levels!–  Many sources of “Verbs” at each level!

•  SMART format!–  Specific!–  Measurable!–  Attainable!–  Realistic!–  Time-Oriented!

3/17/14  

11  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Good Example!Outcome Example: Students will use knowledge of basic science and

research methodology to interpret evidence-based research related to athletic training to answer clinical questions and to guide clinical practice.!

!Objectives of Outcome Example:!1. !Successfully access and accurately interpret a variety of healthcare

related research data. [Classroom]!2. !Demonstrate contemporary skill in comprehensive care of patients with

injuries. [Clinical Education]!3. !Demonstrate contemporary skill in comprehensive care of patients with

illnesses. [Clinical Education] !4. !Develop patient-centered treatment plans that consider the impact of the

condition on the patient’s life (eg, socioeconomic, religion). [Clinical Education]!

5. !Incorporate patient-centered outcome measures to evaluate the quality of care provided. [Clinical Education]!

6. !Develop a comprehensive plan for continuing education. [Classroom]!!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Poor Example!

Outcome Example: Entry-level professional knowledge, comprehension, and proficient skills of an orthopedic clinical examination and diagnosis of the knee!

Objectives of Outcome Example:!!Taking a patient's history; inspection of knee along with observation of gait; palpation of bony and soft-tissues; joint motion assessment using a goniometer; muscle function assessment using manual muscle tests; circulatory assessment; neurological function assessment including sensory, motor, and reflex testing; selective tissue testing utilizing appropriate special tests; clinical diagnosis of common injury, condition, illness, and/or disease based on clinical examination; professional behaviors exhibited during the clinical examination.!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Laurent  &  Weidner  (2002)    

3/17/14  

12  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Good Example!Outcome Example: Students will use knowledge of basic science and

research methodology to interpret evidence-based research related to athletic training to answer clinical questions and to guide clinical practice.!

!Objectives of Outcome Example:!1. !Successfully access and accurately interpret a variety of healthcare

related research data. [Classroom]!2. !Demonstrate contemporary skill in comprehensive care of patients with

injuries. [Clinical Education]!3. !Demonstrate contemporary skill in comprehensive care of patients with

illnesses. [Clinical Education] !4. !Develop patient-centered treatment plans that consider the impact of the

condition on the patient’s life (eg, socioeconomic, religion). [Clinical Education]!

5. !Incorporate patient-centered outcome measures to evaluate the quality of care provided. [Clinical Education]!

6. !Develop a comprehensive plan for continuing education. [Classroom]!!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Developing an assessment plan:!

The Basics!1.  Institutional Mission Statement!2.  Department / Program Mission Statement!3.  Institutional Goals and Outcomes!4.  Department / Program Outcomes!5.  Measureable Objectives!6.  Assessment Strategies / Benchmarks!7.  Administrative Assessment of Results!

3/17/14  

13  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Assessment Strategies / Benchmarks!

•  Strategies may be used to cover multiple objectives!

•  Rubrics must be used to gauge learning!•  Assignments may be used but grades don’t tell

the story!–  Multiple assignments !–  Projects / Presentations!–  Practicum evaluations!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Outcome: Students will demonstrate post-graduation preparedness !

Objectives:!1.  Students will demonstrate their ability to

integrate themselves into a post graduate professional setting.!

2.  Students will be able to practice in a competent and safe manner.!

3.  Students will indicate they are prepared to work as a professional!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

What assessment strategies did you use to measure each objective:!q Alumni Survey!q Case Study!q Class Presentation!q Employer Survey!q Exam!q Exit Interview!q IPE!q Objective Structured Clinical Exam (OSCE)!q Practical Examination!q Preceptor Evaluation!q BOC Exam!

3/17/14  

14  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Assessment Strategies / Benchmarks!•  Determine % of students that earn at least a

minimally acceptable score!•  Compare against national averages or compare

to peer institutions!•  Base some off of CAATE’s standards!

–  BOC Pass rate!–  Job placement!–  Alumni Surveys!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Benchmarks!•  80% of students will score above a 70 on their senior

case study according to the faculty rubric.!

•  Which benchmarks were met / not met?!

•  “We met our benchmark of 80% - no further action needed at this time”!

• Problems??

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

3/17/14  

15  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Benchmarking 2.0!Outcome   Excep,onal   Very  Good   Adequate   Inadequate  

Communica>on   5%   85%   10%   0  

Cri>cal  Thinking   7%   70%   20%   3%  

Wri>ng   4%   60%   30%   6%  

Professionalism   15%   75%   10%   0  

Evidenced-­‐based  Clinical  Prac>ce   6%   40%   50%   4%  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Developing an assessment plan:!

The Basics!1.  Institutional Mission Statement!2.  Department / Program Mission Statement!3.  Institutional Goals and Outcomes!4.  Department / Program Outcomes!5.  Measureable Objectives!6.  Assessment Strategies / Benchmarks!7.  Administrative Assessment of Results!

3/17/14  

16  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Utilizing the results for change!•  Improve curriculum!•  Maximize learning!•  Tutoring!•  Library Services!•  Academic advisement and counseling!•  Technology !•  Co-curricular opportunities!•  Budgeting!•  Academic review!•  Improve relationships / learning at clinical sites!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

2.  OBJECTIVES  What  are  the  

measurable  pieces  that  make  up  our  

outcomes  

3.  ASSESSMENT  (10) Have  we  met  our  benchmarks  and  what  do  our  

students  look  like  

4.  ACTION  PLAN  (12) -­‐  Curricular  Changes  -­‐  Preceptor  Educa>on  -­‐  Modify  Tools  -­‐  Increase  Benchmarks  

1.  OUTCOMES What  we  want  our  graduates  to  look  

like

3/17/14  

17  

CLOSING REMARKS!

March| 2014!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Annual Report Section V: Outcomes!

34. Provide one outcome for your program:!35. Provide the objectives that you measured for this

outcome: (2-7)!36. What assessment strategies did you use to

measure each objective: !37. Provide the benchmark(s) you set to determine if

your students are competent based on your assessment strategy: !

38. Which benchmarks were met/not met: !39. What changes did you make in the program as a

result of the outcomes measurement: !

3/17/14  

18  

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Student Learning Outcome    One row for each SLO  

 

Objectives   (Expected Learning Outcome)  

 List all Objectives of the given SLO in same row (1.1,1.2,1.3).  

Assessment Strategy and Benchmark Note:    The  same  instrument/tool  (e.g.,  an  external  exam,  a  capstone  project,  or  other  cumula>ve  experience)  may  be  used  to  assess  mul>ple  objec>ves.  

Assessment Results    Summarize  the  results  found  for  each  objec>ve,  related  to  the  benchmark.  

Use of Results        Summarize  the  planned  program  changes  intended  to  improve  student  learning.      

   

     

   

           

T R A N S F O R M I N G T H E P R O F E S S I O N T H R O U G H Q U A L I T Y E D U C A T I O N!

Assessment Strategy & Indicators Measured →     Details about Assessment Strategy Methodology ↓  

Assessment Strategy #1:     Objectives Measured:     

Assessment Strategy #2:     Objectives Measured:     

Assessment Strategy #3:     Objectives Measured:     

Assessment Strategy #4:     Objectives Measured:     

Description of Strategy (e.g., test, rating scale, culminating assignment)  

               

Assessment Result yielded (e.g. rubric score, test score means)  

               

Benchmark (Criteria for Success)  

               

Sample Size and Source                  

Administrator                  

Time of Administration of Assessment Strategy  

               

Results maintained/archived where and by whom  

               

Time of Analysis of Results  

               

Analyzed by                  

Feedback to Faculty/ Discussion