+ Academic Program Workshop for Creating an Assessment Plan Facilitated by Pamela Baker, PhD,...

+

Academic Program Workshop for Creating an Assessment Plan

Facilitated by Pamela Baker, PhD, Director, CET&L Laurah Turner, PhD, Assistant Director, CET&L

University of Cincinnati, Center for the Enhancement of Teaching & Learning


+Session Outcomes

Identify and explain to colleagues the purpose and process of assessment.

Identify and evaluate the components of an assessment plan.

Develop a framework for the ideal graduate of your program.

Draft strong program-level learning outcomes.

Understand alignment of outcomes at the program, course, and unit-level.

Conduct a preliminary audit of program-level assessments.

Draft a curriculum map for your program that identifies specific courses where program learning outcomes are addressed.

Develop a plan to gather & analyze assessment data to determine how well student learning matches expectations (with a view to identify action to be taken based on findings).


+Session Timeline

9:00-9:15 Introductions and Program Overview 9:15-9:30 Identify & Explain the Purpose of an Assessment

Plan 9:30-9:45 Identify & Evaluate the Components of an

Assessment Plan 9:45-10:05 Develop a framework for the Ideal Graduate 10:05-10:45 Identify Strong Program Outcomes 10:45-10:55 BREAK 10:55-12:00 Review Program Outcomes 12:00-12:30 LUNCH 12:30- 1:15 Conduct a Preliminary Audit of Program Level

Assessment 1:15-2:15 Draft a Curriculum Map 2:15:2:25 BREAK 2:25-3:25 Develop a plan to gather and analyze assessment

data (with a view to identify action to be taken based on findings).

3:25-3:30 Q & A


+Overview: Identify & Explain the Purpose & Process of Assessment What is the purpose of program assessment within the context of

Semester Conversion & Higher Ed?

What is assessment of a program? Program Effectiveness (e.g. the program will graduate 90% of

students, & 70% of those students will earn degrees within five years)

Program Outcomes (i.e. emphasis on demonstrating what students can do as a result of completing a program)

What is the common purpose of assessment? A tool to make meaningful improvements and appropriate

interventions A tool to ensure efficient allocation of resources A way to highlight strengths within a curriculum Reaccreditation


+Assessment, Continued

SHORTER TERM Ascertain whether what our students have learned is good enough

Identify assessment methods & measures Identify whether these assessment tools are assessing what it is

we expect our students to learn Identify whether required courses/experiences within the

curriculum align with the program level outcomes Determine whether our students are learning at the appropriate

scope, depth, and level

LONGER TERM Use long-term findings to make more significant curriculum revisions

(beyond simple fine-tuning of programs) Use assessment evidence to make informed decisions about your

program within the context of college and institutional priorities Identify external benchmarks to evaluate the learning of our students

Sharing evidence & best practicesAdapted from Linda Suskie, “Why Are We Assessing?,” Inside Higher Ed (October 26, 2010), adapted from her talk at the 2010 Assessment Institute. (Dr. Suskie serves as Vice President of the Middle States Commission on Higher Education.) & Susan Hatfield, “Assessing Your Program-Level Assessment Plan,” Idea Paper #45, The Idea Center.

+


Anatomy of an Assessment Plan

Program Outcomes that can be measured, assessed, or observed in some way

Curriculum/Program Map connections identified between courses and program-level outcomes as well as clear levels of learning at each level (emerging, developing, achieved)

Methods/Measured articulated, which include direct & indirect examples of student learning

Program Outcomes

Curriculum/Program

Map

Methods/Measures

Assessment Infrastructur

e

Findings

Use of Findings


+Anatomy of an Assessment Plan, Continued

Assessment Infrastructure comprised of program faculty charged with overseeing the execution of the assessment plan, with efforts linked to college and institutional assessment efforts & goals

Findings for several years explained, patterns and trends identified (data collected & reviewed on an annual basis)

Use of Findings discussed among faculty; learning outcomes and pedagogical strategies reviewed and revised based on assessment data


+ACTIVITY 1: Envisioning the Ideal Graduate

What does an “ideal” graduate of your program look like?

How will your graduates think and behave?

What are 3-4 important things your students will learn to do in your program?

What do students in your program need to learn to prepare them for a job?

What would you like your students to be doing consistently 5 years from now?

The Task: Brainstorm the “ideal” graduate of your program. Be sure to describe how you imagine your ideal graduates once they become professionals in their field.


+ACTIVITY 1: Continued

If these category listings are not applicable to your program, please insert alternative headings.


+Program Outcomes

Outcomes are achieved results or consequences of what was learned (i.e. evidence that learning took place).

Outcomes should be framed in terms of the program and not individual courses or students.

Outcomes are student-centered and describe what a student should be able to do or understand upon graduation.

Outcomes should describe the abilities, knowledge, values and attitudes expected of students after completion of the program.

Outcomes must be stated in such a way that they can be measured or observed by more than one assessment method.

+


Program Outcomes

Course Outcomes

Unit Outcomes

Class Outcomes

How Program Outcomes Fit Within a Curriculum

Represent broad statements incorporating areas of inter-related knowledge and skills developed throughout a program, across courses and experiences.

Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the course. Course outcomes should align with the program outcomes.

Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the unit. Unit outcomes should align with the course outcomes.

Represent the knowledge, skills, abilities, or attitudes that students should achieve by the end of an individual class session. Class outcomes should align with the unit outcomes.

+


Course Outcomes


Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the course. Course outcomes should align with the program outcomes.

+

Unit Outcomes

Class Outcomes


Represent the measurable and observable knowledge, skills, abilities, or attitudes students should have by the end of the unit. Unit outcomes should align with the course outcomes.

Represent the knowledge, skills, abilities, or attitudes that students should achieve by the end of an individual class session. Class outcomes should align with the unit outcomes.



+Unit & Class-Level Outcomes

Move from content covered to student action

Consider how in-class activities can model or reinforce learning outcomes

Make course calendar more explicitly supportive of learning outcomes

+


Program Outcomes


Represent broad statements incorporating areas of inter-related knowledge and skills developed throughout a program, across courses and experiences.


+What are Program Outcomes?

Program outcomes represent broad statements that incorporate many areas of inter-related knowledge and skills developed over the duration of the program through a wide range of courses and experiences.

They represent the big picture, describe broad aspects of behavior, and encompass multiple learning experiences.


+Program Outcomes…

are achieved results or consequences of what was learned (i.e. evidence that learning took place)

should be framed in terms of the program and not individual courses or students

are student-centered and describe what a student should be able to do or understand upon graduation

should describe the abilities, knowledge, values and attitudes expected of students after completion of the program

must be measurable or observable by more than one assessment method

+

Universtiy of Cincinnati, Center for the Enhancement of Teaching & Learning

Bloom’s Taxonomy Action Verbs


+Program Outcomes, Examples: Undergraduate

Examples: Too generic: Students completing the Engineering program

will be practiced in design skills. Specific/Assessable: Engineering graduates will

demonstrate knowledge of math, science, and engineering fundamentals. Specifically, the student will have the ability to:• Demonstrate general design principles.• Use fundamental engineering techniques, skills and tools

for engineering practice.• Analyze and interpret data to produce meaningful

conclusions and recommendations.


+

Bachelor Degree in Business (BBA) Upon completion of the BBA Program, students will be able to:

1. Apply functional and cross-functional knowledge to critically assess business problems

2. Use analyses to inform and develop integrative solutions that improve business outcomes

3. Express ideas clearly, logically and persuasively in both oral and written formats

4. Recognize ethical and social responsibility issues in a business environment and know how to apply a process of ethical inquiry

5. Show how operating in a global market creates business opportunities and challenges

Program Outcomes: Undergraduate


+Program Outcomes: Undergraduate

Bachelor Degree in Health Science (HLSC):

Upon completion of the HLSC Program, students will be able to:

1. Plan, execute, evaluate, adjust physical activities and programs appropriate for self and clients

2. Provide complete, understandable and accurate information within content areas to clients and community

3. Develop and/or disseminate scientific information to the general public and the academic community

4. Apply information from various basic and applied science disciplines in a manner that provides for efficient, effective and safe physical activities


+Program Level Outcomes Categories for Graduate Level

Need to be progressively more advanced in academic content

Need to address discipline knowledge and research or professional practice

Need to address technology skill development (if appropriate)

Need to be informed & aligned with professional organization standards, licensing bodies, or specialized accreditors

Professional development of graduate students


+Example Student Learning Outcomes at the Graduate LevelBEFORE:

Develop knowledge across the subdisciplines of physics beyond that expected for a baccalaureate degree in physics.

Develop expertise in an area of physics that is appropriate to the doctoral level.

Develop and demonstrate the ability to carry through a substantial research project, with independent intellectual contributions to the conduct and direction of the research.


+Example Student Learning Outcomes at the Graduate LevelAFTER:

Upon completion of the physics PhD program, students will be able to: identify and describe the broad fundamental concepts of physics; analyze physics problems using fundamental physics concepts and

appropriate mathematical models; solve physics problems using analytic, computational, observational, or

experimental methods; present clearly organized oral explanations of physics topics at a level

appropriate to the audience at hand; explain and analyze at an expert level the concepts and the state of

current research in a subfield of physics; compose and defend written presentations of physics research in a

format and at a level appropriate for journal publication; conceptualize, design, and implement new approaches for solving

research problems in physics, including analytic, computational, observational, and experimental methods, as appropriate.


+Example Student Learning Outcomes at the Graduate Level Professional Development

Ethics


+Activity 2: Define Student Learning Outcomes

Focused on the learner, rather than on the content covered

Attainable, rather than aspirational

Describe observable and demonstrable skills and knowledge

Reflect how the learner will use course content now and in the future

Indicate specific and measurable elements that will be assessed using multiple assessment methods

Identify in the curriculum where the PO will be introduced, developed and achieved


+ACTIVITY 2b: Peer Review of Program Outcomes

Gallery Walk


+Program Outcomes and Your Program Assessment Plan PO’s are the foundation of an assessment plan

Program Outcomes

Course Outcomes

Unit Outcomes

Class Outcomes

Program Outcomes

Curriculum/Program

Map

Methods/Measures

Assessment Infrastructur

e

Findings

Use of Findings

The Process of

Assessment


+Activity 3: Curriculum Mapping Matrix

To identify where key learning outcomes are addressed in the curriculum

To identify whether courses are aligned with the curriculum

Specifically, to determine/identify whether course-level student learning outcomes align with program-level outcomes

Note: You’ve already completed the process of mapping course-specific student learning outcomes to program outcomes, when you completed the P-1.


+Activity 3: ContinuedInstructions

List the program learning outcomes identified in the P-1 in the column labeled “Program Learning Outcomes.”

List the required courses and experiences identified in the P-1 across the top row.

NOTE: Do not list courses or experiences controlled by external academic units. Only list courses and experiences that

are controlled by your program. If you identify a program outcome, you must be able to assess it.

Identify which course(s) and/or experiences utilize each program learning outcome and at what level the program learning outcome is addressed (e.g. emerging, developing, achieved)


Curriculum Mapping Matrix: Linking Program Outcomes to Curriculum

Key Required Courses and Experiences Identified in P-1

E: EmergingD: DevelopingA: Achieved

A101 A202

A502 …. … …

OUTCOMES

1 Outcome 1 E D A

2 Outcome 2 E D A

3 Outcome 3 EDA

4

5

6

7

8

9


+Identify & Describe Corresponding Program Outcomes & Assessment Methods: Conduct a Preliminary Audit

Circle any of the measures used in your academic program

+Why Not Just Use Course Grades?

+Why aren’t course grades adequate indicators for program assessment?

Course letter grades are insufficient for program assessment because: Letter grades are nominal values reflecting a summary of a

student’s performance. Grades are not an exact measurement of what students

have and have not learned. Grades do not measure achievement of program outcomes,

learning of specific skills, or the capacity of the curriculum to achieve program outcomes.

+Example: Outcomes for an Undergraduate Economics Program

Students should be able to identify, explain, and use economic concepts, theories, models; and data-analytic techniques.

Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.

Students should use investigative skills necessary for conducting original economic research and participating effectively in project teams.

Students should be able to apply their economic tools to formulate positions on a wide range of social and economic problems and engage effectively in policy debates.

+Example Curriculum Map for an Undergraduate Economics Program

E101

E202

E303

E404

1. Students should be able to identify, explain, and use economic concepts, theories, models; and data-analytic techniques

E D A

2. Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.

E D A

3. Students should use investigative skills necessary for conducting original economic research and participating effectively in project teams.

E D A

4. Students should be able to apply their economic tools to formulate positions on a wide range of social and economic problems and engage effectively in policy debates

E D D A

+Letter grades are nominal values reflecting a summary of a student’s performance

Measurements of Learning in E303

Grade Points Possible

Exam 78 100

Assignment 18 20

Quiz 10 50

Oral Presentation 98 100

Exam 88 100

Attendance 10 20

Total 302 390

Course Grade: 77% C

+PO 1. Students should be able to identify, explain, and use economic concepts, theories, models; and data-analytic techniques


Grade Points Possible

PO 1 PO 2

Exam 70 100

Assignment 15 20

Quiz 10 50

Oral Presentation 98 100

Exam 82 100

Attendance 10 20

Total 285 390 167/220

98/100

Course Grade: 73% C 76% 98%

PO 2. Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.

+PO 2: Students should be able to deliver effective presentations in which they combine visual communication design with oral arguments and/or the written word.

The oral presentation assignment is worth 25% of the course grade.

It is possible for one student to get an exceptional grade on that assignment, and another to fail, with both students receiving a “C” for the final course grade.

The OVERALL course grade does not measure: students’ demonstrated

oral presentation skills the capacity of the

curriculum to achieve PO2

Measurements of Learning

in E303

Student 1

Student 2

Points Possible

Exam 70 70 100

Assignment 15 12 20

Quiz 10 40 50

Oral Presentation

98 50 100

Exam 82 98 100

Attendance 10 15 20

Total 285 285 390

Course Grade:

73% 73% C


Course letter grades are insufficient for program assessment because: Grades reflect the evaluation practices, policies, and

criteria of individual instructors. Faculty teaching the same course may teach different

material. Faculty teaching the same course may emphasize different

course outcomes.

+Grades reflect the evaluation practices, policies, and criteria of individual instructors


Points Possible

Exam 100

Assignment 20

Quiz 50

Oral Presentation 100

Exam 100

Participation 20

Total 390


Points Possible

Exam 100

Assignment 20

Quiz 50

Oral Presentation 100

Exam 100

Attendance 20

Participation 20

Quiz 50

Total 460

Instructor 1 Instructor 2

Difference of 70 points between two sections of the

same course!


Course letter grades are insufficient for program assessment because: Letter grades are nominal values reflecting a summary of a

student’s performance. Grades are not an exact measurement of what students have

and have not learned. Grades do not measure achievement Program outcomes,

learning of specific skills or the capacity of the curriculum to achieve PO2

Grades reflect the evaluation practices, policies, and criteria of individual instructors.

Faculty teaching the same course may teach different material. Faculty teaching the same course may emphasize different

course outcomes.

+What CAN Be Used for Assessment?

Specific grades that reflect a specific program outcome An exam Assignment Oral presentation

Then the corresponding scores can be used to represent students' learning achievement on the program outcome, which can serve as data for program assessment.

+Conditions for Using Course Products as Assessment Measures:1. All sections of the required course must produce the course

product (e.g. an oral presentation)

2. Faculty must agree on product structure (e.g. directions, requirements)

3. Faculty must agree on how the products should be evaluated: Minimum performance standards Rubric

Sect

ion

1

Sect

ion

2

Sect

ion

3

Sect

ion

4

Sect

ion

5

Sect

ion

6

Sect

ion

7

Sect

ion

80

40

8077 80 71

8869 75 85 77

Average Grade on Standard Oral Presentation

Average Program Score: 78/100

+Final Message

Course products (cases, papers, presentations, exercises) may be used as program assessment measures, but course grades, by

themselves, cannot.


+Activity 4: Conduct a Preliminary Audit of Assessment Methods Utilized in Your Program

Using the “Program Assessment Inventory” (also available on the CET&L Website), read through the assessment measures and their associated descriptions. Check off any of the measures currently used in your academic program.


+Activity 5: Complete the Assessment Measures Alignment Matrix


+Example Assessment Measures Alignment Matrix


+Activity 5: Complete the Assessment Measures Alignment Matrix1. List your revised program learning outcomes in Assessment

Measures Alignment Matrix on the following page in the column labeled “Program Learning Outcomes”.

2. Using the Program assessment measures identified in the previous step, match the assessment tool to the Program outcome (please note that that same assessment may be used for multiple program outcomes).

3. Enter these in the table on the following page under the column labeled “Assessment Methods” and next to the program outcome to which they apply.

4. For each assessment measure listed describe in the adjacent columns where (what courses), when/how often the data will be collected (timeline), and who is responsible for collecting, reviewing and making recommendations about the data.


+Activity 6: Developing Your Assessment Infrastructure

Assessment Personnel Who will chair/serve on the assessment committee? What is the responsibility of the committee?

Collecting data, submission process and form Review of data (recommended annually) Recommendation of findings and decimation/discussion of findings/recommendations Collection of course syllabus/discussion with individual faculty regarding required courses in

the curriculum Recognition of service-load

Assessment Schedule How often will data be collected When will faculty be reminded that data will be collected (recommended beginning of term) Remind faculty to submit data at end of term Schedule of data compilation, summary and review by PC Timeline for when these findings with be shared & discussed with department faculty Follow-up discussion with faculty teaching required courses and/or overseeing required

experiences Timeline for improvement strategies identified (both short and long-term)


+Looking Forward: An Example Implementation of an Assessment Plan


+Homework: Revise your plan to gather, analyze, and interpret assessment data

Complete the “Academic Program Assessment Template,” using the following resources: Program outcomes Curriculum mapping matrix Assessment Measures Alignment Matrix Developing your assessment infrastructure


+Tomorrow

Activity 7: Peer evaluation of drafted program assessment plan

+ Academic Program Workshop for Creating an Assessment Plan Facilitated by Pamela Baker, PhD,...

Documents

Transcript of + Academic Program Workshop for Creating an Assessment Plan Facilitated by Pamela Baker, PhD,...