Culture of Assessment CREATING A CULTURE OF ASSESSMENT Dean Kenneth B. Roberts, Ph.D. Heidi M....
-
Upload
adam-griffith -
Category
Documents
-
view
216 -
download
0
Transcript of Culture of Assessment CREATING A CULTURE OF ASSESSMENT Dean Kenneth B. Roberts, Ph.D. Heidi M....
Culture of AssessmentCulture of Assessment
CREATING A CULTURE OF ASSESSMENT
Dean Kenneth B. Roberts, Ph.D.Heidi M. Anderson, Ph.D.
Eleanora Bird, M.S.Jeff Cain, M.S.
Office of Education InnovationUniversity of Kentucky College of Pharmacy
Closing the Loop
Culture of AssessmentCulture of Assessment
2
Outline Literature “Creating a Culture of
Assessment” UKCOP Case Study
Created an office to handle the College assessment efforts
Faculty development and involvement Student involvement Documenting current assessment efforts
Future Directions at UKCOP
Culture of AssessmentCulture of Assessment
Creating a Culture
of
Assessment 3
Culture of AssessmentCulture of Assessment
4
Culture of Assessment1. Leadership must be ‘on board’
Mission, resources, use results for improvement
2. Involve all stakeholders Faculty Students Alumni
Culture of AssessmentCulture of Assessment
5
Culture of Assessment3. Make assessment relevant
Course reviews Surveys, rubrics
4. Focus on teaching and learning rather than assessment Focus conversations not on
assessment, but on helping students learn
Culture of AssessmentCulture of Assessment
6
Culture of Assessment5. Provide opportunities to learn about
assessment6. Minimize the burden of assessment on
faculty and students Provide centralized leadership,
coordination and support for assessment Provide resources Keep paperwork to a minimal
Culture of AssessmentCulture of Assessment
7
Culture of Assessment7. Start small, Set up clear expectations and be
flexible8. Celebrate and reward assessment efforts
Give faculty and staff written recognition Encourage leaders to commend publicly Provide special funding Honor faculty and staff who obtain external
grants that will strengthen assessment
Culture of AssessmentCulture of Assessment
8
Culture of Assessment9. Use assessment results appropriately
Keep faculty, students and staff informed
Don’t penalize faculty whose assessments results are less than positive
10. Don’t expect everyone to get on board
Culture of AssessmentCulture of Assessment
(UKCOP)
9
Culture of AssessmentCulture of Assessment
10
Vision Office of Education Innovation (OEI)
was developed to assist faculty Teaching Learning ASSESSMENT Technology Scholarship of Teaching
Today
Culture of AssessmentCulture of Assessment
11
Mission The mission of the “Office of Education
Innovation” (OEI) is to provide assistance to faculty in developing and evaluating the professional pharmacy and graduate programs within the University of Kentucky, College of Pharmacy (UKCOP).
Culture of AssessmentCulture of Assessment
12
Community Dean’s Leadership Team Office of Academic Affairs Office of Education Innovation (4 FTEs)
Assistant Dean Assessment Coordinator Information Technology Officer Administrative Assistant
Faculty Students Other Participants
TEAM
Culture of AssessmentCulture of Assessment
13www.mc.uky.edu/Pharmacy/oei
Culture of AssessmentCulture of Assessment
14
Who the office works with? The “Office of Education Innovation” (OEI)
works closely with: UKCOP Course Coordinators/Directors UKCOP Individual Faculty UKCOP Departments UKCOP Students External Constituents to College
Culture of AssessmentCulture of Assessment
15
OEI Activities and Services Curriculum
Assessment Instructional Technology Faculty Development (Enrichment) Scholarship of Teaching
Today
Culture of AssessmentCulture of Assessment
UKCOP Faculty Development: How, What and
Why of Assessment?
16
Culture of AssessmentCulture of Assessment
17
Why do assessment? Accountability
ACPE Accreditation University Review Curriculum committee requirements
Enhance student’s educational development Decision-making
Culture of AssessmentCulture of Assessment
18
Accreditation July 2000 accreditation standards require
assessment activities to be aligned with mission and goals: achievement indicators, formative and summative formats, outcomes assessment, and systematic assessment plan.
Culture of AssessmentCulture of Assessment
19
Assessment ProcessDriven by Mission
Used forImprovement Ongoing
Use Multiple Measures
Culture of AssessmentCulture of Assessment
20
What is the Assessment Process? “Systematic collection, review and use of
information about educational programs undertaken for the purpose of improving student learning and development”
Palumbo and Banta, 1999, p. 3
Culture of AssessmentCulture of Assessment
21
Levels of Assessment Institutional
Academic CURRICULUM COURSES Teaching-
learning Methods Assessment
Methods
Individual STUDENTS:
Learning Performance Attitudes Behaviors Skills Satisfaction
Culture of AssessmentCulture of Assessment
22
Levels of Assessment Institutional
UNIV REVIEW ACCREDITATION
Individual Faculty
SELF Teaching Portfolio
Peer Observation
Scholarly Productivity
Culture of AssessmentCulture of Assessment
23
What is Assessed? Student
Student Learning Outcomes
Performance Faculty
Contribution to teaching Scholarly productivity National visibility
Program Graduation rates Licensure pass/fail Curriculum Courses Teaching-Learning
Methods
Culture of AssessmentCulture of Assessment
24
Difference: Student Assessment vs. Program Assessment
Student Assessment is a teaching process designed to improve learning
Program Assessment is an institutional process designed to improve the program
Culture of AssessmentCulture of Assessment
25
Student Assessment Carefully define your student learning
outcomes If you don’t know what you’re looking for,
you will have a difficult time measuring it. Outcome statements should describe
student learning or behavior Outcome: “the result”
Culture of AssessmentCulture of Assessment
26
Student Assessment Start simple. Choose a limited number
of outcomes to measure at first. Begin modestly so that the data
collected can help to define and refresh accomplishments
Culture of AssessmentCulture of Assessment
Faculty Involvemen
t
27
Culture of AssessmentCulture of Assessment
28
Faculty Involvement Faculty learning about assessment:
Website developed to keep them abreast of activities
Faculty participation on committees: Assessment committee Curriculum committee
Culture of AssessmentCulture of Assessment
29
Faculty Involvement Self-study reports
Curricular purposes Curricular Outcomes Course Review (online submission) Course Evaluation (online)
Survey developmentPS Graduate Program
Requested peer evaluation of teaching
Culture of AssessmentCulture of Assessment
30
UKCOP General Outcomes
Foundation Knowledge Critical Thinking Information Management Communications Professionalism Social Interaction, Citizenship, Leadership Life-long Learning
Culture of AssessmentCulture of Assessment
31
UKCOP Professional Outcomes Patient Care
Medication Distribution Human Resources Management and
Operations Research Teaching
Culture of AssessmentCulture of Assessment
32
Faculty Involvement Focus on Teaching/Learning
Generate the appropriate assessment questions
Determine appropriate ways to measure outcomes
Decide on when to assess the students Prepare and utilize the reports
Culture of AssessmentCulture of Assessment
33
Faculty Involvement
Developing Questions to Ask: What do we want students to know, and do? How will we know they are learning what we
hope they will? What is the effect of our curricula, courses,
teaching, styles and educational tools? Did our changes bring about the effect we
desired?
Culture of AssessmentCulture of Assessment
Student Involvemen
t
34
Culture of AssessmentCulture of Assessment
35
Student Involvement Incoming orientation:
Explain the College’s assessment, rationale relating to accreditation standards, role of OEI
Review IRB protocol and informed consent Assessments throughout their 4 years Anonymous (code for tracking) Reports in aggregate fashion
Culture of AssessmentCulture of Assessment
36
Student Participation Curriculum Committee
Student Liaison Committee (SLC) meets 3x per semester
Assessment Committee Course
Performance Assessments Other
Accreditation Visit, Program Review
Culture of AssessmentCulture of Assessment
UKCOP Case Study:Documenting Current Effort
37
Culture of AssessmentCulture of Assessment
38
Documenting Current Efforts
Purpose Determine what assessments were already
taking place within the College
Develop an assessment inventory list and appropriate questions about student learning
Culture of AssessmentCulture of Assessment
39
Self-assessment of outcome abilities Attitudes towards peer/self assessment Performance records from courses CAPP results (integrated abilities) Student portfolios Yearly interviews Post-graduation surveys
Goal:
Develop a Complete Student Profile
Culture of AssessmentCulture of Assessment
40
Three Phases Phase 1: Needs Analysis
Phase 2: Recommendations and Plan Development
Phase 3: Align Assessment Plan, Outcome Indicators
Culture of AssessmentCulture of Assessment
41
Research Literature review was conducted to
determine types of assessment data that might be useful to gather for the program’s assessment functions Student records Program evaluations Student performance and competencies
Needs Analysis
Culture of AssessmentCulture of Assessment
42
Data Audit Class of 2002-2005: Student and educational
outcomes records Data obtained from various offices and files within
the college: Demographic Entry-exam scores Learning styles GPAs
Needs Analysis
Culture of AssessmentCulture of Assessment
43
Data Audit Data portfolios for several student cohorts
were examined Faculty Other sources:
Institutional Research Office Boards of Pharmacy State/local pharmacy associations
Needs Analysis
Culture of AssessmentCulture of Assessment
44
Data Audit Report
Culture of AssessmentCulture of Assessment
45
Success External IR officer requested to use the
college audit form as a model for the campus
Data are being collected from all audiences in a wide variety of venues
Some data are being disseminated in meaningful way
Data AuditNeeds Analysis
Culture of AssessmentCulture of Assessment
46
Lessons Learned from Data Audit
Amount & breadth of information was considerable
Gaps/Duplication
Some areas require more or different data
Culture of AssessmentCulture of Assessment
47
Lessons Learned from Data Audit
Length and frequency of surveys Faculty engagement Incorporate findings into College
assessment plan Reassess
Culture of AssessmentCulture of Assessment
48
Phase 2Recommendations about research
questions to assessment committee
I-E-O data management model used to classify student and program data (Astin, 1993)
I=inputs E=environments O=outcomes
Culture of AssessmentCulture of Assessment
49
Phase 2A. Content Analysis
B. Educational Impact
Which educational experiences/environments maximize student learning during the first year of pharmacy school?
Culture of AssessmentCulture of Assessment
50
Phase 2Ho: Structured Clinical Instructive Models
(SCIMs), prior to students participating in EPPE, statistically improve student critical thinking skills.
Ho: Videotaped counseling sessions statistically improve student communication skills during the first year of pharmacy school.
Culture of AssessmentCulture of Assessment
51
PHASE 3
Align and map curricular outcomes
Disseminate reports and use for improvement
Culture of AssessmentCulture of Assessment
Future Assessment Directions
52
Culture of AssessmentCulture of Assessment
53
Next Steps OEI reviewing literature Formulating appropriate research questions Prioritization and approval of research
questions by assessment committee Identifying appropriate measures Conducting the analysis Reporting and using the results for
improvement
Culture of AssessmentCulture of Assessment
54
Assessing Faculty Teaching
Classroom teaching Portfolio
Student advising Scholarship of
Teaching Publications Presentations
Supervision of clinical experiences
Supervision of graduate students
Consultation about teaching
Next Step:
Culture of AssessmentCulture of Assessment
Other Activities of the Office of
Education Innovation (OEI)
55
Culture of AssessmentCulture of Assessment
56
Instructional Technology Introduce and foster the use of
innovative technologies that enhance teaching, learning and assessment
Consultation and chair side assistance Remove obstacles
Culture of AssessmentCulture of Assessment
57
Assessment Software CourseEval
TrueOutcomes
Culture of AssessmentCulture of Assessment
58
Technology Smart classrooms Blackboard Audience Response System PDAs FLASH animations Assessment hardware/software
Culture of AssessmentCulture of Assessment
59
Faculty Enrichment Education Research Focus Group (2002)
Seminars Technology (primary focus) Teaching (Forthcoming)
Learning (Forthcoming)
Assessment (Forthcoming)
Website: primer materials
Culture of AssessmentCulture of Assessment
60
Scholarship of Teaching Publications Presentations
AACP Annual Meeting poster, presentation Other Presentations
Syllabus International Education Conference Assessment Institute
Culture of AssessmentCulture of Assessment
61
Summary Assessment evaluates the effectiveness of the
program in terms of measurable outcomes The plan consists of:
Student learning outcomes Measures of student achievement of the
outcomes (and objectives) Methods of collecting data Process for using and reporting the results Assessment of the process
Culture of AssessmentCulture of Assessment
62
Summary Fostering a culture of assessment:
Takes time and commitment Is challenging Involves various stakeholders at some
levels Requires resources Continuous
Culture of AssessmentCulture of Assessment
63
http://www.mc.uky.edu/Pharmacy/oei/faculty_assessment.asp
Click to edit Master title style
Click to edit Master text styles Second level
Third levelFourth level
Fifth level
04/18/23 1 64Culture of AssessmentCulture of AssessmentCulture of AssessmentCulture of Assessment
Questions ?
64