1 WASC Evaluator Workshop Spring Visits 2010. 2 Workshop Outcomes Know how to prepare for and...
-
Upload
gabriel-ortega -
Category
Documents
-
view
212 -
download
0
Transcript of 1 WASC Evaluator Workshop Spring Visits 2010. 2 Workshop Outcomes Know how to prepare for and...
1
WASC Evaluator WorkshopSpring Visits 2010
2
Workshop Outcomes
• Know how to prepare for and conduct an effective visit and produce a useful, high-quality team report
• Be prepared to make sound judgments about institutions under the Standards
• Be familiar with resources that support your work on a team
3
Agenda
• Context for the Visit/Accreditation
• Preparing for the Visit
• Conducting the Visit
• Developing Team Recommendations
• Writing the Team Report
4
Context for Accreditation and Visits
• The Continuing Evolution of the WASC Process and Standards
• The Accountability Movement– Retaining Peer Review
• The Impact of the Economy
• Value Added, Collaboration, and Ongoing Efforts to Refine and Improve
5
Recent Changes in the Institutional Review Process
and Standards • Implement 2009 changes to Institutional Review Process re:
Student Success, Program Review and EE Sustainability• Implement 2009 changes to CFRs• Clarify the scope of the CPR visit to review the “infrastructure”
for assessment of student learning• Examine Program Review and Program-Level Student
Learning in a systematic way• Allow teams more time together on visits
Tools: Table A (RB pg. 41); Table B (RB pg. 47)
6
THE THREE-STAGE REVIEW PROCESS
7
THE THREE-STAGE REVIEW PROCESS
1. Institutional Proposal2. Capacity and Preparatory Review3. Educational Effectiveness Review
• This applies to all institutions, regardless of where they are in the accreditation cycle.
• This is intended to be for a maximum of 10 years.
8
TIMELINE FOR THREE-STAGE REVIEW PROCESS
Submitted 2Years beforeCPR review
Capacity And Preparedness
Review
18-24months toprepare for
EER(or less for
Candidacy orInitial Accreditation)
EducationalEffectiveness
Review
Extendedperiod oftime tosustain
Initiatives(7-10 years)
Proposal
9
INSTITUTIONAL SELF-REVIEW
• The heart of accreditation
• Built upon an effective internal process of– Evaluation– Reflection– Recommendations– Plans for Action
10
OUTCOMES OF THE ACCREDITATION PROCESS FOR
THE INSTITUTION-1
• Validation of institution’s presentation of evidence of compliance and improvement
• Effective use of indicators of institutional performance and educational effectiveness
• Greater clarity about institution’s educational objectives
• Improvement of institution’s capacity for self-review and quality assurance
11
OUTCOMES OF THE ACCREDITATION PROCESS FOR
THE INSTITUTION-2
• Deeper understanding of student learning
• Use of assessment results to improve program and institutional practices
• Systematic engagement of the faculty on issues of assessing and improving teaching and learning processes
12
THE INSTITUTIONAL PROPOSAL
13
THE ROLE OF THE INSTITUTIONAL PROPOSAL
• Guides the entire accreditation review process
• Connects institution’s context and priorities with the Standards of Accreditation
• Provides primary basis for both institution self-review and team evaluation
• Allows alignment of accreditation activities to institutional strategic plan and key areas chosen for improvement
14
THE INSTITUTIONAL PROPOSAL-1
• Section 1– Sets the institutional context– Relates the proposal to the Standards– Describes how the proposal was developed– Describes extent to which the process
generated broad institutional support
15
THE INSTITUTIONAL PROPOSAL-2
• Section 2– Frames the review process to align the two
reviews– Describes a coherent vision and specific
outcomes for the entire review cycle as a single connected process
– Specifies what institution intends to accomplish and how the CPR and EER are aligned to achieve those outcomes, framed as results not as activities to be undertaken
16
THE INSTITUTIONAL PROPOSAL-3
• Section 3– Demonstrates a feasible work plan and
engagement of key constituencies– Includes:
• A work plan and milestones• The effectiveness of data gathering and analysis
systems• A commitment of resources to support the review
17
THE STRUCTURE OF THE INSTITUTIONAL PROPOSAL-4
• Section 4– A set of Appendices that include:
• Data exhibits• Information about off-campus and distance
education degree programs• Institutional stipulations signed by the Chief
Executive Officer
18
THE INSTITUTIONAL LETTER OF INTENT
• Submitted by institutions seeking Candidacy or Initial Accreditation, the LOI serves the same purpose as the proposal
• Includes suggestions from Eligibility Approval Letter
• Submitted to Assigned WASC Liaison, one year in advance of Capacity Review
• Instructions are in “How to Become Accredited” on WASC website
19
THE CAPACITY AND PREPARATORY REVIEW
20
PURPOSE OF CAPACITY AND PREPARATORY REVIEW
• Review and verify the information in the institutional presentation (report and data)
• Evaluate key institutional resources, structures, processes in light of Standards
• Evaluate institution’s infrastructure to support student and program learning
• Assess institution’s preparedness to undertake Educational Effectiveness Review
21
THE EDUCATIONAL EFFECTIVENESS REVIEW
22
PURPOSE OF EDUCATIONAL EFFECTIVENESS REVIEW
• Invite sustained engagement by the institution on the extent to which it fulfills its educational objectives
• Enable the Commission to make a judgment about extent to which institution fills its Core Commitment to Educational Effectiveness
23
THE TWO CORE COMMITMENTS
24
CORE COMMITMENT #1
“The institution functions with clear purposes, high levels of institutional integrity, fiscal stability, and organizational structures to fulfill its purposes.”
25
CORE COMMITMENT #2
“The institution evidences clear and appropriate educational objectives and design at the institutional and program level. The institution employs processes of review, including the collection and use of data, that ensure delivery of program and learner accomplishments at a level of performance appropriate for the degree or certificate awarded.”
26
THE FOUR STANDARDS
27
STANDARD 1:Defining Institutional Purposes and Ensuring Educational Objectives
• Institutional Purposes
• Integrity
28
STANDARD 2:Achieving Educational Objectives
Through Core Functions
• Teaching and Learning
• Scholarship and Creativity
• Support for Student Learning
29
STANDARD 3:Developing and Applying
Resources and Organizational Structures to Ensure Sustainability
• Faculty and Staff
• Fiscal, Physical, & Information Resources
• Organizational Structures & Decision Making Processes
30
STANDARD 4:Creating an Organization Committed to
Learning and Improvement
• Strategic Thinking and Planning
• Commitment to Learning and Improvement
32
INTERIM REPORTS AND SPECIAL VISITS
33
INTERIM REPORTS AND SPECIAL VISITS
• May or may not be connected to a sanction• Intended to monitor institutional issues identified
by Commission or to assess how institution will move into compliance
• Both focus on only a few specified areas of concern
• Interim Reports are reviewed by a panel of peer reviewers (IRC)
• Special Visit length and number of team members are determined by number, depth, and complexity of issues
34
Expectations for Two Reviews
Capacity and Preparatory
• Preparatory = readiness for the Educational Effectiveness Review
• Capacity = purposes, integrity, stability, resources, structures, policies, processes
Educational Effectiveness
• Demonstrating student learning
• Demonstrating institutional learning
• Demonstrating evidence-based decision-making
Tool: Expectations for Two Reviews (RB pg. 20)
35
The CPR and EER as a Whole
• The CPR evaluates what an institution has for infrastructure (staff/faculty, resources, processes, facilities, systems, structures).
• The EER evaluates how well that infrastructure works and the results that the institution achieves.
36
Navigating Multiple Purposes-1
Focusing on the institution Applying the Standards and CFRs
Focusing on Proposal themes/topics
Evaluating capacity and effectiveness under Standards; addressing team-identified issues
Reviewing the whole institution Focusing on specifics, e.g., distance education, samples of program reviews
Advancing institutional development
Addressing “compliance” matters
37
Navigating Multiple Purposes-2
Allowing flexibility and experimentation on visit
Ensuring consistency and fairness among visits and quality control of visits and reports
Using the CPR to evaluate EE readiness
Leaving evaluation of educational effectiveness until EER
Supporting institutional creativity and excitement
Reporting to the Commission and serving the public
38
Covering the Impact of the Financial Recession on Institutions
Questions to ask the institution:
• How has the financial recession affected your institution?• How has your institution responded?• What plans are in place in case the current state of affairs
becomes permanent?
39
Preparing for the Visit(Visit Guide, Part II, pp. 29-52)
40
Timeline For CPR/EER Reviews
12 weeks 2 months
Institution mails report to team and
WASC
Team holds conference
call
Site visit held and team report
written
Institution responds to
errors of fact in team report
Institution responds to final team
report
Commission acts at
February or June meeting
Tool: CPR or EER Timeline (VG, pg. 29)
41
Roles and Responsibilities of Team Members and Staff
• Role of Team Chair (RB pg. 193)
• Role of Team Assistant Chair (RB pg. 195)
• Role of assigned WASC staff liaison (VG pg. 7)
• Team assignments
42
Pre-visit Preparation
• Read all the documents from WASC– Standards, CFRs, policies, visit guide, rubrics– Background documents re: institution and purpose of
the visit, including Proposal and/or last action letter/team report
• Read the institutional report• Review the data portfolio and exhibits
– What to look for and highlight?
Tools: Timeline (VG pg. 8, VG pg. 29)
43
Reviewing the Exhibits
• Enrollment data– Headcounts and FTE
• Graduation data• Faculty data• Key financial indicators• Inventory of Educational Effectiveness Indicators• Inventory of Concurrent Accreditation and Key
Performance Indicators
Tool: How to Review WASC Data Exhibits (RB pg. 61)
44
Reading the Report
• Has the institution done what it said it would do in its Proposal?
• Has it collected and analyzed data effectively?• Are its conclusions supported by evidence?• Are there serious problems or potential areas
of noncompliance?• Does the report contain recommendations for
further institutional action?
45
Worksheet for Team Conference Call
• Organizes team’s responses to institutional materials
• Helps team make preliminary evaluation under the Standards
• Provides basis for team to work toward consensus
• Should be submitted in advance of call
Tool: Team Worksheet (VG pg. 42)
46
Team Conference Call
• Evaluates quality of institutional report and alignment with Proposal and previous action letter(s)
• Identifies areas of good practice, improvement, and further inquiry
• Identifies issues, strategies, evidence needed• Identifies persons and entities to be interviewed • Makes or refines team assignments• Plans visit logistics
47
Off-Campus Sites and Distance Education Programs
(special requirement for some visits)Prior to Visit: Sites will be identified and assignments made
• Review substantive change action letters to determine if issues have been identified
• Develop plan for the review of the programs and/or sites During Visit
• Interview faculty, administrators and students• Evaluate facilities OR online infrastructure• Observe classes• Document visit and findings in appendix • Discuss important findings with team for inclusion in report, as
appropriateTools: Protocols (RB pg. 157, RB pg. 162)
Forms (RB pg. 55, RB pg. 58)
48
Compliance Audit (special requirement for some visits)• Required for:
– Institutions seeking Candidacy and Initial Accreditation
– Some institutions under sanction• Additional report submitted by institution in
advance of the visit—with links to documents
Tool: Compliance Audit Checklist (RB, pg. 51)
49
Determining Strategy for CPR Visit
• What evidence is provided to show capacity and readiness for EE?
• Why was it chosen?• What are the strengths and weaknesses of the
evidence?• What other evidence do you want to review to evaluate
capacity and preparation for EE?• Do any issues arise with regard to the Standards?• Meetings: format/methodologies
50
Determining Strategy for EER Visit
• What evidence is provided to show EE?• Why was it chosen?• What are the strengths and weaknesses of the
evidence?• What other evidence do you want to see to evaluate
effectiveness?• Do any issues arise with regard to the Standards?• Meetings: format/methodologies
51
Drafting in Advance of the Visit
• Assistant Chairs draft outline of team report and Section I
• Team members draft outline or text for which they are responsible, using institution’s report and data portfolio, with space for additional evidence, analysis and conclusions
Tool: Team Reports (VG pg. 54)
52
Team Executive Session
• Discuss preliminary findings• Identify major issues for exploration• Refine lines of inquiry• Confirm team assignments• Discuss use of tools and rubrics• Review draft team report• Discuss options for confidential team recommendation• Review schedule
53
Conducting the Visit
54
Process of Visit
• Team meets at start of visit to confirm roles, assignments, logistics, and agenda
• Team meets frequently re: observations, emerging recommendations, and issues
• Team members complete draft sections of report and turn them in to Assistant Chair on the last day
• Team agrees on report recommendations and confidential recommendation to Commission
55
Visit Schedule
• Executive sessions and debriefings with team only
• Meetings and interviews with key individuals and groups
• Open meetings with students, faculty and staff• Document review • Time for drafting report sections• Final exit meeting
Tool: Sample Visit Schedule (VG, pg. 40; SV, pg. 68)
56
Confidential Email Account
• Set up by WASC as extension of open meetings• Checked by Assistant Chair during visit• Important emails shared with team and
investigated• Comments included in team report only if the
institution has a chance to address them
Tool: Sample Notification re: Confidential Email Account (RB, pg. 151)
57
Approaches Used on Visits
• Document review• Interviews and meetings
– Mini-questionnaires– Techniques for small and large meetings– Fishbowl exercises
• Audits
Plan visit methodologies in advance as part of schedule.
58
Document Review
DO as much as possible in advance:
Use to:• Check compliance• Evaluate the level of institutional engagement• Examine the evolution of a policy or process• Identify direct and indirect evidence of
student and organizational learning• Confirm report claims
59
Interviews
Use to:• Gather information• Explore issues• Build relationships with members of the
institution• Validate impressions and observations
60
Tips for Good Interviews
• Decide on a protocol for interview• Prepare questions and lines of inquiry in
advance • Ask questions that elicit information, stimulate
discussion, or require judgment• Avoid interrogation, leading questions, or
loaded language• Avoid consultation, giving solutions, or talking
about your institution• Let them do the talking
61
Alternative Forms of Interview
• Fishbowl• Brainstorm/free discussion on a salient topic• Go-round• Bundling• Audit
62
Evaluating Program Review and Student Learning
on EER Visits
Tool: Suggested Approaches for Evaluating Program Review (RB, pg. 156)
63
64
65
66
67
Rubrics re: Assessment of Student Learning
1. Academic Program Learning Outcomes
2. Use of Portfolios in Assessing Program Outcomes
3. Use of Capstones in Assessing Program Outcomes
4. Integration of Student Learning Assessment into Program Review
5. General Education Assessment Process
Tool: Rubrics (VG, pp. 77-88)
68
Educational Effectiveness Framework
• Use with team to evaluate institution’s “place” • Use language of rubric to describe the institution in
the report• Ask the institution to evaluate itself and discuss• Confer with team toward end of visit to mark a copy
of the EEF • Submit the marked EEF confidentially to WASC
Tool: Educational Effectiveness Framework (RB, pp. 160-161)
69
The Exit Meeting
• Team chair communicates commendations and key recommendations that will be included in report
• Chair may ask team members to participate• The meeting is not a dialog, discussion or
debate
70
Developing Team Recommendations
71
Two Kinds of Recommendations
• Team recommendations at the end of team report, delivered at the exit meeting
• Confidential Team Recommendation to the Commission for action
Tools: Commission Decisions on Institutions
(Visit Guide pg. 89; SV Guide pg. Append. F)
Commission and Team Decision Matrix (RB pg. 177)
72
Team Report Recommendations
Should be:• Overarching and important• Supported by evidence • Linked clearly to Standards and CFRs• Supported by text in the report
- Distinguish recommendations from suggestions and observations embedded in the report
Tool: Educational Effectiveness Framework (RB pg. 160)
73
Confidential Recommendation to Commission-1
• CPR– Proceed to EER or reschedule EER visit– Conduct a Special Visit (not preferred)– Add time to EER visit– Issue a notice of concern or impose
a sanction (rare)
Tool: Confidential Team Recommendation Form (VG pg. 71)
74
Confidential Recommendation to Commission-2
• EER– Grant Candidacy, Initial Accreditation or
Reaffirmation of Accreditation for specified term
– Sanction or Notice of Concern– Interim Report or Special Visit
Tool: Confidential Team Recommendation Form
(VG pg.71)
75
Confidential Recommendation to Commission-3
• Special Visit– Varies with status of institution – Next steps– Removal or continuation of sanction
(note two-year limit on sanctions)
Tool: Confidential Team Recommendation Forms (SV Guide, pg. 48)
76
Producing Effective Team Reports
77
Report Preparation Logistics
• Follow report template
• Start writing before the visit
• Complete your sections on site and give to Assistant Chair for editing together
Tool: Team Report Templates (VG, pg. 55; SV Guide, Appen. K)
78
Using Evidence in Team Reports
• Use qualitative and quantitative evidence • Select evidence carefully and purposefully• Connect evidence to an assertion or question • Analyze information; do not just set forth data• Let evidence suggest improvements• Use evidence that speaks to the institution’s
themes and the team's questions
79
Team Use of the Standards and CFRs
• Team judgments must be linked to specific Standards and CFRs
• CFRs must be cited in reports • Standards and CFRs form the basis for
Commission decisions• Standards and CFRs provide a context for
continuous quality improvement
80
What is an effective team report?
Reflects a thorough assessment of the institution’s capacity, preparation, and/or effectiveness
Is evidence basedCites the Standards and CFRsProvides the basis for a sound and supportable
Commission decision Identifies important areas for institution to address
81
New requirements in the Institutional Review Process
• Institutions will cover the following in their reports:– Student Success (at CPR and EER)– Program Review (at EER)– Sustainability of EE (at EER)
• Teams should address in the team report
Tool: Table B (RB pg. 47)
82
Tips for Writing Team Reports• Consider multiple audiences: institution,
Commission, and next team• Know your areas of responsibility, including length
and depth of your section• Start writing before you arrive on campus• Address priorities and goals set by the institution• Address Commission’s concerns (last action letter)• Make commendations, but don’t overdo it• Use praise that doesn’t send wrong or mixed signal
83
More Tips on Team Reports….
• Be sure to check facts
• Support findings and recommendations with evidence --and tie them to CFRs
• Ensure evidence is sound and valid
• Distinguish recommendations from suggestions or observations
• Use formal language and tone (e.g., not “we/they”)
• Don’t mention personnel by name
• Don’t prescribe solutions
84
After the Visit
85
What happens next?
• Assistant Chair prepares draft for Chair, team and staff review; changes as needed
• Chair sends to institution for corrections of fact
• Chair finalizes draft and submits to WASC• Chair sends Confidential Team
Recommendation and completed EEF to WASC
• WASC sends report to institution
86
Then…
• Staff prepares draft action letter, which is reviewed by team Chair
• Commission Panel reads report and documentation including institution’s written response, meets with institutional representatives at Commission meeting
• Panel makes recommendation to Commission, and Commission acts
• Staff finalizes draft action letter on behalf of Commission
87
Also after the visit….
• Team members send reimbursement forms to WASC within 30 days– Hotel arranged and paid directly by institution– Travel / food reimbursed – Rental car must be approved in advance by
WASC staff– Spouse or assistant costs not covered– See policy for more details
• Team members should not have any contact with the institution – About the visit OR– Consult with the institution for one year
88
The Team’s Impact
• Peer review is the foundation of accreditation. • The team report forms the basis for the
Commission action and its letter.• The team report and action letter inform the
work of the institution for years to come.• Why were you chosen for a team?
89
Resources for Teams
• Appendices of Visit Guide
• Team Materials and Institutional Report mailed 10-12 weeks in advance of visit
• WASC Website: www.wascsenior.org
• WASC Email Advisory (sent prior to visit)
• WASC Staff
90
Thank youfor your service to the
region
91
Announcements
The materials presented during this webinar and a recording of this session will be posted at: http://www.wascsenior.org/evatrainingSpring2010