Using Institutional Effectiveness to Build a Culture of Performance Improvement 2007 SACS-COC Annual...

Using Institutional Effectiveness to Build a Culture of Performance

Improvement

2007 SACS-COC Annual Meeting

Department of Institutional Research and EffectivenessSt. Petersburg College

P.O. Box 13489 St. Petersburg, FL 33733

(727) 341-3059FAX (727) 341-5411

Revised version for the 2011 SACS Working Group

Using IE to Build a Culture of Performance Improvement

2007 SACS-COC Annual Meeting 2December 10, 2007 (Revised 04-27-2011)

Issues with IE

SACS’s Peer Review Research Project revealed that in the area of Institutional Effectiveness (IE) 62% of institutions in off-site reviews were

deemed non-compliant and 27% of institutions in on-site visits were

deemed non-compliant IE was the second most identified area

behind faculty qualifications



SPC Experience with IE

As part of the recent SACS reaccreditation process, SPC spent a considerable amount of time and effort documenting and detailing their IE processes

The result…SPC had “zero” compliance issues in the area of IE during the off-site reviews



SACS Requirements for IE

Core Requirement 2.5: Integrated Planning & Evaluation Processes Institutional Effectiveness “…ongoing, integrated, and institution-wide research-based

planning and evaluation processes…” Comprehensive Standard 3.3.1: Institutional Effectiveness

“…identifies expected outcomes…,assesses whether the extent to which it achieves these outcomes, and provides evidence of improvement…”

Comprehensive Standard 3.5.1: General Education Core College Level Competency “… identifies college-level general education competencies and

the extent to which graduates have attained them. within the general education core and provides evidence that graduates have attained those competencies.”



SACS Requirements for IE

Additional IE related requirements include: Comprehensive Standard 3.1.1: Institutional Mission Federal Requirement 4.1: Evaluating Success

Student Achievement Federal Requirement 4.2: Program Curriculum



IE Evidence

SACS Suggested Documentation for CR 2.5 Evidence of linkage of IE to institutional mission Institutional plans and budgets that demonstrate linkage of

assessment findings to planning at all levels Minutes of appropriate (IE related) unit, committee, task

force meetings… Documentation that relates to IE, such as budget

preparation instructions, minutes of budget presentation meetings, annual reports, annual assessment updates, IE reports

Samples of specific actions taken to improve the IE process and/or results from that process



Performance Improvement

From Compliance to Performance Improvement

Data Information ImprovementKnowledge

Improved OutcomesSPCReportsAssessment Data

CommunityCollege LeadershipProgram Directors

FacultyStudents



SPC’s Definition of IE

“At SPC, Institutional Effectiveness is the integrated, systematic, explicit, and documented processes of measuring performance against the SPC mission for purposes of continuous improvement of academic programs, administrative services, and educational services offered by the college.”

Operating ResultsBudget PlanningStrategic & Unit

PlanningEvaluations/ AssessmentsMission

“Closing the Loop”



IE Phases

IE Processes are yearly processes that consist of three distinct phases: Evaluation/assessment phase Strategic/budget planning phase Implementation/operation phase



Evaluation/Assessment Phase

Reflective look back at the prior year’s assessment and evaluation results. educational outcomes institutional surveys departmental-specific assessments

Individual assessments results are reviewed by the respective oversight group which in turn aggregates the information and presents recommendations to the President’s Cabinet.

These recommendations are evaluated and are routinely summarized in the form of the Institutional Objectives (IOs).



Strategic/Budget Planning Phase

Transition from assessment/evaluation results to planning for the upcoming operational year

Includes assessment-driven unit planning Unit managers state their intent for the upcoming year

in measurable objectives Unit managers ensure alignment with the institution’s

mission and goals, Strategic Directions and Institutional Objectives (SD-IOs)



Implementation/Operation Phase

Execution of the developed unit plans

Unit managers are required to provide periodic reports of their progress to ensure the fidelity and stability of these unit plans

Periodic reports include a “mid-year status report” and an “end-of-year final results report.” The end-of-year report includes assessment results in addition to the unit’s use of these results in improving their unit.



Individual IE Processes

Individual IE processes are designed to result in continuous improvement and comprise the following: a. Mission statement processb. Evaluation and assessment processes for academic

programs, student services, and administrative servicesc. Strategic Directions and Institutional Objectives (SD-

IOs) processd. Unit / budget planning processe. Collegewide operating budget development process f. Budget approval, implementation, and annual operating

results



Mission Statement

IE process begins and ends with the College Mission statement

College’s Mission is formally reviewed every five years by a committee representing faculty, administrators, students, community leaders, and members of the Board of Trustees (BOT)

Mission is also reviewed annually by President’s Cabinet before the new strategic planning cycle begins

a. Mission Statement Process



Assessing the Mission

1. On-going

2. On-going

3. Annual

1. Environmental Scans

2. Trends Analysis

3. WorkNet Surveys & Exit Interviews

Economic & Workforce Development

AS Degrees & PSAV Programs

WorkNet One-Stop Ops.

1. Annual

2. Annual

3. Annual

4. Annual

5. Annual

6. Annual

7. Annual

8. Annual

9. Annual

10. Annual

11. Annual

1. Entering Students Survey

2. Enrolled Students Survey

3. Graduating Students Survey

4. Employer Survey

5. Recent Alumni Survey

6. Budget. Planning, IR Survey

7. Technology Resources Allocation Survey

8. Other Department-specific Surveys

9. Administrative Services Oversight (ASOG) Reports

10. Student Services Oversight (SSO) Report

11. Education Support Services (ESS) Reports (Libraries, Learning Support Centers)

Partnerships / Services Student Services

Community Services

Other Institutions

1. Annual

2. Every 3-years / program

3. Annual


5. Annual

1. Education Oversight Group (EOG) Reports

2. Academic Program Assessment Reports (APARs)

3. Academic Program Viability Reviews (APVRs)

4. Comprehensive Academic Program Reviews (CAPRs)

5. End-of-year SD-IOs Assessment Report

Accessible Education BS, BAS, AA, AS Degrees

Technical Certificates

Applied Tech. Diplomas

Distance Education

Developmental Education

Continuing Education

Recurring CycleEvaluations / AssessmentsSPC Mission Components

1. On-going

2. On-going

3. Annual

1. Environmental Scans

2. Trends Analysis

3. WorkNet Surveys & Exit Interviews

Economic & Workforce Development

AS Degrees & PSAV Programs

WorkNet One-Stop Ops.

1. Annual

2. Annual

3. Annual

4. Annual

5. Annual

6. Annual

7. Annual

8. Annual

9. Annual

10. Annual

11. Annual

1. Entering Students Survey

2. Enrolled Students Survey

3. Graduating Students Survey

4. Employer Survey

5. Recent Alumni Survey

6. Budget. Planning, IR Survey

7. Technology Resources Allocation Survey

8. Other Department-specific Surveys

9. Administrative Services Oversight (ASOG) Reports

10. Student Services Oversight (SSO) Report

11. Education Support Services (ESS) Reports (Libraries, Learning Support Centers)

Partnerships / Services Student Services

Community Services

Other Institutions

1. Annual


3. Annual


5. Annual

1. Education Oversight Group (EOG) Reports

2. Academic Program Assessment Reports (APARs)

3. Academic Program Viability Reviews (APVRs)

4. Comprehensive Academic Program Reviews (CAPRs)

5. End-of-year SD-IOs Assessment Report

Accessible Education BS, BAS, AA, AS Degrees

Technical Certificates

Applied Tech. Diplomas

Distance Education

Developmental Education

Continuing Education

Recurring CycleEvaluations / AssessmentsSPC Mission Components

SPC Mission – Assessments - Closing the Loop

a. Mission Statement Process



Evaluation and Assessment

Evaluation and assessment processes at SPC are centered on mission-driven outcomes in the following program and service areas: (i) 26 Organizational units comprising 40 Academic programs

(lower division); (ii) 8 Colleges and Schools comprising 20 Majors (upper

division), and (iii) key administrative and educational support services.

Analysis of outcome results is on-going and captured in various assessment reports.

SPC evaluates all student services to ensure students are provided the best support possible to ensure student success.

b. Evaluation and Assessment Processes



Assessment Areas

Academic Outcomes General Education BAS/BS AAS/AS

Key Administrative and Educational Support Services Viability Importance Satisfaction




Assessment Focus

Continually evaluate assessment instruments and current processes

Integrate assessment with educational process Focus on fewer, yet meaningful improvements (action

items) Use multiple assessment methods Involve faculty in process Integrate timing of assessments

(3-year cycle)

Assessments

Reviews Surveys

Gen

Ed




Direct and Indirect Measures

Direct Measures Internally Developed General Education Externally Developed General Education (e.g., MAPP, iSkills) AAS/AS Academic Program Assessment Report (APAR) BAS/BS Program Assessment Comprehensive Academic Program Review (CAPR)

Indirect measures Entering Student Survey Enrolled Student Survey Graduating Student Survey Recent Alumni Survey Employer Survey Community College Survey of Student Engagement (CCSSE)




Three-year Assessment Cycle

Academic Program Viability Reviews

(APVRs) conducted yearly

APARs

CAPRs



21December 10, 2007 (Revised 04-27-2011)

Key Assessments

Academic Program Viability Review

(APVR)

Academic Program Assessment Review (3)

(APAR)

Comprehensive Academic Program Review (3) (CAPR)

Oversight Group Report

(Multiple Programs)

1) Program Description (1)

2) Number of Graduates – by year

3) Actual Enrollments – by last six semesters

4) SSH Productivity – by last six semesters

5) Program Profitability – RPI Index by year

1) Introduction w/ use of past results (1,2)

2) Major Learning Outcomes – program specific (1, 2, 3)

3) Assessment Methodology (3)

4) Criteria for success (3)

5) Summary of assessment findings

6) Discussion & analysis

7) Action plan & time table

8) Budgetary & planning implications

9) Follow-up report

1) Key APAR findings

2) Key APVR findings

3) Environmental scan of stakeholders – program specific (1)

4) Enrollment / Graduation trends

5) Employment / wages trends (1)

6) Discussion & analysis

7) Action plan & time table

8) Annual follow-up report

9) Cabinet Review & Approval

1) Outcomes Asmt. Sources (1)

2) Previous Years Plans & Results (1,2)

3) Current Year Asmt. Results (1,2)

4) Key Areas Requiring Improvements (not meeting criteria / expectations)

5) Recommended Action Plans

6) Cabinet Review & Acknowledgement

Notes: 1) SACS Compliance Requirement2) Florida DOE Recommendations3) BOT Rule 6Hx23-3.04 Requirement




Academic Program Assessment

Academic Program Assessment Reports (APARs) Prepared for the Gen Ed, A.S., and A.A.S. programs

according to the three-year cycle Include a summary of assessment findings and a

discussion and analysis of those assessment findings (including use of results, action plan items, and budgetary implications)

Process was initiated college-wide in 1999 Program data is gathered and analyzed on a

continuous basis




Academic Program Viability Review

Academic Program Viability Review (APVR) Published yearly Included measures:

Program Graduates Course Enrollment Student Semester Hour

(SSH) productivity Relative Profitability Index

(RPI-T) Economic Trend Data




Program Review Process

Comprehensive Academic Program Review (CAPR) is a summative evaluation that includes multiple measures such as: program-specific performance profitability measures academic outcomes, and stakeholder perceptions




Program Review Objectives

CAPR was designed to meet three objectives:

To provide a comprehensive report that summaries all elements of the program’s viability and productivity from a 360-degree perspective,

To provide comprehensive and relevant program specific information to key College stakeholders such as the President Cabinet members in order to make critical decisions regarding the continued sustainability of a program, and

To provide program leadership a vehicle to support and document actionable change for the purposes of performance improvement.




Ed Outcomes Online

To provide a medium for completing the draft educational assessment reports as well as establishing a repository for program specific information, SPC developed an Educational Assessment Web site (https://it.spcollege.edu/edoutcomes/)

College administration and instructional staff are provided access to “completed” assessment reports including the APAR (Program & Gen Ed) and the CAPR

Online access further encourages the use of assessment data as well as highlighting “best practices” across the college


https://it.spcollege.edu/edoutcomes/



College Access




Program Director Access




Student Survey Process

Data collection Increased student awareness of survey process and

accessibility Minimized use of class time to complete surveys Use multiple points of student contact Online survey submission and scanning of paper

forms Quality of survey forms

Professional “look and feel” of survey format Items designed to gather pertinent data Included online branching models to reduce survey

length




Strategic Directions

College’s SD-IOs are comprised of longer-term directional statements and shorter-term, annually measurable objectives.

Strategic Directions are: Updated on a three-year cycle and are based on

achieving the College’s mission. Developed to make improvements in learning

outcomes, quality, productivity, and other environmental-scan considerations, as well as inputs from IE Team, President’s Cabinet, faculty and student representatives, the BOT, and external/community stakeholders.

c. Strategic Directions and Institutional Objectives Process



Institutional Objectives

Institutional Objectives are Reviewed and updated annually, with key on-going

initiatives updated based on progress made during the prior year and carried forward, where appropriate.

Designed for the purpose of achieving the corresponding longer-term strategic direction.




SD-IO Alignment

Key ComponentsSPC Mission

Evaluation/ Assessments

2005-2006 Strategic PlanSD & IOs (87 Total; A few

selected below)

Operating / Results(See full details: 2005-06 Year-End Report)

Accessible EducationBS, BAS, AA, AS DegreesTechnical CertificatesApplied Tech. DiplomasDistance EducationDevelopmental EducationContinuing Education

1.Education Oversight Group (EOG) Reports2.Academic Program Assessment Reports (APARs)3.Academic Program Viability Reviews (APVRs)4.Comprehensive Academic Program Reviews (CAPRs)

A.2: Use educational outcome data to improve General Education OutcomesA.11: Incorporate service learning into Honors Program coursesI.4: Increase baccalaureate degree opportunities w/ targeted new programs

A.2: Gen. Ed. outcomes for all A.S. programs implementedA.11: Service learning now required for Honors studentsI.4: Int’l Business, O&P, Paralegal Studies added

Mission – Assessments - Closing the Loop Examples




Unit / Budget Planning

SPC’s unit/budget planning is centered on: more than 150 Unit / Budget Managers, development of departmental unit plans and budget

requests, and use of the national award-winning Strategic Planning

Online (SPOL) system. Unit / Budget Managers are urged and

encouraged to access, review, and make use of all available assessment outcome data (i.e., College’s IE website, oversight group reports, etc.)

d. Unit / Budget Planning Process




Budget guidelines are prepared by the Collegewide Budget Committee (CBC), which provides Information about the College’s expectations for State

appropriations, Collegewide budgetary outlook, and Salary package and other key strategic expectations.

Budget guidelines also provide details on how capital expenditure requests are to be entered into the SPOL system

SPOL system requires the unit / budget managers to select one of the SD-IOs for each of the unit plan objectives entered





Supporting tasks, expected outcomes, and related budget funding are also captured in the SPOL system

After all unit plans and budget requests are entered, they undergo management review in which refinements and adjustments may be made based on funding availability and overall College priorities

Approved Unit plans (by College President’s direct reports) are aggregated and submitted for further review and evaluation by the CBC




Collegewide Budget Committee

Each year, the President confirms the membership of a CBC (SVPs, VPs, provosts, and other key functional leaders)

Certain members of the committee are rotated on an annual basis for cross-training purposes.

The charge of the CBC includes the following: Develop budget guidelines, Review of all budget requests approved by department heads,

vice presidents, and senior vice presidents, Ensure that the SD-IOs are funded according to their priorities

and funding availability, and Develop the College’s Operating Budget.

e. Collegewide Operating Budget Development Process



Budget Approval

Operating Budget is approved by the President’s Cabinet and recommended to the BOT

Once approved by BOT, the budget is communicated and uploaded into the College’s central financial system

College financials and operating results are monitored on a monthly basis and reported quarterly to the BOT

f. Budget Approval, Implementation, and Annual Operating Results



Operating Performance and Results

Operating performance and results are centered on: Executing SD-IOs Supporting unit plans and quality improvement initiatives Adhering to the operating budget

Operating performance and results are accomplished through the following key steps: Implementing budgets and action plans Monitoring achievements and related circumstances Evaluating mid-year unit plan reviews Review end-of-year unit plan reports

f. Budget Approval, Implementation, and Annual Operating Results



IE Next Steps

Ultimate goal is to provide stakeholders ‘timely’, ‘relevant’, ‘accurate’, and ‘interpretable’ data through: Formatted (dashboard) style reports, and On-demand customizable reporting, with Valid, reliable, and standardized measures.



Links to Additional IE Materials

http://www.spcollege.edu/Central/IE/2007_SACS_Presentation.htm

http://www.spcollege.edu/Central/IE/2007_SACS_Presentation.htm


2007 SACS-COC Annual Meeting 41

Key Points

IE is more than assessment A lot has changed since 2007

IE is less centralized New planning system has been implemented Assessment activities have expanded

December 10, 2007 (Revised 04-27-2011)


2007 SACS-COC Annual Meeting 42

Next Steps

Need to capture/define the current processes

Need to ensure alignment among current processes

Need to incorporate the new Student Achievement Committee Structure

Finally, need to tell the story…

December 10, 2007 (Revised 04-27-2011)



Questions

Using Institutional Effectiveness to Build a Culture of Performance

Improvement

2007 SACS-COC Annual Meeting

Department of Institutional Research and EffectivenessSt. Petersburg College

P.O. Box 13489 St. Petersburg, FL 33733

(727) 341-3059FAX (727) 341-5411

Revised version for the 2011 SACS Working Group

Using Institutional Effectiveness to Build a Culture of Performance Improvement 2007 SACS-COC Annual...

Documents

Transcript of Using Institutional Effectiveness to Build a Culture of Performance Improvement 2007 SACS-COC Annual...