Agenda Introduction Benchmarks Benchmarking Survey data and benchmarking.
-
Upload
molly-ryan -
Category
Documents
-
view
237 -
download
4
Transcript of Agenda Introduction Benchmarks Benchmarking Survey data and benchmarking.
Agenda
Introduction
Benchmarks
Benchmarking
Survey data and benchmarking
Benchmarks
Benchmark:
Used to establish “industry standards” based on external and internal comparisons
Comparisons to similar institutions establish benchmarks
CCSSE measures what students are doing
Benchmarks
An example from the corporate world
An interview with Jeff Immelt, CEO of General Electric, identified Toyota, Dell, and Procter & Gamble as the three companies that GE has benchmarked the most
GE looks at Toyota and Dell to learn from their process excellence
GE looks at Procter & Gamble to learn from their marketing and commercial excellence
Benchmarks
Opportunities abound in educational research
Corporate benchmarking tends to be sharing between partners
Corporate benchmarking limited to a few competitors in contrast to education that has a much larger universe with which to compare
Are these approaches really that different?
Should institutions attempt to find similar or different partners for benchmarking?
Benchmarks
What are the most important characteristics of a benchmark?
Benchmarks
Elements Credibility—reliability and validity
Comparative—results examined relative to peers
Comprehensive—measures key elements according to the experts
Performance & Importance—what has the greatest impact
Confidentiality—it takes courage to assess oneself
Continuous—it takes time for improvement
-Joseph A. Pica, CEO of Educational Benchmarking Inc. in About Campus
Benchmarks
The CCSSE survey:
is administered directly to community college students at CCSSE member colleges in randomly selected classes.
is based on research, asking questions about institutional practices and student behaviors demonstrated to promote student learning and retention.
uses a sampling methodology that is consistent across all participating colleges.
Benchmarks
The five CCSSE benchmarks:
Active and Collaborative Learning
Student-Faculty Interaction
Academic Challenge
Support for Learners
Student Effort
Benchmarks
Active and Collaborative Learning:
Worked with other students on projects during class
Worked with classmates outside of class to prepare class assignments
Tutored or taught other students (paid or voluntary)
Participated in a community-based project as a part of a regular course
Made a class presentation
Asked questions in class or contributed to class discussions.
Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.)
Criterion Benchmarking
How do you determine which measure you use to compare yourself with other institutions?
Benchmarking
Normative Compare your college with the mean
CriterionCompare your college with a predetermined value
Benchmarking
Normative Benchmarks Provide contextDetermine what the mean you would like to be compared with is
Normative Benchmarks Situate Your ResultsWhat does it mean to have 80% of your students satisfied?
A good place to start, but not necessarily the end point
Benchmarking
Normative Benchmarking with CCSSE
•Look for Differences of 5 points (a standardized effect size of .2)
•Is .2 a noteworthy difference?
Criterion Benchmarking
Criterion Benchmarking with CCSSE
•What is the college mission?
•What are the college’s accreditation goals?
•Are all students equally engaged?
Benchmarking
Five ways to colleges can reach for excellence using CCSSE Benchmarks:
Compare themselves to national average
Compare themselves to high-performing colleges
Measure their performance against their least-engaged group
Gauge work in areas most strongly valued
Compare now to where they want to be
Benchmarking
Comparisons yourself with high-performers
Benchmarking
Measure performance against least-engaged group
Breakout by race, gender, enrollment status, parental education, traditional vs. non-traditional age
At risk students vs. other students
Define at-risk at your college
BenchmarkingDensity Curves of Student v. Institutions benchmarks (within institution v. between institution variation)
Benchmarking
Gauge work in areas most strongly valued
Focus On Current College Initiatives
Sharing thoughts about how to use CCSSE data to evaluate different programs
Examine institutional mission, vision, and values
Benchmarking
Understand values by sharing results
Share results with others to determine what is most strongly valued
Faculty, students, and administrators will likely have different opinions on what it is that accounts for particular phenomena
Hopefully, the questions that are created from benchmarks are more focused questions than the original question
Benchmarking
Compare now to where you want to be
Survey Data as Benchmarks
Should benchmarks derived from surveys be used to rank colleges?
Are there fundamental differences in how census benchmarks (e.g., graduation rates) and benchmarks derived from surveys should be used?
Survey Data as Benchmarks
Ewell’s distinguishes between ‘hard’ statistics and ‘second-order’ statistics
‘Hard’ statistics are clearly enumerated and based on census-type data, such as numbers of students, graduates, and degrees awarded
‘Second-order’ statistics measure phenomena that cannot be directly counted, such as student satisfaction and students’ self-assessments of their behavior, and as such contain some statistical instability
‘Hard’ statistics are preferable for performance funding because they are more statistically stable than ‘second-order’ statistics
Source: Ewell, P. T. (1999). Linking performance measures to resource allocation: Exploring unmapped terrain. Quality in Higher Education, 5 (3), 191-209.
Survey Data as Benchmarks
Texas Community Colleges Performance Measures:
1.The rate at which students completed courses attempted.
2. The number and types of degrees and certificates awarded.
3. The percentage of graduates who passed licensing exams related to the degree or certificate awarded, to the extent the information can be determined.
4. The number of students or graduates who transfer to or are admitted to a public university.
5. The passing rates for students required to be tested under the Section 51.306.
6. The percentage of students enrolled who are academically disadvantaged.
7. The percentage of students enrolled who are economically disadvantaged.
8. The racial and ethnic composition of the district’s student body.
9. The percentage of students contact hours taught by full-time faculty.
Source: http://www.thecb.state.tx.us/reports/DOC/1197.DOC
Survey Data as Benchmarks
Texas State-Level Benchmarks for Higher Education:
• Percent of recent high school graduates enrolled in a Texas public college or university
• Percent of first-time, full-time freshmen returning after one academic year
• Percent of first-time, full-time freshmen who graduate within four years
• Percent of first-time, full-time freshmen who graduate within six years
• Percent of two-year college students who transfer to four-year institutions
• Percent of two-year transfer students who graduate from four-year institutions
• Percent of population age 24 and older with vocational/technical certificate as highest level of educational attainment
• Percent of population age 24 and older with two-year college degree as highest level of educational attainment
Source: http://www.thecb.state.tx.us/reports/DOC/1197.DOC
Survey Data as Benchmarks
If performance funding is based on verifiable hard statistics, what role does survey data have in benchmarking?
Or, why should we concern ourselves with the student experience?
Input and outcome versus process measures
To achieve outcomes, we need to understand the process by which they are obtained
Survey Data as Benchmarks
“We can tell people almost anything about education except how well students are learning.”
Patrick M. Callan, President, National Center for Public Policy and Higher Education
Survey Data as Benchmarks
Input -> Process -> Outcome Model
Inputs include costs, numbers admitted, etc.
Outputs include graduation rates, retention, graduate satisfaction
How do we measure the Process component?
Survey Data as Benchmarks
Input and Ranking
Inputs are heavily emphasized in media rankings and potentially serve to maintain an establishment
Inputs and outputs are naturally correlated
The challenge for institutions is to maximize process to improve on the ability of inputs to predict outputs
Survey Data as Benchmarks
Input -> Process -> Outcome Model
Inputs include costs, numbers admitted, etc.
Outputs include graduation rates, retention, graduate satisfaction
How do we measure the Process component?
Survey Data as Benchmarks
Is there a danger of impacting results by raising the stakes?
Increasing the outcome without increasing the process
To achieve outcomes, we need to understand the process by which they are obtained
Increasing the outcome may not reflect improvement, but increasing the process won’t hurt
Survey Data as Benchmarks
CCSSE does not rank
There is not a single criteria or set of criteria that can be used universally
Institutional characteristics matter
Institutional missions differ
Benchmarking with CCSSE data is best when presented in a non-threatening manner
Improvement requires an understanding of the process
Understanding the process in an institution will require hearing different voices and different perspectives on the same issues
Survey Data as Benchmarks
Stability of CCSSE Benchmarks
Correlate 2005 and 2006 benchmarks for colleges that participated both years
45 institutions
55,903 students
Survey Data as Benchmarks
Variable
Academic Challenge
2006
Active & Collaborative
Learning 2005
Active & Collaborative
Learning 2006
Student Effort 2005
Student Effort 2006
Student-Faculty
Interaction 2005
Student-Faculty
Interaction 2006
Support for
Learners 2005
Support for
Learners 2006
Academic Challenge 2005 0.80 0.55 0.44 0.65 0.56 0.59 0.36 0.24 0.10
Academic Challenge 2006 0.49 0.58 0.56 0.67 0.57 0.53 0.34 0.21
Active & Collaborative Learning 2005 0.78 0.58 0.55 0.44 0.30 0.15 0.16
Active & Collaborative Learning 2006 0.47 0.61 0.49 0.61 0.28 0.27
Student Effort 2005 0.74 0.39 0.29 0.53 0.46
Student Effort 2006 0.44 0.42 0.49 0.48
Student-Faculty Interaction 2005 0.77 0.47 0.33
Student-Faculty Interaction 2006 0.52 0.41
Support for Learners 2005 0.90
Summary
Summary
Survey results are ‘second order’ data not ideal for performance funding and
Understanding ‘hard’ statistics naturally leads to a discussion of processes.
Survey data present an opportunity to understand processes and impact hard, outcome measures
As such, survey data presents opportunities for non-threatening discussions of