Assessing Student Attainment in the Undergrad. Major:
description
Transcript of Assessing Student Attainment in the Undergrad. Major:
Volkwein- Penn State
Assessing Student Attainment in the Undergrad. Major:
Examples of Discipline-Specific Strategies•Senior Thesis or Writing Project•Senior Research Project•Capstone Course with Assessment Embedded in the
Course•Comprehensive Examination or Proficiency Test•Student Portfolio, Performance, or Exhibit
-- with expert or peer assessment•Senior Essay followed by Focus Group Discussions
with Faculty•Student/Faculty Retreat for Collective Assessment•Internship or Field Work•Analysis of Historical, Archival, or Transcript Data•Alumni and Employer Surveys and Interviews
Volkwein- Penn State
Should I purchase an instrument or create my own??
If you develop your own instrument:
• can be less expensive (at least in dollars)• can address institutionally-specific issues• can be very time consuming• must attend to formatting, ease of completion
for respondents, printing, on-site scoring• may require pilot testing• can require extensive commitments for
reliability, validity checks or can jeopardize credibility
• Has less external credibility, can increase chances for criticism
Volkwein- Penn State
Literature Review
and Collection of Survey
Items
Item Reduction & Revisions by
Research Team
Pilot Test to Determine
Psychometric Properties &
Revise
Final Survey Items
Vet with Faculty, Employers, Alumni, and Administrators
AlumniFaculty
Development of Local Instruments
Seniors
Program Chairs
Employers
Volkwein- Penn State
Tips for Improving Response Rates
Explain the Importance of the Survey Explain How the Results will be Used Assure Confidentiality Thank Participants Make the Instrument Interesting Offer Incentives, if possible Offer to Share the Results Be sure the Survey is easy to understand and complete Offer multiple modes of response (web, paper, telephone)
Volkwein- Penn State
Should I purchase an instrument or create my own??
If you purchase an instrument:
Requires less time for design, printing, etc. Likely has established reliability and validity Often includes norms groups for inter-
institutional comparisons May include scoring and reporting services Likely to have external credibility, Increases
face validity May not fit your curriculum & goals Can be expensive in dollars
Volkwein- Penn State
Advantages of Standardized Tests
Validity Established through content specification expert panel reviews for fairness pilot testing
Minimal Staff Time Involved Comparative Data and User Norms Available Short Turn-around Time for Results Access to Professional Expertise
Why not use the GRE for assessment? GRE Exams have the following deficiencies. 1. GRE scores are relational and only answer the comparison question.
You don’t know what a 600 means in terms of student achievement because you don’t know how many questions were answered correctly and incorrectly.
2. The GRE uses the wrong comparison group. Instead of comparing each GRE score against all college graduates, student GRE scores are in relation to a graduate school-bound population.
3. There are no GRE subfield scores within each major field, so you cannot tell if student performance is congruent with the curriculum.
The ETS Major Field Exam was constructed in response to these weaknesses in the GRE. The Major Field Exam scores not only are normed on populations of graduating seniors, and not only indicate the number answered correct (non relational), but also report scores by subfield, thus providing useful information for analyzing the curriculum. I recommend that the faculty send off for a specimen set, take the exam, discuss and decide.
Volkwein- Penn State
Validity of Self-Reported Data
Self-reports vs. actual scores/tests in various studies shown to correlate (r = .50 to .90) with:
SAT & GRE Verbal and Quantitative scores National Teacher Examination scores Criterion-referenced achievement tests in
English, math, science, and social studies College and High School GPA Course material tests Behaviors/activities consistent with reported
gains
Volkwein- Penn State
When Self-Reports are Reasonable Proxies for Objective Measures
The information requested is known to the respondent Questions are phrased clearly and unambiguously A moderate-to-high degree of overlap in content exists
between the measures The instruments measure the same constructs, and
questions refer to recent activities Respondent believes questions merit a serious, thoughtful
response Answering won’t threaten, embarrass, or violate the
respondent’s privacy, nor encourage socially desirable responses
Self-reported gains/proficiencies are for groups (averaged), rather than for individuals
(Based on Kuh, 2005 and Pike, 1995)
Volkwein- Penn State
Strengths of Surveys
Great Flexibility and Universal Applicability
Ease of Construction Application to all Students Results do not Require Expert
Interpretation Relatively Inexpensive
Volkwein- Penn State
Weaknesses of Surveys
Better for Measuring Group Responses than for Individual Responses
Best if Supplemented by other information from Interviews & Focus Groups.
To Construct Good Surveys avoid vague items Construct multi-item scales Link to constructs in your model
or to educational goals Effective visual layout
Volkwein- Penn State
Value of Alumni Studies(Volkwein, 1990)
Alumni provide a ‘Janusian’ perspective - Internal and external experiences.
Alumni and employers have legitimacy with both internal and external stakeholders.
Internally, Alumni Studies can assess important outcomes and provide info for enhancing
curriculum, programs, and policy. Externally, Alumni studies can support accreditation,
accountability, recruitment, and fund raising. Such studies provide an opportunity for faculty and
administrative collaboration. Centralized collection, decentralized uses
Volkwein- Penn State
Outcomes Approach to Alumni Assessment
Assessment of alumni rests on the premise that institutional quality and effectiveness can be measured by examining:
Educational & Job satisfaction Income & Socio-economic Status Occupational Attainment Engagement in civic and political activities Values like Tolerance for diversity Collegiate experiences General Satisfaction with Institution Quality of Instruction received Preparedness for employment & Grad School Willingness to enroll again & Donate $
(Volkwein, 1990, 1998; Dellow & Romano, 2002)
Volkwein- Penn State
Selected Alumni Surveys
ACT Alumni Survey (2-year & 4-year) SUNY Alumni Outcomes Survey Appalachian Region Alumni Outcomes SUNY-Albany Alumni Survey HEDS Alumni Survey HERI Alumni Survey NCHEMS Alumni Assessment Survey Clemson University Alumni Survey Georgia Tech Alumni Survey Penn State Engineering Alumni Survey
Volkwein- Penn State
ACT Alumni Outcomes Survey Demographics/Background/Career (15) Employment History and Experiences (18) Educational Outcomes (22)
Critical Thinking & Problem Solving (4) Life-long Learning (2) Social/Moral Values/Ethics (3) Multi-Cultural/Diversity (4) Citizenship/Global Issues (2) Team Work (2) Communication (2) Overall (3)
Educational Experiences/Satisfaction (31) Activities and Organizations (11) Additional Questions (30 spaces) Space for Comments/Suggestions
Volkwein- Penn State
Penn State - CSHE Engineering Alumni Survey
Demographics/Background/Career (13) Undergraduate Engineering Outcomes/Experiences (72)
Technical Skills and Abilities (15) Professional Skills (23) Analytical/Thinking Skills (10) Nature of Engineering Courses (15) Diversity & Tolerance (9)
Additional Information (9) Satisfaction (1) Test performance (1) Degrees, field, and minor (4) Planned and Actual employment (3)
Surveys available at Surveys available at http://www.ed.psu.edu/cshe/abet/instruments.htmlhttp://www.ed.psu.edu/cshe/abet/instruments.html
Percent of Alumni That Would "Attend All Over Again" and Select Same Major and Same Career (N=1743)
Attend All Over Again
Select Same Major
Select Same Career
020406080100
121114
2620
34
1916
21
Probably & Definitely No
All Alumni Career Related to Major Career Not Related to Major
0 20 40 60 80 100
6667
63
5461
41
5862
52
Probably & Definitely Yes
Mean Response on a 5 point Scale
3.93.93.8
3.53.7*3.1
3.63.7*3.5
* = These mean responses are significantly higher than those in careers not related to the major.(p=<.01)
Perceived contribution of the College ExperienceTo Alumni Development as Undergraduates
Intellectual Growth
Personal Growth
Social Growth
Effective Preparation
Effective Preparation
020406080
44
912
1110
913
12
19
Percent responding 1 or 2 on a 5 point scale
Pre-2000
Post-2000
0 20 40 60 80100
68
70
6371
58
70
6251
5642
Percent responding 4 or 5 on a 5 point scale
* Means are significantly different
from pre-2000 (p=<.01).
for Graduate School
for Employment
Mean Response(Scale of 1 to 5)
3.84.2*
3.73.9*
3.63.9*
3.73.5*
3.63.3*
Volkwein- Penn State
Abilities and Skills Rated by Engineering Employers as Highly Important for Success
99%99%97%97%95%94%
92%90%85%73%70%
Communicate effectively*Problem solving*Apply math, science, and engineeringUse modern engineering toolsTeamwork*Understand professional and ethical
responsibilitiesDesign a system to meet needsLife-long learning*Design and conduct experimentsKnowledge of contemporary issuesEngineering In global and social contexts
Abilities and Skills Rated by Non-Engineering Alumni as Highly Important for Success
Function independently, without supervisionExercise personal responsibilityListen effectively*Exercise self-disciplineExercise problem-solving skills*Maintain openness to new ideas*Speak effectively*Evaluate and choose between alternative actions*Think analytically and logicallyAcquire new skill and knowledge on your own*Possess clear goalsCope with conflictUnderstand myselfWrite effectively*Lead and supervise tasks and groups of peopleFunction effectively as a member of a team*Learn how to learn*
96%96%93%91%91%90%90%89%89%87%84%82%80%80%78%75%75%
Consensus Liberal Education Outcomes Over the past ten years, the AAC&U appears to have forged a national consensus among business, government, and accreditation leaders about the liberal education outcomes that all undergraduates should possess, summarized in Our Students’ Best Work: A Framework for Accountability Worthy of Our Mission (2004):
•Strong analytical, communication, quantitative, and information skills.•Deep understanding or hands-on experience with the inquiry practices of disciplines that explore the natural, social, and cultural realms.•intercultural knowledge and collaborative problem-solving
skills.•A proactive sense of responsibility for individual, civic, and
social choices.•Habits of mind that foster integrative thinking and the ability to transfer skills and knowledge from one setting to another.
Volkwein- Penn State
The Value of a Model
•Encourages Clarity of Purpose
•Stands as a Road Map or Guide for Developing•Research Questions•Assessment Design•Data Collection•Hypothesis Testing•Statistical Analysis
•Serves as a Logic Chart (Otherwise Causality is not Clear)
•Assists Workload Conservation and Fog Dissipation (concentrates energy and attention, streamlines research design, reduces the amount of data collection, data storage, analysis, and reporting)
2. Five Questions:
1. Are you meeting your goals?2. Are you improving?3. Do you meet the standard?4. How do you compare?5. Are your efforts cost-effective?
4. Collect & Analyze Evaluation Evidence
3. Select Model, Methods, Measures
For Academic Effectiveness: Student Learning, Research & Scholarship
For Administrative Effectiveness:Planning & Resource Management
3. Select Methods & MeasuresFor Academic Effectiveness: Student Learning, Research & Scholarship
For Administrative Effectiveness:Strategic Planning & Resource Mgmt
1. Janusian Duality
Inspirational Purposes-for Internal Improvement
Pragmatic Purposes- for External Accountability
Volkwein Effectiveness Model
5. Communicate Results
Take Academic & Administrative Action
Improve & StrengthenPrograms
For Each Level --Institution
--Program --Individual
Volkwein- Penn State
Today’s Effectiveness Approach
Undergraduate Focus, Goal driven, Evidence based, Improvement oriented
Volkwein- Penn State
Assessment Resources
•Patrick Terenzini’s article, “Assessment with Open Eyes” (JHE, 1989).
•Middle States 2003 Handbook, Student learning assessment: Options and resources.
•Linda Suskie’s 2004 book, Assessing Student Learning: A Common Sense Guide
•Several books by Trudy Banta and her associates constitute helpful resources for campus assessment efforts, Assessment in practice: putting principles to work on college campuses (Banta, et al. 1996), Assessment essentials: planning, implementing, and improving assessment in higher education (Palomba & Banta 1999), and Building a scholarship of assessment (Banta et al., 2002).
•Additionally, there have been several constructive national attempts to develop guidelines and standards for good assessment practices. The most significant of these are the AAHE Nine Principles of Good Practice for Assessing Student Learning, and the recommendations of the AAC&U (from Our Students’ Best Work, 2004).