VT University Libraries: Identifying, Teaching, and

Assessing What Matters Most

Office of Academic AssessmentRay Van Dyke, rvandyke@vt.eduSteve Culver, sculver@vt.edu 231-6003

November 13, 2008

Think about it…

What is the most important indicator in determining the

effectiveness of the library’s programs?

What is Assessment of Student Learning Outcomes?

“Assessment of student learning is the systematic gathering of information about student learning, using the time, resources, and expertise available, in order to improve the learning.” – Walvoord

“Assessment is a systematic way of paying attention to our curriculum.” – Nancy Metz

What is a student learning outcome?

A student learning outcome states a specific skill/ability, knowledge, or belief/attitude students are expected to achieve through a program.

Example: Students will be able to match information needs to information resources and organize an effective search strategy.

Other outcome examples

Students will be able to distinguish between catalogs, databases, indexes, and the internet as examples of the complex nature of the information environment.

Students will be able to conduct an advanced search on the WWW.

Setting the Context for Outcomes Assessment at Virginia Tech

Why are we talking so much about assessment of learning outcomes

right now?

Setting the Context

Federal Government -- Spellings Commission Report – “Institutions are asked, ‘Are you measuring student learning?’ And they check yes or no. That must change. Whether students are learning is not a yes-or-no question. It’s how? How much? And to what effect?” (Chronicle, 11-4-06)

SACS – The institution identifies Student Learning outcomes in its educational programs, assesses the extent to which it achieves these outcomes, and provides evidence of improvement based on analysis of the results. (SACS Standard 3.3.1.1)

Setting the Context

Professional Accrediting Agencies (Example: ABET)– “…describe what students are expected to know

and be able to do by the time of graduation, – …identify, collect, and prepare data to evaluate the

achievement of program outcomes, – …results in decisions and actions to improve the

program.” (ABET website) SCHEV

– “Ensure that the institution’s academic programs and course offerings maintain high academic standards by undertaking a continuous review and improvement of academic programs…” (Institutional Performance Standards – IPS)

Setting the context

• VT Strategic Plan -- “Establishing a culture of continuous improvement -- Through benchmarking, assessment, evaluation of academic and administrative goals, and a commitment to process improvement and change, Virginia Tech will achieve greater national and international prominence.”

• The Most Important Reason for Assessing student learning outcomes?

Most Important Reason for Assessing Student Learning Outcomes…

What is the most important indicator in determining the effectiveness of the libraries’ programs?

Could it be, should it be what students have learned through their experiences with the libraries’ programs?

Our focus then becomes our students’ learning and our teaching.

The goal then becomes to find direct evidence that shows us what our students have learned.

Thus, The process of learning outcomes assessment provides us with meaningful information for improving teaching and learning.

What is The Process for Assessing Student Learning Outcomes?

2. Gather and AnalyzeInformation About

StudentAchievementOf Outcomes

3. Use InformationGathered

To ImproveStudent Learning

1. IdentifyAnd Articulate

StudentLearning

Outcomes

Step 1: Identifying and Articulating Student Learning Outcomes

Owned by Library faculty What would an ideal graduate of VT know,

understand, and be able to do because of participation in our programs?

Identify a list of possible outcomes, knowing the list will change.

Narrow to 3 - 5 key outcomes to consider first. Understand that you can look at others next year or the year after…

Step 2: Gathering and Analyzing Information About Student Achievement of Outcomes

Direct and Indirect Measures Most effective way to assess: Commonly Used

Student Assessments That May Be Used for Program Assessment (Course-embedded)– Direct assessments such as Comprehensive Exams,

Internships, Essays, Portfolios, etc.– Capstone Courses– Aggregate data

ePortfolio as a tool for gathering data

Step 3: Using Information Gathered to Improve Learning

What recommendations would you make based on the data from the analysis of the assessment?– Alignment of student learning outcomes with course

content with assessment tool– Course changes

Introduction/emphasis/reinforcement of outcomes

Duplication/repetition– Faculty development about a topic– Rethink Learning outcomes

Develop and implement plans to improve

Indirect measures of student outcomes

Interviews with students (either exit as graduates or during as continuing student)

Alumni, current student, employer surveys Focus groups Percentage of students who go on to graduate school

(and other such numbers related to graduates) Job placement numbers Enrollment trends Magazine ratings Grades

Direct evidence measures of student outcomes

Student work samples (ideally embedded in course work)

Capstone assignments Search log Observation of student behaviors Performance on national licensure examinations Externally reviewed internship Standardized tests Collections of student work (portfolios, eportfolios*)

Key elements in measuring these outcomes

Need to provide high-quality, timely feedback that gives clear and meaningful directives for improvement

Clear criteria before measurement will guide expectations

Multiple measures of an outcome are ideal

Key questions to consider when choosing a measurement approach

How much money do I have? How much time do I have? Is the measure important or just interesting? What are my analysis capabilities? How easily can I fit this into my regular work? Who needs to make decisions with the data? How will I document the evidence and any

decisions made from the evidence?

Practical tips

Don’t have one person do all the work. Cooperate with other departments, even at different

schools. Borrow methods and instruments (if they fit). Use existing data if possible (IR, Registrar’s Office, etc.) Do as much as possible in the context of what you are

already doing. Make your instrument, assignment, observation as

short as it can be and still provide the information you want.

A data collection plan

From whom can we get this information? (sampling issues are part of this)

How will it be collected? When is the best time to collect the data? How will the data be evaluated? How can the data be used? How can the data be reported?

WEAVE Online & Evidence

WEAVEonline helps us provide evidence of our students’ achievement of student learning outcomes.

documents the continuous efforts to establish a culture of evidence.

Provides a concrete record of systematically “paying attention to our curriculum.” (Metz)

http://www.weaveonline.net/subscriber/vt/

In summary

Think of data collection as circumstantial evidence in a court case.

Though each source of data may have questions (reliability and validity) associated with it, the preponderance of evidence indicates that your students can do (or can’t do) what you say they can.

Sources

Hatfield, S. R. (1999). Department level assessment: Promoting continuous improvement. Idea paper #35. Manhattan, KS: Idea Center.

Hawthorne, J. (2004). General education outcomes scoring session report. http://www.und.nodak.edu/dept/oid/Documents/Gen%20Ed%20Outcomes%20Scoring.pdf

Links of Interest

Spellings Report– http://

www.ed.gov/about/bdscomm/list/hiedfuture/reports.html

Virginia Tech’s Scholar Site:– http://scholar.vt.edu

Home of OSP (Open Source Portfolio):– http://www.osportfolio.org/

Reflection…

“We are being pummeled by a deluge of data and unless we

create time and spaces in which to reflect, we will be left with only our reactions.”

--Rebecca Blood

Now for your area : A way to begin…

1. brainstorm at least three student learning outcomes for your area.

2. Take an inventory of measures already used in your program (in courses, at entry, at exit, etc.) that might relate to these outcomes.

3. Think of other measures you might use.