Assessment of Student Learning From Reference

Service

Jill Gremmels and Karen Lehmann

Wartburg CollegeWaverly, Iowa

RUSA Reference Research Forum 2005 Chicago, June 25, 2005

Survey of the Literature on Reference Evaluation

Overview by Green & Peach (2003) Reference as a teaching & learning activity

Review essays: Bunge (1994), Smith (1991), Campbell (2000)

Traditional techniques: Tally questions, such as length, time, date,

etc. User interviews (Mendelsohn, 1997) Focus groups (Massey-Burzio, 1998) Survey/Observation (Norlin, 2000)

Category 1: Reference Evaluation Literature

“55 percent school” Hernon & McClure (1986) Reference service evaluated on accuracy of

responses to predetermined questions. Questions come from “unobtrusive” people

posing as patrons. Critics: Reference is more than right and

wrong answers. Critics: If reference is communication,

delivery can be as important as answers.

Category 2: Reference Evaluation Literature

Interpersonal communication Durrance (1989; 1995); Jardine (1995) More complex model. Focuses on willingness of patrons to return

to the same staff member for future needs. Critics: Wrong answers are still wrong, even

from a librarian you liked.

Category 3: Reference Evaluation Literature

User satisfaction + conditions of the reference transaction Bunge; Murfin (1995); Stalker & Murfin (1995) Based on Wisconsin-Ohio Reference

Evaluation Program. 2 part evaluation form: patron and librarian. Allows variables (# of resources used, is

library busy, subject area of inquiry, training of librarian).

Externally validated assessment instrument (can match to other libraries).

Shortcomings of all 3

Correctness: Don’t reflect reality of academic reference

work. Performance:

Approaches don’t offer proof of teaching activity of librarians.

Assessment: Don’t connect to learning outcomes.

Assessment Applied to Learning

Outcomes Assessment is: Knowing what you are doing. Knowing why you are doing it. Knowing what students are learning as a

result. Changing because of the information.

Debra Gilchrist, Pierce Colleges, Lakewood and Puyallup, WA

History of Assessment in Higher Education (Ewell)

AAHE Assessment Conferences ran 1985-2004.

Assessment is still not a “culture of use” (Ewell, 2002), embedded into lives and work.

Dueling agendas: institutional change vs. “accountability for results.”

Summative or formative? Basic (and still unanswered) question:

Is there any real evidence that assessment actually promotes better learning?

Tying Academic Libraries into Campus Assessment

Quantitative vs. qualitative By Services:

Adams (1996); Wallace (2001) By Learning:

Iannuzzi (1999); Breivik (1998); Boyer Commission Report

Gremmels & Ruediger (2003); Lopez (2002)

If information literacy library culture is strong, it will influence campus culture.

Building Foundations for Assessment

Reference Course-Related Credit Courses

Information Literacy Outcomes

Information Literacy Definition

Mission Debra Gilchrist

Pierce College Philosophy

Reference Mission Statement

Vogel Library’s mission is to educate information-literate lifelong learners. We strive to make each reference encounter an educational experience that reinforces information literacy concepts by building upon prior instruction and giving further opportunities for guided practice.

The Teaching Role of Reference: Can it be Assessed?

Elmborg (2002) Librarians need pedagogy for the reference

desk. This implies reference is a form of teaching. Academic librarians should approach reference

transactions as academic conferences where teaching and learning occur.

Green & Peach (2003) Assumed this connection and wanted to

validate it for purposes of librarian evaluation.

The Wartburg Study

Wanted to know WHAT the student learned.

Does reference reinforce classroom instruction?

Based on outcomes (Info Lit Competency Standards) They set our agenda Create our vision Focus our teaching Provide our common purpose Guide our students

Debra Gilchrist, Pierce Colleges, Lakewood and Puyallup, Washington

Methodology

Paired and numbered surveys. Librarian and student both

answered after a reference encounter.

Questions developed by reference librarians.

Based on information literacy concepts taught in the classroom.

Methodology

Teaching “intentions” vocabulary: choosing good search terms database selection search strategy evaluating information how to use a specific tool other (specify)

No limits on taking survey multiple times.

Methodology

Student questionnaire Spring 2003: Tried to tie reference encounters to

prior instruction. 143 returned (85% response rate).

Librarian questionnaire Spring 2003: Six choices for instructional outcomes.

Only one answer allowed. 169 returned.

Methodology

2003-04 Academic Year: Student questionnaire:

Added demographic questions Asked student to describe instruction and

assign a category 121 returned (78% response rate).

Librarian questionnaire: Only change was allowing multiple answers

if ranked. 156 returned.

Methodology

January-May 2003 141 useable

responses (143 returned).

Four librarians administered the survey.

3 of the 4 participate in ILAC classroom instruction.

2003-2004 Academic Year

121 useable responses

Same four librarians administered the survey.

Data Analysis – First Survey

Surveys re-paired. Answers entered into spreadsheet

and narrative list and tallied. Two librarians independently judged

congruence as “related,” “not related,” or “inconclusive.” High inter-rater reliability.

Discussed and resolved differences.

Data Analysis – Second Survey

Surveys re-paired. Answers entered into spreadsheet and

Qualrus Category matches identified. Researchers independently analyzed

forms for “strong,” “acceptable” or “no” description matches. Inter-rater reliability high.

Discussed and resolved differences.

Description Match Examples

Student: how to use the catalog Librarian: iPac to find music CDs

Student: how to cite CQ Researcher

Librarian: citation with CQ

Student: how to find literary criticism

Librarian: Literature Resource Center and Contemporary Literary Criticism

Non-Match Examples

Student: Showed me different places I could look for the information I was seeking.

Librarian: Oxford Reference Online; defining "liberal arts"

Student: He told me some important information about companies on websites recommended by the college.

Librarian: LexisNexis Business and Business Source Elite

Demographics

2003-2004 Survey

35%

39%

26%

Male Female No Answer

2003-2004 Survey

23%

16%

15%

24%

22%

1Y 2Y 3Y 4Y No Answer

Q. 1: Did the librarian who just helped you teach you anything while answering your question?

Spring 2003 Survey

94%

6%

Yes No

2003-04 AY Survey

98%

2%

Yes No

Q. 3: Did a librarian meet with your class and teach your class how to find information for this

assignment?Jan.-May 2003

Survey

33%

62%

5%

Yes No No Response

2003-2004 Survey

34%

65%

1%

Yes No No Response

Q. 4: Did what the librarian taught you relate to or build on anything a librarian taught your

class?Jan.-May 2003

Survey

89%

11%

Yes No

2003-2004 Survey

95%

5%

Yes No

Q. 5: Did what the librarian taught relate to or build on a

previous lesson?2003-2004 Survey

74%

21%

5%

Yes No No Response

Jan.-May 2003 Survey

77%

20%3%

Yes No No Response

Students made the link – sometimes.

2003-2004 Survey

36%

21%

21%

22%

Category DescriptionBoth No Match

Jan.-May 2003 Survey

60%20%

20%

Related Inconclusive Not Related

Link stronger with tools

2003-2004 Survey

42%

22%

16%

16%4%

Tool TermsStrategy DatabaseOther

Jan.-May 2003 Survey

62%4%

16%

10%8%

Tool Terms Strategy Database Other

What We Learned

Students understand reference to be an instructional activity.

Information Literacy instruction does seem to yield results.

Reference helps students practice and reinforce information literacy knowledge.

Reference Applications

Began a three-tier reference model. 1--Frontline: Tool instruction on

demand 2--Backup: Concepts on call 3--In-Depth: Consultations as

scheduled All staff participate.

Reference Applications

Focus on reference as guided practice.

Embrace role as facilitator.

Be more explicit about links to classroom instruction. Brief questions about

process and concepts. Graphic organizers.

Reference Applications

Have declined to participate in consortial virtual reference project.

Beginning a campaign to promote consultations: PSA Faculty awareness

New librarian will create online tutorials.

Classroom Applications

Continue to seek information literacy opportunities in classes. especially as survey data reveal

which courses students are working on.

Have all staff sit in on information literacy lessons. so they know what is being taught

and can reinforce better.

Further Research

Other academic libraries adapt our form and do their own assessments?

Assess consultations at Wartburg

Contact Information

Jill Gremmels

(319) 352-8462

jill.gremmels@wartburg.edu

Karen Lehmann

(319) 352-8460

karen.lehmann@wartburg.edu