Robin Grant, Felicia Haywood, and Dana Casper* · 20.12.2016 · student in FYE following library...

506

The Proof is in the Worksheets:Tying Library Instruction Assessment to ACRL Information Literacy StandardsRobin Grant, Felicia Haywood, and Dana Casper*

When Middle Georgia State University Libraries decided to assess our instruction, the First Year Experience program’s mandatory library sessions gave us the perfect opportunity. Us-ing learning outcomes based on ACRL information literacy standards, we scored every student in FYE following library sessions—approximately 2,100 students over a two-year period. We will share what we learned from this large pool of data, and how we are using this knowledge to improve both FYE and other undergraduate library instruction.

IntroductionMiddle Georgia State University was formed in 2012 when the Board of Regents voted to consolidate Macon State College and Middle Georgia College. The new institution incorporates all five campuses of the two former colleges, in five different Georgia towns.

The Macon State College Library (before consolidation) had been using an online survey at the end of most library instruction sessions to assess student learning and make improvements. In these surveys, students answered questions giving their impression of the usefulness of the class and providing suggestions for chang-es—many of which were put into practice. However, after reading studies that show a gap between students’ perception of their research competence and their actual skills,1 we wanted a more concrete way to measure their learning. We decided to develop an assessment of our library instruction based on measurable learning outcomes, not just student impressions of the class or how they felt about their abilities.

We investigated how other libraries were doing this kind of assessment and were interested in a program at the University of North Carolina Wilmington, in which students completed activities on a worksheet during the sessions. Items on the worksheet were tied to specific learning outcomes and scored for assessment.2 We con-tacted Anne Pemberton, Assistant Director, Library Instructional Services at the University of North Carolina Wilmington, and discussed these ideas in more detail. Ms. Pemberton provided us with samples of their work-sheets, outlines, and rubrics for scoring the worksheets.

For the past three years, Macon State College had been conducting two library instruction sessions for each section of our new First Year Experience class, and we were using a standardized instruction outline and work-

* Robin Grant is Electronic Resources Librarian, Middle Georgia State University, email: [email protected]; Fe-licia Haywood is Assistant Director of Library Services—Macon Campus Library, Middle Georgia State University, email: [email protected]; Dana Casper is Reference and Instruction Librarian, Middle Georgia State Univer-sity, email: [email protected].

mailto:[email protected]



sheet for each of those classes already. For other classes, such as English 1101 or English 1102, there is far more difference from one library instruction session to another because we teach based on the students’ specific as-signments and often do them workshop-style. In FYE, however, we teach broad, basic concepts that are the same for each class section. It therefore appeared that the FYE program would be the ideal place to try a pilot program of assessment based on consistent learning outcomes and student activities across all sessions.

In the fall of 2013, as the newly-consolidated Middle Georgia State, we would be expanding this standard-ized FYE instruction program to four of the five campuses and training librarians at all these locations to con-duct FYE sessions. This seemed the perfect opportunity to tweak the existing outline and worksheet to use for a pilot program of assessment of FYE library instruction sessions.

The prep work was done in summer of 2013. We conducted the assessment during the 2013–14 fall and spring terms and again in 2014–15 fall and spring.

Developing Learning Outcomes, Class Outlines, and Rubrics based on ACRL StandardsACRL Information Literacy Competency Standards for Higher Education (2000) stated in the Information Lit-eracy and Assessment section:

In the following competencies, there are five standards and twenty-two performance indicators. The standards focus upon the needs of students in higher education at all levels. The standards also list a range of outcomes for assessing student progress toward information literacy. These outcomes serve as guidelines for faculty, librarians, and others in developing local methods for measuring student learning in the context of an institution’s unique mission.3

Based on ACRL’s Information Literacy Competency Standards for Higher Education (2000), we identified three desired learning outcomes for the first FYE session and four for the second session.

Session I:• Demonstrate an understanding of the concept of keyword searching to broaden or narrow a topic.• Demonstrate an understanding of evaluating a book for its appropriateness for their information need.• Demonstrate an understanding of how to locate books in the library.Session II:• Demonstrate the ability to evaluate a website for trustworthiness and appropriateness for their infor-

mation need.• Demonstrate the ability to locate an article in an online library database.• Demonstrate the ability to evaluate an article for its appropriateness for their information need.• Demonstrate an understanding of the value of scholarly information sources.Based on the ACRL standards and these learning outcomes, we developed instruction outlines, activity

worksheets, and scoring rubrics for each session. (See appendices for examples of learning outcomes and class activities based on ACRL standards, as well as our scoring rubrics.)

Teaching the ClassesIn addition to having standardization of the classes from the library side, we were fortunate that, when the FYE program was being developed in 2010, librarians were on the curriculum committee and helped make infor-mation literacy an important goal. Consequently, information literacy and library instruction sessions were a

MARCH 22–25, 2017 • BALTIMORE, MARYLAND

The Proof is in the Worksheets 507

required part of the FYE curriculum, and the Library was able to take part in training sessions for FYE faculty. We were also able to secure a requirement that FYE faculty assign a research project, so their students would im-mediately put into practice the concepts they would be learning in the library. Studies have shown that students are more engaged in learning information skills when instruction is tied to a class research assignment, making it relevant4 and requiring them to actively practice what they are learning.5 Therefore, assigning the students a research project in conjunction with library instruction was seen to be critical. Faculty members of each FYE section were, however, free to devise their own research project.

Because there are so many sections of FYE, every librarian was called upon to teach at least one of the classes, so it was important to have a standard teaching outline as well as the worksheets and scoring rubrics. Each librarian was allowed some leeway as far as sample books or websites chosen for each session, but oth-erwise were asked to present the classes and guide the students through the worksheet exercises according to a standard outline. A list of broad sample topics was also provided to each librarian, as well as suggestions for keywords to help guide students through the processes of broadening and narrowing topics. We began using all these materials in the fall of 2013.

After each class, the teaching librarian would score the worksheets according to the rubric and send the scores to the FYE professor. The professors could use these scored sheets as they desired, and most chose to give the students at least a quiz grade. This had the added benefit of helping the students be more attentive in class and see the session as a part of their class grade, as supported by earlier research from Robinson and Schlegl (2004). This research showed that students were more likely to use the skills demonstrated in library research sessions when doing so was tied to an enforceable penalty,6 which would in this case be the potential loss of a quiz grade.

One type of problem often found with studies is “nonsampling error,” which involves issues in data process-ing, how the observations were made or interpreted, etc.7 To reduce the possibility of nonsampling errors such as scoring discrepancies or librarian bias in our study, the original worksheets were scored again by a second librarian. Both scores were entered into an Excel worksheet and the average of the two was the final score for the assessment study.

First Year Results and Next StepsFeeling confident that our FYE students had been getting a good foundation in information literacy concepts from these library sessions, we set our initial measure of success high. Our measure of success would be 80 per-cent of students scoring three or better on each learning outcome/performance indicator.

We were unpleasantly surprised at the end of the first semester (fall 2013) to find that our students had failed to make the 80 percent goal in all learning outcomes except for one: locating an article in an online database.

In response, librarians held a debriefing session to take a closer look at where students were having prob-lems and to discuss ways for improvement. Neither the learning outcomes nor the rubrics were changed, but the worksheet questions were tweaked and rearranged in areas that seemed to be causing the students confusion. For example, on the “book evaluation” section of the original worksheet, identification of the book’s author had been separated from a question about author credentials by questions about book location. Additionally, stu-dents did not seem to be certain what we were asking for on other book-related evaluation questions, so wording was fine-tuned.

Based on our end-of-term debriefing session with all librarians, we also discovered that the library tour near the beginning of session I, during which students were allowed to select a topic and book for the class, was cutting into worksheet and evaluation time. We therefore started assigning topics and bringing in pre-selected

ACRL 2017 • AT THE HELM: LEADING TRANSFORMATION

Robin Grant, Felicia Haywood, and Dana Casper508

FIGURE 1Book Evaluation Section of Original Worksheet

FIGURE 2Revised Book Evaluation Section of Worksheet



books that students could choose from. This not only allowed us more time for evaluation activities, but also al-lowed us to select books with a variety of “issues” we could discuss as a class—a dubious author, lack of currency, lack of information about the author, etc.

As can be seen from table 1, scores did improve from fall to spring in all areas of session I.

Second Year’s ResultsThe Library decided to repeat the assessment for a second year, encompassing fall of 2014 and spring of 2015. At the end of the fall semester, librarians once again had a debriefing session to examine the scores. For the second year, we also set a more realistic goal that 70 percent of the students would achieve a score of three or better on all the worksheet questions.

At the end of the fall term, we achieved our goal that 70 percent of students would achieve a score of three or better on all the worksheet questions/learning outcomes, except for these two, both on the worksheet for the second session:

TABLE 1FYE Session I Worksheet Totals Fall 2013 and Spring 2014



• Demonstrate the ability to evaluate an article for its appropriateness for their information need.• Demonstrate an understanding of the value of scholarly information sources.We noted that both of those learning outcomes had to do with recognizing the difference between scholarly

and other types of sources. It is sometimes hard for information professionals to understand that the idea of scholarly, peer-reviewed sources is new and foreign to many students. Our worksheets and scores graphically demonstrated that students often have difficulty grasping exactly what scholarly, peer-reviewed materials are, what their purpose is, and how to identify them. In response, we decided to make some changes to our spring 2015 teaching outline to address those two learning outcomes specifically.

After analyzing the data, it appeared that one librarian had the best average scores from her students on those two questions—and that she was doing some specific things in her teaching that were unique to her ses-sions:

• Bringing in samples of scholarly, peer-reviewed journals for the students to examine;• Tying the website evaluation to the article evaluation.

◊ Specifically, when evaluating a website at the beginning of the class, students were asked to note one claim the website made;

◊ Then, when they reached the part of class dealing with locating and evaluating an article, she led them through a keyword search and helped them locate a scholarly, peer-reviewed article verifying (or disproving) the website’s claim.

Since the data indicated that students were having more success with the evaluation of scholarly resources when exposed to this method, the teaching outline for the second FYE session was revised to follow this gen-eral outline. Again, we did not change the worksheets, learning outcomes, or rubrics, but adjusted the teaching outline to give more weight and clarity to the issue of scholarly resources. We again gave individual librarians the option of choosing their own websites and article sources for the evaluations, but also provided two topic/website samples.

At this point, we also asked librarians not to use “hoax” web pages for the website evaluation section of their classes. Some librarians had been successfully using hoax websites in library instruction to make classes more fun and get students’ attention. One popular example was “Save the Pacific Northwest Tree Octopus.”8 However, hoax websites would not work as well with our new outline, because we wanted students to be able to follow the process of seeing original research reported in a scholarly journal, then interpreted and reported secondhand on a website.

As seen in table 2, after adjusting our teaching outline for session II, scores did improve from fall of 2014 to spring of 2015.

Results/Conclusions/Next StepsWe conducted library instruction sessions and scored and analyzed worksheets for 1,215 students in 2013–14 and for 1,109 students in 2014–15. Most of those students attended two library sessions, producing two work-sheets, and all of the worksheets were scored by two different librarians. This large pool of data collected on four different campuses across four semesters and many class sections helped give us a detailed picture of our students and information literacy: their understanding and skills at the very beginning of their college careers; how we were communicating with them; what they were learning; and the weaknesses that remained.

One benefit of this assessment is having hard data demonstrating students scoring well on questions about lo-cating a certain resource, such as a book in a catalog or an article in a database. Once pointed in the right direction, they were able to figure out where to click, where to enter search terms, etc. In other words, the mechanics of online



searching, according to worksheet scores, was far more intuitive for students than the evaluation of information—and in particular, the difference between scholarly and other types of information. Faculty members sometimes as-sume the opposite—that their students understand what kind of resource they need and will know it when they see it, and only need librarians to show them the mechanics of the databases and “where to click.” Having these figures allows us to demonstrate that students need much more to help them become information literate.

Another unplanned advantage we have discovered in using the worksheets instead of a survey is that we don’t have to wait for another college department to send us results from our online form at the end of a semester in order to make adjustments to our teaching. We are able to look at our worksheets immediately after classes and spot possible trouble spots, or where students are not grasping the material as readily. We can then tweak the next sessions to give more emphasis and clarity to the weak areas.

One issue we have had in standardizing these FYE library sessions has been that each FYE professor has been responsible for developing his or her own research project for their classes. We found that the assigned

TABLE 2FYE Session II Worksheet Totals Fall 2014 and Spring 2015

Note: the rows highlighted in yellow show the learning outcomes we focused on improving by adjusting our teaching outline from fall to spring of that year.



Robin Grant, Felicia Haywood, and Dana Casper

projects varied greatly from one class to another, and many did not actually contain a satisfactory research component. In response, we have developed a suggested research assignment and were allowed to share it with FYE instructors starting at the beginning of fall 2015. We are hopeful that we will eventually be able to have a standard, required research assignment across all sections to go along with the FYE library instruction sessions.

Although ACRL now has a new Framework for Information Literacy in Higher Education (2015) available, institutions interested in replicating our project should be able to use this same general process with other stan-dards or learning outcomes.

Since we were already teaching all these FYE students and scoring their worksheets as a regular part of our instruction, we chose to use all the students’ worksheets in this assessment, rather than a sample. We felt more confident of the validity of our results since this allowed us to avoid sampling error, in which our chosen sample may not have accurately reflected our whole population.9 However, conducting the second scoring to acquire an average and working with so many worksheets and scores did involve a great deal of work hours. We might consider including only a sample of the students if we do the study again.

Notes1. Melissa Gross and Don Latham, “What’s Skill Got to Do With It?: Information Literacy Skills and Self-Views of Ability Among

First-year College Students,” Journal Of The American Society For Information Science & Technology 63, no. 3 (2012): 574–583, doi: 10.1002/asi.21681; Melissa Gross, “The Impact of Low-Level Skills on Information-Seeking Behavior: Implications of Com-petency Theory for Research and Practice,” Reference & User Services Quarterly 45, no. 2 (2005): 155, http://ezproxy.mga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edsjsr&AN=edsjsr.20864481&site=eds-live&scope=site.

2. Anne Pemberton, “Getting More Bang for Your Buck: Using a Template to Create Rubrics and Worksheets for Library Instruction Assessment,” YouTube video, 31:55, posted by the University of Chicago August 22, 2011, https://youtu.be/O4C3opGb_8A.

3. “Information Literacy Competency Standards for Higher Education,” American Library Association, last modified September 1, 2006, accessed December 20, 2016, http://www.ala.org/acrl/standards/informationliteracycompetency.

4. Janet Webster and Lorett Rielly, “A Library Instruction Case Study: Measuring Success from Multiple Perspectives,” Research Strat-egies 19, no. 1 (2003): 16–32, http://dx.doi.org/10.1016/j.resstr.2003.11.001.

5. Susan E. Cooperstein and Elizabeth Kocevar-Weidinger, “Beyond Active Learning: A Constructivist Approach to Learning,” Refer-ence Services Review 32, no. 2 (2004): 141–148, http://ezproxy.mga.edu/login?url=http://search.proquest.com/docview/200546638?accountid=12418; Ann Roselle, “Preparing the Underprepared: Current Academic Library Practices in Developmental Education,” College & Research Libraries 70, no. 2 (2009): 142–156, http://ezproxy.mga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=eft&AN=502972691&site=eds-live&scope=site.

6. Andrew M. Robinson and Karen Schlegl, “Student Bibliographies Improve When Professors Provide Enforceable Guidelines for Citations,” Portal : Libraries and the Academy 4, no. 2 (2004): 275–90, http://ezproxy.mga.edu/login?url=http://search.proquest.com/docview/216172087?accountid=12418.

7. Peter Hernon, “Determination of Sample Size and Selection of the Sample: Concepts, General Sources, and Software,” College & Research Libraries 55, no. 2 (1994): 171–179, doi:10.5860/crl_55_02_171.

8. “Save the Pacific Northwest Tree Octopus,” http://zapatopi.net/treeoctopus/.9. Hernon, “Determination of Sample Size,” College & Research Libraries 55, 173.



http://10.1002/asi

http://ezproxy.mga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edsjsr&AN=edsjsr.20864481&site=eds-live&scope=site

http://ezproxy.mga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=edsjsr&AN=edsjsr.20864481&site=eds-live&scope=site

https://youtu.be/O4C3opGb_8A

http://www.ala.org/acrl/standards/informationliteracycompetency

http://dx.doi.org/10.1016/j.resstr.2003.11.001

http://ezproxy.mga.edu/login?url=http://search.proquest.com/docview/200546638?accountid=12418


http://ezproxy.mga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=eft&AN=502972691&site=eds-live&scope=site

http://ezproxy.mga.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=eft&AN=502972691&site=eds-live&scope=site



http://10.5860/crl

http://zapatopi.net/treeoctopus/

APPENDIX 1Teaching Outlines and Activities Based on ACRL Standards

FYE SESSION I

FYE Teaching Outline ACRL Info Lit Standard

ACRL Performance Indicator/Outcome

FYE Activity for Assessment/Learning

Outcome

Search Strategy • Keyword basics• Facebook as a

database and keyword search example

• Narrowing and broadening topics

Standard One (I): The information literate student determines the nature and extent of the information needed.

Standard Two (II): The information literate student accesses needed information effectively and efficiently.

(I) 1d. Defines or modifies the information need to achieve a manageable focus.

(I) 1e. Identifies key concepts and terms that describe the information need.

(II) 2b: Identifies keywords, synonyms and related terms for the information needed.

On worksheet:Student identifies main topic.Identifies keywords to narrow or broaden.

Search Strategy• Start with general

info like reference, then go more specific

• Intro to library book (GIL) catalog and library locations


(I) 2c. Identifies the value and differences of potential resources in a variety of formats.

(I) 2d. Identifies the purpose and audience of potential resources.

On worksheet, student identifies books as reference or general.

Evaluating Sources• Discuss

importance of evaluating information

• Demonstrate evaluation of a book for currency; author’s credentials; intended audience; factual information vs. bias.

Standard Three (III): The information literate student evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system.

(III) 2. The information literate student articulates and applies initial criteria for evaluating both the information and its sources

(III) 2a. Examines and compares information from various sources in order to evaluate reliability, validity, accuracy, authority, timeliness, and point of view or bias.

On worksheet, student discusses and evaluates a sample book for currency, author’s credentials, intended audience, factual information vs. bias.







Outcome

Evaluating Sources• Discuss

importance of evaluating information

• Demonstrate evaluation of a book for currency; author’s credentials; intended audience; factual information vs. bias.


(III) 2. The information literate student articulates and applies initial criteria for evaluating both the information and its sources


On worksheet, student discusses and evaluates a sample book for currency, author’s credentials, intended audience, factual information vs. bias.

When possible, give a tour of the campus library. Show students how to locate books in the library during the tour. If tour is not possible, show them how to find books based on locations in GIL catalog.


(II) 3a. Uses various search systems to retrieve information in a variety of formats

(II) 3b. Uses various classification schemes and other systems…to locate information resources within the library…

On worksheet, identifies book’s call number and describe its location. Identifies GIL as the library’s book catalog.

FYE SESSION II




Outcome

Lead students in evaluation of a sample website for • Currency• Appropriateness

for project (scholary v. popular, reference, news, etc.)

• Authority• Information v. bias• Accuracy


(III) 2. The information literate student articulates and applies initial criteria for evaluating both the information and its sources.


On worksheet, student identifies and discusses sample website’s date/currency; author and credentials; identity of website owner/organization; purpose of website and possible bias; accuracy of information on site.







Outcome

Demonstrate Advanced Google searching to achieve more focused results.


(II) 2e: Implements the search strategy in various information retrieval systems using different user interfaces and search engines, with different command languages, protocols, and search parameters.

Hands-on student practice, but no worksheet item.

Demonstrate Subject Guides and article databases as a way to most efficiently find scholarly information. Ask the student to locate an article on a given topic in Academic Search Complete.

Standard Two (II). The information literate student accesses needed information effectively and efficiently.

(II)1. The information literate student selects the most appropriate investigative methods or information retrieval systems for accessing the needed information.

(II) 1c. Investigates the scope, content, and organization of information retrieval systems.

(II) 1d. Selects efficient and effective approaches for accessing the information needed from the investigative method or information retrieval system.

Demonstrates the ability to locate an article in an online library database.

Demonstrate and discuss popular articles vs. scholarly, peer-reviewed.


(I) 2d. Identifies the purpose and audience of potential resources (e.g., popular vs. scholarly, current vs. historical)

Student identifies the article they located as popular or scholarly and discusses the intended audience.

On the worksheet, student identifies scholarly, peer-reviewed journals as the most authoritative sources of information for their papers.



517





Outcome

Demonstrate how to evaluate an article found in Academic Search Complete. Ask students to evaluate the article they located.


(III) 2. The information literate student articulates and applies initial criteria for evaluating both the information and its sources.


Student discusses the article they located in terms of currency, authority, intended audience, and scholarly or popular.



APPENDIX 2Scoring Rubric for FYE Session I

Learning Outcome

Excellent (4) Good (3) Fair (2) Weak (1)

Demonstrate an understanding of the concept of keyword searching to broaden or narrow a topic.

Provides at least two terms that would broaden AND two to narrow their assigned topic.

Provides at least one term that would broaden AND one to narrow their assigned topic.

Provides at least one term that would broaden OR narrow their assigned topic.

Provides no terms to broaden or narrow their assigned topic.

Demonstrate an understanding of evaluating a book for its appropriateness for their information need.

Provides all of the following: Sound discussion of the book’s currency; author’s credentials; intended audience; factual information vs. bias

Provides most of the following: sound discussion of the book’s currency; author’s credentials; intended audience; factual information vs. bias

Provides some of the following: sound discussion of the book’s currency; author’s credentials; intended audience; factual information vs. bias

Provides little or none of the following: sound discussion of the book’s currency; author’s credentials; intended audience; factual information vs. bias

Demonstrate an understanding of how to locate books in the library.

Does all of the following:

Identifies GIL as the resource to look for books; physically locates a call number in the library and/OR fully describes on the worksheet where the book would be located.

Does most of the following:


Does some of the following:


Does little or none of the following:




APPENDIX 3Scoring Rubric for FYE Session II

Learning Outcome

Excellent (4) Good (3) Fair (2) Weak (1)

Demonstrate the ability to evaluate a website for trustworthiness and appropriateness for their information need.

Provides all of the following: sound discussion of the website’s currency, accuracy, source, bias (or lack thereof), and authority.

Provides most of the following: sound discussion of the website’s currency, accuracy, source, bias (or lack thereof), and authority.

Provides some of the following: sound discussion of the website’s currency, accuracy, source, bias (or lack thereof), and authority.

Provides little or none of the following: sound discussion of the website’s currency, accuracy, source, bias (or lack thereof), and authority.

Demonstrate the ability to locate an article in an online library database.

Provides all three of the following: Title of an article; name(s) of author; and name of database.

Provides at least two of the following: Title of an article; name(s) of author; and name of database.

Provides at least one of the following: Title of an article; name(s) of author; and name of database.

Provides none of the following: Title of an article; name(s) of author; and name of database.

Demonstrate the ability to evaluate an article for its appropriateness for their information need.

Provides all of the following: sound discussion of the article’s currency; authority; and intended audience; and identification as scholarly or popular.

Provides most of the following: sound discussion of the article’s currency; authority, and intended audience; and identification as scholarly or popular.

Provides some of the following: sound discussion of the article’s currency; authority; and intended audience; and identification as scholarly or popular.

Provides little or none of the following: sound discussion of the article’s currency; authority; and intended audience; and identification as scholarly or popular.

Demonstrate an understanding of the value of scholarly information sources.

Identifies journal articles as the most authoritative source of information for a research paper and gives at least two sound supporting reasons.

Identifies journal articles as the most authoritative source of information for a research paper and gives at least one sound supporting reason.

Identifies journal articles as the most authoritative source of information for a research paper and gives no sound supporting reasons.

Does not identify journal articles as the most authoritative source of information for a research paper and gives no supporting reasons.



Robin Grant, Felicia Haywood, and Dana Casper* · 20.12.2016 · student in FYE following library...

Documents

Transcript of Robin Grant, Felicia Haywood, and Dana Casper* · 20.12.2016 · student in FYE following library...