Measuring GALILEO’s Performance:

2014 GALILEO User Survey Report

November 2014

Table of Contents

I. Executive Summary ..........................................2

II. Methodology ......................................................2

III. Demographics ...................................................2

IV. About the Data ..................................................3

V. The Survey ........................................................9

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 2

Executive Summary

The annual GALILEO survey is conducted to ascertain user satisfaction and usage habits of GALILEO

users. Annual surveys have been conducted since 1996, one year after GALILEO went online. The

survey has been available only in an online format since the start. The 2014 GALILEO User Survey was

conducted November 10-16, 2014.

The survey included questions that have been asked in previous years. Below is a summary of these

questions:

Six statements rated by the respondent on GALILEO performance and value (questions included every year)

Multiple choice question: “I would improve GALILEO by . . .” (questions included every year)

Six qualities of the discovery search (previously “federated search”) tool rated by the respondent (questions included 2011-2014)

Two questions on social media preferences (included 2013-2014, slightly changed to address current GALILEO social media profiles)

Demographic questions about user and institution types, usage frequency, and where the respondent learned about GALILEO (questions included every year)

A final question asking for further feedback was added in 2014. The complete survey questionnaire is

included later in this document for reference.

Methodology

Participants were invited to participate in the survey through visiting the GALILEO website and through

direct and secondary communications, i.e., retweets and other communication from the libraries to their

end users. Communication channels to promote the survey included the GALILEO Listserv, password

contacts, social media profiles, and the GALILEO news feed. Survey announcements encouraged staff in

libraries and media centers to promote the survey to their end users. A flyer with a QR code for mobile

users that libraries could display on site was included in list announcements.

After the removal of incomplete and duplicate entries, 1885 surveys were submitted. Incomplete surveys

were defined as those without answers to the first six response-required questions. Data analysis was

completed using tools in the QuestionPro survey program and Excel.

Demographics

Questions on the survey asked users to identify themselves by their user type (e.g., undergraduate

student, faculty, or public library user) and by their institution type (e.g., state college or university,

technical college, or public K-12 school). The categories in both these questions have changed little since

the first survey in 1996; for example, the private K-12 school category was added to institution types

when the first independent school consortium joined GALILEO.

User Types Of the 1885 respondents, 276 were librarians, library staff, or media specialists, and 1444 were end users (students, faculty or teachers, public library users, or users who identified themselves as

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 3

"other"). The remaining 165 did not self-identify. The largest number of responses came from undergraduate students (35%), followed by 2-year/technical college students (17%). Institution Types The highest percentage of responses came from USG institutions (42%), then technical colleges (25%).

About the Data

Rating Performance and Value Every year, the survey includes five statements designed to measure satisfaction with different aspects of GALILEO using a Likert scale. Responses are required for these statements. Trend analysis over the years shows that satisfaction levels remain steady even when significant changes, such as the

0% 5% 10% 15% 20% 25% 30% 35% 40%

Student, Graduate

Student, Undergraduate

Student, 2-Year/Technical

Student, K-12

Librarian/Library Staff

Media Specialist

Faculty

K-12 Teacher

Public Library User

Other

User Type

0% 10% 20% 30% 40% 50%

State (USG) College/University

Private College/University

Technical College

Public Library

K-12 Public School

K-12 Private School

Other

Institution Type

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 4

implementation of a discovery service or an interface redesign, has been implemented. For example, this year 94% agree/strongly agree that GALILEO is a valuable service, the same percentage as 2011-2013 and only slghtly different from previous years.

The two charts above show the numbers and percentages for all respondents and also for end users only (librarians and media specialists removed). There is little variation when librarians and media specialists are removed. Rating the GALILEO Search Function A set of questions asking respondents to rate the search/discovery tool in GALILEO has been asked the past four years. In 2011, the tool was a federated search tool powered by 360 Search. In 2012, Discover GALILEO, powered by EBSCO Discovery Service, had been in production for one month before the annual survey was conducted, so respondents had little or no experience with the new service. Enhancements were implemented in 2013 and 2014, including the addition of library catalogs and local resources; however, a change in satisfaction levels was not apparent in the survey data. For example, in 2014, 85% were very satisfied/satisfied with the overall functionality of Discover GALILEO (84% in 2012 and 2013, 86% in 2011 before EDS was implemented). Discovery services represent a significant

0%10%20%30%40%50%60%70%80%90%

100%

Summary with Librarians and Media Specialists Included

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree

0%10%20%30%40%50%60%70%80%90%

100%

Summary with Librarians and Media Specialists Removed

Strongly Disagree

Disagree

Neutral

Agree

Strongly Agree

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 5

improvement in searching across resources compared to federated search, but user satisfaction with the relevancy of results has changed less than 1%. It should be noted that the annual user survey does not measure usability or success using the tool, so factors that contribute to the consistent ratings could include a general response to GALILEO on the whole, response bias, or other considerations not directly related to the Discovery service.

“Improve” Responses

Respondents were given multiple choices to answer the question, "I would improve GALILEO.”

Historically, “more content” and “simpler interface and navigation,” very general categories, have been the

top concerns. Respondents in 2014 also saw these as the top concerns, but the number who rated “more

content” as the top concern has been dropping a little each year since 2011. No other category changed

more than 2%. The same set of categories will be included in the 2015 survey to watch the “more

content/full text” trend. The next most-selected categories were “simpler interface and navigation” at 26%

and “better searchability” at 19%.

0%10%20%30%40%50%60%70%80%90%

100%

Rating Discover GALILEO with Librarians and Media Specialists Included

Very Dissatisfied

Dissatisfied

Neutral

Satisfied

Very Satisfied

0%10%20%30%40%50%60%70%80%90%

100%

Rating Discover GALILEO with Librarians and Media Specialists Removed

Very Dissatisfied

Dissatisfied

Neutral

Satisfied

Very Satisfied

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 6

Social Media Responses

Questions about social media were included to help determine the tools respondents used and the kinds of information they would like to receive through those channels. The questions included links for liking/following GALILEO on Twitter, Facebook, and the GALILEO news feed. The spread of responses to the kinds of information respondents would like to receive was not wide, suggesting the content of GALILEO news and posts matches the respondents’ expectations. Other suggestions for posts included GALILEO tips and new content in GALILEO.

0% 5% 10% 15% 20% 25% 30% 35%

More content/full text

Simpler interface and navigation

Better searchability

Easier access (password access)

Improved functionality

Improved help information

I would improve GALILEO by —Librarians and Media Specialists Included

0% 5% 10% 15% 20% 25% 30% 35%

More content/full text

Simpler interface and navigation

Better searchability

Easier access (password access)

Improved functionality

Improved help information

I would improve GALILEO by—Librarians and Media Specialists Removed

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 7

Further Feedback The final question on the survey was a general request for feedback. The question was answered by 20%, or 380, of the respondents. Many respondents commented on the relevance of results. Individual databases and the Discover search offer tools for limiting and refining results, but there is not enough data to infer whether the respondents were familiar with those options or not. Below are sample comments relating to search results.

Discover greatly improved access to regular users to find what they need. Still more simplification of filters would be helpful to beginning users (Public library librarian)

Everything about GALILEO is great. The only thing I could recommend to improve is the search feature. Sometimes it can fetch what I'm looking for instantly. Other times, I have to scroll through many pages to find the title for something that I searched for, even though I type it word for word (Technical college student)

Galileo is great but it gets a little tricky to use because the relevance of the results isn't always the best (Undergraduate)

The GALILEO interface was the topic of some responses, with users suggesting a redesign (“The user interface of the site could be updated more for the 21st century” –K-12 media specialist) or specific changes (“Is it possible to add a quick browse instead of having to do drop-down menu after drop-down menu?” – K-12 media specialist). Some comments were related to navigation or ease of use (“Galileo should have easier navigation for students” –Technical college student). Some respondents commented on the frustration of finding article citations without full text available

“It gets frustrating when you finally find the prefect source with all of the information you are looking for and it only has the abstract” (K-12 student)

Availability for full text online for more articles would be great! Very often, I've had search results that lead to nothing that is even available in print, which is very frustrating. Why have it in the database if no one has it? (Undergraduate)

0% 2% 4% 6% 8% 10% 12% 14%

Password change alerts

Fun facts/trivia

GALILEO contests

Trending search terms

Trending journals

Today in Georgia history

Database spotlights

Quick polls (example: What did you search for…

New databases/removed databases

Outages and status messages

GALILEO training announcements

National library events (e.g., Banned Book…

Book award announcements

Other

What information do you like to receive through social media?

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 8

Many respondents took the opportunity to comment on the value of GALILEO to their studies or to their students’ and patrons’ research.

As a biology major, GALILEO has helped me find primary and secondary articles to write summaries, papers, and also create projects with. It is a very organized system that you just do not find with Google or Bing. I will continue to use GALILEO during and after school (Undergraduate)

At first I did not even consider using GALILEO because I just thought of it as complicated but as of a year ago, it is always a reference source to me now. GREAT INSTALLMENT! (Undergraduate)

For a long time I never used it; however, when I tried it for the first time it saved me a lot time and worry because I found just what I needed (Technical college student)

I love sharing with patrons about their accessibility to Consumer Reports! (Public library librarian)

For college graduate school use, GALILEO is extremely easy to navigate and an invaluable research tool! (Graduate student)

GALILEO got me through college! It is the first thing I show college students who come into my library to do research for papers (Public library librarian)

GALILEO is a fantastic service and a model to other consortia in the country. Keep up the good work (Academic librarian)

GALILEO is an amazing website to find articles on. It's free and also provides the cited page in any format! Love, Love, Love Galileo! (Undergraduate)

Happy to have GALILEO through my library. Thank you (Public library user)

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 9

The Survey

Thank you for participating in the Annual GALILEO User Survey. The survey should take only a few minutes to complete. If you have any questions about the survey or procedures, you may contact GALILEO staff at http://www.galileo.usg.edu/contact.Your responses to the survey are confidential. If you wish to be entered into the prize drawing, you will be asked to enter your name and contact information. Your contact information will remain confidential and be strictly used to contact you should you win a prize. Thank you very much for your time and support. Please start the survey by clicking on the Continue button below. Please indicate your opinion about GALILEO.

Strongly Agree

Agree Neutral Disagree Strongly Disagree

I found it easy to use GALILEO. ❏ ❏ ❏ ❏ ❏

Using GALILEO databases saved me time. ❏ ❏ ❏ ❏ ❏

I think GALILEO is a valuable service. ❏ ❏ ❏ ❏ ❏

I think GALILEO response time is acceptable. ❏ ❏ ❏ ❏ ❏

GALILEO met my information needs ❏ ❏ ❏ ❏ ❏

I would recommend GALILEO to a friend. ❏ ❏ ❏ ❏ ❏

I would improve GALILEO by (select the single response that best describes the improvement you would like to see):

1. More content/full text 2. Simpler interface and navigation 3. Better searchability 4. Easier access (password access) 5. Improved functionality 6. Improved help information

Please rate your satisfaction with Discover GALILEO, the discovery service that powers the search launched from the search box on the GALILEO website.

Very Satisfied

Satisfied Neutral Dissatisfied Very Dissatisfied

Relevance of results ❏ ❏ ❏ ❏ ❏

Quality of results ❏ ❏ ❏ ❏ ❏

Relevancy of databases included ❏ ❏ ❏ ❏ ❏

Access to full text ❏ ❏ ❏ ❏ ❏

Speed ❏ ❏ ❏ ❏ ❏

Overall functionality ❏ ❏ ❏ ❏ ❏

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 10

In which of the following GALILEO communication channels or social media sites do you participate/follow/like? Check all that apply.

1. Twitter (@GalileoLibrary) 2. Facebook (facebook.com/galileolibrary) 3. GALILEO News Feed (http://about.galileo.usg.edu/news/feed) 4. GALILEO Status Feed (http://status.galileo.usg.edu/site/feed/ 5. GALILEO E-Mail List (http://about.galileo.usg.edu/news/list/)

What information do you like to receive through social media sites? Check all that apply.

1. Password change alerts 2. Fun facts/trivia 3. GALILEO contests 4. Trending search terms 5. Trending journals 6. Today in Georgia history 7. Database spotlights 8. Quick polls (example: What did you search for today?) 9. New databases/removed databases 10. Outages and status messages 11. GALILEO training announcements 12. National library events (e.g., Banned Book Week announcements) 13. Book award announcements 14. Other

Today, I am using GALILEO as

1. Student, Graduate 2. Student, Undergraduate 3. Student, 2-Year/Technical 4. Student, K-12 5. Librarian/Library Staff 6. Media Specialist 7. Faculty 8. K-12 Teacher 9. Public Library User 10. Other

Please indicate your primary institution affiliation. (Upon selection, participants are asked to choose their institutions from a drop-down list.)

1. State (USG) College/University 2. Private College/University 3. Technical College 4. Public Library 5. K-12 Public School 6. K-12 Private School 7. Other

How often do you use GALILEO?

1. Daily 2. Weekly 3. Monthly 4. First time used

Where did you find out about GALILEO?

1. I am a long-time user 2. From a media specialist or librarian 3. From a teacher 4. From a friend

MEASURING GALILEO’S PERFORMANCE: 2014 GALILEO USER SURVEY REPORT – MARCH 1, 2015 11

5. From a link on my school or library web page 6. From a session at a conference or workshop 7. Other

Do you have any further feedback to share?

Thank you for participating in the Annual GALILEO User Survey.