Users and Metasearch Applications:New Challenges for Usability Assessment

William E. Moen, Ph.D.<wemoen@unt.edu>

Texas Center for Digital Knowledge

University of North Texas

Denton, Texas

Presented at Access 2003: Extending Our AbilitiesOctober 3, 2003 Vancouver, British Columbia, Canada

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 2

Access personal journey

• 1999: The Z Texas Profile: Improving Z39.50 Interoperability with a Focus on User Requirements

• 2000: Z39.50 Update: Profiles and Semantic Interoperability

• 2001: Bath Profile Update: An International Z39.50 Specification for Library Applications and Resource Discovery

• 2001: Z-Interop: The Z39.50 Interoperability Testbed

• 2002: Z39.50 for Finding It All• 2003: Users and Metasearch Applications

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 3

Standards in service of applications• Preparing technical infrastructure for

broadcast searching / metasearch

• Profiles specifying Z39.50 implementation

• Defining searches

• Solving interoperability problems

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 4

Where are we now?

• Some support among library technology vendors for Z39.50 profiles

• Many candidate search targets don’t support Z39.50 profiles or Z39.50 at all

• Commercial metasearch products and applications emerging in the past several years

• NISO Metasearch Initiative• Assessing what we’ve got and how to

improve

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 5

Metasearch

• Applications with:– Single search interface – Broadcast searches to multiple targets– Search targets may be

• Relatively homogeneous (e.g., library catalogs)• Very diverse (e.g., format, topical coverage, etc,)

• Applications are faced with search targets– That are accessible via standard protocols (e.g., Z39.50)

or proprietary APIs or screen-scraping– That support few or many common access points– That support various record formats for exchange

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 6

The challengeIf our users make it to the library's web site at all, chances are they are confronted with library terminology they don't understand and a long list of databases they have to decipher and choose among. The result? Libraries are losing potential users. Librarians license valuable and costly full-text databases that we know contain the information researchers are seeking. But in a three-click world, each vendor's database remains a separate silo of information that our users don't find. Even if patrons are familiar with searching the OPAC, that won't help them retrieve articles. Library services that require training or require the user to come to the library undermine the advantages of licensing electronic content.

Judy Luther -- 10/1/2003, Trumping Google? Metasearching's Promise, Library Journal

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 7

Metasearch issues

• User interface

• Collection selection (search targets)

• Reliable access and connection

• Search functionality

• Presenting results

• Navigating and subsequent searching

• Performance

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 8

Library of Texas

A Virtual LibraryReaching Across Boundaries to

Extend Access to Information Resources

• Collection Boundaries

• Organizational Boundaries

• Format Boundaries

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 9

ZLOT project

• Z Implementation Component of the Library of Texas (ZLOT)– Develop standards-based approach for the virtual library

• Project Focus– Identify requirements for resource discovery service– Demonstrate feasibility and utility of resource discovery

tools– Assist in implementation and testing

• Phase 1 – 2002: Design and specifications• Phase 2 – 2003: Implementation

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 10

Service-centric virtual library

Virtual Library Services

UserGroup

UserGroup

UserGroup

UserGroup

LibraryResources

LibraryTechnology

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 11

Virtual library services

Standard Information Interchange Among Services

Financial

Authorization

ResourceSharing

PatronAccount

LibraryInformation

EducationResourceDiscovery

Reference Personalization

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 12

Resource discovery: Supporting user tasks

Browse

SelectCollection

Evaluate

Search

Common User Interface

Personalize

Save

Acquire

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 13

Resource discovery:Operational view

Content

Content

Content

Content

Content

OAIRepository

Standards-basedSearch & RetrievalProtocols

Proprietary or Non-standard

Search & Retrieval Protocols

Content

Content

Content

CommonSearch

Interface

UserInterface

Texas Library Directory Database

Relational Database To Power the RDS Application

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 14

LOT RDS:Summary of key requirements

• Customizable interface– For local library presence– For personalization

• Common interface for search and retrieval – Simple and advanced searching– Single database searching– Broadcast searching across similar/dissimilar databases

• Tools to help users select search targets• Common display for results• Indications of availability and “how to get”

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 15

Database-driven customization• Texas Library Directory Database (TLDD)

– Multi-use relational database

• RDS functionality driven by TLDD– Association of user with home library– Access/authentication mechanisms– Preliminary set of search targets

• Geographic proximity• Strength of collection

– Interlibrary loan request initiation

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 16

TLDD overview• General Organization Details• Person Details• Address Details• Collection Details

– Collection – describes Collections of resource items or resource item descriptions

– Collection Strength – describes the strength of a Collection

• Service Details– Service – contains information about the Services delivered by an

Organization– Access Policy – describes Access Policies for a given Service– Z Protocol Information - records information about the level of support of a

given Z39.50 Service delivery process

• Common Use– Alias – Supports the recording of names, abbreviations, acronyms, standard

numbers, and symbols used to identify an entity

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 17

Optimizing the RDS

• Assumptions– Multiple user groups – Diversity of information needs, behaviors, and

expectations– Users may not understand distributed search

applications– User interface presents complex functionality

• Strategy to optimize– Usability assessment– Helping users develop new mental model

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 18

Informing the RDS product

• Heuristic evaluation by usability expert

• Acceptance testing by ZLOT staff– Use of original technical specifications– Identified usability issues

• User testing

• Ongoing testing by developers, ZLOT and TSLAC staff

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 19

Usability considerations

• Users– Categorization of users based on demographics,

skills, needs, behaviors– Current library users vs. new users

• Interface is really about functionality– Presents application and its components– Components support various user tasks

• Usability criteria, measures, and procedures

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 20

Usability criteria

• Jakob Nielsen – Learnability– Efficiency– Memorability– Errors– Satisfaction

– Other quality attributes (e.g., utility)

• Whitney Quesenbery– Effective– Efficient– Engaging– Error tolerant– Easy to learn

Nielson, J. (2003). Usability 101.http://www.useit.com/alertbox/20030825.html

Quesenbery, W. (2002). Getting Started: Using the 5Es to understand users. http://www.wqusability.com/articles/getting-started.html

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 21

Application & components

• Technical components

• Interoperability components

• User interface components

• Task components – Collection selection components– Searching components– Retrieval components– Personalization components

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 22

Usability framework

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 23

Priority usability issues

• Do user understand the service?• Are user tasks supported?

– Tasks represented by application functionality– Selection of search targets– Searching

• Simple• Advanced• Refining searches

– Reviewing results– Navigating– Accessing/acquiring

• Is system performance adequate?

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 24

5 Es of usability revisited

ResourceDiscovery

Service

Effective

Efficient

Easy to Learn

Engaging

Error Tolerant

Adapted from Quesenbery, W. (2002). Getting Started: Using the 5Es to understand users.

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 25

Next steps

• Prepare a formal usability assessment • Develop measures based on 5 Es to assess RDS

components• Develop scenarios for users to exercise all

functionality• Conduct user testing• Inform enhancements to the product

• Generalize usability assessment methodology other for metasearch applications, resource discovery services, and virtual libraries

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 26

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 27

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 28

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 29

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 30

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 31

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 32

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 33

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 34

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 35

RDS example

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 36

DC proceedings/Seamark

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 37

DC proceedings/Seamark

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 38

Metasearch usability

• Selecting appropriate search targets– Default set of targets– User selection from target list– Getting contextual info from users for search– Intermediate search against representations of targets

• Making sense of search results– Linear lists of results– Using metadata to categorize results and show

relationships– Hierarchical faceted metadata

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 39

References

• NISO Metasearch Initiative<http://www.niso.org/committees/MetaSearch-info.html>

• Library of Texas Initiative<http://www.tsl.state.tx.us/lot/><http://www.unt.edu/zlot>

• Hierarchical Faceted Metadata and Search– The Flamenco Search Interface Project<http://bailando.sims.berkeley.edu/flamenco.html>

– Faceted Metadata Search and Browse<http://www.searchtools.com/info/faceted-metadata.html>

Moen Access 2003 -- October 3, 2003 -- Vancouver, British Columbia 40

William E. Moen

wemoen@unt.edu

Library of Texas<http://www.tsl.state.tx.us/lot>

ZLOT Project<http://www.unt.edu/zlot/>

Texas Center for Digital Knowledge<http://www.txcdk.org/>