Architectures forExpanded Usability: A

Research Agenda

Sheila O. DennDoctoral Candidate

School of Information and Library ScienceUniversity of North Carolina at Chapel Hill

December 8, 2005

The The GovStat GovStat ProjectProjectFind what you need,understand what you find.

Integration of Data and Interfaces to Enhance HumanUnderstanding of Government Statistics:Toward The Statistical Knowledge Network

Outline

Usability as an evolving concept Software usability Information usability

Information usability research perspectives Metadata Human-computer interfaces

Information usability evaluation An ongoing research agenda

Usability

Defined in various ways, but usuallyconcerned with the following qualities (fromNielsen): Learnability Efficiency Memorability Errors Satisfaction

Usability (cont.)

Traditionally has been a software engineeringconcern

Emphasis largely on software functionality . . . But what of the Web, where the software

(the browser) is the same for eachinformation source, but the content isdifferent?

Information Usability

My definition: the extent to which contentencountered by a user is able to beunderstood and manipulated by that usersuch that she can make correct inferencesabout the meaning of that content

Content InterfaceMetadata User

Represented in Rendered by Presented to

Requests fromQueriesRetrieves

Content InterfaceMetadata User

Represented in Rendered by Presented to

Requests fromQueriesRetrieves

Metadata Research “Integration of Data and Interfaces to

Enhance Human Understanding ofGovernment Statistics: Toward the NationalStatistical Knowledge Network” (NSF GrantEIA 0131824) aka The GovStat Project

“Criminal Justice Statistics IntegratingMetadata Application Profile” (US Bureau ofJustice Statistics Grant 2005-BJ-CX-K016) aka The CJS I-MAP Project

Goals of the GovStat Project

To create an integrated model of user accessto and use of US government statisticalinformation (The Statistical KnowledgeNetwork)

To design and test prototype interface toolsto support finding and using statistics

To support integration (technical andintellectual) of statistical data

GovStat SKN Data FlowAgencies

SKN RegistryActions

Contribute Find

Display Annotate

UnderstandManipulate Collaborate

…..

………….

Objects Actions

Private Work Space

Objects Actions

Private Work Space

Ontology Rules & Constraints

SKN Consortium

…..

Objects Reports metadataTables metadataPeople metadata

GlossaryAnnotations

Objects Actions

Private Work Space

Objects Actions

Private Work Space

Metadata in the GovStatProject Conducted initial user study to investigate how metadata

affects the use of US government statistical websites Modeled metadata based on results of user studies Built initial data model based on DDI and ISO/IEC 11179 Developed a strategy to “test” and further develop the schema Tested mark-up via a scenario Through the markup process, determined that there was too

much complexity in the data model for representing tabulardata; developed a streamlined data model in response

GovStat Metadata User Study

Goals and research questions How do experts and end users undertake

information integration activities? What kinds of metadata do experts and end users

need to complete these activities? What are the implications of what we learn about

these activities on developing tools and interfacesto support integration?

Methodology

Scenario-based design to develop tasks Interviews with 14 expert users, 9 student

users where they were given tasks tocomplete

Think-aloud protocols, follow-up interviews Extraction of themes and stories to illustrate

metadata needs

Results

User Knowledge Surveys and Statistics Interpretation of Information Date Issues Geographic Issues Navigation Content Layout Terminology

Integration Activities Make comparisons across

geographic units definitional differences within concepts and variables units of time different information sources index values

Note discrepancies Manipulate statistics

Integration Challenges Lack of definitions or source information Lack of user knowledge of

Appropriate strategies for using statistics Nature of index values and their use Nature of survey/census purpose and approach Domain and social science statistics

processes/approaches/definitions Interface design problems including interface

inconsistency Inconsistent data across sources User inability to determine if statistics were available Terminology differences across sources

Can Metadata Help? Common user problems that can be addressed

through metadata Mapping of agency terms to user terms Definitions of statistical/survey terms Comparability of statistics Help with finding and interpreting statistics Information about recency of statistics, update schedules,

and propagation of updates to websites Geographic levels at which statistics available Navigation by means other than large lists of text links

Metadata SchemaDevelopment Process Identified and modified/expanded existing elements within

standards (such as the Data Documentation Initiative (DDI)) tofit our needs

Discussed our initial element set and created DTD Used scenarios to choose sample tables for markup Revised based on discussion and samples Held a metadata symposium with metadata experts to help

refine our process and our elements Converted the DTD to an XML schema Revised schema to accommodate aggregate data in a more

streamlined fashion

Current State of the GovStatSchema The current metadata model includes:

Effort to balance complexity with functionality Removal of elements designed to align data values

and row/column headings with survey variables Ability to add on to the model to represent

additional information using a “hierarchy ofintegration” (which will be discussed shortly)

Schema and documentation can be found athttp://ils.unc.edu/govstat/metadata

GovStat Metadata Philosophy

To provide sub-document level access andintegration across documents and agencies.

To provide a minimal set of metadataelements necessary while allowing forextensibility.

To achieve these goals in a manner thatenables efficient transfer to agencies.

Hierarchy of Integration

Low level ofintegration

High level ofintegration

• Searchable table titles

• Searchable row and column headings

• Linking of data values to row andcolumn headings

• Linking of row and column headingsto underlying survey variables

• Linking of analysis units, universestatements, concept definitions, acrossdocuments and agencies

• Linking of contextual information(such as footnotes) to tables,row/column headings, or data values

Our schema canprovide the itemsbeneath this dottedline.

Hierarchy of Integration

Low level ofintegration

High level ofintegration

• Searchable table titles

• Searchable row and column headings

• Linking of data values to row andcolumn headings

• Linking of row and column headingsto underlying survey variables

• Linking of analysis units, universestatements, concept definitions, acrossdocuments and agencies

• Linking of contextual information(such as footnotes) to tables,row/column headings, or data values

Our schema canprovide the itemsbeneath this dottedline.

Organizations can determine wherethey “sit” on the hierarchy in terms ofeffort required and level of integrationdesired.

Goals of the CJS I-MAP Project

Integration and synchronization of criminaljustice statistical information between BJSand its agency partners through thedevelopment of a metadata application profile

Emphasis on facilitating end user discoveryof relevant information items across disparatesources

CJS I-MAP Deliverables A Metadata Application Profile (MAP) to facilitate

metadata sharing and reuse Primarily directed towards supporting users’ discovery of

data Evaluation of the MAP

Technical review User-oriented studies

Final report Methodology for MAP development User studies report

Recommendations for implementing and using theprofile at BJS and partners

CJS I-MAP Project Approaches

Partnering with agencies Theoretical perspectives/approaches

Supporting statistical information integration andthe role of metadata

Usage of metadata: key findings and questions Strategies for building a metadata application

profile

Partnering with Agencies Agency partners include:

Bureau of Justice Statistics (BJS) Federal Bureau of Investigation (FBI) Federal Justice Statistics Resource Center (FJSRC) National Archive of Criminal Justice Data (NACJD) National Center for Juvenile Justice (NCJJ) Office for Juvenile Justice and Delinquency Prevention

(OJJDP) Sourcebook for Criminal Justice Statistics

Partnering with Agencies(cont.) Project goal is to develop a “real-world” solution for

the agencies that enables us to extend the state ofthe art/knowledge in metadata design andimplementation

Through these partnerships we will achieve: Exchange of expertise Ongoing conversations (including some real-time, face-to-

face) Intermediate products/insights on which we will ask for

feedback Joint development of user study research questions

MAP Strategies Identify existing schemas from which we may

be able to draw elements: Data Documentation Initiative (DDI) Federal Enterprise Architecture Data Reference

Model (FEA DRM) Global Justice XML Data Dictionary (GJXDD) GovStat ISO/IEC 11179 Statistical Data and Metadata Exchange (SDMX)

MAP Strategies (cont.)

Identify relevant elements from each schema Identify any necessary elements that cannot

be pulled from other schemas Integrate elements from schemas, fully

documenting any new elements created Process of ongoing iteration with partners Validate with user studies

Future Directions for MetadataResearch Achieve better understanding of trade-offs between

costs of metadata creation and user/agency benefitsin various tasks

Assess hierarchy of integration for Supporting agency decision making Supporting user understanding

Understand extent to which metadata usage isdomain-specific Or when and how can we generalize about metadata

usage, metadata system design

Human-Computer InterfaceResearch

“Toward More Effective Information Use:Fostering Cognitive Integration via theHuman-Computer Interface”

Dissertation project Proposal approved September 2005

Background Access to more information than ever before Effective use of information more difficult Information use requires information

integration What do we know about cognitive process of

information integration? Can we use what we know to improve

information interfaces? What does “improve” mean?

Theoretical Framework Educational psychology

Bloom’s Taxonomy of Educational Objectives (Bloom, etal., 1964) One source of evidence of “understanding” is that learner

can combine information at hand with previous knowledgeto create a new understanding

Cognitive psychology Cognitive Load (Feinberg & Murphy, 2000; Sweller, et al.,

1990) Scaffolding Theory (Clark, 1997) Distributed Cognition (Hollan, et al., 2000; Hutchins, 1995;

Norman, 1993.)

Methodology Extensive literature review to identify promising

interface components Securing agreement to use a particular interface

component (the Hunter Gatherer interface built byschraefel, et al.)

Creation of tasks for user study Pilot testing of experimental interface component

and tasks Revision of tasks as necessary Full testing of experimental interface vs. regular

browser interface

Experimental Interface Hunter Gatherer interface (schraefel et al., 2002) Designed as an integrated module to a Web

browser that allows the user to cut and paste piecesof a web page into a “collection”

Idea was that this could be done within a browserwindow, without the user having to exit the browserto use a text editor

Acts like a combination bookmarking tool and editor

Subject Tasks

Subjects will be recruited through UNCCareer Services

Tasks will require students to imaginethemselves as career consultants forstudents of a particular major, andrecommend one of two cities based onstatistical information that can be found onthe Web (such as at the Bureau of LaborStatistics)

Study Artifacts and Analysis Subjects will be asked to complete a brief pre-study

questionnaire about basic demographic characteristicsand experience with the Web

Subjects will be asked to write a brief statement outliningwhich city they would recommend, annotated with datafound in their search

Subjects will be given a usability questionnaire, askingabout the efficacy of the interface they used and theirsatisfaction with the results

Subject statements will be analyzed by the researcher interms of Bloom’s synthesis and evaluation levels

Experts in UNC Career Services will analyze subjectstatements, and these analyses will be compared withthe researcher’s analyses to ensure inter-rater reliability

Expected Results Results should show if the Hunter Gatherer interface

has any advantage over the standard browserinterface in terms of subjects’ understanding of thedata they find.

If there is a positive effect, then this could be a basisfor recommending the Hunter Gatherer module ininterfaces to complex information.

This research should also demonstrate whether theuse of the concepts of synthesis and evaluation fromBloom’s taxonomy are useful for evaluatinginformation usability.

The Research Agenda Integration is key

In more and more contexts, there is a need for theability to bring together data from disparatesources. Data Integration refers to those activities that occur

on the back-end, such as aggregating selected datafrom distributed sources into a centralized repository.

Information Integration refers to the cognitiveprocesses undertaken by an individual user tosynthesize disparate bits of information intounderstanding.

The Relationship betweenMetadata and Integration

Standard metadata schemas can enhancedata integration by allowing crawlers orother kinds of agents to pull materials fromsources based on the values of metadataelements.

Standard metadata schemas can alsoenhance information integration byallowing users to group data according to thevalues of metadata elements.

The Relationship betweenInterfaces and Integration

Interfaces can be used to provide portals toinformation from disparate sources, or indifferent formats, so that the informationappears integrated from the user’sperspective.

Interfaces can be used to providefunctionality for users to integrate informationfrom disparate sources, or in differentformats, themselves.

Future Directions Explore and describe the relationships between

human cognition and human-information interaction. Explore and describe the relationships between

information architectures and human-informationinterfaces.

Expand the notion of usability to include informationusability. Explore what it means for information to be usable. Explore how different visualizations of information impact

information usability. Move toward greater theoretical foundations for

human-information interaction.

For More Information

Sheila O. Denndenns@ils.unc.edu

http://ils.unc.edu/~denns/http://ils.unc.edu/govstat/

The The GovStat GovStat ProjectProjectFind what you need,understand what you find.

Integration of Data and Interfaces to Enhance HumanUnderstanding of Government Statistics:Toward The Statistical Knowledge Network