Methods for Improving Organizational Website IA by Barbora Batokova

Methods for Improving Organizational Website IA

WORLD IA DAY PITTSBURGH, PA • 2.20.2016

@WIAD_PGH

BARBORA BATOKOVACARNEGIE MELLON SOFTWARE ENGINEERING INSTITUTE

@bbatokova

DM-0003306 This material has been approved for public release and unlimited distribution. © 2016 Carnegie Mellon UniversityMethods for Improving Organizational Website IA • Barbora Batokova 2 of 42

Copyright 2016 Carnegie Mellon University

This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.

NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.

[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.

This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at [email protected].

DM-0003306


Goals

2 Introduce you to tree testing and how to analyze the results.

CASE STUDY

1 Provide a brief overview of a 3-stage design process for improving a site IA.

PROCESS OVERVIEW

OVERVIEW

Could be Improved Good Example Bad Example


THE CHALLENGE


Fill in the GapsSurvey • Expert Interviews • Competitive Analysis

Assess Current IATree Testing•Think Aloud Protocol•Heatmaps/Click Tracking1

2Create New IAButcher Paper IA•Card Sorting•Click Testing•Tree Testing3

3-Stage Process

OVERVIEW

Developed for SEI’s website redesign project.


Assess Current IA

1


Pies

Lines

Went down the right pathWent down the wrong pathWent backNominated as correct answerSkipped question

Correct pathIncorrect path

Home

Work Areas

Cyber-Physical Systems

Overview

Research

Functional Verification

Real-Time Software

Collaborative Autonomy

Our People

Performance & Dependability

Overview

Tools & Methods

Risk Management

Overview

Tools & Methods

Additional Materials

Security & Survivability

Overview

Getting Started

Research

People

Software Architecture

Tools & Methods

Software Product Lines

Tools & Methods

Systems of Systems

Engage with Us

ConnectConsult

Solutions

Combating cyber security threats from inside and outside the organization

Sponsor

Sponsor

Develop

Products and Services

SEI TrainingTools & Methods

Research

Consulting

Library

News

Overview

SEI Article

News RSS Feed

News Stories

Press Releases

Media Coverage

SEI Bulletin

About Us

SEI Overview

SEI Overview

Why Work with an FFRDC

Year in ReviewOur Organization

Software Solutions Division

Tree Testing


Methodology

ConsiderationsSite Tree Size => No. of Tasks => No. of Tests

Participants• Min 50 for 10 tasks• Site Visitors / Recruitment Service

1: ASSESS CURRENT IA » TREE TESTING

Software• Treejack• User Zoom• C-Inspector


Site Tree6-Level NavGlobal > Main > Drop-Down Menus > Standardized Local > Local > Sub-Local

1

2

3

4

5

6

600+ items}



Organizational FocusTasks (15)

# TASK TASK COMPLETION

9. Find out about what our organization does. 81%4. Get involved with us on a research project. 50%5. Fund a research and development (R&D) project. 42%7. Find out about our upcoming conferences. 37%2. Find out about our latest research. 35%15. Find out about the history of the organization. 0%

Based on website objectives and basic organizational information.

37% Overall Success Rate Usable75%

Good 75% and above

Very Poor 0–49%

Poor 50–74%



Good 75% and above

Very Poor 0–49%

Poor 50–74%

# TASK TASK COMPLETION

3. Download software packages and tools the SEI has created. 76%10.

Find out how the SEI is helping to solve the nation’s cybersecurity challenges. 36%

17.Learn more about SEI participation at approved targeted academic conferences. 26%

2. Find out about the outreach and impact the SEI has had. 18%1.

Find information on what high priority problems the DoD faces and how we help to solve them. 6%

4.Find out how the SEI enables new or extended capabilities in existing military systems. 0%

Tasks (18)Based on web objectives and technical staff input.

20% Overall Success Rate Usable75%

Technical Focus



TREE TESTING

Analyzing Results

Analyzing Results For each question, we created a Results Sheet that summarized the findings.


Results Sheets

Download: http://bit.ly/1RzDIRz

TASK RESULTS

DESTINATIONS

PIE TREE

IMPLICATIONS

FIRST CLICK PATHSDirect Failure

QUESTION#%

Indirect Failure

Indirect Success

Direct Success

Overall Success RateHow many people completed the task successfully? 75% above is acceptable.

Direct FailureIf a participant goes directly to a wrong answer, without ever clicking back up the tree. When someone goes directly to the wrong information, it suggests they have confidence they’re on the right track.

Indirect FailureIf a participant has gone back through the tree at least once before they’ve selected the wrong answer. When a task receives high numbers of indirect fails, it gives a clear message that the labeling and organization of that part of your tree is confusing people.

Direct SuccessIf a participant goes directly to the correct answer.

Indirect SuccessIf a participant has gone back through the tree at least once before they’ve selected the correct answer. This means that organization and labeling can be improved.

Task ResultsBreakdown of success, directness, time taken, and the overall score calculated for each task.

First ClickShows which branches were clicked first for

each task, and what % of participants did so.

DestinationsWhere did participants end up? / Where did they think they would find the answer to the

question? The more there is, the more confused participants were.

Pie TreeShows which way participants went at each

junction in the site tree and what they selected as their final answers. Useful for determining

the breakdowns.

ImplicationsWhat does this mean? What are the takeaways?

How can we improve the site tree based on these results?

Destination Across All TasksNumber of participants that picked this item as

the correct answer for multiple tasks. Each column corresponds to questions. If an

item is picked as the correct answer for multiple questions, it means the item label is vague and

applies to too many things.



Task ResultsBreakdown of success, directness, time taken, and the overall score calculated for each task.



Shows which branches were clicked first for each task, and what percentage of participants did so.

First Click



Shows which way participants went at each junction in the site tree and what they selected as their final answers.

Home

Work Areas

Acquisition Support

Our People


Research

Processor Performance


Overview

Engage with Us

Connect

CollaborateCases Examples: Collaboration ResultsSoftware Assurance Curriculum


SEI Training

SEI Partner Network

SEI Certification

Overview

Calendar of EventsTools & Methods

Research

About Us

SEI Overview

SEI Overview


SEI Statistics and History

Year in Review

Our Organization

Our Organization

Director's Office

CERT Division

Our People

Pie Tree

Pies

Lines

Went down the right pathWent down the wrong pathWent backNominated as correct answerSkipped question

Correct pathIncorrect path

Home

Work Areas

Acquisition Support

Overview

Tools and Methods

Methods

Consulting on Acquisition Issues and Needs

Strategic PlanningCase Studies

Cyber-Physical Systems

Overview

Research

Measurement Analysis


Research

Consulting

Case Studies

Military Aircraft Systems

Weapons Systems Management

Process & Performance Improvement

Getting Started

Research

Accelerated Improvement Method (AIM)

Consulting at the SEI

Process Case StudiesIEEE Computer Society/Software Engineering Institute Watts S. Humphrey Software Process Achievement Award

Risk ManagementResearch

Tools & Methods

ConsultingSecurity & Survivability

Overview

Research


Systems of Systems

Overview

SEI Training in Systems Interoperability Integration of Software-Intensive Systems

Research

Identity Management in SOA Environments

Tools & Methods

ConsultingCase Studies

Ultra-Large Scale Systems

Overview

Engage with Us

ConnectConsult

ConsultSolutions

Effective Use of Software Architecture Provides a Blueprint for Successful Products

Cases

The Army Strategic Software Improvement Program (ASSIP)

Sponsor

SponsorCase Examples: R&D Results

Collaborate

CollaborateCases Examples: Collaboration Results

Cyber Intelligence Tradecraft Project

Architecture Evaluation

Develop


SEI Training

SEI Partner Network

Overview

Find an SEI Partner

Partner Organizations

Find a Partner - Search Results

Partner Detail

SEI Certification

Calendar of Events

Tools & Methods

Research

Consulting Library

SEI Digital Library

SEI Podcast Series

Browse by Category

Cybersecurity Engineering

SEI Webinar Series

News

OverviewNews Stories

Press ReleasesSEI Bulletin

CareersAbout Us

SEI Overview

SEI Overview

Director's MessageYear in ReviewOur Organization

Our Organization


Architecture Practices



Paths

Direct Failure

Indirect Failure

Indirect Success

Direct Success



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15TASK #

Where did participants end up?

Destinations



Visualizing the Results

WORK AREAS

ACQUISITION SUPPORT

Overview

Getting Started

Research

Tools & Methods

Consulting

Our People

CYBER-PHYSICAL SYSTEMS

Overview

Research

Our People

MEASUREMENT & ANALYSIS

Overview

Getting Started

Research

Tools & Methods

Consulting

Case Studies

ABOUT US

SEI OVERVIEW

DIRECTOR’S MESSAGE

OUR ORGANIZATION

Director’s Offi ce

CERT Divsion

Emerging Technology Center


OUR PEOPLE

Sponsoring and Oversight

Leadership Team

SEI Fellows

YEAR IN REVIEW

PRODUCTS & SERVICES

SEI TRAINING

Overview

Find a Course

Classroom

Onsite Training

E-Learning

Registration

FAQ

Certifi cates

SEI PARTNER NETWORK

SEI CERTIFICATION

CALENDAR OF EVENTS

TOOLS & METHODS

RESEARCH

CONSULTING

LIBRARY

SEI DIGITAL LIBRARY

SEI PODCAST SERIES

SEI WEBINAR SERIES

NEWS

NEWS STORIES

ANNOUNCEMENTS

PRESS RELEASES

MEDIA COVERAGE

SEI BULLETIN

CAREERS

OVERVIEW

AFFILIATE PROGRAM

ENGAGE WITH US

OVERVIEW

CONNECT

CONSULT

Solutions

Cases

SPONSOR

Case Examples: R&D Results

COLLABORATE

Case: Examples: Collaboration Results

DEVELOP

CMU SEI CERT DIVISION DIGITAL LIBRARY INSIGHTS

Ambiguous Meaning Problematic Placement

Global > Main > Drop-Down Menus …

Ambiguous Meaning Problematic Placement



Think AloudProtocol


Methodology

1: ASSESS CURRENT IA » THINK ALOUD PROTOCOL

Softwareusertesting.com, User Zoom, Usability Tools, etc.


What does this mean? What are the takeaways? How do we fix it?

Implications

1: ASSESS CURRENT IA » THINK ALOUD PROTOCOL & TREE TESTING


Heatmaps &Click Tracking


1: ASSESS CURRENT IA » HEATMAPS / CLICK TRACKING

Softwarecrazyegg, ClickTale, ClickHeat, etc.

sei.cmu.edu • 60 days 48 mins • 25,000 visits • 15,698 clicks

Methodology


Fill in the Gaps

2


Survey

Distribution Statement A: Approved for Public Release; Distribution is Unlimited © 2016 Carnegie Mellon UniversityMethods for Improving Organizational Website IA • Barbora Batokova 27 of 42DM-0003306 This material has been approved for public release and unlimited distribution.

© 2016 Carnegie Mellon UniversityMethods for Improving Organizational Website IA • Barbora Batokova 27 of 42

To gather types of information that are currently not represented on the site.• What important content is missing?• What content areas are obsolete and should be archived?• Are our clients looking for information that we don’t have online?

Benefits• Quick way to gather a lot of useful data.• Involve the entire organization and clients in the design process.

Why do a Survey?

2: FILL IN THE GAPS » SURVEY


ExpertInterviews



To identify business needs not being met within the structure.• Where is the organization headed and how does the website fit in?• How can the website IA help express the direction of the organization?

Benefits• Cheap in terms of effort and budget.• Quick way to get expert information directly from the source.• Ensure business needs are being met.

2: FILL IN THE GAPS » EXPERT INTERVIEWS

Why do Expert Interviews?


CompetitiveAnalysis



MethodologySteps1. Identify your competitors.2. Define evaluation framework.3. Set up spreadsheet to record findings.4. Visit each site, record findings and grab screenshots.5. Analyze and document results.6. Create recommendations.

Participants• Project Team

CompetitorsIdentify your competitors.

Screenshots

FrameworkWhat are you evaluating?

Spreadsheet

2: FILL IN THE GAPS » COMPETITIVE ANALYSIS


CreateNew IA

3


ButcherPaper IA


3: CREATE NEW IA » BUTCHER PAPER IA

MethodologySet Specific Goals


CardSorting


MethodologyItems Categorization

• Open or closed card sort?• Hybrid?

Software• OptimalSort• User Zoom• UsabilityTools ...

ResultsSimilarity Matrix

3: CREATE NEW IA » CARD SORTING

Dendrograms


ClickTesting


Measuring ImprovementProposed IA

3: CREATE NEW IA » CLICK TESTING


Measuring ImprovementProposed IA + Heatmap

Compare: Original First Click 72%

118%Improvement33%News

3: CREATE NEW IA » CLICK TESTING


Home

Work Areas

Acquisition Support

Our People


Research

Processor Performance


Overview

Engage with Us

Connect

CollaborateCases Examples: Collaboration ResultsSoftware Assurance Curriculum


SEI Training

SEI Partner Network

SEI Certification

Overview

Calendar of EventsTools & Methods

Research

About Us

SEI Overview

SEI Overview


SEI Statistics and History

Year in Review

Our Organization

Our Organization

Director's Office

CERT Division

Our People

TreeTesting


Additional SourcesWeb Usability: http://webusability.com/firstclick-usability-testing/

Boxes and Arrows: http://boxesandarrows.com/tree-testing/

Universal Methods of Design, Bella Martin and Bruce Hanington

Optimal Workshop Blog: http://www.optimalworkshop.com/blog

Communicating Design, Dan M. Brown

Software Tools Used• UserTesting• Optimal Workshop

• Treejack• Optimal Sort

• crazyegg• UsabilityHub

Other Software Tools• ClickHeat• ClickTale• Loop11• Optimizely• Qualtrics

• SurveyMonkey• UsabilityTools• UserZoom

RESOURCES

http://bit.ly/1RzDIRzResults Sheet Diagram


Thanks!Do you have any questions?

BARBORA BATOKOVA [email protected] @bbatokova

Methods for Improving Organizational Website IA by Barbora Batokova

Design

Transcript of Methods for Improving Organizational Website IA by Barbora Batokova