Test Plan - Usability Research

08

USABILITY TESTING TTU CURRENT STUDENTS WEB PAGE

USABILITY Report

The A-Team

USABILITY TESTING 2008

of 90

Contents Executive Summary ....................................................................................................................................... 4

Introduction .................................................................................................................................................. 5

Purpose ..................................................................................................................................................... 5

Problem Statement and Test Objectives .................................................................................................. 5

Discovery ................................................................................................................................................... 6

Methodology ................................................................................................................................................. 6

User Profiles .............................................................................................................................................. 6

Tasks .......................................................................................................................................................... 7

Think-Aloud Protocol ................................................................................................................................. 8

Web page Evaluation Tasks ...................................................................................................................... 8

Test Environment ...................................................................................................................................... 9

Evaluation Site .......................................................................................................................................... 9

Observation Area ...................................................................................................................................... 9

Evaluation Methods ...................................................................................................................................... 9

Results ......................................................................................................................................................... 10

Taxonomy ................................................................................................................................................ 10

Navigation ............................................................................................................................................... 10

Information Hierarchy............................................................................................................................. 11

Content ................................................................................................................................................... 12

Prototype Analysis .................................................................................................................................. 12

recommendations ....................................................................................................................................... 12

Appendix A- Pre-Test Survey ....................................................................................................................... 13

Appendix B- Texas Tech University Fall 2007 Fact Book ............................................................................. 14

Appendix C- Student Informational Survey ................................................................................................ 15

Appendix D- Post Test Questionnaire ......................................................................................................... 16

Appendix E- Observation Logs .................................................................................................................... 17

Appendix F- Task List ................................................................................................................................... 18

Appendix G- Think Aloud Protocol Warm-up ............................................................................................. 19

Appendix H- Consent Form ......................................................................................................................... 20

Appendix J- Equipment ............................................................................................................................... 22


of 90

Test Equipment ....................................................................................................................................... 22

Lab Equipment ........................................................................................................................................ 22

Appendix K- Photos of Lab Set-up............................................................................................................... 23

Appendix L- Student Informational Surveys ............................................................................................... 24

Appendix M- Observation Logs ................................................................................................................... 48

Appendix n- Pre-Test surveys ..................................................................................................................... 68

Appendix O- Consent Forms ....................................................................................................................... 73

Appendix P- Post test Questionnaires ........................................................................................................ 78

Appendix Q- Prototype Photographs .......................................................................................................... 83


of 90

EXECUTIVE SUMMARY


of 90

INTRODUCTION

PURPOSE

This study evaluated the usability of the existing TTU “Current Students” web page. The Texas Tech

University Office of Communications and Marketing would like to assess the ease with which current

students access and navigate the page. Additionally, the office also seeks to understand whether the

existing content meets the needs of the users. For this test, the primary client is Senior Editor of Web

Communications, Lisa Low.

PROBLEM STATEMENT AND TEST OBJECTIVES

In order to address the client’s concerns, the goals for this study of the Current Students web page are

to determine navigation efficiency, improve site taxonomy, and provide analysis for possible redesign.

The following areas are identified as problematic:

Users experience difficulty navigating throughout the page

Users express difficulty locating links to their desired destination

The client expressed interest in improving the aesthetic appeal of the page to users

In order to effectively address these issues, the objective of the study is to provide qualitative and

quantitative data gathered from representative users testing the page’s usability. In response to the

aforementioned problems we present the following objectives:

Assess the difficulty of the TTU “Current Students” web page navigation

Categorize web page information to improve efficiency of use

Gather information on content preferences from representative users

Gauge user satisfaction with web page experience


of 90

DISCOVERY

For this test our job was to evaluate the usability of the “Current Students” web page at TTU. The first step of the testing process was to contact the client and determine current issues and information on the site. Our client, Lisa Low, identified her concerns as being the ease of navigation, lack of content organization, and user satisfaction with the page. The client also noted that the page is the second most visited page on the Texas Tech web page, gathering 125,839 hits in the period from May 28, 2008 to August 24, 2008, as compared to the main Texas Tech index (www.ttu.edu) with 1,049,544 for that same period.

In order for the team members to familiarize themselves with the object of the study, each member of our team explored the page and the links to see what they contained. During this observation, group members generalized that the page is merely an alphabetical list of links to other pages, providing no textual information. Team members also created and distributed an informational survey (Appendix C) to Texas Tech students to gather data about how they currently use the page and with what frequency. Findings from the survey suggest that while a large percentage of the student population used the page, less than ten of the listed links were being utilized with great frequency.

Team members met to discuss the findings of these two activities. We identified specific issues and drafted a paper prototype test to further investigate the usability of the “Current Students” page. During the paper prototype test, the user was asked to look at the current page and then, using paper and writing utensils, create the type of page they would like to see geared towards current Texas Tech students. The results of the test were combined with the other discovery activities to gain better understanding of the users and the page content.

The team leader conducted another meeting to assess all the data gathered. From this information we identified possible user groups and tasks appropriate for the usability test. To make our task set, team members generated original ideas and upon further discussion narrowed the list down to final tasks we believe typical of the majority of users. We also decided to create a task that allows the user to create and design his or her idea of the perfect “Current Students” web page. To determine the representative user, the group analyzed demographic information from the Texas Tech University Fact Book for Fall 2007 (Appendix B) which was suggested by the client. The representative user was also determined based on the results of the informational survey (Appendix C).

METHODOLOGY

USER PROFILES

Indicative of the web page’s title, the users can be any person enrolled at Texas Tech University.

Through communication with the client, in addition to data gathered from the university fact book, we

identified several diverse groups of potential web page users:


of 90

Current Students

o Undergraduate

o Graduate

o Transfer Students

o Law Students

o International Students

o TTUHSC Students

After researching all possible user groups, we determined it best to create a representative user based

on demographic information from the university fact book, the Internet, and data gathered from the

survey. Therefore, the most represented user of the TTU “Current Students” web page displays the

following characteristics:

Full-Time Undergraduate

Ranging in age from 20-22

Surfs the internet at least 5 hours a week

Has previously used the “Current Students” page

Visits the Current Students page for at least three of the following:

o TechSIS

o Tech Mail

o Libraries

o University Calendar

o Financial Aid

o Athletics

We will verify the users embody the previously stated characteristics through a pre-test survey

(Appendix A).

TASKS

Testing were conducted on November 4th and 6th, 2008 in the Usability Research Lab located on the

third floor of the English Building at Texas Tech University campus. While users are not testing the web

page, they will waited either in an area outside the lab or in a classroom in the English Building. By

completing the following tasks the team was able to evaluate the taxonomy and aesthetics of the page.

Although we recognize that these exercises may not be typical tasks performed by the users, we feel

that they accurately gauged both user satisfaction and page usability. From start to finish, the test will

took about 30 to 45 minutes for each participant.


of 90

THINK-ALOUD PROTOCOL

So that team members can have insight to the users’ thoughts and reasoning process, the test facilitator

employed Think-Aloud Protocol. Because this technique may be unfamiliar and seem unnatural to

participants, they al completed a think-aloud warm-up activity. The user was seated at a desk with a

computer. There was a web browser opened to the Google search engine. The facilitator will explain the

following task to the user:

“You are currently a student in a fiction literature class. Your teacher requested that you purchase To Kill

a Mocking Bird, but forgot to order it from the book stores in the surrounding area. Locate a web page

to purchase the book, and then locate your desired item. Finally add the book to the cart. When you are

done, please say ‘Finished.’”

During the warm-up exercise, the facilitator prompted the user to think out loud while completing the

tasks. The facilitator asked questions including but not limited to: What thoughts are going through your

head right now? Why did you choose that link? What are you looking for on this page? Why did you

search those key words? Where did you expect this link to take you?

Having users walk through this basic, plausible scenario while employing think-aloud protocol ensured

that our users will be able to successful think-aloud throughout testing procedure.

WEB PAGE EVALUATION TASKS

To meet the needs of the client and to successfully accomplish the test objectives, users were asked to

complete a series of tasks. The first set of tasks instructed the user to navigate through the existing

“Current Students” web page having them locate specific information from different links currently

listed. The last task asked the user to construct their ideal “Current Students” page, using the prop set

(Appendix K) provided. The tasks will be as follows:

1. You are interested in attending the Celtic Christmas celebration, but do not know the date and

time. Use the “Current Students” web page to locate this information.

2. Return to the “Current Students” web page. You want to be involved in more student

organizations. Because you recently learned to play billiards, you would like to get in touch Tech

Billiards Club to see when the group’s next meeting is. Locate the relevant information.

3. Return to the “Current Students” web page. You have a test coming up and want to study at the

library. Find out when it is open.

The purpose of these first three tasks is to measure efficiency, errors, learnability, and satisfaction.

These will be measured by: time taken to complete the task, errors made while completing the task,

think-aloud responses, and facial expressions.

The fourth and final task asked the user to create their own version of the “Current Students” web page

based on their personal preferences and needs. The user was provided with a white board on which to

create the page. They had at their disposal different magnets symbolizing links, pictures, and graphics.

Users also had the tools provided to create any other magnet that they want but was not provided. This


of 90

task measured memorablity of the existing page by comparison of the “Current Students” page versus

their creation. Satisfaction was also be measured by how many changes the user made in their re-

designed page.

TEST ENVIRONMENT

The test took place in the Usability Research Lab at Texas Tech University. The lab consists of two rooms; one for observation and another for conducting tests. For this test, we utilized both spaces. In addition to the equipment provided in the lab, the team also brought in additional items to facilitate the tasks.

EVALUATION SITE

For the first part of the test, the user sat at a desk that faces a two-way mirror. The desk had a computer for the user to complete the tasks. On the opposite side of the room there was a white board for the user to create a personal prototype of a current students page. Next to the board, there was a table with various items (Appendix J) brought in by the group to represent links, pictures, graphics, etc. After the user completes the test and has left the room, the Observer photographed the prototype. Throughout the test, think aloud protocol was recorded on microphones in the room (Appendix K).

OBSERVATION AREA

Behind the two-way mirror, all the recording equipment is housed. During the test, the Observer and Technician were in the Observation Area. The Observer recorded relevant information on logs (Appendix E) created specifically for this test. The output for all cameras, microphones, and other recording devices are located here. During the first part of the test, the screenshot of the user was recorded. In the second part of the test, the overhead ceiling camera recorded the user’s activity (Appendix K).

EVALUATION METHODS

To gather the information we utilized qualitative and quantitative methods of evaluation. Our

evaluation methods are constructed based on MEELS mentioned above.

Qualitative Evaluation

Think-Aloud Protocol

Personal Prototype Content

Post-Test Questionnaire

Observation Logs (Appendix E) using Dumas Redish scale

Qualitative data interpretation sheet (Appendix L)

Quantitative Evaluation

Number of graphics and links on the user prototype

Time spent navigating the page


of 90

Pre-Test Survey

Number of errors made completing the task

Time when user gives up on any task

RESULTS

As a result of the testing, we discovered 19 issues that were categorized into four problem areas:

Taxonomy, Navigation, Information Hierarchy, and Content. Taxonomy examines the categorization and

structure of information on the web page. Navigation explores how the participant goes through the

page while completing the tasks. Information Hierarchy discusses how participants ordered information

on their personal prototypes. Content examines user satisfaction with the amount of information on the

web page.

TAXONOMY

As it stands, the Current Students web page is laid out

so that links are alphabetically listed in two columns. In

post-test interviews participants often commented

that lists were easier to navigate. In creating

prototypes, all participants organized information in

list format, but did not organize them in alphabetical

order.

Also in creating their prototypes, only two of the five

participants made headings for information. In post-

test interviews, those participants who did create

heading on their prototype commented that with less

information headings are not necessary.

NAVIGATION

While participants were completing the computer-based tasks, many avoided using the content on the page to gather the information necessary to complete the task. Participants frequently used the search toolbar located in the upper left corner of the page. One participant became so frustrated with the first task, that after searching within the TTU web site for more than two minutes she abandoned the TTU web site in favor of Google and eventually abandoned the task all together. Ultimately the search toolbar was used for 1/3 of the computer-based tasks.

Table 1.1 Occurrence of Participants Using Search Toolbar

User Task 1 Task 2 Task 3

Photo 1.1 Example Prototype

Appendix


of 90

1 X X

2

3 X

4 X X

5

Users used the search toolbar for 1/3 of the tasks

The completion rate for the computer based was sub par. Only two out of five participants completed all

of the tasks. Based on completion and time taken for the task, the first task had the lowest completion

rate with three of the five participants failing the task while the third task had the highest rate of

completion with all participants completing the task.

Table 1.2 Task Completion

Participant Task 1 Task 2 Task 3

1 X X

2 X X X

3 X X

4 X

5 X X X

Participants were not able to efficiently navigate through the page on the more challenging tasks and opted to utilize the search toolbar.

INFORMATION HIERARCHY

Based on post-test questionnaires, interviews, and comments participants made during the construction o f their prototypes, participants ordered information differently than it is organized on the current page. We found that participants ordered their preferred content according to their frequency of use and relative importance. After analyzing the personal prototypes, the most popular links for the participant were: Library, Calendar, TechMail, and Techsis. These findings are congruent with the responses to the Student Informational Survey administered prior to testing


of 90

(Appendix ) During prototype construction, a majority of users commented that it was important to have the most frequently used links at the top of the page. CONTENT

The final area of suggested improvement is the amount of content on the page. The majority of participants, when first starting the computer-based tasks, commented on the general aesthetics of the page. In addition to calling the page “messy” and “overwhelming,” one participant was so disconcerted with the amount of content on the page that she said “I don’t even know where to start” (Appendix ). In the post-test questionnaires, three out of the five participants disagreed with the statement that said there was an appropriate amount of content on the current page. This claim is substantiated by the problems participants had completing the first task of finding the date and time of a TTU event. Three of the five participants used the Academic Calendar link instead of the appropriate Campus Calendar link. We understand the need for separate calendars and feel that the issue can best be solved with renaming the Campus Calendar link to Events. PROTOTYPE ANALYSIS

From the prototypes the participants created, there were four key areas of difference (Appendix ).

Photographs: On average participants used 2.6 photographs, contrasting the zero photographs currently on the page

Links: Participants utilized an average of 11.4 links compared to the 59 links on the current page

Graphics: All participants used graphics, with an average of 1.4 graphics among the prototypes

Headings: Two of the five participants created headings for information on their prototype

RECOMMENDATIONS

Based on the findings of the test, we have derived two possible recommendations to improve the

aforementioned issues.

The first option for improvement is to categorize the links on the page according to importance of

information and frequency of use. This is based on the users frequently commenting that they

themselves organized the links on their prototype with these criteria. The results of the Student

Informational Survey and those of the test reveal that there is already a set of links that that fit these

criteria.

The second option to improve the page is reduce the amount of links so that only those most important

to the user and those that are most frequently used appear the page. This recommendation is based on

participants’ comments during the tasks and their responses on the post-test questionnaires and

interviews. The results of the Student Informational Survey and the links used in the prototypes show a

clear pattern as to which links the representative user group feels are most important.


of 90

APPENDIX A- PRE-TEST SURVEY

Pre-Test Survey

Name:

E-mail:

Age:

Please Check One:

Undergraduate Student Graduate Student

Full-time Part-time

I am available:

Thursday, October 30, 2008 at 4 p.m.

Tuesday, November 4, 2008 at 4 p.m.

If you are unavailable either of these dates, please disregard the rest of this survey.

1. How many hours a week are you on the Internet?

0-4 5-7 more than 7

2. Have you been to the “Current Students” web page?

Yes No

3. If so, why do you visit the page?

TechSIS

TechMail

Library

University Calendar

Financial Aid

Athletics Information

Other:


of 90

APPENDIX B- TEXAS TECH UNIVERSITY FALL 2007 FACT BOOK


of 90

APPENDIX C- STUDENT INFORMATIONAL SURVEY

TTU Current Students Informational Survey

Age: Major: Classification: Gender: □ M □ F

Please check all that apply:

□ Transfer Student

□ International Student

□ Graduate Student

□ Undergraduate

□ Full-time

□ Part-time

1. How many hours a week do you spend on the internet?

□ 0-1 □ 2-4 □ 5-7 □ More than 7

2. How many times a week do you visit the www.ttu.edu web site?

□ 0-1

□ 2-4

□ 5-7

□ More than 7

3. Have you visited the “Current Students” web page?

□ Yes □ No

4. If yes, how many times a week do you visit the page?

□ 0-1

□ 2-4

□ 5-7

□ More than 7

5. What do you access the page for?

6. How much time do you spend on the page?

APPENDIX D- POST TEST QUESTIONNAIRE

Post Test Questionnaire

Please circle the response that best reflects how much you agree or disagree with each of the following

statements:

The existing “Current Students” web page was populated with useful information.

Strongly Agree Agree Neutral Disagree Strongly Disagree

It was easy to locate information I needed on this page.


The existing page has information that current students need.


The page has an appropriate number of links to be convenient to use.


Please answer the following questions:

1. What is most important for you to have on a current students page?

2. What was the most significant change between your page and the existing one? Why did you

make t his change?

3. What do you think would make the page easier for students to use?

4. Is there anything else you would like to have on the current students page you created that was

unavailable to you during the test?


of 90

APPENDIX E- OBSERVATION LOGS

Dumas Redish Scale User Number: 1. Prevents completion of a task 2. Creates significant delay/frustration 3. Has a minor effect on usability 4. Subtle problem; points to future enhancement

Task Action DR # Time Comment


of 90

APPENDIX F- TASK LIST

Please complete the following tasks. Please let me know when you have completed each task.

1. You are interested in attending the Celtic Christmas celebration, but do not know the date and

time. Use the “Current Students” web page to locate this information.

2. Return to the “Current Students” web page. You want to be involved in more student

organizations. Because you recently learned to play billiards, you would like to get in touch Tech

Billiards Club to see when the group’s next meeting is. Locate the relevant information.

3. Return to the “Current Students” web page. You have a test coming up and want to study at the

library. Find out when it is open.

For the fourth task please use the white board and the materials provided on the table to create your

ideal version of the “Currents Students” web page. I want to stress to you that there are no restrictions

to the page you create. You are free to put anything on or take anything off the page that you want.

Please let me know when you have finished creating your idea of the perfect page.


of 90

APPENDIX G- THINK ALOUD PROTOCOL WARM-UP

Think Aloud Protocol Warm-up

You are currently a student in a fiction literature class. Your teacher requested that you purchase To Kill

a Mocking Bird, but forgot to order it from the book stores in the surrounding area. Locate a web page

to purchase the book, and then locate your desired item. Finally add the book to the cart. When you are

done, please say ‘Finished.’


of 90

APPENDIX H- CONSENT FORM

Consent Form

Purpose: We are asking you to participate in a usability study for the “Current Students” web

page. By participating in this study, you will help us make the web page easier to use and help

populate it with relevant content

Testers: This test is being conducted as part of a class project for ENG 3367: Usability testing.

The people conducting the test are all enrolled in this course.

Environment: The study takes place at TTU’s usability laboratory, where you will be observed

as you use the site.

Information collected: We will record information about how you use the web page. We will

ask you to fill out questionnaires and surveys. We will use the information you give us, along

with the information we collect from other people, to recommend ways to improve the web

page.

Videotape Waiver: Your work with the site and the interviews will be videotaped. By signing

this form, you give your consent to use your voice, verbal statements, and videotaped pictures

for the purpose of evaluating the site and showing the results of our testing. However, your

data will always be reported anonymously, i.e. your name will not be revealed.

Comfort: You may take a break at any time you wish. Simply inform the facilitator that you

would like to do so. Also, remember that we are testing the web page NOT you.

Freedom to withdraw: Helping us today is up to you. You may withdraw from this study at any

time, without any negative consequences. However, we do appreciate any help you are able to

give.

Compensation: For helping us out today you will be provided with refreshments.

Freedom to ask questions: If you have any questions, you may ask the test administrator now

or at any time during the study.

I agree to participate in this research. I have read this form. My questions were answered.

Signature: _________________________________

Printed Name: ___________________________________

Date: ___________________________


of 90


of 90

APPENDIX J- EQUIPMENT

TEST EQUIPMENT

Magnets

Dry erase board, erasers & markers

Printed graphics

o Double T

o Photographs

o Campus shots

Buildings

People

Tape

Snacks

o Pizza

o Candy

Supply Table

LAB EQUIPMENT

Ceiling camera

Lab microphones

Windows Movie Maker®

Computer Stopwatch


of 90

APPENDIX K- PHOTOS OF LAB SET-UP

Prototype Area

Computer Task Area

APPENDIX L- STUDENT INFORMATIONAL SURVEYS


of 90


of 90

APPENDIX M- OBSERVATION LOGS


of 90


of 90

APPENDIX N- PRE-TEST SURVEYS


of 90


of 90

APPENDIX O- CONSENT FORMS


of 90


of 90

APPENDIX P- POST TEST QUESTIONNAIRES


of 90


of 90

APPENDIX Q- PROTOTYPE PHOTOGRAPHS

Participant 1 – Photo 1.1


of 90

Photo 1.2

Photo 1.3


of 90



of 90

Photo 2.2

Photo 2.3


of 90


Photo 3.3


of 90


of 90


Photo 4.2


of 90


Test Plan - Usability Research

Documents

Transcript of Test Plan - Usability Research