Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....

Prepared by: ADGS Consulting - Monash University

Usability Test Client Report

Project: Website Usability Assessment

Client: Victorian Deaf Society (Vicdeaf)

Team: Adrian Caceres

Greg Ortega

Steve Remington

Di Zhang

ADGS Consulting - Monash University Usability Test Client Report

Victorian Deaf Society (Vicdeaf) Website Usability Assessment Date: 12/02/2014 Version: 1.0

Commercial In Confidence

DOCUMENT INFORMATION

Document Details

Document Title: Usability Test Client Report

Project Name Website Usability Assessment

Document Version: 1.0

Document Date: 12/022014

File Name: Client Report - Victorian Deaf Society (Ver 0.3).Docx

Document Authorisation

Prepared by: Adrian Caceres; Greg Ortega; Steve Remington; Di Zhang

Authorised by: Manoj Kathpalia; Julie Fisher

Prepared for: Ms Lidia Risicato - Victorian Deaf Society (Vicdeaf)

Company Details

Name: ADGS Consulting - Monash University

Address: Monash University Caulfield Campus

Caulfield East, Victoria, 3165




EXECUTIVE SUMMARY A usability test was conducted on the Auslan course enquiry and enrolment, and online

donation functionality of the Vicdeaf website. These two areas of functionality were

chosen after a discussion with Michael Parremore about concerns Vicdeaf had about the

website. After a review of the website content and functionality two user scenarios and a

set of tasks related to the areas of concern were developed.

The usability tests were conducted in the period 03/02/2014 to 08/02/2014. The test

protocol required the participants to complete a small number of tasks related to each of

the scenarios. The tests were observed by members of the project team. At the completion

of the test the participants completed a questionnaire that sought their feedback on the

overall usability of the functions tested and suggestions on how the functions of the site

could be improved. Six participants were used for the test. The participants were

members of the Victorian general public known to the project team members. Five of the

participants completed the usability test for both scenarios and one participant only

completed the tasks for the Auslan course scenario.

The test results indicated that the participants’ opinion of the usability of the online

donation functionality was moderately positive (i.e. 3.6 out of 5.0), while the opinion of

the Auslan course functionality was slightly positive (i.e. 3.4 out of 5.0). The responses to

open ended questions about the online donation process showed that participants thought

the process was generally easy to use however the information on the left side of the

donation page is difficult to read, and that it would have been better if there was more

obvious access to information about how Vicdeaf uses donations. The responses to the

open-ended questions about the Auslan course information and enrolment process

showed that participants found it difficult to find information about courses because the

information is spread across many pages. Moreover, the layout of the course enrolment

page made it difficult to find the particular course they wanted to enrol in.

Based on the test findings and best-practice website usability guidelines a five key

changes to the Vicdeaf website are recommended. Additional minor recommendations are

presented in the body of this report. The key recommendation are outlined below.

Recommendations – Auslan Course Registration

Create a single Auslan course information page that outlines the course levels,

duration, typically available days / time, costs and available locations. The

information should ideally be presented in the form of a set of “frequently asked

questions” about Auslan courses.

Reorganise the Auslan course timetable so that it clearly shows level, location, date,

time of day, and open or closed status of a course and present the information in a

tabular format that make it easy for the user to scan. The format of the list should

allow users to sort and filter the courses by each of the categories.

Simplify the Auslan course main page so that it only has three links below the video

and introductory text, namely; “Course Information”, “Find a Course and Enrol” and

“Gift Vouchers”

Recommendations – Online Donations

On the home page near the donation link add a link titled “Learn More” which will

take the user to a page that briefly outlines how donations are used. The “Learn

More” page should include a “Donate Now” link that takes the user to the existing

online donation page.

Change either the font colour to the tone of the background image in the donation

page to increase the readability of the information on the left side of the page.




TABLE OF CONTENTS EXECUTIVE SUMMARY ................................................................ 3

TABLE OF CONTENTS ................................................................. 4

LIST OF TABLES ...................................................................... 4

1 Introduction .............................................................. 5 1.1 Purpose .................................................................... 5 1.2 Outcomes ................................................................. 5 1.3 Scope ....................................................................... 5

2 Description of the Test ............................................... 5

3 Summary of Results and Findings ................................ 6 3.1.1 Usability Test ............................................................. 6 3.1.2 Heuristic Evaluation .................................................... 7 3.1.3 General Observations ................................................. 7

4 Recommendations ..................................................... 8

Appendix 1 – Usability Test Detailed Results ......................... 10

LIST OF TABLES Table 1: Scenarios and task used in the usability tests ............. 6

Table 2: Recommended usability changes and benefits ............. 9

Table 3: Summary of participant demographics ..................... 10

Table 4: Summary of participant experience statement responses ............................................................................. 10

Table 5: Summary of key themes mentioned in participants’ responses to open-ended questions ........................... 11

Table 6: Summary of results of heuristic evaluation of the Auslan course enrolment functionality based on Nielsen’s 10

Usability Heuristics for User Interface Design .............. 13




1 INTRODUCTION

1.1 Purpose This document briefly describes the usability test on the Auslan course enquiry and

enrolment and the online donation functionality on the Vicdeaf website, summarise the

results and findings of that usability test and document recommended changes that will

improve the usability of those areas of functionality.

1.2 Outcomes The outcome of this document is Vicdeaf receiving a set of actionable recommendations

that will improve the usability of the Auslan course enquiry and enrolment, and the online

donation part of the Vicdeaf website.

1.3 Scope The scope of this project is limited to two specific use scenarios on the Vicdeaf website,

namely; Auslan course enquiry and enrolment, and online donation parts of the Vicdeaf

website. All other parts of the website were not tested in this project.

2 DESCRIPTION OF THE TEST The usability test method used for this project was a hybrid of the hallway testing and

think-aloud protocols of usability testing. The hallway testing approach recruits a small

random group of five or six participants who loosely match the profile of the core group

of users of the website. Research has shown that the hallway testing approach can identify

approximately 80% of the usability problems with a given website scenario. The think-

aloud protocol is a variation of the typical usability testing approach in which the

participants are asked to talk through the actions they are performing on the site. The

think-aloud protocol helps the observer more clearly understand what the user is looking

at, thinking, doing, and feeling as they go about the task.

The usability test involved two scenarios and related tasks. The choice of the

functionality on the Vicdeaf website on which the scenarios were based resulted from a

phone meeting discussion with Michael Parremore on Friday 30 January. Michael said

that a recent review of web search analytics showed that many users were finding the

Vicdeaf Auslan course page after searching for terms related to learning sign language in

Melbourne, but only a small percentage of those users were enrolling in a course. Michael

suggested that it would be good to test if there were any usability problems with the

Auslan course part of the site. Michael also suggested that online donations were a

growing source of income for Vicdeaf and that it would be useful to know if there are any

usability problems in that part of the site.

After a review of the website functionality two scenarios and related tasks corresponding

to the two areas of concern were developed. The specific text of the scenarios and the

tasks is outlined in Table 1.

At the completion of the scenario tasks the participant was asked to fill in a questionnaire.

The questionnaire was divided into three sections, namely; a demographics questions

section that captured a few personal attributes of the participant, a website experience

statements section that allowed the participant to indicate their overall rating of the

usability of the site for the scenario, and an open-ended questions section that allowed the

participant to give his or her opinion about and offer suggestions to improve the site.




Scenario Tasks

Online Donation

You have a friend who is deaf. In a recent

conversation she told you how much the

services provided by Vicdeaf have helped

her through her life. You recently received

a good pay rise at work and decided that

you would regularly donate part of the pay

rise you received to a charity that you like.

After hearing about the work that Vicdeaf

has done for your friend you decide to

choose them as the charity you will

support.

1. Find out how Vicdeaf uses the money

from donations.

2. Establish a monthly donation of $35

using a credit card.

Enrolling for Auslan Course

A new colleague at your workplace is deaf

and primarily communicates using “sign

language”. You would like to communicate

more effectively with your new work

colleague and as a result you have decided

to enrol for a course in sign language. Your

work colleague told you that in Australia

sign language is called “Auslan”

(Australian Sign Language) and that

Vicdeaf regularly runs Auslan courses in

various suburbs in Melbourne. You decide

to visit the Vicdeaf website to find out

about and enrol on an Auslan course.

1. Find out what is the duration of the

Auslan courses and how much do they

cost.

2. Find out if there are any courses

specifically related to learning Auslan to

communicate with a work colleague.

Make an enquiry to find out more

information about the course.

3. Register for an Auslan class that runs on

a weekend at a venue closest to where

you live.

Table 1: Scenarios and task used in the usability tests

The usability test were conducted in the period 03/02/2014 to 08/02/2014. The tests were

observed by members of the project team. Six participants used for the test. The

participants were members of the Victorian general public known to the project team

members. Five of the participants completed the usability test for both scenarios and one

participant only completed the tasks for the Auslan course scenario. A breakdown of the

participant demographics is shown in Table 3.

3 SUMMARY OF RESULTS AND FINDINGS

3.1.1 Usability Test

The ratings of the usability experience for each scenario showed that the participants,

overall had a more positive experience with the online donation scenario compared to the

Auslan course scenario. The key points from the experience ratings were:

In both scenarios the participants’ responses were moderately negative about the

screens generally being too much information on the screen. This indicates that

there is an opportunity to review the amount of content on these the pages.

The participants’ responses were slightly positive about the ease of use and visual

appeal of the Auslan course pages. This indicates that there is an opportunity to

improve the clarity of the content and layout of these pages.

For a more detailed breakdown of the usability experience ratings refer to Table 4 in

Appendix 1.

The participants generally had positive comments about the usability of the online

donation scenario. It was generally considered easy to make a donation. A few users,

however, commented on the difficulty they had finding information about how Vicdeaf




uses money from donations. On participant summarised it thus: “You must first make the

decision to donate to understand / obtain information on how Vicdeaf use the donations”.

The participants’ comments about the Auslan course scenario were on the other hand

more generally more negative. The common themes raised in the comments were:

The information about the Auslan courses is unnecessarily spread across too

many pages, and sometimes pages on a similar topic present inconsistent

information.

It was difficult to find out if the desired type of course, is available at the desired

location on the desired day.

Even when the desired course was found the participant often clicked on the

registration link only to find that the course was full.

The general feeling of the participants about the usability of the Auslan course scenario

was summarised thus: “Couldn’t find all the information together. Not all links were

working”. A summary of the themes raised by the participants’ responses to the open-

ended questions is presented in Table 5 in Appendix 1.

3.1.2 Heuristic Evaluation

In light of the usability issues identified in the Auslan course scenario a heuristic

evaluation of the Auslan course scenario pages was conducted to provide an alternate and

wider ranging assessment of the usability issues on the relevant web pages. The results

confirmed the findings of the participants’ usability test. In particular the heuristic

evaluation identified the following key issues:

Related information (e.g. course types, location and cost) is spread over many

different pages which requires the user to rely too heavily on remembering what

is on each different page to fully understand the details of the course offered.

The current course list page does not advise the user if the course is full until the

user clicks on the course resulting in unnecessary clicks and potential frustration

for the user.

The course registration page allows the user to submit the page before all

required field are filled in. A better user experience would be achieved if the form

submission is prevented until all required fields are filled in.

The fully documented results of the heuristic evaluation, including severity rating of the

issues found is presented in Table 6 in Appendix 1.

3.1.3 General Observations

While watching the participants conducting the usability tests the observers identified a

small number of technical issues with the website that should be addressed. The key

items are outlined below:

When the user is directed to the online donation and course registration pages

provided by bCommerce the links in the top menu bar and the search field return

an error page from bCommerce web server when clicked.

A number of links on the Auslan course pages (e.g. course timetable link, and

some of the links in the page “breadcrumb” trail) redirect the user to pages served

from the Bliss Media staging web server.

The page “breadcrumb” trail on some of the Auslan course pages do not

accurately reflect the position of the page in the site hierarchy.




4 RECOMMENDATIONS Based on the findings outlined in the previous section and best-practice website usability

guidelines we make the recommendations as outlined in Table 2 below.

Recommended Change Benefit of Change

Scenario: Enquiring About and Registering for Auslan Courses

Create a single Auslan course information

page that outlines the course levels,

duration, typically available days / time,

costs and available locations.

Present the information in the form of a

set of “frequently asked questions” about

Auslan courses. Using this format would

present users with information is a format

that they are familiar with from other

websites.

Presenting information about Auslan

course to user this way means that it is

possible to find all information from a

single page.

A single course information page will

provide a more targeted page for search

engines to land on when search terms such

as “learn sign language Melbourne” are

used.

Reorganise the Auslan course timetable so

that it clearly shows level, location, date,

time of day, and open or closed status.

Present the information in a clear tabular

format that make it easy for the user to

scan. Ideally the list of courses should be

(a) sortable by the user on each of the

categories, and; (b) filterable by the user

(e.g. only show me level 1 course on the

weekend).

Do not show other course information on

this page, but rather provide links to the

above course information page.

Presenting the course timetable in this

format with sorting and filtering options

will make it easy for the users to find and

register for exactly which course they

need.

If it is easy for users to find the course and

then register then there will be a reduced

level of user frustration and site

abandonment.

Present the information about and

purchase of the Auslan course gift

vouchers on a separate page and remove

information about gift vouchers from the

course list page.

Presenting the Auslan course gift voucher

information on a separate page will make

it clear that a person is paying for a course

sometime in the future rather than

enrolling in a specific course.

Reduce the number of image links below

the welcome video and introductory text

on the Auslan course. The reduced set of

image links should be:

o Course Information

o Find a Course and Enrol

o Gift Vouchers

Reducing the image links on the Auslan

course main page will not only align the

links with the above recommend changes,

but also simplify the page and bring it

more in line with how most users would

approach obtaining information about and

enrolling for courses.

Scenario: Making and Online Donation

On the home page near the donation link

add a link titled “Learn More” which will

take the user to a page that briefly outlines

how donations are used. The “Learn

More” page should include a “Donate

Now” link that takes the user to the

existing online donation page.

Adding this page will allow potential

donor to quickly find out how their money

will be used without the perceived

pressure of the online form. If the user

likes what they read he/she can in one

click go directly to the donation page.




Recommended Change Benefit of Change

Change either the font colour to the tone

of the background image to increase the

readability of the information on the page.

Making this change will make it easier for

a potential donor find out more detail

about how donations to Vicdeaf are used.

Moreover, research has shown that a user

will read more information on a web page

and are more likely to complete a

transaction if the page is well laid out easy

to read.

General Site Issues

Change the Vicdeaf site main menu bar

that is displayed on the bCommerce e-

commerce pages (e.g. course registration

and donations) so that the links return the

user to the correct pages on the Vicdeaf

site.

Correct the small number of links on the

site that are pointing to the Bliss Media

staging server.

Update the format of the “breadcrumb”

trail links so that it more accurately reflect

the position of the page on the site.

Making these changes, particularly

correcting the e-commerce links will

reduce the chance of users seeing

unnecessary and confusing error

messages. Moreover eliminating these

errors will increase the trust that the user

has in the Vicdeaf site generally but in the

e-commerce parts of the site specifically.

Table 2: Recommended usability changes and benefits




APPENDIX 1 – USABILITY TEST DETAILED RESULTS

Participant Demographic

Attributes

Scenario

Auslan Course (Count) Online Donation (Count)

Gender Male (2) | Female (4) Male (2) | Female (3)

Age Group 18-34 (3) | 35-49 (3) 18-34 (3) | 35-49 (2)

Internet Experience Some (1) | Extensive (5) Some (1) | Extensive (4)

Vicdeaf Experience None (5) | Limited (1) None (5)

Table 3: Summary of participant demographics

Participant Experience Statements

Scenario

Auslan Course

(Average Score)

Online Donation

(Average Score)

1. I found the layout of the website easy to

understand. 3.2 4.0

2. The design of the website is appealing. 3.2 4.0

3. When assessing the information, the

number of steps required was acceptable. 3.8 4.0

4. It was easy to navigate through the

website. 3.5 3.4

5. I did not get lost when completing the

task. 3.8 3.6

6. The information of the page was

effectively laid out. 3.3 3.8

7. Generally there was too much information

on the screen. 3.2 2.6

Note: Shaded scores are two worst scores for each scenario.

Table 4: Summary of participant experience statement responses




Open-Ended Question Auslan Course Online Donation

1. Were you able to easily complete

the task set? If yes, what

aspect(s) of the site helped you

complete the task? If no, what

aspect(s) of the site made it

difficult complete the task?

Yes

o It is easy to find course related information

the home page

No

o Purpose of gift voucher is confusing

o Auslan page very confusing. There is a lot of

information in spread across many places

o Hard to find a weekend course

Yes

o Easy to find donation and make an online

donation

No

o It is not easy to find how donation money is

used without directly to the donate page

o Could not find link to donation page on menu

2. What was the one element of the

website that you feel made it

easier for you to complete the

task(s)?

There was none

Radio buttons to select payment options

The main Auslan page is specific

The donation image link was easy to find

The site menu is useful after the home page

Filling donation details was easy

3. If you could change one thing

about the website that would

make your experience better,

what would it be?

Improve layout of the course timetable

Group related course information together

Reduce size of banner on home page as it distracts

from main image links below it

Focus less on appearance more on content

Change white text on light coloured background

on donation page

Too many distracting colours on the home page

Make more obvious on home page link to

information on how donations are used

4. Were there any elements of the

website that caused confusion or

slowed down your retrieval of

information when completing the

task? If yes, please explain what

it was and why.

Inconsistent link destinations

Repeated but inconsistent information

The menu links on all four edges of the pages

Lack of clarity on status of courses

It is not clear what types of course are available

Distracting content on the home page

The frequency options

It is possible to set an expiry date that in the past

No instructions on the donation page

Not clear on pages other than the donation page

how donations are used.

5. Are there any other comments or

suggestions about the website

that you would like to add?

Move main link images on home page to above

banner

Clearer presentation of course information

Make the site less fancy

Search option in multiple place on home page

Inconsistent colour of donate image on home page

and other pages.

No images on the home page makes it feel

impersonal

Table 5: Summary of key themes mentioned in participants’ responses to open-ended questions




Nielsen’s Usability Heuristic Issue Location Issue Details Issue

Severity

Visibility system of status Course registration form

Auslan courses list

Form field error messages should be displayed next to the

fields that causes the error, not as a list at the top.

User should not be able to click on a course that is not

available or full.

1

2

Match between the system

and the real world

Auslan courses list Grammar mistake, “To register in class”, reads as one can

register yourself on the day of the course.

1

User control and freedom Information for a specific

Auslan course and location

The breadcrumb trail does not show the previous page 2

Consistency and standards When an Auslan course is

full

When an Auslan course is

full

When clicking on “back to events” link, the page layout of

the course list changes. Register and enrol buttons confuse

the user

The contact us and make an enquire page have a different

page layout.

2

2

Error prevention Course Registration form

Course Registration form

Auslan course list

Proceed to payment button should not be active if the fields

are not filled in

There is no confirmation dialog before registering and

payment. When alternate payment options.

There is no map for the new St Kilda course location.

2

2

1

Recognition rather than recall Enquire or contact about a

specific course that is full or

not available

The form should be pre filled with the course information

that the user is enquiring about.

2

Flexibility and efficiency of

use

No issue identified Not relevant to this case 0

Aesthetic and minimalistic

design

Auslan courses list Button colours for new course locations and gift vouchers

should be different because they serve different purposes.

1

Help users recognise,

diagnose and recover from

errors

Course Registration form Form field error messages should be displayed next to the

fields that causes the error, not as a list at the top.

1




Nielsen’s Usability Heuristic Issue Location Issue Details Issue

Severity

Help and documentation Course Registration form

Auslan courses list

Auslan Course Information

Message reads click “Enquire” for more payment options,

but there is no enquire button or link.

The course list should not show courses that are not

available, or that are full.

Related information (e.g. course types, location and cost) is

spread over many different pages

2

1

3

Issue Severity Rating

0. There is no usability problem.

1. Cosmetic problem only: Need to fix only if time permits.

2. Minor usability problem: fixing this should be given low priority.

3. Major usability problem: important to fix, so should be given high priority.

4. Usability catastrophe: imperative to fix this before product can be released

Table 6: Summary of results of heuristic evaluation of the Auslan course enrolment functionality based on Nielsen’s 10 Usability

Heuristics for User Interface Design

Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....

Documents

Transcript of Web Site Usability Test - Client Report - Victorian Deaf Society (Ver 1....