Download - R25 Usability Testing

Transcript

Usability Test of the

Missouri Western

R25 WebViewer

Yaser Khani

Michael Nelson

Stewart Hutchison

Reserve 25 (R25) WebViewer

The R25 WebViewer is widely used by college campuses,

primarily used for event management & scheduling, allowing

students to:

View course information, events and activities

Access the schedule of indoor and outdoor facilities

Reserve campus space for student organizations or personal use

Business Goals

Encourage students to become more involved with campus

activities:

Initiate and join campus organizations

Fostering an active and cooperative campus environment

Two Locations:

“Calendar of Events” “WebViewer / Campus Calendar”

Target

Users A full-time undergraduate student who spends the

majority of their time on campus.

A highly motivated and active student who initiates

activities with their peers.

A student who belongs to an organization on

campus or regularly attends athletic events.

Ideal participants for

our study

Participant Screener

How many credits are you currently enrolled?

Which of the following age ranges are you in?

Are you currently living on campus?

Have you ever accessed our online Calendar of Events to view an activity or search

course/activity information?

Have you ever initiated or created an event on campus with the Center for Student

Engagement?

Do you currently belong to a student organization?

Profile Summary of Five Users

Participant Introduction

“Thank you for taking the time to meet with us today.

We’re a graduate student group interested in better understanding how people interact with the R25

WebViewer Scheduling software and have asked for your help. So we can explore the software together,

you will receive specific directions for each task scenario. You can withdraw from this study at any time, and

will still receive the extra credit points from your instructor.

If you have any questions, please feel free to ask the moderator. No personally identifying information is

required and only the graduate student team will have direct access to your responses. You are encouraged

to think aloud as you complete the scenarios, describing your thoughts and feelings to the extent you feel

comfortable.

The Missouri Western website is often busy and may take time to load new pages, so there may be some

significant “clicking-and-waiting”.

We thank you again for your time and patience.”

Task Scenarios

You are curious about today’s events on campus…

You have decided to initiate an activity on campus and would like to submit an event request…

You are a softball player and would like to participate in a tournament game on February 1, 2014. You will need to contact the event organizer by obtaining their phone number…

You have enrolled in a college writing course (ENG 104 01), which begins Jan. 14 on Tuesday and Thursdays. You still need to know the time and location of your class…

You are planning to organize a conference meeting in one of the Fulkerson rooms the last week of April, 2014. You require a room that will seat 125 people…

15 Tasks

1. Locate today's date on the calendar to view the most current event listings.

2. Select any event for more information.

3. Locate the option to view all requests that you have already made.

4. Select the option to submit a request for an event.

5. Locate the ability to view only events held at the Griffon Indoor Sports Complex.

6. Navigate to the specified date of the softball tournament.

7. Locate the event for more information and confirm the requestor’s phone number.

8. Locate the option to display both events and course information.

9. Navigate to your course using the calendar of events.

10. Select your class to determine the time and location.

11. Locate the option to display building locations only.

12. Navigate to the desired week to schedule your conference.

13. Find and select one of the Fulkerson rooms for more information.

14. Select the room again above the calendar to determine the seating capacity.

(Repeat this step until you have found a suitable room for your audience).

15. Determine an available time and select the time slot to request a conference.

Testing Procedure

Two minutes anticipated per task; participants given 30 minutes total.

One assist was given for every minute spent on a singe task.

All testing was completed on a laptop computer (Toshiba Satellite P855).

Screen movements and audio were recorded with Camtasia Studio 8.1.

Remote testing was conducted via Google Hangouts.

Scenario 1

Scenario 2

Scenario 3

Scenario 5

Time on Task (seconds)

Scenario 4

#5: Locate the ability to view only events

held at the Griffon Indoor Sports Complex

#11: Locate the option to display

building locations only

Mean and Variance

#4: Select

the option to

submit a

request for

an event

#14: Select

the room

above the

calendar…

Task #4“Select the option to submit a request for an event”

Video 1/3

3:06

Mean = 4 seconds

Variance = 3 seconds

Task #4 Summary

The first two users began scrolled-down,

where the top menu was not visible.

Locating “My Requests” took considerably

longer for users #1 and #2 (72 and 103

seconds).

Once accessed, all users located “New

Request” in 6 seconds or less.

Task #5“Locate the ability to view only events held at the Griffon Indoor Sports Complex”

Video 2/3

3:14

Mean = 62.8 seconds

Variance = 279 seconds

Task #5 Summary

User #4 took considerably longer to

complete task 5 (283 seconds).

After selecting “Locations”, user #4 did not

scroll down to view “Filter” options until

prompted by moderator.

User #3 also selected “Locations” first, but

recovered quickly by navigating the “Filter”

options (16 seconds).

X

“The ability to view” = “Filters”?

Task #11“Locate the option to display building locations only”

Video 3/3

2:26

Mean = 63.6 seconds

Variance = 189 seconds

Task #11 Summary

All users began task scrolled-down, where

“Locations” was not immediately visible.

Users #2 & #3 required a prompt to look away

from the “Filters” option

Users #1, #4, and #5 completed the task under

17 seconds.

Possible learning effect at this point, where the

“Locations” option had been previously

explored.

“The option to display” = “Locations”?

X

SUS – System Usability Scale

1. I think that I would like to use this website frequently.

2. I found this website unnecessarily complex.

3. I thought this website was easy to use.

4. I think that I would need assistance to be able to use this website.

5. I found the various functions in this website were well integrated.

6. I thought there was too much inconsistency in this website.

7. I would imagine that most people would learn to use this website very quickly.

8. I found this website very cumbersome/awkward to use.

9. I felt very confident using this website.

10. I needed to learn a lot of things before I could get going with this website.

All questions are scaled 1-5 (1 = Strongly disagree, 5 = Strongly agree)

Scoring the SUS

For odd items: subtract one from the user response.

For even-numbered items: subtract the user responses from 5.

This scales all values from 0 to 4 (with four being the most positive response).

Add up the converted responses for each user and multiply that total by 2.5.

This converts the range of possible values from 0 to 100.

http://www.measuringusability.com/sus.php

SUS Scores:

User 1 = 50

User 2 = 47.50

User 3 = 67.50

User 4 = 70

User 5 = 55

A score of 68 is considered average (50% rank)

http://www.measuringusability.com/sus.php

Measure of perceived usability

Interpretation of SUS Scores

http://www.measuringusability.com/sus.php

User

#1

User

#5User

#2

User

#3

User

#4

Based on Jeff Sauro’s 500 studies

Conclusion of SUS Analysis

Participant’s average SUS Score = 58%

This is below Sauro’s average: 68%

Measure of perceived usability percentile ranking = 22%

Severity Scale – Dumas & Redish

Level 1: Prevents task completion

Level 2: Creates significant delay and frustration

Level 3: Problems have a minor effect on usability

Level 4: Subtle and possible enhancements/suggestions

Local (single page) vs. Global (entire website)

Strengths

No major usability issues

Participants were able to complete all tasks

Strength – Location of Filters

Users responded positively to the left-hand navigation menu

Areas to Improve – Text to Describe Event

MWSU does not have descriptive text within the time slots

Recommendations

Include descriptive text

Event A

Event B

Event H

Event C

Event D

Event F

Event EEvent G

Event I

Areas to Improve – Disappearing Menu

Recommendation:

“Locations” and “My Requests”

A persistent view of the menu bar while

scrolling down the calendar of events

An example is given by Binghamton

University…

Most users did not initially see the top menu

bar

Areas to Improve – Aesthetics

Sorting through bolded

information is strenuous

The color of the top

navigation menu makes it

easy to overlook

Viewing information as a

grid

(Global 3)

Recommendations

Bold times instead of descriptions

Stronger color contrast to make menu more noticeable

Note about locations/additional features

Viewing information as a list (have it be default setting)

Example Aesthetics – Navigation Note

Areas to Improve

Old Information (Global 3)

Scheduling information dates from 2005

This may be the cause of significant lag and slow response times

Recommendations

Remove the old data and information from the system

Back up the data on another server/system

Priorities

Navigation bar at the top of the screen (Global 2)

Aesthetics (Global 3)

Text to describe event (Global 3)

Old Information (Global 3)

Questions / Comments?