Milestone Five prototype...

Jeff BazerHCI 598X - Capstone Project

Iowa State UniversityFall 2015

prototype testingMilestone Five

SUMMARY

The current AgDirect Quote center is a tool that lets dealers run quotes for customers,

compare those quotes and be able to print, email and save them to be given to the customer

or used at a later time. The current tool is outdated, runs slow and has a poor user experience.

Issues with the current tool include data being lost after timing out, no ability to compare

quotes in the quote center and no easy way to email quotes to customers.

There are three types of users that will interact with the AgDirect Quote Center. The first type of

users are internal sales representatives. They are on the phones most of the day talking to

sales people at dealerships. The second type of users are sales people at dealerships. They

spend a couple hours a day in front of the computer and are typically outside of the office on

the sales lot talking and interacting with customers. The last type of users are external users

visiting the public facing website which can be divided into an older generation, Generation X

and Baby Boomers, and a younger generation, Generation Y.

Users who visit the AgDirect Quote Center should be able to complete the following three

tasks. The first task is to be able to run a basic or advanced finance or lease quote. The user

should be able to select between a finance or lease quote and be able to add in many other

options. The second task is to be able to compare that quote to multiple other quotes and

compare the differences in an easy to view side by side format. The third task is the quote(s)

should be able to printed, emailed and saved.

EVALUATION METHODS

Ferre, Juristo and Moreno (2004) define usability as “The extent to which a product can be

used by specified users to achieve specified goals with effectiveness, efficiency, and

satisfaction in a specified context os use.” What is usability testing? Usability testing is defined

�

AGDIRECT QUOTE CENTER �2

as requiring five characteristics: that the goal is to improve a product’s usability, that the

participants represent real users, that they do real tasks, that the testers observe and record

the participants and that they then analyze the data and recommend changes to fix problems

(Dicks 2002). Why should we test? Design decisions are informed by data gathered from

representative users to expose design issues so they can be remedied, thus minimizing or

eliminating frustration for users (Rubin and Chisel 2008). The evaluation method that was

chosen for this was a formative evaluation including doing a usability study and a

questionnaire.

Rubin and Chisel (2008) state that doing formative evaluation and early analysis with research

is important. This point in time is when critical design decisions set the stage for what will

follow and if certain elements are wrong it can cause many issues down the road for the whole

project. Using this and the usability test with a feedback form and questionnaire will allow more

useful data for the prototype of the AgDirect Quote Center.

An important aspect stated by Katner (1994) is “usability testing is in the details.” She states

that “details overlooked during preparation for a usability test become gaping holes in the data

collected during the actual test (Katner, 1994). This is why it is critical that the usability test is

set up correctly and efficiently so that the most accurate data is collected.

As research suggests, approximately 80% of the usability problems identified are found using

four to five users (Virzi, 1992). Virzi (1992) also states that additional subjects are less and less

likely to reveal new information and by running fewer subjects in each iteration it is a valuable

time and money saving approach. For this usability test four users will be tested - two internal

sales representatives and two AgDirect dealer sales people. Rubin and Chisel (2008) suggest

screening potential users beforehand to make sure they are representative of the correct user

profile for the usability test. The AgDirect internal sales representatives were selected after

�


meeting with their leader to select individuals that are representative of the usability test

scenarios and then asking them if they would be willing to help out. Rubin and Chisel (2008)

state that a good way to find existing customers is through sales representatives. Each state

has an AgDirect Territory Manager and they were an excellent resource to sit down with and

select two dealerships that met the usability test scenarios. An informal phone screen was then

conducted and two sales people at two different dealerships agreed to volunteer.

Compensation was provided in the form of lunch for each dealership.

The two internal sales representatives that were selected were younger female users who are

computer savvy and create, edit and compare a minimal amount of quotes (0-5) in a single day.

They will be referred to as “Tester 1 and Tester 2.” The two AgDirect dealer sales people

selected include one who works for a John Deere dealership and one who works for a Case IH

dealership. They are both not very computer savvy and each creates, edits and compares

many quotes (10-15) for customers in a single day. They will be referred to as “Tester 3 and

Tester 4."

As soon as the participant enter the usability session, Rubin and Chisel (2008) mention starting

with an orientation script. Parts of this script include introductions, explaining why the

participant is there, what is expected of the participant, assuring the participant that they are

not being tested and that it is ok to ask for questions at any time. When each participant enters

the usability session, they will be given a questionnaire to fill out which includes their name, job

title along with these three questions: 1. On average, how many new quotes to do you create in

a day?, 2. On average, how any quotes do you edit in a day? and 3. On average, how many

times do you compare quotes in a day? The form then includes information at the bottom

stating that this not a test of your abilities but our ability to make something that can be useful

to you - there are no right or wrong answers. See Appendix A, figure 1 for reference.

�


Each participant will go through two different scenarios with various tasks for each scenario.

They will be given a sheet that includes both scenarios to work on at their own pace. The tasks

that are to be completed for the first scenario include: participants will be instructed to

calculate the payment for a new quote with the following information: Quote Description: 2013

Planter 5 year, Amount: 215,000.00, Rate: 4.15, Term: 5 years, and Frequency: Annual. The

participant will then add the quote for comparison and run a new quote with the same

information but for a seven year term and also add it for comparison. The last step for the first

scenario is to edit the payment frequency for both quotes to be “Semi-Annual."

The tasks that are to be completed for the second scenario include adding a quote to

an existing saved quote. The participants will be instructed to open the quote “Jack Jones

2012 Tractor.” They will then add a new quote for comparison with the following

information: Quote Description: 2013 Tractor 5 year, Amount: 150,000.00, Rate: 4.15, Term: 7

years, and Frequency: Annual. The final step for this scenario is to duplicate that quote and

remove the trade-in amount and the trade-in payoff and save the changes.

This form can be found in Appendix A, figure 2 for reference.

Rubin and Chisel (2008) define a post-test questionnaire as gathering information from the

participants after the usability test in order to understand more the strengths and weaknesses

of the tool. Accordingly, a post-test questionnaire was distributed right after each session. It

includes seven questions - three questions that ask for a rating from one to ten and why and

four open ended questions. In retrospect, this would have been better done with a five point

scale as Rubin and Chisel (2008) point out.

�


The questions are included below:

1. How would you rate the navigation of the new Quote Calculator Tool? (1-10, 1 = very

difficult and 10=very easy) Why?

2. How would you rate the design (screen layout and colors) of the new ADO website (1-10, 1

= hideous and 10=looks great) Why?

3. How would you rate your experience entering and editing quotes through the new

quoting told design? (1-10, 1 = very poor and 10 = perfection) Why?

4. What did you like about creating or editing quotes in the new ADO website?

5. What did you not like about creating or editing quotes in the new ADO website?

6. What additional features would you like to see when creating or editing quotes?

7. Additional Comments?

This questionnaire can be found in Appendix A, figure 3 for reference.

The testing environment for the internal AgDirect sales representatives will take place using a

simple single-room setup. Rubin and Chisel (2008) define this as the most basic type of testing

setup, both in terms of resources and the amount of space required. It will take place in a

conference room in the office and each participant will be brought in separately to run the

usability test. An advantage to using this setup is that the moderator and observers have an

excellence sense of what is going on with the participant but a disadvantage is that

the observers behavior can affect the behavior of the participant.

The testing environment for the AgDirect dealer sales people will take place at each of their

offices. Items that will be brought to each location include scripts, forms and questionnaires,

laptops with prototype on it, projector to project the participants screens for observers to take

notes, and pens to write with. This list is based off of what Rubin and Chisel (2008) suggest to

bring.

�


See Appendix B for photos of each setup.

RESULTS

The results from the usability sessions ended up with interesting themes. There were a total of

four participants, two internal AgDirect sales representatives and two AgDirect Dealer sales

people. Overarching themes from the participant and observer forms that were positive include

easy flow and navigation, the payment box with illustration of payment and interest and

user friendly design.

Below are quotes from participants that did the usability study that are positive. These quotes

varied between all four participants and were collected from the post-test questionnaire.

- "Easy navigation with user friendly design. Easily can add/compare quotes when needed."

- "The payment box on the right side makes it easy to see and the ability to have the principal/

interest is very handy.

- "Like the illustration of principal and interest."

- "Smooth flow. Love the large payment."

The overarching theme that shows where improvement is needed is in editing and

updating the quotes and duplicating the quotes. Each user had some kind of issue with editing

and updating a quote.

Below are quotes from participants that did the usability study that show room for

improvement. These quotes varied between all four participants and were collected from the

post-test questionnaire.

�


- "Initially editing the quote took a minute to find "edit" button. Seamless with rest of the

process though."

- "Scrolling to the bottom of the compared quotes to find "edit" was the only issue I had."

- "I needed some explanation on how to update quotes. Most of it was intuitive if your use to

the previous version but it'll take some training for salesmen to get up to speed."

- "Not sure about the compare and edit features."

It is worthy to note that while there are many positive quotes and feedback, the most helpful

feedback is the needs improvement feedback. Without this feedback, the AgDirect Quote

Center will not be able to mature and gain a better user experience from this usability study.

Even with each participant having the issues with editing the quotes and duplicating the

quotes, they still rated each of the three questions fairly high. See figure 1 for details on each

question and rating. While there were issues, each participant did not think that those issues

were bad enough to majorly affect the design.

�


Figure 1: Post-Questionnaire Survey Responses

See Appendix C for feedback forms from each participant.

DISCUSSION

Overall there was very positive feedback. All the participants had positive feedback about the

flow of the AgDirect Quote Center and how each page was laid out. They also thought the

�


quote section with the large payment along with the graph and the payment and interest

beneath it was extremely useful.

Items that have room for improvement include the edit at the bottom of the compared quotes.

Participants had a hard time finding that - partially due to screen size and participants not

scrolling to the very bottom - but good feedback as it has shown that the link needs to be

moved. A possible fix for this is to include the edit link beneath the description instead of at the

bottom. Participants also had a hard time finding duplicate at the bottom - one participant

didn’t even notice the duplicate link. A possible fix for this is to also move it to the top next to

edit so users will see that right away also. Having these links separate from the delete and view

amortization will also reduce the users cognitive load since they will not have to look through

four links at the bottom to find what they are looking for.

The site seems to be on the right track. It just needs to have some cleanup of the items listed

above. One other comment that users mentioned was that when they had more than a couple

quotes added to the compared section that it was difficult to see the key information which

includes interest rate, term and payment. These fields and data will be bolded to stand out to

the user more.

Surprising findings that come out of the usability sessions were items that hadn’t been

originally thought about. One of those items was one participant asked if they could rearrange

the quote order in the comparison. It wasn’t feasible in the prototype, but it wasn’t something

that was originally thought of so that will be a good addition. Another surprising finding is that

users had a difficult time finding the edit link at the bottom of the compared quotes. As this

was being designed, the thought process was that a user would look through the quote and

once they reach the bottom, they would see edit and click it, but the testing shows that is not

�


the case. As discussed above, a possible fix to this is move it up beneath quote description so

it is easier for a user to see and it is not mixed up with the other links at the bottom.

One issue that may cause implications is if a user opens a collection of quotes, the compared

quotes could be below the fold on smaller screen resolutions and users may see a blank quote

at the top and not realize they need to scroll to see their saved quotes. This will need to be

revisited to see if there is a way to make it noticeable that there are quotes towards the bottom.

One test that we did not test yet due to the limitations of the prototype software is that if there

are more quotes added to the compare section than will fit on the screen - how to display that.

The current design has large arrows showing up on each side like an image carousel to let the

user know that they can scroll left and right to see more quotes.

The process of doing usability testing proved to be extremely useful. Without doing this

process, there were design flaws that would have been pushed out to users that could have

resulted in calls with questions and feedback. There would have had to been more time and

money spent on fixing those design flaws and having to push it out to users again - by doing

this usability test it eliminated all of that. The usability test also showed some ideas that had

not been thought of before and will be incorporated into the design to help future users of the

site. The test also help with feedback knowing that the site design was moving in the

right direction and that it was not off-base.

�


References

Rubin, Jeffrey and Chisnell, Dana (2008) Handbook of Usability Testing: How to Plan, Design,

and Conduct Effective Tests. Indianapolis, Indiana: Wiley Publishing, Inc.

Virzi, Robert A. (1992). Refining the Test Phase of Usability Evaluation: How Many Subjects Is

Enough? Human Factors: The Journal of the Human Factors and Ergonomics Society,

34 (4). Retrieved from

http://hfs.sagepub.com.proxy.lib.iastate.edu/content/34/4/457.short

Dicks, R. Stanley. (2002). Mis-usability: on the uses and misuses of usability testing. ACM: New

York, NY Retrieved from http://dl.acm.org.proxy.lib.iastate.edu/citation.cfm?id=584960

Brown, Kenneth and Gerhardt, Megan. (2002) Formative Evaluation: An Integrative Practice

Model and Case Study. Personnel Psychology, 55 (4) Retrieved from http://

onlinelibrary.wiley.com.proxy.lib.iastate.edu/doi/10.1111/j.1744-6570.2002.tb00137.x/

abstract

Katner, Laurie. (1994) Techniques for managing a usability test. Professional Communication,

37 (3) Retrieved from http://ieeexplore.ieee.org.proxy.lib.iastate.edu/xpls/abs_all.jsp?

arnumber=317479&tag=1

Ferre, Xavier, Juristo, Natalia and Moreno, Ana (2004) Improving Software Engineering Practice

with HCI Aspects. Software Engineering Research and Applications. Springer

Berling Heidelberg.

�


http://hfs.sagepub.com.proxy.lib.iastate.edu/content/34/4/457.short

http://dl.acm.org.proxy.lib.iastate.edu/citation.cfm?id=584960

http://ieeexplore.ieee.org.proxy.lib.iastate.edu/xpls/abs_all.jsp?

APPENDIX A

Figure 1: Usability Test Pre-Questionnaire

�


Figure 2: Usability Test Scenarios

�


Figure 3: Usability Test Post-Questionnaire

�


APPENDIX B

�


�


APPENDIX C

Figure 4: Tester 1 Pre-Questionnaire

�


Figure 5: Tester 1 Post-Questionnaire

�



�


Milestone Five prototype...

Documents

Transcript of Milestone Five prototype...