Act like someone’s life depends on what we do.

UNPARALLELED

COMMITMENT & SOLUTIONS

U.S. ARMY ARMAMENT RESEARCH, DEVELOPMENT & ENGINEERING CENTER

Rethinking User Interface Design and Evaluation Presented to: NDIA Armament System Forum 3 May 2017

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

UNCLASSIFIED

3 MAY 2017 2

• Classification: UNCLASSIFIED • Type of Briefing: INFORMATIONAL • Distribution A: Approved for Public

Release.

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

UNCLASSIFIED

3 MAY 2017 3

• When completely redesigning a Graphical User Interface, what is the best way to collect user preferences to make the best system possible?

PROBLEM

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

UNCLASSIFIED

3 MAY 2017 4

BACKGROUND: CURRENT INTERFACE

1993

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 5

PROTOTYPES FOR NEW INTERFACE

• 3 Prototypes

• Different look and feel

• Users choose the best from each

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 6

TESTING THE PROTOTYPES

The User Jury • A usability test where

actual intended users perform tasks and give feedback

Multiple: Prototype Options Locations User Experience

Levels

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 7

RESULTS AND LIMITATIONS

• Surveys were open ended or asked to rate the interface or specific features

• Some useful feedback on features that were generally liked or

disliked by the majority of the users. • The tests failed to provide actual metrics on usability, entry

time

Limitations: • 1-2 Trials • Subjective feedback • Underused data

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 8

IMPROVEMENTS

How intuitive is the interface and who will be using it?

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 9

TESTING INTUITIVENESS

The Test: • Brand new users • Run through the same

process up to ten times and collect times

• Analyze

Test the interface for intuitiveness before bringing to the user jury

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 10

ANALYSIS: DASHBOARD

0 20 40 60 80

100 120 140

0 2 4 6 8 10 12

Seco

nds

Trial

Dashboard

Average

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

1 2 3 4 5 6 7 8 9 10

Perc

ent o

f Tim

e 1

Trial

Percent Improvement

50%

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 11

ANALYSIS: KIOSK

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

1 2 3 4 5 6 7 8 9 10

Perc

ent o

f Tim

e 1

Trial

Percent Improvement

50%

0

50

100

150

200

250

0 2 4 6 8 10 12

Seco

nds

Trial

Kiosk

Average

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 12

ANALYSIS: BUTTONS

0 20 40 60 80

100 120 140 160

0 2 4 6 8 10 12

Seco

nds

Trials

Buttons

Average

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

100%

1 2 3 4 5 6 7 8 9 10

Perc

ent o

f Tim

e 1

Trial

Percent Improvement

65%

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 13

ANALYSIS

0

50

100

150

200

250

0 2 4 6 8 10 12

Seco

nds

Trial

Average Times per Trial

Dashboard Kiosk Buttons

Trial 5

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 14

APPLYING TO THE USER JURY

• Get uninvolved volunteers to run

through the process • Find your “burn time” • Use that as your minimum trial

amount at your user jury

• Gather the best feedback from your users

Befor

e you

leav

e At

the u

ser ju

ry

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

3 MAY 2017 15

• Normal user jury testing (one or two runs through the system) may not select the best design when the target user should be a well-trained expert

• A better design may not be apparent until the user conducts

repeated trial runs through a process and gets a feel for how the system will actually be used

• Find your “burn time” of each prototype and use that in your user jury

• Your test may take longer, but it will pay off in accuracy

CONCLUSIONS

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

UNCLASSIFIED

3 MAY 2017 16

QUESTIONS

Distribution A: Approved for public release. Distribution unlimited UNCLASSIFIED

UNCLASSIFIED

Contact: Korinne Dobosh Paladin Fire Control RDAR-WSF-I Korinne.e.dobosh.civ@mail.mil 973.724.7554 Co-Author: David Musgrave David.m.musgrave2.civ@mail.mil