Making Connections

July 11, 2005July 11, 2005 IPCC ConferenceIPCC Conference 11

Making ConnectionsMaking Connections

Teaming Up to Connect Users, Developers, Teaming Up to Connect Users, Developers, and Usability Expertsand Usability Experts

A presentation by The Usability Center at A presentation by The Usability Center at Southern Polytechnic and Ipswitch, Inc.Southern Polytechnic and Ipswitch, Inc.


Your presentersYour presenters

Carol Barnum, Co-Director of Usability Carol Barnum, Co-Director of Usability Center, Southern PolytechnicCenter, Southern Polytechnic

David Dayton, Usability Center AssociateDavid Dayton, Usability Center Associate Kevin R Gillis, Dir Product Mgt, IpswitchKevin R Gillis, Dir Product Mgt, Ipswitch Ben Henderson, Chief Technologist, Ben Henderson, Chief Technologist,

IpswitchIpswitch Joe O’Connor, Information Development Joe O’Connor, Information Development

Manager, IpswitchManager, Ipswitch


usability.spsu.edu

Evaluation Room

Control Room

Exec Viewing Room

The Usability Center at SPSU.edu


Our agendaOur agenda

BackgroundBackground Collaboration – process and productCollaboration – process and product The WS_FTP product history/issuesThe WS_FTP product history/issues Goals of testingGoals of testing FindingsFindings Iteration/RetestingIteration/Retesting Measuring results – Internal/ExternalMeasuring results – Internal/External What’s next What’s next


The WS_FTP product: history & The WS_FTP product: history & issuesissues

The productThe product– 15+ years old, 40 million users ranging from 15+ years old, 40 million users ranging from

students and eBay users up to Fortune 500, students and eBay users up to Fortune 500, Government and Public SectorGovernment and Public Sector

– Historically, 2 products - commercial Historically, 2 products - commercial (WS_FTP Pro) and a free education version (WS_FTP Pro) and a free education version (WS_FTP LE)(WS_FTP LE)

– In 2004, launched WS_FTP Home and pulled In 2004, launched WS_FTP Home and pulled LELE

– Never put through formal usabilityNever put through formal usability


How usability entered the How usability entered the picturepicture

Adding more features to the applicationAdding more features to the application Technical support incidents were risingTechnical support incidents were rising

– User base appears to be less technicalUser base appears to be less technical How do people “use” the product?How do people “use” the product? How do we build an interface that is How do we build an interface that is

usable by novice and professional usable by novice and professional users?users?


Background for Background for collaborationcollaboration BackgroundBackground

– Ipswitch selects Southern PolytechnicIpswitch selects Southern Polytechnic– Changing focus to early and iterativeChanging focus to early and iterative

Collaboration processCollaboration process– Team approachTeam approach– Co-development for testing/analysisCo-development for testing/analysis– Commitment from Ipswitch to participate Commitment from Ipswitch to participate

fullyfully– Iteration and retestIteration and retest


Findings: startup Findings: startup wizardwizard Walks through creating a connectionWalks through creating a connection Evolution from create site wizard to Evolution from create site wizard to

startup wizardstartup wizard Changes:Changes:

– Wizard text and examples improvedWizard text and examples improved– Walks through creating a connection and Walks through creating a connection and

connects when wizard is completedconnects when wizard is completed– Improved feedback for failuresImproved feedback for failures


Location: Create site wizard within the start dialog


Location: Wizard displayed on first startup


Help text and layout: Before


Help text and layout: After


Error text and layout: Before


Error text and layout: After


Getting started tourGetting started tour

Novices did not understand file transfer Novices did not understand file transfer concepts: concepts: – What is an FTP site?What is an FTP site?– What do I connect to?What do I connect to?– How do I connect?How do I connect?

Answers have always been in the help Answers have always been in the help system (text explanation)system (text explanation)

Tour represents a quick demo … show me, Tour represents a quick demo … show me, but don’t explainbut don’t explain


Tour starts with files on desktop (known concept)

User can control the tour


Imparts new concepts via the “story”


Findings: Multiple Findings: Multiple Document Interface Document Interface (MDI)(MDI) End-user feature requestEnd-user feature request

– Manage multiple sites simultaneouslyManage multiple sites simultaneously First tested version used MDI to First tested version used MDI to

display multiple sitesdisplay multiple sites Users could not navigate the MDI Users could not navigate the MDI

setup (for site-to-site transfer)setup (for site-to-site transfer) Time wasted setting up multiple file Time wasted setting up multiple file

views so they could navigate views so they could navigate betweenbetween


Before: A separate window for each site


After: Use tabs to switch between sites and local views


Findings: PGP setup Findings: PGP setup (for secure transfer)(for secure transfer) Advanced usersAdvanced users In initial testing, no advanced user In initial testing, no advanced user

successfully completed the PGP successfully completed the PGP scenarioscenario

Users did not understand concepts:Users did not understand concepts:– Creating a key pairCreating a key pair– ““Signing” a keySigning” a key– ““Trusting”Trusting”

Users drawn to the PGP mode button Users drawn to the PGP mode button in toolbarin toolbar


Before: Import keys


Before: Sign keys


Before: Import keys


After: Sign keys


PGP mode: beforePGP mode: before

Start Selectpublic key

Is theselected key

certified?

EnablePGP Mode

Prompt to use un-certified key

CancelPGP Mode

YesYes

No


PGP mode: afterPGP mode: after

Start Selectpublic key

Is theselected key

certified?

EnablePGP Mode

Prompt to use un-certified key

CancelPGP Mode

Yes

YesNo

Default: Use key & certify

Verifycertification

request

Does signingkey exist?

Prompt forsigning keypassphrase

Create orImport?

Launch CreateOr Import

Wizard


Measuring results - Measuring results - internalinternal

Survey of attitudes Survey of attitudes – 96 Ipswitch employee responses96 Ipswitch employee responses– strongly positive view of usability strongly positive view of usability

testingtesting– some expressed desire to do moresome expressed desire to do more– negative views small percentagenegative views small percentage

– observing tests effective in changing observing tests effective in changing views about usability testingviews about usability testing

– many hadn’t heard about the testingmany hadn’t heard about the testing


A web-delivered A web-delivered surveysurvey Initial email from Ipswitch president inviting Initial email from Ipswitch president inviting

employees to fill out web questionnaireemployees to fill out web questionnaire Two reminder emails over following weekTwo reminder emails over following week IRB-approved consent form to startIRB-approved consent form to start 96 respondents total; ~64% response rate96 respondents total; ~64% response rate Basic demographic and job informationBasic demographic and job information Likert-scale statements to elicit opinions Likert-scale statements to elicit opinions

about usability testingabout usability testing


Respondents profileRespondents profile

Avg 9.8 yrs software industry experience (from Avg 9.8 yrs software industry experience (from < 1 year to 35 years)< 1 year to 35 years)

Avg 3.7 years with IpswitchAvg 3.7 years with Ipswitch(from < 1 year to 11 years)(from < 1 year to 11 years)

AgeAge Job Function Job Function

18-2418-24 3 3 30 development30 development

25-3425-34 4343 30 sales & marketing30 sales & marketing

35-4435-44 2525 22 tech & adm support22 tech & adm support

45-5445-54 2323 12 upper & product mgmt12 upper & product mgmt

55 & up55 & up 1 1 2 no answer2 no answer


Measuring attitudesMeasuring attitudes

Likert-scale opinion items adapted from a Likert-scale opinion items adapted from a survey of software developerssurvey of software developers– 9 opinion items for all respondents9 opinion items for all respondents– 15 additional opinion items for those who 15 additional opinion items for those who

had observed usability testshad observed usability tests Several open-ended questions solicited Several open-ended questions solicited

opinions about usability testing opinions about usability testing


Strongly positive view:Strongly positive view:9 Likert statements9 Likert statementsOpinion Statements in Descending Order of Mean Score on 1-5 Scale; 5 = Most Positive

Usability testing . . .

pos - negs

Mean(n=94)

should be used more often. +70 4.46

has made dev. teams more concerned about usability

+70 4.36

has not been worth the effort* +61 4.27

has improved ease-of-use of products tested +65 4.19

has improved usefulness of products tested +57 4.07

has made dev. teams more customer-centered +47 3.82

has hurt cost-effectiveness* +30 3.79

has slowed down product development* +7 3.20

has made extra work for developers* -6 3.03

*reverse coded: higher score reflects pro-usability-testing attitude


Strongly positive view:Strongly positive view:open-ended answersopen-ended answers "Sum up your opinions about the "Sum up your opinions about the

effects of usability testing on effects of usability testing on product development and product product development and product quality at Ipswitch."quality at Ipswitch."

About three-fourths responded (76)About three-fourths responded (76) Positive statements outnumbered Positive statements outnumbered

negative 56 to 7 (8 to 1)negative 56 to 7 (8 to 1)


Breakdown of the 76 Breakdown of the 76 open-ended responsesopen-ended responses

22 strongly negativestrongly negative

55 mildly negativemildly negative

2323 general, positive statementsgeneral, positive statements

2424 strongly positive statementsstrongly positive statements

99 positive & wanted stronger usability effortpositive & wanted stronger usability effort

1111 didn't know enough to give an opiniondidn't know enough to give an opinion

22 no opinionno opinion


Significant? We like to Significant? We like to think sothink so 3 of 13 respondents3 of 13 respondents (23%)(23%) who observed who observed

usability tests before Ipswitch saidusability tests before Ipswitch saidobserving u-tests had changed their opinions observing u-tests had changed their opinions about usability testing.about usability testing.– These did These did notnot observe tests at Southern observe tests at Southern

PolytechnicPolytechnic

To the same question: To the same question: 8 of 20 respondents 8 of 20 respondents (40%) (40%) who observed usability testing for the who observed usability testing for the first time at Southern Polytechnicfirst time at Southern Polytechnic answered answered affirmatively.affirmatively.


Surprising finding: Surprising finding: many had little infomany had little info 13 of the survey’s respondents said they 13 of the survey’s respondents said they

didn’t know impact of testing on product didn’t know impact of testing on product designdesign– including two senior software developersincluding two senior software developers– a member of upper managementa member of upper management– two members of the marketing team.two members of the marketing team.

24 of 68 respondents (35%) who had not 24 of 68 respondents (35%) who had not observed testing at Southern Polytechnic observed testing at Southern Polytechnic said they didn’t know much about it — said they didn’t know much about it — beyond that it had taken placebeyond that it had taken place


Measuring results - Measuring results - externalexternal Success metricsSuccess metrics

– conversion rates of product trial conversion rates of product trial evaluationsevaluations

– support callssupport calls shorter durationshorter duration smaller numbersmaller number


Measuring results - Measuring results - externalexternal Success metricsSuccess metrics

– conversion rates of product trial conversion rates of product trial evaluationsevaluations

1.1.After 2 years of declining WS_FTP client After 2 years of declining WS_FTP client unit sales, experienced 3% growth.unit sales, experienced 3% growth.

2.2.For 12 months since WS_FTP For 12 months since WS_FTP Professional 9 and WS_FTP Home 9 Professional 9 and WS_FTP Home 9 shipped, conversion rates of WS_FTP shipped, conversion rates of WS_FTP Evaluators increased nearly 20%.Evaluators increased nearly 20%.


Measuring results - Measuring results - externalexternal


R&D process changesR&D process changes

Improvements in design of other Improvements in design of other productsproducts

Usability added to “bug report Usability added to “bug report database”database”


R&D process changesR&D process changes

Ingrained in development methodologyIngrained in development methodology– At least one usability round factored into dev At least one usability round factored into dev

scheduleschedule– “ “Broken as designed” replaced with a defect Broken as designed” replaced with a defect

categorize under Usabilitycategorize under Usability– Internal and impromptu usability occurs Internal and impromptu usability occurs

midstreammidstream Improvements in design of other productsImprovements in design of other products Usability added to TestTrack – Ipswitch Usability added to TestTrack – Ipswitch

R&D’s Defect Tracking databaseR&D’s Defect Tracking database


R&D Process ChangesR&D Process Changes

“Usability”


Lessons learnedLessons learned

Internal, informal user tests to Internal, informal user tests to professional usability testingprofessional usability testing

Prototyping (we went with live Prototyping (we went with live software versus paper or software versus paper or mockups)mockups)

Iterative - refinements to test Iterative - refinements to test based on walkthrough and pilotbased on walkthrough and pilot


What’s next?What’s next?

Collaboration continuesCollaboration continues– latest product never tested beforelatest product never tested before– very few issues uncoveredvery few issues uncovered– transition to web-based application successfultransition to web-based application successful– Executive viewing room stays packedExecutive viewing room stays packed

Nearly ½ of the company attends 1 or more sessionsNearly ½ of the company attends 1 or more sessions

– Ipswitch continues to grow and respond to its Ipswitch continues to grow and respond to its usersusers

customer visitscustomer visits contextual inquirycontextual inquiry


Questions?Questions? Carol BarnumCarol Barnum Kevin R Gillis Kevin R Gillis Ben HendersonBen Henderson Joe O’ConnorJoe O’Connor

Making Connections

Documents

Transcript of Making Connections