Good hypothesis testing is surprising!

GOOD HYPOTHESIS

TESTINGIS SURPRISING

Timothy Fitz (.com)

Timothy Fitz

Coined Continuous DeploymentTechnical Lead at IMVU

CTO at CanvasSoftware Consultant

TimothyFitz.comSystemsLive.com

@TimothyFitz

http://TimothyFitz.com

http://SystemsLive.com

WHICH SHADE OF BLUE?

Buy Buy Buy

Buy Buy Buy

Buy Buy Buy

Buy Buy Buy

MULTI-ARMED BANDIT

GOOD HYPOTHESIS

• Documented

• Testable

• Actionable

• Interesting

• Surprising

DOCUMENTEDNCSU Libraries Mobile Redesign for 2012

Objective The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience enhancements to the existing set of mobile library services.

Requirements ● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort ● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework ● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with

the NCSU Libraries website ● Perform a comprehensive review of the usage stats; use this information to remove underutilized services ● Maintain or improve usage data collection; improve usage data reporting tools ● Retire legacy MoboiLIB service

Out of Scope ● Development of brand new apps ● Deep redesign of existing apps ● Custom iPad mobile website ● Native app deployment (iOS/Android)

Team Project Manager: Tito Sierra UI/UX Lead: Adam Rogers Tech Lead: Cory Lown

Schedule Launch: 2012.12.08

DOCUMENTED

Hypothesis Students want to use their phones to track their outstanding library books.

The details really matter.

Tiny code changes can change results dramatically.

TESTABLE

• Are you collecting the data?

• Can you collect and analyze it quickly?

• Is it statistically significant?

HIGH TEMPO TESTING

3 experiments per week!

Hypothesis:

Adding Group Chat (a 12-month effort) will result in a 25% increase in 3-month retention.

Hypothesis: Users want to chat with strangers.

Hypothesis: Users want to chat with strangers about specific topics.Hypothesis: Users want a directory of group chats.

Hypothesis: Users who group chat will have higher retention.




✓

ACTIONABLE

The results of testing your hypothesis cause action.

ACTIONABLE

Hypothesis: Adding a new payment method will increase overall revenue by 5%

• If confirmed, we will keep the payment method

INTERESTING

The action you take is important and

dependent on the outcome.



• If denied, we will keep the payment method

INTERESTING+DOCUMENTED

GOOD PRODUCT MANAGERS

Can rationalize ANY action from ANY outcome.

:( :( :(



• If denied, we will DELETE the payment method

INTERESTING+DOCUMENTED

SURPRISING

Bayesian definition of surprise:

“The total number of wows experienced when simultaneously considering all models is obtained

through the integration over the model class.”

http://ilab.usc.edu/surprise/

SURPRISING

surprise

=

change in beliefs

SURPRISING

The results of your hypothesis tests

should frequently and significantly

change your beliefs about your product!

Don’t let a single hypothesis test invalidate

your whole vision.

GOOD HYPOTHESIS

• Documented

• Testable

• Actionable

• Interesting

• Surprising

Good hypothesis testing is surprising!

Software

Transcript of Good hypothesis testing is surprising!