Changing the Game: Always On Testing with Artificial Intelligence
Transcript of Changing the Game: Always On Testing with Artificial Intelligence
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Peep LajaFOUNDER
The New World of AI-Based Testing
WEBINAR
Jonathan EpsteinCHIEF MARKETING OFFICER
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
• What are artificial intelligence and evolutionary computation?
• How Sentient Ascend™ automates your CRO efforts
• How Sentient Ascend is able to get results faster using AI
• Results and case studies from Sentient Ascend customers
• Q&A with Peep Laja
Agenda
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Artificial Intelligence and Evolutionary Computation
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
• A field of computer science focused on building “intelligent” machines
• An ideal “intelligent” machine
• Perceives its environment
• Takes actions that maximize its success against some goal
• Learns from the results of prior actions and adapts
• AI is all around you already!
• Siri, Waze, Airline pricing
• Is it smarter than me?
• Depends on what you mean by smart. But it is a lot faster.
What is Artificial Intelligence?
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
AI comes in many shapes and forms…
…each of which excels at solving certain classes of problems.
DEEP LEARNING EVOLUTIONARY ALGORITHMS
NEURO-EVOLUTION
BAYESIAN
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Evolution: A Proven Efficient Design System
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
FITNESSWhat makes one candidate (website, trading strategy, animal) more viable than another?
COMBINATIONBy combining successful candidates, you can achieve greater fitness, and also prune out bad attributes.
MUTATIONAn element of randomness helps discover new successful combinations and ensures you find the best solution—not just a good one.
The 3 Core Principles of Evolutionary Algorithms: AI Based on Natural Selection
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
How Sentient Ascend™
Automates Your CRO Efforts
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Ideas In, Results Out
Develop Hypotheses and How to Test Them
Enter Test Variants
into Ascend
QA Ascend Automated
Evolutionary Testing
Review Results
Deploy Winner
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Demo: Setting Up a Test in Sentient Ascend
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
After the operator inputs the various test hypotheses, Ascend tests a first generation – each individual change is measured against the goal…
Each box represents a designEach letter represents a hypothesis
Ascend Applies Evolutionary Principles to CRO
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
How Sentient Ascend Works
+ 0 - + - + 0 + +
…and determines which ones have good “genetic material.”
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
How Sentient Ascend Works
The system then combines the successful candidates into a new generation to test…
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
How Sentient Ascend Works
…and again selects the winners in that population.
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
How Sentient Ascend Works
The breeding continues, with mutations to ensure we explore all corners.
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
How Sentient Ascend Works
Within 4 generations you see massive impact; by 8 to 10 generations Ascend converges on the winning design
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Faster Results with Sentient Ascend™
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
The New Era of Massively Multivariate Testing
• Test dozens of site improvements at once
• Determine the best combination of those changes to achieve your growth goals
• Evolutionary algorithms can determine the best performer from 1 million designs while only testing 500–600 of them, in generations of 20–40 candidates at a time
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Always-On, Continuous OptimizationAscend always tests in the background and adapts your site presentation to provide the best results if your audience changes for any reason.
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Results & Case Studies
Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Ascend Customer Successes160Test combinations
4Funnel pages tested
38.4% Increase in conversions (best design)
Goal: Increase checkout conversion rate
90% Increase in conversions (best design)
21.5% Cumulative lift across all pages
1,152 Test combinationsGoal: Increase checkout
conversion rate
48% Increase in conversions (best design)
11% Cumulative lift across all pages
380,000 Test combinationsGoal: Increase lead-form
submits
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Q&A with Peep LajaRanked Most Influential CRO Expert in the World by PPC Hero
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Thank you!LEARN MORE
sentient.ai/[email protected]