Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI...
Transcript of Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI...
![Page 1: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/1.jpg)
AI and Machine Learning for Testers
Jason Arbon, CEO @Appdiff
![Page 2: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/2.jpg)
Ai for Test Automation
2
Relevant Context
Testing Neural Net Ranker
Personalized Web Search and Chrome Test Automation
AI for Mobile Test Automation
![Page 3: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/3.jpg)
Ai for Test Automation
Agenda
AI For Testing
Testing AI
Future
![Page 4: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/4.jpg)
Appdiff Presentation4
The Real Data Scientists
Joanne TsengData Scientist @appdiff.com
Appdiff mission:
Transform app
development with
automation & insightsFrancis Iannacci
Lead Data Scientist @appdiff.com
![Page 5: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/5.jpg)
Testing is much like AI5
Testing: Apply Inputs, checking outputs.
![Page 6: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/6.jpg)
https://www.extremetech.com/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they
6
AI: Apply Inputs, check outputs
![Page 7: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/7.jpg)
AI for App Testing
Features
Complexity increases
exponentially as new features and
states interact with existing
featuresTests
Test coverage grows linearly
because tests can only be added
one at a time
Time
Co
mp
lexi
ty /
Co
vera
ge
COVERAGE GAP
7
Testing Needs AI
![Page 8: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/8.jpg)
Our Story8
Definition
![Page 9: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/9.jpg)
When will AI Start Testing?
![Page 10: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/10.jpg)
Our Story10
AI for Testing
![Page 11: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/11.jpg)
What We Do11
ML : Subjective or complex labels via Humans
![Page 12: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/12.jpg)
Our Story12
AI for Testing: Input
Reduce input space
150 actions per page
35 steps -> 150^35 = 10^78 paths
#atoms in universe10^78...
![Page 13: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/13.jpg)
Our Story13
AI for Testing
![Page 14: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/14.jpg)
![Page 15: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/15.jpg)
Our Story15
AI for Testing
![Page 16: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/16.jpg)
Our Story16
AI Driven Test Flows
![Page 17: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/17.jpg)
Our Story17
AI Driven Test Flows
![Page 18: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/18.jpg)
Our Story18
AI Driven Test Flows
![Page 19: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/19.jpg)
Our Story19
Coverage
![Page 20: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/20.jpg)
Coverage
![Page 21: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/21.jpg)
What We Do21
ML : App Changes Don’t Break Bot Testing!
![Page 22: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/22.jpg)
Our Story22
AI bots can test almost any app. 10,000+ apps tested.
![Page 23: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/23.jpg)
Our Story23
Testing the Conference App
![Page 24: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/24.jpg)
Our Story24
Benchmarked Performance
![Page 25: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/25.jpg)
portfolio
![Page 26: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/26.jpg)
AI for Testing26
Current AI Coverage: ~2/3rds Existing Tests
Long Sequences of Dependant Actions and Verifications
Basic Tasks (Login, Search, Create Account, Add items to Card, etc.)
Specific Sequences of Events with Specific Input (search for ‘beanie babies’, etc.)
![Page 27: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/27.jpg)
AI for Testing27
Automation Coverage: Bots soon 100%
![Page 28: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/28.jpg)
AI for Testing28
Automation Coverage: Bots... soon after, 10X
Canonical Tests
Learning Tests Cross-app
![Page 29: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/29.jpg)
Our Story29
Testing AI: Test Graph Generation and Traversal
![Page 30: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/30.jpg)
What We Do30
Testing Flow with AI
![Page 31: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/31.jpg)
What We Do31
Abstract Intent Language: AIT
Focus on Intent, not “how”
Human Readable
App-Independant
![Page 32: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/32.jpg)
Our Story32
AI for Testing
![Page 34: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/34.jpg)
Appendix
![Page 35: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/35.jpg)
What We Do35
ML: Page Label Training
![Page 36: Learning for Testers AI and Machine - PNSQCuploads.pnsqc.org/2017/slides/AI-and-Machine... · 26 AI for Testing Current AI Coverage: ~2/3rds Existing Tests Long Sequences of Dependant](https://reader036.fdocuments.us/reader036/viewer/2022071216/6048263313ff1e6a186818a6/html5/thumbnails/36.jpg)
Our Story36
Appdiff Represents the Evolution of Software Quality
1
AD HOC TESTINGReactively test
2
MANUALTESTINGProactively test
3
TEST AUTOMATION
Automate repetition
4
AI-DRIVENAPPROACHAccelerate coverage
App
QA
App
Automation
QA
App