WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage...
Transcript of WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage...
![Page 1: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/1.jpg)
Measuring Performance with WebDriver
![Page 2: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/2.jpg)
Laba diena!
Christian Bromann
Senior LeadSoftware Engineer at
Sauce Labs
christian-bromann@bromann
![Page 3: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/3.jpg)
3034 kb
3.21 sIs the average load of a webpage (Pingdom/2018)
of mobile site visits are abandoned if pages take longer than 3 seconds to load. (Study by DoubleClick owned by Google)
is the average web page size in 2018, trend: increasing(https://speedcurve.com/blog/web-performance-page-bloat/)
53%
![Page 4: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/4.jpg)
“How fast your website loads is critical but often a completely ignored element in any online business and that includes search
marketing and search engine optimisation.”—Google
![Page 5: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/5.jpg)
#perfmatters
![Page 6: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/6.jpg)
“Performance stands out like a ton of diamonds. Nonperformance can always
be explained away.”
—Harold S. Geneen.
![Page 7: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/7.jpg)
BrowserPerformance
How fast does my application load ?
![Page 8: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/8.jpg)
8
responseStart-
fetchStart=
time to first byte (TTFB)
![Page 9: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/9.jpg)
9
First Paint(FP)
First Contentful Paint (FCP)
First Meaningful Paint (FMP)
First Paint (FP)first render to the screen
First Contentful Paint (FCP)is triggered when any content is painted – i.e. something defined in the DOM
First Meaningful Paint (FMP)measures how long it takes for the most meaningful content to be fully rendered on the site.
Time To Interactive (TTI)number of seconds from the time the navigation started until the layout is stabilized
Time ToInteractive (TTI)
![Page 10: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/10.jpg)
10
(Source: Google)
computes an overall score for how quickly the content painted
Speed Index
![Page 11: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/11.jpg)
Score Based
Other Metric Types
Milestone Based
Describing a duration between two events
Resource Based
Describing performance based on a score
Describing certain resource limits
![Page 12: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/12.jpg)
Are all these metrics important?
Yes!
![Page 13: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/13.jpg)
Mapping Metrics toUser Experience!
Is it useful?Has enough content rendered that users can engage with it?
Did the navigation start successfully? Has the server responded?
Is it happening?
Can users interact with the page, or is it still busy loading?
Is it usable?
Are the interactions smooth and natural, free of lag and jank?
Is it delightful?
?
?
?
?
https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics
![Page 14: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/14.jpg)
14
![Page 15: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/15.jpg)
“Fast forward to today and we see that window.onload doesn’t reflect the user
perception as well as it once did.”
—Steve Souders
![Page 16: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/16.jpg)
BrowserTracing
How the browser knowsabout all this?
![Page 17: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/17.jpg)
17
![Page 18: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/18.jpg)
18
Contains a list of events from different types that happened during the capturing process, e.g.
Duration Events (B - begin, E - end)Complete Events (x)Instant Events (i)Counter Events (C)Sample events (P)Metadata Events (M)Memory Dump Events (V - global, v - process)Other… (see Trace Event Format)
Trace data representations can be processed by a Trace Viewer tool like DevTools or Catapult
{ "name": "myName", "cat": "category.list", "ph": "B", "ts": 12345, "pid": 123, "tid": 456, "args": { "someArg": 1, "anotherArg": { "value": "my value" } }}
Event Descriptions:
![Page 19: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/19.jpg)
{ "pid": 41316, "tid": 775, "ts": 170385299237, "ph": "I", "cat": "devtools.timeline", "name": "UpdateCounters", "args": { "data": { "jsEventListeners": 31, "nodes": 4089, "documents": 9,
"jsHeapSizeUsed": 11140520 } }, "tts": 20811400, "s": "t"}
![Page 20: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/20.jpg)
{ "pid": 579, "tid": 775, "ts": 170383426118, "ph": "O", "cat": "disabled-by-default-devtools.screenshot", "name": "Screenshot", "args": { "snapshot": "..." }, "tts": 2879188825, "id": "0x1"}
![Page 21: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/21.jpg)
21
Google Lighthouse
![Page 22: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/22.jpg)
22
![Page 23: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/23.jpg)
WebDriver
How browser getautomated today?
![Page 24: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/24.jpg)
24
![Page 25: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/25.jpg)
const elem = $("#myElem")
elem.click()
ChromedriverGeckodriver
IEDriverEdgeDriverSafariDriver
AppiumSelendroid
WebDriverAgent
HTTP
Selenium Grid
![Page 26: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/26.jpg)
V
Webdriver.io
![Page 27: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/27.jpg)
ChromedriverHTTP
browser.url("https://webdriver.io")
browser.startTracing()
WebSockets
WebSockets
WebdriverIO using @wdio/devtools-service
![Page 28: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/28.jpg)
DEMO
Capture Performance Datawith WebdriverIO
![Page 29: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/29.jpg)
TestingPerformance
Using Sauce Labs new Performance Feature
![Page 30: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/30.jpg)
const elem = $("#myElem")
elem.click()
ChromedriverGeckodriver
IEDriverEdgeDriverSafariDriver
AppiumSelendroid
WebDriverAgent
HTTP
Selenium Grid
![Page 31: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/31.jpg)
const elem = $("#myElem")
elem.click()
HTTP
![Page 32: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/32.jpg)
Shift Testing To The Left
![Page 33: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/33.jpg)
Development Staging Production
Performance in the Lab Performance in the Real World
Existing SolutionsExisting Solutions
![Page 34: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/34.jpg)
SPEEDO
![Page 36: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/36.jpg)
DEMO
Capture Performance Data with Speedo
![Page 37: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/37.jpg)
import { remote } from 'webdriverio';
let browser
(async () => { browser = await remote({ user: process.env.SAUCE_USERNAME, key: process.env.SAUCE_ACCESS_KEY, capabilities: { browserName: 'chrome', platformName: 'Windows 10', browserVersion: 'latest', 'sauce:options': { extendedDebugging: true, capturePerformance: true, name: “Performance Test” } } })
await browser.url('https://www.instagram.com/accounts/login')
const username = await browser.$('input[name="username"]') await username.setValue('performancetestaccount')
const password = await browser.$('input[name="password"]') await password.setValue('testpass')
const submitBtn = await browser.$('button[type="submit"]') await submitBtn.click()
await browser.deleteSession()})().catch(async (e) => { console.error(e) await browser.deleteSession()})
$ speedo analyze “Performance Test” \ -p https://www.instagram.com/ \ --all
Check Performance forInstagram Login
![Page 38: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/38.jpg)
pipeline { agent none stages { stage('Linting') { ... } stage('Unit Tests') { ... } stage('Functional Tests') { ... } stage('Performance Tests') { agent { docker { image 'saucelabs/speedo' } } steps { sh 'speedo run https://google.com -u ${SAUCE_USERNAME} -k ${SAUCE_ACCESS_KEY} -b ${BUILD_NUMBER}' } } }}
Ready For CI/CDSpeedo was build to run within your continuous integration pipeline!
variables: SPEEDO_IMAGE: saucelabs/speedo
stages: - lint - test - performance - deploy
# ...
# run performance testsperformance: stage: performance image: $SPEEDO_IMAGE script: - speedo run https://google.com -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY -b $BUILD_NUMBER
# ...
![Page 39: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/39.jpg)
const submitBtn = await browser.$('button[type="submit"]') await submitBtn.click()
const result = await browser.assertPerformance( 'My Performance Test',
['speedIndex', 'timeToFirstInteractive'])
expect(result.pass).toBe(true)
Test Performance within a WebDriver test
/session/:sessionId/sauce/ondemand/performance WebDriver Extension
JS Executor (Selenium Python)
driver.execute_script('sauce:performance', {“metrics”: [...]”})
![Page 40: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/40.jpg)
/** * Test performance of hard page transition */browser.url('https://postmates.com')let result = await browser.assertPerformance(JOB_NAME, ['score'])assert.equal(result.result, 'pass', 'Performance test for opening main page did not pass')
/** * Test performance of soft page transition */const username = await browser.$('#e2e-geosuggest-input')await username.setValue('San Francisco')const submitBtn = await browser.$('#e2e-go-button')await submitBtn.click()
result = await browser.assertPerformance(JOB_NAME, ['score'])assert.equal(result.result, 'pass', 'Performance test for the feed did not pass')
![Page 41: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/41.jpg)
Jankiness
The Browser A JavaScript Powerplant
![Page 43: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/43.jpg)
https://discuss.httparchive.org/t/cpu-time-breakdown/1510
![Page 44: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/44.jpg)
“Jank is any stuttering, juddering or just plain halting that users see when a site or app isn't keeping up with the refresh rate. Jank is the result of frames taking too long for a browser to make, and it negatively impacts
your users and how they experience your site or app.”—jankfree.org
![Page 45: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/45.jpg)
![Page 46: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/46.jpg)
![Page 47: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/47.jpg)
Performance Best Practices
● Functional vs. Performance Testing● Don’t worry about other browser / versions too much● Keep it simple!● Maintain one job name for one performance test● Know what you want to test
○ Scoring based metrics are the best generalised metrics○ Use others if you have more specific requirements
What to do and what not to do?!
![Page 48: WebDriver Performance with Measuring - TestCon · 3034 kb 3.21 s Is the average load of a webpage (Pingdom/2018) of mobile site visits are abandoned if pages take longer than 3 seconds](https://reader034.fdocuments.us/reader034/viewer/2022042310/5ed7854f7bbb9f68866aa9a0/html5/thumbnails/48.jpg)
Thanks!Does anyone have any questions?
https://speakerdeck.com/christianbromann/automated-performance-testing-with-webdriver
christian-bromann@bromann