Post on 28-Mar-2015
TestIstanbul Conferences 2012
Test Automation at the Scale of Google
Baran Ozgul
Sr. Test Engineer - Google AdSense
Olivier Gaillard
Sr. Software Engineer in Test - Google AdSense
Guessing Game
How many test cases are executed per year at Google?
100 x 10^9
~14 000 years of test execution per year
TestIstanbul Conferences 2012
Agenda
• Case study: AdSense Frontend
• Google's Innovation Culture
• The Google Scale
• Developer Engagement
• Testing Infrastructure
• Web Testing at Google
• Key learnings
• Q&A
GOOGLE'S INNOVATION CULTURE
TestIstanbul Conferences 2012
Google's Innovation Culture
"Launch early, launch often"
"The only true failure is not attempting the audacious."
• Take risks
• Launch and iterate
TestIstanbul Conferences 2012
Launch and iterate
Google+: 100 new features in first 90 days
Search: Idea-to-user in less than 48 hours
Chrome: A new version every 6 weeks
THE GOOGLE SCALE
TestIstanbul Conferences 2012
The "Google Scale"
• 60 offices, 30+ countries
• 1 giant code base everything built at latest check-in
• 11K people in R&D
• 10s of thousands of changed code lines / day
• 10s of products, 100s of feature teams
• Weekly to sub-daily release cycles
How is this possible?
TestIstanbul Conferences 2012
Paradigm Shifts!
• Developer owns code
• Test engineering = quality assurance
• Reliance on people to scale
• Test to prevent bugs
• Spread thinly
• Developer owns code + test • Test engineering =
Productivity enablers
• Reliance on common test infrastructure to scale
• Capacity to fix bugs fast
• Staff high impact projects
DEVELOPER ENGAGEMENT
TestIstanbul Conferences 2012
Developer engagement
TestIstanbul Conferences 2012
Developer engagement
TestIstanbul Conferences 2012
Developer engagement
TestIstanbul Conferences 2012
Test Engineering
"Engineering productivity"
• Test leadership• Process automation• Code health signals• Test evangelism
TestIstanbul Conferences 2012
Test Leadership
• Make existing code testable
• Implement the first N tests, lead by example
• Use data to define testing priorities
• Remove testing bottlenecks
• Make it faster to write and run the tests
TestIstanbul Conferences 2012
Process automation
• Continuous integration
• Automated releases
• Release tracking
• Production monitoring
• Automated code quality checks
TestIstanbul Conferences 2012
Code health signals
Latency (ms)
Code coverage
If you don't measure it, you can't control it.
TestIstanbul Conferences 2012
Metrics: Incremental Code Coverage
TestIstanbul Conferences 2012
Test Evangelism
• Codelabs & Techtalks ("yt: code health talks")
• Testing on the Toilet (TotT)
• Internal & external test automation events (GTAC)
• Testing intergrouplet
• Test certification
Make testing part of the development culture
TestIstanbul Conferences 2012
Test Certified Programme
• An indication test automation maturity• Show value every stage
• “Test Certified” Levels
1: Setup tools – get familiar with testing
2: Write tests for all new code
3: Get all code under test
4: Improve test coverage
5: Invest in testable code
ADSENSE FRONTEND
Case Study
TestIstanbul Conferences 2012
Case Study: AdSense Frontend
Before
• Time until next release: months
• Build broken for weeks
• 5+ release candidates
• Reliance on manual testing for release confidence
• Near absent test automation coverage
• One man to release them all
TestIstanbul Conferences 2012
Case Study: AdSense Frontend
Now
• Developers write tests for their features
• Affected tests run at every code change
• Manual testing: only exploratory
• Build gets fixed in hours
• weekly releases
• <2 release candidates per release
• 6 auto-staging releases per day
• Push-button releases
TEST INFRASTRUCTURE
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
minutes hours days
code review
Push to production
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
code ready for submission
Submit code through presubmit queue
tests fail?
minutes hours days
Push to production
code review
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
code ready for submission
Submit code through presubmit queue
tests fail?
code submittedimpacted tests run
minutes hours days
Push to productionbuilcops/build breakers
notified
code review
TestIstanbul Conferences 2012
Life of a code change (AdSense Frontend)
code ready for submission
Submit code through presubmit queue
tests fail?
code submittedimpacted tests run
minutes hours days
Push to productionbuilcops/build breakers
notified
push to production dry runcode review
TestIstanbul Conferences 2012
Test enablers
• test engineers worked on optimizing this process
• makes it easier to create/maintain testso developers are more likely to create more tests
• lots of these steps are using a common infrastructureo search for "continuous integration google"
WEB TESTING
TestIstanbul Conferences 2012
Browser Automation
Selenium 2.0/WebDriver
• Core browser automation platform (others exist)
• A collection of language specific bindings to control a browser
• Emulate true user actions
• Support for major browser manufacturers
TestIstanbul Conferences 2012
Browser Automation
WebDriver driver = new FirefoxDriver(); driver.get("http://www.google.com"); WebElement element = driver.findElement(By.name("q")); element.sendKeys("Cheese!"); element.submit(); assertTrue(driver.getTitle().contains("Cheese"));
How does this scale?
TestIstanbul Conferences 2012
Browser automation
Test Runner
Shard 1Browser 1
Test Browser
Shard 1Browser 2
Test Browser
Shard NBrowser M
Test Browser
Shard NBrowser1
Test BrowserOutput
Aggregator
TestIstanbul Conferences 2012
Browser automation (AdSense Frontend)
Test Runner
Shard 1Chrome
Test(s) Browser(s)
Server
Database
...
Shard 1Firefox
Test(s) Browser(s)
Server
Database
...
Shard 1InternetExplorer
Test(s) Browser(s)
Server
Database
...
TestIstanbul Conferences 2012
Browser Automation: Test enablers
• Test engineers worked on making it easy to write/maintain browser testso developers are more likely to more tests
• Browser tests run at every affecting code changesome costly tests are throttled
KEY LEARNINGS
TestIstanbul Conferences 2012
Key Learnings
• Push testing up-streamo Early developer testing brings biggest winso Position ourselves as productivity enablerso Make/Hire developers passionate about testing
• Invest in a great testing infrastructureo Reduce the cost of writing tests to zeroo Automate as much as possible
TestIstanbul Conferences 2012
Thanks
"Google Testing Blog""Google Test Automation Conference (GTAC)""Google Engineering Tools""WebDriver" or "Selenium"
"GTAC""Clean code talks""Web Testing at Google GTAC""Build Software at Google Scale GTAC"