Selenium RC
-
Upload
praveensel -
Category
Documents
-
view
54 -
download
1
description
Transcript of Selenium RC
![Page 1: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/1.jpg)
Selenium RCoff of your desktopand onto the grid
Jennifer Bevan & Jason Huggins{jbevan, hugs}@google.com
Google, Inc.August 24, 2007
![Page 2: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/2.jpg)
2
What we have to share…
• Experience report on extending Selenium RC from desktopuse to running within a grid.
• Demo: “gridified” (© jason) Selenium RC for the masses.
– You don’t have to be Google to parallelize your selenium tests
• Future directions & addressing needs.
![Page 3: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/3.jpg)
3
Background
• Selenium RC (org.openqa.selenium) uses javascript to interactwith browsers.
– Provides ability to inspect contents of pages programmatically.
– Executes only within the scope of a browser session.
• Can’t handle all modal dialog windows:
– File upload
– File download
– Browser crash report
– Supports injection “of large chunks of badly written javascript”[S. Stewart, GTAC’07] into the page.
• Google has created a “farm” of Selenium RC machines thatfacilitates parallelized testing of multiple configurations.
![Page 4: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/4.jpg)
4
High-level Architecture
![Page 5: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/5.jpg)
5
A couple of usage statistics…
• Over 10 projects’automated/continuous test systems,and many(uncounted)individual users.
• Gmail tests,running at 1thread/test, wentfrom taking over40 minutes to 3.5minutes
![Page 6: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/6.jpg)
6
Experience Report
• We discovered many different issues while deploying andduring adoption of the Google Selenium RC farm.
– Browser/OS variances in configuration and capabilities
– Reliability and scalability issues
– Limitations that are out of the execution context of SeleniumRC
• Changes we make to Selenium RC to support our needs getpropagated to the open source repository.
![Page 7: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/7.jpg)
7
Browser/OS Issues
• Top issue: test isolation
– Firefox: RC generates a user profile specific to a test session.
– IE: RC modifies registry settings and configures the LANconnection settings directly.
• Therefore, either IE tests must be externally isolated(individual VMs, single-tracked, etc.) or RC needs to manageaccess to the shared resources.
– The latter approach not used in current RC code.
![Page 8: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/8.jpg)
8
Browser/OS Issues (cont.)
• Javascript evaluation within the browser
– Firefox (*chrome) allows tests to bypass security checks inspecific situations
– IE (*iehta) is expected to perform similarly, but isexperimental within Selenium RC.
• Therefore, we’ve only deployed *iexplore
• Basic tests (no https, etc.) are not a problem.
• Complex tests that work in *chrome are not immediatelyusable on *iehta.
• Limits ability to immediately test multiple configurations(requires test modification)
![Page 9: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/9.jpg)
9
Reliability Issues
• When we initially deployed our Selenium RC farm, thedeadlock in the current RC code was not known.
– Concurrently, ongoing work on RC increased deadlockrate, resulting in a wider awareness.
• Is not related to browser type or test complexity, so allusers experience spurious test failures.
– Defensive strategy: retry failed tests
• Google and the Selenium RC development team areactively addressing this problem (top priority).
![Page 10: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/10.jpg)
10
Reliability Issues (cont.)
• Selenium RC has a memory leak and a connection leak.
• We adopted a two-prong approach for this.
– Defensive strategy: periodically drain off tests and restart RC
– Offensive strategy: search and destroy source of leaks.
• Defensive strategy works, so for right now this is not our toppriority.
![Page 11: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/11.jpg)
11
Reliability Issues (more)
• Current Selenium RC regression test suite contains bothfunctional and unit tests.
– And yet, most of them are “happy path” (© patrick) tests.
– All of the issues we’ve found were in code that passed theRC regression tests.
– Resulting uncertainty affects ability to quickly deploy newversions (with new RC features) on the farm.
• We are contributing tests to the open source repositorythat better exercises our new code and the surroundingpre-existing code.
![Page 12: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/12.jpg)
12
Scalability Issues
• Session identification
– Current RC code uses a method of identifying sessions that isbased solely on time.
• For our purposes, this was not unique enough.
• Patch to improve uniqueness made, will be given to opensource repository.
• Multiple tests per RC instance
– Unofficial assumption by RC development team that eachsession is in an isolated VM.
• We are considering tradeoffs between this model and addingfull support for concurrent tests in one machine.
![Page 13: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/13.jpg)
13
What about Performance?
• Sure -- making RC faster would be great.
– We’re focusing on fixes that affect RC’s “gridability”: reliability andscalability
– Also, the point of creating a grid is to make all of your tests run inthe amount of time it takes your longest test to run.
• At which point performance fixes get more attention.
![Page 14: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/14.jpg)
14
Other contributions…
• Exposed bug in handling of InterruptedException withinSelenium RC.
– A waitFor…(elementName, 45000) statement “timed out” inunder 4 seconds, not 45 seconds.
– Added fix and tests for timeout calculation.
• Exposed tendency of Selenium RC to leave browsers openafter test session has ended.
– Usually when a test enters a deadlock or when a javascript “eval”stops responding.
– Current effort focused on eliminating deadlock; followup work willstrengthen browser shutdown method.
![Page 15: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/15.jpg)
15
Out-of-scope RC limitations
• A grid of Selenium RC machines is not something you want tomaintain manually.
– So if IE “encounters an error” and gets wedged to the pointwhere no new IE session can be started…” -- do you reallywant to VNC in and click “Don’t Send” to clear the IE state?
• Out-of-band communication with a farm manager to handleplatform-specific browser error handling is necessary for large-scale deployment.
– And once you have the requisite ‘watchdog’ client on the RCmachine, you can also use it for metric collection, managingconfiguration recovery, etc…
![Page 16: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/16.jpg)
16
(and now we switch to Jason…)
![Page 17: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/17.jpg)
17
Which is faster? (1 server, sequential tasks)
Time
Servers
1
2
3
4
![Page 18: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/18.jpg)
18
Which is faster? (4 servers, parallel tasks)
Time
Servers
1
2
3
4
![Page 19: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/19.jpg)
19
How do you add more servers?
Thank this dude
![Page 20: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/20.jpg)
20
Make computing a true utility (for anyone)
![Page 21: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/21.jpg)
21
Pricing is cheap, but it’s not free.
$.10 per hour
==
~$74 per month
![Page 22: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/22.jpg)
22
Demo…
![Page 23: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/23.jpg)
23
Questions?
1. Are you going to open-source this?
2. Who is Paul Hammant?(http://paulhammant.com/)
![Page 24: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/24.jpg)
24
Creator of Selenium Driven aka “Remote Control”
![Page 25: Selenium RC](https://reader031.fdocuments.us/reader031/viewer/2022012316/5465dcadb4af9f301e8b4afb/html5/thumbnails/25.jpg)
25
Thank you, Paul. :-)