Post on 14-May-2015
description
<Insert Picture Here>
How the Oracle JDeveloper Team Test JDeveloperGerard Davison : Senior Principle Software EngineerGeoff Waymark : Principal QA Engineer
<Insert Picture Here>
Presentation Agenda
• Introduction• Automation• Tools• Abbot• Selenium• Conclusion
<Insert Picture Here>
Introduction
Introduction
• Admission– We are not testing saints– We do not have 100% code coverage– We probably never will
• But– We have tried and succeeded in testing the ‘hard bits’– We have learned some lessons– Turns out a little can go a long way
Introduction
• JDeveloper– We have everything from Swing to Excel• Our focus is on the Java parts today
– We tend to be using technologies before you• We create some of the technology• This means we are ahead of test tools
– JDeveloper 8mloc in 35k source files over 650 modules• 4 continents, at least 6 versions of English
– Two builds• Sanity – verify no broken code• Nightly - full release build ready for a customer
<Insert Picture Here>
“To build may have to be the slow andlaborious task of years. To destroy canbe the thoughtless act of a single day ”
BuildingSir Winston Churchill
Introduction -Terminology
• Unit testing– The testing the Java Classes directly
• Integration testing– Testing it all working together
• Automation– Making it all work whilst you are in bed / lunch / pub
• Poking tooling– Pretending to be you whilst you are in bed / lunch / pub
<Insert Picture Here>
Automation
Automation
What to automate?The big question
What to automate?
• What is it for?– Cost saving• Humans - expensive, quite squishy – not entirely reliable• Machines - comparatively cheap, don’t get bored
– Build verification• Is today's build worse than yesterdays?
– Ready to use• Most worn path works• What would have to work to demo to your boss works
– Dull things– Performance
What to automate?
• Where to begin?– Fit to test / DFT• Earlier rather than later• Handover automation from Dev to QA– Unit testing– Smoke testing with poking tools
• When to begin?– Unit testing, straight away, if not sooner– Integration testing takes place later in the cycle• Less code churn• Fewer UI quakes
What to automate?
• What type of Automation?– Unit testing – can only take you so far• Verifies that what the developer thinks should happen
happens– Integration testing – better, but you need more complicated
tools• Verifies that what the rest of the tool thinks should happen
happens
How do we automate?
• Calling Java APIs– Unit testing - in complicated situations it can be hard to set up
an environment.– Often requires mock objects as things rarely work in isolation
• Using a Poking tool– Control swing using a java.awt.Robot based tool– Running a HTML based service from a web browser– Likely native to the OS– Better representation of what the user will actually see
What to automate?
• How much Automation?– Build verification is limited by the time it takes between builds
– Use code coverage to meet a set value• Lines in the sand are easy to measure– Less value than you’d think early on.
– Less is more• Don’t have more results than you can analyze• Can you fix the tests In good time if they break
<Insert Picture Here>
Tools
Build tooling
• Build tools– Needs to• run automatically– Generally as a result of SCM operation
• notify you when something goes wrong– SMS alerts and the like*
• provide nice reporting, rss feeds and the like• be beneficial for the developer
* Important for “Testing whilst at the pub”
Build tooling
• Build tools– We have lots of custom code built up over years– Get an answer as quickly as possible• Sanity builds• Run tests in parallel
Build 1S Build 1N Build 2S
SRG 1S SRG 1N
LRG 1N
SRG 2S
Build tooling
• Cruise Control– Ant wrapper– Email notification
• Hudson– Lots of web 2.x UI– Using Webstart you can add machine to build environment• Useful if you have many time zones
• cron or Windows scheduler– Not really a build tool but can assist
<Insert Picture Here>
“Consistency is contrary to nature,contrary to life. The only completely
consistent people are the dead..”
ConsistencyA. Huxley
Automation Environment
• Work on a Dev machine the same as on theAutomated test machine
• Make sure your test environment runs on all platforms– We have to hand test on a Mac still
• Just like Java - write once, debug everywhere– Platform specifics like file separators, install locations
• Select which browsers you want to run in– Firefox, IE, Chrome? What version?
• Size is important - what display size should you runyour tests in?– Do you have a minimum display size for your app
Automation Environment
• Quick set ups– Don’t want to test the setup 100 times• We use zips to store project context• Start tool and run inside rather than once for
each test• Collect forensics– Make a copy of the logs - ideally for each test– If doing UI testing, take a picture when it fails
• Keep developers happy– It you don’t make it easy tests won’t get run– Assign someone smart to set this up.
Testing Tools Environment
• Integration testing gets asynchronous real quick,– Harder than nice linear JUnit testing– “Wait don’t delay”– Make sure your testing tool understands this
otherwise find a new one
Testing Tools Environment
Other tools worth mentioning
• Free– FEST– JFCUnit– HtmlUnit–WebTest
• Expensive–WinRunner– SilkTest– QTP
<Insert Picture Here>
Abbot
Abbot
• Automation of Swing using java.awt.Robot– Also SWT version, not used it though
• Provides a Java API• We use the ability to record and playback xml scripts– Reduces the skill level required to create a test
• Is extensible so can handle custom components• Gerard is a committer so we can fix and extended
where necessary
Costello
• Watches that you are doing• Record actions steps• Creates references to components– Not brittle, will survive UI code changing
• UI a little bit special• Does the job, can be a lot quicker than writing code• You have to do some tidying up later though
Integration into JDeveloper
• Invoke tests / Costello inside of JDeveloper– For faster turnaround
• Can invoke tests from command line• Some small tweaks to make Costello work better– Integration with our source control system– Better default scripts
<Insert Picture Here>
Abbot Demo
Abbot tips
• Use the “Name” property to distinguish similarcomponents
• Wait don’t delay– Ignore progress bars, test for something more reliable
• Run tests in “read-only” displays– VNC useful for this on Unix– Fast user switching for the same effect on Macs– Windows users just have to go for a cup of tea
• Disable all popup notification– Nothing more annoying than an IM ruining your test run– Screensavers are great but won’t help the tests
• Don’t record mouse movements, wont run again
Abbot tips
• Call– Invoke static Java methods
• Script– Run BeanShell script
Abbot in JDeveloper facts and figures
• An example– DB team (Our best team – test coverage speaking)• Connections = 70% code coverage (block)• Modeller = 68% code coverage (block)• 86 Abbot tests• 889 JUnit tests• 40% coverage is JUnit on it’s own• Abbot on its own would also be 40% showing the
intersection of the tests
<Insert Picture Here>
Selenium
Selenium
• Suite of tools
– Remote Control• Integrates tests with your build
– IDE• Exclusive to Firefox
– Grid• Not used by Oracle– We have an internal tool that predates Grid
Selenium Remote Control
• Comes in two parts
– Server• Starts and stops the target browser• Acts as an http proxy for requests• Also includes the selenium core
– Client libraries• Client libraries for your favourite language• Flexible
– Integrates nicely with existing reports JUnit,etc…
Selenium IDE
• Creates tests– Extension to Firefox
– Simple to use• Record and playback• Insertion for hand coding• Looks like a nicer Costello
– Save tests in your preferred language• HTML – {Default}• Java – Good for us!• C# ,Ruby ,PHP ,Python ,Perl – Good for everybody
<Insert Picture Here>
Selenium Demo
Selenium tips
• Maximize your browser
• Has to be verification for any action performed– If you clicked a button, what did it do?
• Assertions should have meaningful messages
• Document the test– Clearer for everybody– Test spec– JavaDoc for Java
Selenium tips
• Limited to the DOM tree– Stylesheet or Style classes are not inherently testable
• Use clear ID’s– ‘table1:0:openPopup’ is better than ‘table1:0:link1’
• Improve your test to get more from it– Refactor IDE recorded tests• Reliability• Internationalization• Data driven tests
<Insert Picture Here>
Conclusion
Conclusion
• Automated tests
– Have some – it helps
– Not too much though – it takes time
– Build and Test – have to be linked
– Development and QA in partnership
For More Information
• Abbot– http://abbot.sourceforge.net/
• Selenium– http://selenium.seleniumhq.org/
• Your speakers– gerard.davison@oracle.com• http://kingsfleet.blogspot.com/
– geoff.waymark@oracle.com