How the JDeveloper team test JDeveloper at UKOUG'08

<Insert Picture Here>

How the Oracle JDeveloper Team Test JDeveloperGerard Davison : Senior Principle Software EngineerGeoff Waymark : Principal QA Engineer


Presentation Agenda

• Introduction• Automation• Tools• Abbot• Selenium• Conclusion


Introduction

Introduction

• Admission– We are not testing saints– We do not have 100% code coverage– We probably never will

• But– We have tried and succeeded in testing the ‘hard bits’– We have learned some lessons– Turns out a little can go a long way

Introduction

• JDeveloper– We have everything from Swing to Excel• Our focus is on the Java parts today

– We tend to be using technologies before you• We create some of the technology• This means we are ahead of test tools

– JDeveloper 8mloc in 35k source files over 650 modules• 4 continents, at least 6 versions of English

– Two builds• Sanity – verify no broken code• Nightly - full release build ready for a customer


“To build may have to be the slow andlaborious task of years. To destroy canbe the thoughtless act of a single day ”

BuildingSir Winston Churchill

Introduction -Terminology

• Unit testing– The testing the Java Classes directly

• Integration testing– Testing it all working together

• Automation– Making it all work whilst you are in bed / lunch / pub

• Poking tooling– Pretending to be you whilst you are in bed / lunch / pub


Automation

Automation

What to automate?The big question

What to automate?

• What is it for?– Cost saving• Humans - expensive, quite squishy – not entirely reliable• Machines - comparatively cheap, don’t get bored

– Build verification• Is today's build worse than yesterdays?

– Ready to use• Most worn path works• What would have to work to demo to your boss works

– Dull things– Performance

What to automate?

• Where to begin?– Fit to test / DFT• Earlier rather than later• Handover automation from Dev to QA– Unit testing– Smoke testing with poking tools

• When to begin?– Unit testing, straight away, if not sooner– Integration testing takes place later in the cycle• Less code churn• Fewer UI quakes

What to automate?

• What type of Automation?– Unit testing – can only take you so far• Verifies that what the developer thinks should happen

happens– Integration testing – better, but you need more complicated

tools• Verifies that what the rest of the tool thinks should happen

happens

How do we automate?

• Calling Java APIs– Unit testing - in complicated situations it can be hard to set up

an environment.– Often requires mock objects as things rarely work in isolation

• Using a Poking tool– Control swing using a java.awt.Robot based tool– Running a HTML based service from a web browser– Likely native to the OS– Better representation of what the user will actually see

What to automate?

• How much Automation?– Build verification is limited by the time it takes between builds

– Use code coverage to meet a set value• Lines in the sand are easy to measure– Less value than you’d think early on.

– Less is more• Don’t have more results than you can analyze• Can you fix the tests In good time if they break


Tools

Build tooling

• Build tools– Needs to• run automatically– Generally as a result of SCM operation

• notify you when something goes wrong– SMS alerts and the like*

• provide nice reporting, rss feeds and the like• be beneficial for the developer

* Important for “Testing whilst at the pub”

Build tooling

• Build tools– We have lots of custom code built up over years– Get an answer as quickly as possible• Sanity builds• Run tests in parallel

Build 1S Build 1N Build 2S

SRG 1S SRG 1N

LRG 1N

SRG 2S

Build tooling

• Cruise Control– Ant wrapper– Email notification

• Hudson– Lots of web 2.x UI– Using Webstart you can add machine to build environment• Useful if you have many time zones

• cron or Windows scheduler– Not really a build tool but can assist


“Consistency is contrary to nature,contrary to life. The only completely

consistent people are the dead..”

ConsistencyA. Huxley

Automation Environment

• Work on a Dev machine the same as on theAutomated test machine

• Make sure your test environment runs on all platforms– We have to hand test on a Mac still

• Just like Java - write once, debug everywhere– Platform specifics like file separators, install locations

• Select which browsers you want to run in– Firefox, IE, Chrome? What version?

• Size is important - what display size should you runyour tests in?– Do you have a minimum display size for your app

Automation Environment

• Quick set ups– Don’t want to test the setup 100 times• We use zips to store project context• Start tool and run inside rather than once for

each test• Collect forensics– Make a copy of the logs - ideally for each test– If doing UI testing, take a picture when it fails

• Keep developers happy– It you don’t make it easy tests won’t get run– Assign someone smart to set this up.

Testing Tools Environment

• Integration testing gets asynchronous real quick,– Harder than nice linear JUnit testing– “Wait don’t delay”– Make sure your testing tool understands this

otherwise find a new one

Testing Tools Environment

Other tools worth mentioning

• Free– FEST– JFCUnit– HtmlUnit–WebTest

• Expensive–WinRunner– SilkTest– QTP


Abbot

Abbot

• Automation of Swing using java.awt.Robot– Also SWT version, not used it though

• Provides a Java API• We use the ability to record and playback xml scripts– Reduces the skill level required to create a test

• Is extensible so can handle custom components• Gerard is a committer so we can fix and extended

where necessary

Costello

• Watches that you are doing• Record actions steps• Creates references to components– Not brittle, will survive UI code changing

• UI a little bit special• Does the job, can be a lot quicker than writing code• You have to do some tidying up later though

Integration into JDeveloper

• Invoke tests / Costello inside of JDeveloper– For faster turnaround

• Can invoke tests from command line• Some small tweaks to make Costello work better– Integration with our source control system– Better default scripts


Abbot Demo

Abbot tips

• Use the “Name” property to distinguish similarcomponents

• Wait don’t delay– Ignore progress bars, test for something more reliable

• Run tests in “read-only” displays– VNC useful for this on Unix– Fast user switching for the same effect on Macs– Windows users just have to go for a cup of tea

• Disable all popup notification– Nothing more annoying than an IM ruining your test run– Screensavers are great but won’t help the tests

• Don’t record mouse movements, wont run again

Abbot tips

• Call– Invoke static Java methods

• Script– Run BeanShell script

Abbot in JDeveloper facts and figures

• An example– DB team (Our best team – test coverage speaking)• Connections = 70% code coverage (block)• Modeller = 68% code coverage (block)• 86 Abbot tests• 889 JUnit tests• 40% coverage is JUnit on it’s own• Abbot on its own would also be 40% showing the

intersection of the tests


Selenium

Selenium

• Suite of tools

– Remote Control• Integrates tests with your build

– IDE• Exclusive to Firefox

– Grid• Not used by Oracle– We have an internal tool that predates Grid

Selenium Remote Control

• Comes in two parts

– Server• Starts and stops the target browser• Acts as an http proxy for requests• Also includes the selenium core

– Client libraries• Client libraries for your favourite language• Flexible

– Integrates nicely with existing reports JUnit,etc…

Selenium IDE

• Creates tests– Extension to Firefox

– Simple to use• Record and playback• Insertion for hand coding• Looks like a nicer Costello

– Save tests in your preferred language• HTML – {Default}• Java – Good for us!• C# ,Ruby ,PHP ,Python ,Perl – Good for everybody


Selenium Demo

Selenium tips

• Maximize your browser

• Has to be verification for any action performed– If you clicked a button, what did it do?

• Assertions should have meaningful messages

• Document the test– Clearer for everybody– Test spec– JavaDoc for Java

Selenium tips

• Limited to the DOM tree– Stylesheet or Style classes are not inherently testable

• Use clear ID’s– ‘table1:0:openPopup’ is better than ‘table1:0:link1’

• Improve your test to get more from it– Refactor IDE recorded tests• Reliability• Internationalization• Data driven tests


Conclusion

Conclusion

• Automated tests

– Have some – it helps

– Not too much though – it takes time

– Build and Test – have to be linked

– Development and QA in partnership

For More Information

• Abbot– http://abbot.sourceforge.net/

• Selenium– http://selenium.seleniumhq.org/

• Your speakers– [email protected]• http://kingsfleet.blogspot.com/

– [email protected]

How the JDeveloper team test JDeveloper at UKOUG'08

Technology

Transcript of How the JDeveloper team test JDeveloper at UKOUG'08