Maximizing UI Automation – A Case Study

Maximizing UI Automation

– A Case Study

Noam Kfir | ALM Senior Consultant | Sela Group

Copyright © Sela Group 2012

The Company

Hi-tech security industry

Hundreds of employees

Offices and customers all over the world

Customers include governments and large corporations


Process Characteristics

Waterfall methodology

10 years in development

30 programmers

12 manual testers

No automated tests

Very few automated processes


Product Overview

Intelligence – gathering, managing, dispersing, analyzing

Mission-critical

More than 800 concurrent users

Real time client/server application

Many complex input forms and reports

Multi-lingual and bi-directional


The Product is Complex

More than a million LOC

3rd party control libraries (Telerik, Janus, Infragistics, and more)

Very complex application flow, many screens

WinForms embedded in WPF

Clients communicate directly with the DB instead of via the server

Integration with other company products

Difficult to debug due to security considerations

The company cannot examine client data

Limited logging


The Large Hadron Collider (LHC) is a particle accelerator located at CERN, near

Geneva, Switzerland.

It lies in a tunnel under France and Switzerland.

The Big Problem

The end users report too many bugs

The product fails to handle the expected load

The client is dissatisfied and threatens to end

the relationship

The company does not trust their product

Every QA cycle is very long (lots of “full

regressions”)

The company reevaluates its quality control

strategy


Problem SolvingWe can’t solve problems by using the same kind of

thinking we used when we created them…

Short Term Solutions

On-site QA

On-site development

On-site fixes – not synchronized with

source control

Lots of hacks and improvised solutions

Endless “lessons learned”

Off with their heads…


Long Term Solutions

Full transparency

Improve communication with the customer

Reports – more, richer, live

Improve and add automated builds

Gated check-ins

Nightly builds

Full builds

Automated deployment

Much more testing


New Testing Strategy

Unit testing

Use TFS lab environments

Easier to reproduce bugs

Snapshots

Traceability

Manual testing

End-to-end scenarios

Sanity and full regressions

Load testing – realistic conditions

Server API testing

Automated UI testing


Attempt 1: Coded UI (2010)

The Good

Intention-aware recording

Support for multiple UI technologies

Full TFS build integration including full

support for lab environments

Extensible object recognition

Very low learning curve for programmers

Same language as the product – C#

The Bad

Incomplete tooling

Object recognition needs improvement

(Janus and pure graphical controls)

Very difficult for manual testers


Coded UI (2010) ROI – After 2 Man-Months

Included with VS Ultimate

0 end-to-end tests

Excellent infrastructure


Attempt 2: TestComplete

Better object recognition

Designed for manual testers, good for

building simple keyword-driven tests

Inconsistent object recognition

Very difficult scripting platform, weird

language, very poor documentation

Not the same language as the product,

C# possible with poor COM Interop

implementation

Poor integration with TFS

Steep learning curve


The Good The Bad

Test Complete ROI – After 2 Man-Months

$15,000 for several IDE licenses

$400 per build agent

30 end-to-end tests

No infrastructure or reusable scripts

Agents too expensive for load and stress testing


Attempt 3: Ranorex

Excellent object recognition

Designed for both testers and

programmers, has IDE and .NET API

Partial TFS lab build process integration

Unlimited agents usable for load and

stress tests

TFS integration needs some work

Tests need maintenance

Medium learning curve


The Good The Bad

Ranorex ROI – After 2 Man-Months

$22,000 for 5 IDE licenses

Unlimited agents

60 end-to-end tests

Rich reusable infrastructure

After 3 man-months 120 end-to-end tests


Comparison Matrix

Coded UI TestComplete Ranorex

Target Programmers Testers Both

Cost Part of VS Ultimate $15,000 $22,000

Agents Unlimited $400 per agent Unlimited

Learning Curve Easy Difficult Medium

TFS Integration Yes No Partial

After 2 Man-months 0 tests (due to Janus)

Good infrastructure

30 tests

No infrastructure

60 tests

Good infrastructure


The Ranorex Solution

Excellent object recognition for multiple UI technologies

Load and stress tests use same UI automation platform

Reporting under one umbrella

Bugs are caught much sooner

Better collaboration between testers and programmers

Customers can see and run the UI automation tests

Fewer defects and deployment errors in production

Fewer on-site team members and sessions

Adopted by other product teams


DEMO

UI Automation with Ranorex


Thank You!

Email: [email protected]

Blog: http://noam.kfir.cc

Twitter: @NoamKfir


Maximizing UI Automation – A Case Study

Software

Transcript of Maximizing UI Automation – A Case Study