408 ebooklet dominic mass final

Dominic Maes Sr. Test Consultant

Testing ‘Slow Flows’ FastAutomated End-2-End Testing Using Interrupts

In an ‘online’ world, everything seems to be going faster and faster. Too often, we forget about ‘slower’ processes that need to be tested in our information services. Automated end-2-end testing on these flows, which use independent systems and timers, is not that difficult. Using a record and playback tool with a framework is all it takes. You’ll simply have to add an ‘action queue’ to your framework and you’re back on track for testing thousands of test cases.

20112011

20112011

MAnchesTer , UK21 - 24 november 2011

Testing ‘Slow Flows’ Fast - Automated End-2-End testing using interrupts

1

Contents

Introduction 2

Goal 2

The Challenge 2

The Quest for a Solution 3

A Short Evolution of Record - and Playback Tools 4

Setting up the Solution 5

Let’s Rock ’n Roll! 8

Drawbacks 9

The Summary 10

References 11

Biography 11

PAG

e

w w w . e u r o s t a r c o n f e r e n c e s . c o m


2

PAG

e

Introduction We live in a world where instant fulfillment is a keyword when doing business. Everything seems to be going faster and faster in our ‘online’ world. Customers don’t want to wait for services and products anymore: ‘Wow, I need that iPhone right now!’, ‘I need those tickets for your concert tomorrow’, etc… Modern economics are based on these consumer principles. Too often, we forget about ‘slower’ processes that need to be automated in our information services. And let me be clear, ‘slower’ is still relatively fast. We are not talking days or months here, but minutes and hours.

Test – automating those processes can be a bit tricky. Why even bother using test automation tools to complete a flow that can take several hours? Automated end-2-end testing on these ‘slow flows’, that use independent systems and are perhaps sensitive to different timers, is not that difficult. Using a record and playback tool combined with some scripts is all it takes. You simply need some kind of scheduler and you’re back on track for testing thousands of test cases.

By using an ‘action queue’ in your automated test framework, in combination with a scheduler, you can easily adapt your automated scripts to perform many actions while the tool is waiting for processing to finish. This kind of ‘interrupt processing’ is a solution for every application flow that can take up many hours in total before having finished and where interaction between independent systems is reflected in a database. This framework can even check timers and delays of processes without much hassle. To me, this type of framework has proven its reliability and usability. It has become a standard for automated testing of high volumes in ‘slow processing’.

Too often these kind of ‘slow processes’ are considered difficult and impossible to test and are often only tested in a system testing phase. With this approach, these often critical processes can be tested end-2-end which will give you the extra security, by reducing the risks involved in using multiple independent systems with time critical processing.

Goal This paper is the representation of my experience in my quest in finding a ‘fit for purpose’ framework in automated testing and dealing with time-related issues while testing. It describes a short evolution of record and playback tools and the final solution I implemented.

The Challenge Some 10 years ago, I was working in a company where I got a special challenge to automate test cases with a record and playback tool so that we could execute 10,000 test cases a day. As I had already quite some experience in developing test automation frameworks, it was not a big challenge for me. Creating a test data file, setting up the framework to repeat input and check database contents would do the trick. The pitfall came in the process-flow to test: as the real operational flows unfolded, I discovered that processing time for a single test case could vary strongly according to the

TWeeTABLe

Too often these kind of ‘slow processes’ are considered difficult and impossible to test and are often only tested in a system testing phase.


3

PAG

e

context in which the test case was executed. Due to the great number of communications with different subsystems, execution time of a test case could be between 3 minutes and 8 hours. In a worst case scenario, you could end up with only 3 test case executions per day, hardly worth automating. But the project manager was pretty clear: aim was to execute 10,000 test cases a day.

To protect the privacy of my client, I changed the context and the business, but kept the challenges in the examples that I have used in this paper.

The Quest for a Solution Why use automated testing?

A first question you should always ask in any project you’re testing is: ‘Do I really need test automation for my project?’ The answer can go either way. Initially, test automation tools were seen as a way to speed up the test process, once testing got on the critical path of the project. In a world of rapid application development and ‘releases that should have launched before the project started’, a tool that shortens the project lifecycle would come very handy. As there are many different testing tools, taking care of many aspects of testing (i.e. test management, test data creation, comparators, record- and playback, etc...) we will only focus on test execution tools.

If you look closer to the testing work inside a project, you cannot deny the advantages

that come with the use of a test automation tool:

a) A tool is less error prone than a human tester: i.e. it will do exactly what you have told it to do, time after time, with a great accuracy.

b) It brings more work enjoyment: Executing test cases over and over again under time-pressure is most of the time a real tedious job. A tool will never complain.

c) As modern test execution tools have possibilities to ‘speed up’ test case execution, a higher volume of test cases can be processed in the same timeframe (compared to human test execution).

d) By using a tool, you will have the opportunity to monitor more aspects of your test execution, which were otherwise unable to follow (manually). E.g. in stress testing, CPU and memory usage can be closely monitored while executing test cases.

Unfortunately, test automation has pitfalls, waiting to counter your test automation efforts:

a) ‘If you automate chaos, you’ll only get more chaos’. Make sure you have a structured way of organizing your test process before you start automating it.

b) ‘A fool with a tool is still a fool’: Be smart in what you automate. Avoid replicating your application logic in your test automation framework. A test automation framework should be there to input and check test data. Nothing more, nothing less.

c) Test automation is a separate process/ activity. Don’t think you can automate your test cases ‘in between’ your normal testing activities. Consider

TWeeTABLe

A first question you should always ask in any project is: “Do I really need test automation in my project?”


4

PAG

e

a test automation project as a development project, needing proper people to handle it. After all, you’re developing/updating a framework.

d) Test execution tools cannot test every application. Look for a balance in possibilities of the tool(s) at hand and the effort you need to automate your test case execution.

e) Test tools are not ‘magical’. Unlike what vendors might tell you, testing with a press on the button is not possible unless you invest a great deal of time to program the framework behind the button.

f) Test tools are not cheap. Ideally test execution automation should be used mostly for regression testing. Try to avoid using them in situations where new applications are being developed or where your framework will only be used once. Calculate an initial cost of at least 3 times the effort of creating and executing manually your test cases when you plan to automate them (with or without a framework). Ideally test execution automation should be used mostly for regression testing.

g) And don’t forget to calculate maintenance costs for your automation framework. Your automated tests are also sensible to changes.

So think wisely at the beginning of your project whether or not you will use test automation tools. In our example, the decision was taken to go ahead and start the development of a test automation framework.

A Short Evolution of Record - and Playback Tools As I had experience in building test automation frameworks for different platforms, I went on a search of existing frameworks that would allow me to complete my challenge. It is nice to know that also automation frameworks have evolved in time. Here is a short introduction:

Plain Record and playbackIn the beginning, there were record and playback tools. They were just tools to record key strokes (and later mouse movements) that re-played them in the exact same sequence, kind of like an old VHS tape recorder. In more advanced earlier tools, there were extra’s present like fast re-play and some very basic editing functions (as these tools worked with script-like scenarios) and you could of course save the recorded scenarios.

Advantages were that test cases could be replayed more accurately and faster than a tester could do. Disadvantages were: the difficulty in maintaining the scripts created by the record and playback tool, especially when there was a need of extensive scripts, which manifested the lack of adaptability of the scripts to new test data.

Data drivenIn a later stage, tools were equipped with the possibility to use variables in the recorded scripts, so that changing test cases/test data could be done very easily. The base recorded script is kept, but variables are used to replace the hard coded test data. This functionality allowed re-using the same base-script with different test data sets. It was the first step towards a test harness or test automation framework. Although this was a step in the right direction, it was far from an ideal situation. Maintenance on the scripts remained difficult.


5

PAG

e

Modular frameworkJust as in development languages, the scripting languages for testing tools were also evolving towards more modular programming. The ability to make small chunks of code that could be re-used made this framework very flexible and maintenance-friendly. These small modules of code were then used in a hierarchical fashion to construct larger tests, realizing a particular test case.

Keyword drivenIn the search of a framework that would be easy to maintain but could also give the possibilities to less experienced testers to create test cases at an abstract level, the keyword driven approach was born. The strength of this type of automation framework is its reusability and the fact that once created (by experienced test automation engineers) it can be used by testers with no specific automation experience. All you need as a tester is a list of keywords to create a complete scenario for test execution. The framework then translates this into understandable actions performed on the application under test. The biggest disadvantage is that for every new function or action to be executed by the tool, an appropriate new keyword section needs to be added in the framework.

Model BasedIn search of a more efficient testing approach, model based testing was born. In model based testing, the testers start from a set of rather abstract test cases, derived from a (sometimes mathematical) model. The model is a partial representation of the behavior of the application under test. Finally the abstract test cases are transformed into real test cases to test the application. In some model-based testing tools, the abstract test cases are automatically transformed to an executable test set, provided that there is enough information available. In other cases, transformation to a usable test case set has to be done manually.

Hybrid testing frameworkThe Hybrid Test Automation Framework is what most frameworks evolve to over time and after repeated use. The most successful automation frameworks generally accommodate both keyword-driven testing as well as data-driven testing. This allows data driven scripts to take advantage of the powerful libraries and utilities that usually accompany a keyword driven architecture. In these kinds of frameworks, a careful balance between maintainability and completeness needs to be drawn. Too often, these frameworks become almost a copy of the applications they are bound to test, but that should never be the purpose of a testing tool.

None of the above mentioned Frameworks would be sufficient to fulfill the challenge given to me; I needed to find my own solution.

Setting up the Solution Preparing the business flow

As a first step, it is necessary to divide the (business) flow into little (sub-) processes. The best way to do that is to look at points in the process where you expect responses, or need to wait. It is not really necessary to have a screen or graphical user interface, although you can use screens to check your test case status instead of going directly into the database. By simply using a status screen (i.e. for checking the order status in the example) you avoid the use of queries

TWeeTABLe

Too often, these frameworks become almost a copy of the applications they are bound to test, but that should never be the purpose of a testing tool.


6

PAG

e

to dive into the databases. The business flow would look like this, once divided into (sub -) processes:

Suppose this is the workflow that every order has to pass. There are 10,000 test cases. Total testing time can run up to 10.000 times 3h and 45 min. This is impossible to do manually because of the cost and the time of the test execution.

The test cases to execute are listed in a file containing the start time of each work flow, i.e. every test case workflow starts with Process A.

A complete flow would contain the following sub-processes:

A) Online Purchase

B) Check Supply

C) Check Client status

D) Order missing items

E) Send Items

F) Invoice

The interrupt automation framework

The framework I usually start from is a hybrid form. It’s a keyword and data-driven framework that is setup in a modular way for maintenance purposes. Due to my interest in hardware and electronics, I searched here for answers to my challenge. I remembered interrupt requests or IRQ’s, a way to stop the normal action of a computer component and get it to do something ‘in between’. Interrupts are a way to achieve multi-tasking in real time computing and it contained similarities to my challenge.

If we look a bit closer to a typical interrupt mechanism, it will look like the following diagram overleaf:

‘System Startup’: The initial blocks are there for initialization of the framework and loading initial data. At this point the action queue is filled for the first time with the test data of the test cases that we want to execute. The initial test data set can come with timings prepared, i.e. each test case has its scheduled time already included, or the System startup module can take care of automatically spreading the test cases in time. It will however respect the order in which the test cases have been offered.

Then the control is handed over to the ‘Central system’, containing a scheduler. The scheduler is comparing the system clock with the timing of the first action in the queue. If the system time is greater than or equal to the timing mentioned in the first action’s data set, it will start processing the test case. The different parts of a process flow are represented in the diagram by a ‘pile’ centralized around a ‘Task block’.

Each ‘Task Block’ contains a ‘context restore’ before and a ‘context save’ after the task. A task in this diagram is representing a (sub-) process of the entire flow to test. The context restore is taking data out of the test data set to go back to the previous point in


7

PAG

e

the test cases’ flow or the initial test data arguments for the first (sub-process) of the flow to test. The context save of the block is responsible for saving the status of the test case back in the action queue, ready to process the next part of the flow in the test case.

The ‘Switch block’ is deleting the executed

action out of the list and sorts the action list again to prepare it for the next loop.

The ‘Calls block’ on the right contains modules to address specific functions in reporting and error-handling. Also a big part of generic functions (e.g. date handling, database access and querying, etc.) is handled in this block.

System startup

System initialization

Central system

Scheduler

Context restore

Context save

Task : Process A

Switch

Context restore

Context save

Task: Process B

Context restore

Context save

Task : Process E

Context restore

Context save

Task : Process F

calls

Reporting

Generic functions

Error Handling

Action Param 1 Param 2 ….

Add Order 12455 Mr. Smith ….

Add Order 12456 Ms. Simpson …. Add Order 12457 Mr. Doe …

Execution Timestamp Action Param 1 Param 2 …. 01/12/2010 - 08:00:00 Add Order 12455 Mr. Smith …. 01/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

Execution Timestamp Action Param 1 Param 2 …. 01/12/2010 - 08:00:00 Add Order 12455 Mr. Smith …. 01/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe … 01/12/2010 - 08:10:15 Check Process A 12455 …. ….

Execution Timestamp Action Param 1 Param 2 …. 01/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson …. 01/12/2010 - 08:10:15 Check Process A 12455 …. …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe …

The Action QueueThe second important part of the framework is built around the action queue. We start from a regular keyword based test data file. This could look like this:

By adding an execution time/date stamp, you’re turning it in a simple action queue. After loading the initial list of test cases, the action queue can look like this:


8

PAG

e

Let’s Rock ’n Roll!It’s time to roll up your sleeves. Here is where the action (queue) starts, let’s ‘Play Computer’ (or better ‘Automation Framework’):

Suppose that inserting an order into Process A (the start of the flow) takes as much as 15 seconds. Then the order will take about 10 minutes before it has completely passed the whole Process A.

In a conventional framework, the record and playback tool would wait before continuing. When using an action queue, the tool can perform actions instead of waiting within the time freed up. The only thing we have to add at the end of the ‘Add Order’ action in our tool is the creation of a new action to check the result of Process A some 10 minutes later. So after the initial insertion of an order (Add Order 12456), the action queue could look like:

The 15 seconds are from the processing of (sub-) Process A, while the 10 minutes come from the expected finishing time of the process. So the automation framework will check whether the result is correct (or the order has the correct state)

at 8:10:15. To finish off it will delete the action (order) that was just executed.Before taking the next action out of the queue we sort the action queue by the Execution Timestamp (ascending) to look like this:













When 08:01:00 has passed (scheduler will check with system clock), Order 12456 can be processed. And a similar

‘Check Process A’ action can be added to the queue.

Execution Timestamp Action Param 1 Param 2 …. 01/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson …. 01/12/2010 - 08:10:15 Check Process A 12455 …. …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe … 01/12/2010 - 08:11:15 Check Process A 12456 …. ….

Execution T imestamp Action Param 1 Param 2 …. 01/12/2010 - 08:10:15 Check Process A 12455 …. …. 01/12/2010 - 08:11:15 Check Process A 12456 …. …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe … 01/12/2010 – 08:10:30 Start Process B 12455 …

s.

-F in the �ow. The queue will �ll itself after the initial load of test cases and will control its own processing. When all cases have �nished, the queue will be empty and automated testing will stop. Every task can be monitored by integrating appropriate reporting functionality inside, so that the tester can, at any time, follow -up the progress of the test execution. Additionally he has a proof of the order of execution of the di�erent test cases.

Drawbac ks This ‘Interrupt -driven’ test automation framework has proved already its merits in several occasions, but it is far from perfect. These are some important drawbacks to this framework:

1) Sometimes, you just can’t cut your business �ow into (sub -)process es. Most of the time, this is the case for online applications. It does not mean that the framework will not work with this kind of applications, but it’s just not designed to unleash its full potential in these circumstances.

2) Timing accuracy can be a bit tricky.

a. The system is not designed to take into account the individual processing time of each action. I.e. if the framework is executing an action and in the meantime there is an action due at the top of the action queue, th

-


9

PAG

e

At 08:10:15 ‘Check Process A’ for order 12455 will be done. This can be anything from checking databases to inserting

new info into the system. At the end of this, a new action can be inserted in the queue ‘Start Process B’ for order 12455.

The strength of this action queue is that you can use ‘fixed’ checking times (see Process A in the example flow, that will have a processing time of 10 minutes) or ‘variable’ checking times (see Process B, C and D in the example flow, that are unknown processing times, but with a upper limit in processing time). You can even check Processes B, C and D intermediately to see whether they have already been finished. If they have not yet finished, you can add the same ‘Check Process’ action again with an interval of a few minutes.

We can implement this for every Process A-F in the flow. The queue will fill itself after the initial load of test cases and will control its own processing. When all cases have finished, the queue will be empty and automated testing will stop. Every task can be monitored by integrating appropriate reporting functionality inside, so that the tester can, at any time, follow-up the progress of the test execution. Additionally he has a proof of the order of execution of the different test cases.

DrawbacksThis ‘Interrupt-driven’ test automation framework has proved already its merits in several occasions, but it is far from perfect. These are some important drawbacks to this framework:

1) Sometimes, you just can’t cut your business flow into (sub-) processes. Most of the time, this is the case for online applications. It does not mean that the framework will not work with this kind of applications, but it’s just not designed to unleash its full potential in these circumstances.


a. The system is not designed to take into account the individual processing time of each action. i.e. if the framework is executing an action and in the meantime there is an action due at the top of the action queue, the latter one will have to wait till the framework has finished the execution of the first one. This is the trade-off that has to be made between ‘Quantity’ (of test cases to execute) and ‘Accuracy’ in timing.

Execution Timestamp Action Param 1 Param 2 …. 01/12/2010 - 08:01:00 Add Order 12456 Ms. Simpson …. 01/12/2010 - 08:10:15 Check Process A 12455 …. …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe … 01/12/2010 - 08:11:15 Check Process A 12456 …. ….

Execution T imestamp Action Param 1 Param 2 …. 01/12/2010 - 08:10:15 Check Process A 12455 …. …. 01/12/2010 - 08:11:15 Check Process A 12456 …. …. 01/12/2010 - 08:20:30 Add Order 12457 Mr. Doe … 01/12/2010 – 08:10:30 Start Process B 12455 …

s.

-F in the �ow. The queue will �ll itself after the initial load of test cases and will control its own processing. When all cases have �nished, the queue will be empty and automated testing will stop. Every task can be monitored by integrating appropriate reporting functionality inside, so that the tester can, at any time, follow -up the progress of the test execution. Additionally he has a proof of the order of execution of the di�erent test cases.

Drawbac ks This ‘Interrupt -driven’ test automation framework has proved already its merits in several occasions, but it is far from perfect. These are some important drawbacks to this framework:

1) Sometimes, you just can’t cut your business �ow into (sub -)process es. Most of the time, this is the case for online applications. It does not mean that the framework will not work with this kind of applications, but it’s just not designed to unleash its full potential in these circumstances.


a. The system is not designed to take into account the individual processing time of each action. I.e. if the framework is executing an action and in the meantime there is an action due at the top of the action queue, th

-


10

PAG

e

b. You can try to manipulate execution time stamps in such a way that test cases won’t impact each other too much, but it will come at a price: the number of test cases that can be executed in a given timeframe will be lower.

c. Fine-tuning the timings in your framework is necessary to optimize execution speed of your test cases. Some iterative runs with the same test data will give you enough information to adjust the parameters of your framework to work as optimized as possible.

3) Processing order of your test cases can be different, even if you use the same set of test data multiple times. It all depends on the reaction time of the (sub-) processes you have in your business flow. This is not necessarily a drawback, as we are focusing on completing a number of test cases anyway, rather than executing them in a specific order. The exact execution order of the different tasks (actions) is logged in the test reports generated by the automation framework.

The SummaryBy using an action queue in your automated test framework, you can easily adapt your existing test automation framework to perform many actions while the tool is waiting for processes to finish. This kind of ‘interrupt processing’ is a solution for every application process flow that can take up many hours in total before having finished and where interaction between independent systems is reflected in a (or several) database(s). It can even check timers and delays of processes without much hassle. To me, this type of framework has proven its reliability and usability several times. It has done miracles in automated testing of high volumes of ‘slow processes’. It’s a simple and straightforward asset for your hybrid testing framework with a lot of added value.

Too often these kind of ‘slow’ processes, being time-sensitive or using asynchronous communication are considered difficult and impossible to test and are often only tested in a system testing phase. With this approach, these often critical processes can be tested end-2-end and will give you the extra security, by reducing the risks involved in using multiple independent systems with time critical processing.

TWeeTABLe

As a first step, it is necessary to divide the (business) flow into little (sub-) processes.


11

PAG

e

References- Practical Model-Based Testing: A Tools Approach, Mark Utting and Bruno Legeard, Morgan-Kaufmann 2006.

- Improving the maintainability of Automated Test Suites, CemKaner, Paper presented at Quality week 1997.

- Automatisering van de testuitvoering, Bart Broekman, Christiaan Hoos, Mark Paap, 2001.

- TMap Next, for result-driven testing, Tim Koomen, Leo van der Aalst, Bart Broekman, Michiel Vroon.

Biography Graduated as Master in Commercial and Financial sciences, Dominic started his IT-career in 1997 as one of the first, specially trained, professional testers in

Belgium. He has worked his way up through several dozens of projects in different domains and economic sectors. He grew from junior tester to test expert, specialized in test methodology and testing tools. Dominic has gained experience in sectors such as retail, telecom, finance and government in Belgium as well as in the Netherlands. Dominic has an excellent knowledge of TMap Next® and is ISEB Practitioner and TMap® certified. He is experienced in coaching and supporting, giving training, test and people management, developing test strategies for projects and large programmes, writing down (master-) test plans and the pragmatic implementation of test execution and test automation. Dominic is communicative, diplomatic and pragmatic. He is a known speaker at universities, seminars and conferences. Dominic is member of the board of the Discussion group Software Testing (TI KVIV) and is co-founder of the Belgian Testers.

Join the EuroSTAR Community… Access the latest testing news! Our Community strives to provide test professionals with resources that prove beneficial to their day-to-day roles. These resources include catalogues of free on-demand webinars, ebooks,

videos, presentations from past conferences and much more...

Follow us on Twitter @esconfsRemember to use our hash tag #esconfs when tweeting about EuroSTAR 2011!

Become a fan of EuroSTAR on Facebook

Join our LinkedIn Group

Contribute to the EuroSTAR Blog

Check out our free Webinar Archive

Download our latest ebooks

w w w. e u r o s t a r c o n f e r e n c e s . c o m

The EuroSTAR BlogWritten by Testers for Testers

http://www.eurostarconferences.com/community/member/ebook-library.aspx

http://www.eurostarconferences.com/community/member/webinar-archive.aspx

http://www.eurostarconferences.com/Blog.aspx

http://www.linkedin.com/groupRegistration?gid=1798888

http://www.facebook.com/pages/EuroSTAR-Conferences/87674624681

http://www.twitter.com/esconfs

408 ebooklet dominic mass final

Documents

Transcript of 408 ebooklet dominic mass final