17_performance Testing of Batch Jobs

7/28/2019 17_performance Testing of Batch Jobs

1/16

12/11/20

Confidential 2009 Wipro Ltd

2010 Wipro Ltd - Confidential1

PERFORMANCE TESTINGOf

BATCH JOBS

ByRupesh Garg

Performance Testing Practise

2010 Wipro Ltd - Confidential2 2010 Wipro Ltd - Confidential2

Contents

1.0 INTRODUCTION1.1 BATCH PROCESSING1.2 IMPORTANCE OF BATCH PERFORMANCETESTING2.0 BATCH PERFORMANCE TESTINGPROCESS2.1 REQUIREMENTS COLLECTION

2.2 PERFORMANCE TESTING SETUP2.3 GENERAL PERFORMANCE ISSUES3.0 PERFORMANCE TESTING APPROACH FORBATCH


2/16

12/11/20



Contents

4.0 PERFORMANCE TUNING BATCH JOBS4.1 REDUCING THROUGHPUT4.2 SLOW THROUGHPUT4.3 INFREQUENT DIPS IN CPU UTILIZATIONDELAYING BATCH4.4 OUT OF MEMORY ISSUES4.5 SQL QUERIES TIMING OUT5.0 CONCLUSION6.0 REFERENCES


Batch Processing :

Batch jobs are meant for processing huge volumes ofdata without human intervention and are generallyscheduled to run during off hours so that resourcesare utilized to the best. Majority of the batch jobs areMainframe based. But they can also be written usingUNIX scripts and DOS commands. Data for the batch

job is generally present in the database. Other

methods include, fetching data by script from somefile or passing data as command line arguments.

Introduction


3/16

12/11/20



Performance of batch jobs is equally or in manycases, more important than Web based jobs dueto factors such as:

Batch jobs should be tuned to utilize all theresources to the maximum during off hours.

Irrespective of the volume of the data,performance of the batch should be goodenough to complete the processing within the

allowed time frame. Execution of batch job should never affect the

performance of other processes.

Importance of Batch Performance testing


Batch performance Testing Process

Compared to other applications, batch processesfollow a different approach cum methodology forperformance testing, points for which are listedbelow:

Requirements CollectionPerformance Testing SetupGeneral Performance issues


4/16

12/11/20



There are many factors to be observedand collected before startingPerformance test for a batch job.

Timings of the batch run - When is thebatch expected to run in production?

Time duration (hours) for which batch

would run.

Architectural design of a batch job. The

different methods and DB calls involvedwould need to be known. For eg: Servermappings to each of the Methods orPOJO or Web Service or DB calls.

Requirements Collection


Requirements Collection

Sample Architectural Diagram


5/16

12/11/20



An environment similar to productionenvironment has to be set up to performancetest the batch job. This will include a similarsystem configuration set up, all the expectedconcurrent processes running and a similar datasetup as in production.

In most of the cases since replicating productionlike environment is expensive a relatively less

configured system setup is made for testing andproportionate changes are made in the expectedthroughput during the test

Performance Testing Setup


All the system resources have to be efficientlyand optimally utilized to obtain best performancefrom batch process. So, all the system levelstatistics has to be monitored. In addition to thisall the application level statistics also will bemonitored as required.

The main monitoring counters include:

ThroughputCPU & Memory Utilization

DB Query Response Time

Application level statistics

Counters Monitored


6/16

12/11/20



No. of records processedper minute (second)statistics has to becollected. A script can beexecuted or a DB querylookup can be made tocollect to collectthroughput metrics of thebatch process over thegiven time period.

Counters Monitored - Throughput


These metrics should be always withinthe limit of what the batch job isexpected to consume. Unlike otherapplications, high CPU of 90% isacceptable for batch process providedno other concurrent tasks will run andthe application is finely tuned. This

implies optimal utilization of resources bybatch thus resulting in maximumthroughput. In case, CPU utilization ishigher than expected, its alwaysadvisable to collect server level statistics

Counters Monitored -CPU & Memory

Utilization


7/16

12/11/20



This is one of the most important monitoringcounters to be captured while doing batch jobperformance testing. Most of the Batch jobs doDB updates. If the Query response time is low,any type of code or configuration changes wontbe able to bring any commendable improvementin the batch performance.

Counters Monitored - DB QueryResponse Time


Need to keep track of no. of concurrent threadsor sessions during the batch run. These valueshave to be increased or decreased as situationdemands.

Counters Monitored - Application level

statistics


8/16

12/11/20



For applications withWindows OS perfmon isused to collect theSystem statistics. Thisincludes, total CPUutilization, Memoryutilization, process levelbreakdown of processorutilization, contextswitches, queue lengthetc.

Monitoring Tools


For UNIX and AIX therewill be scripts executingcommands in regularintervals to collect all themetrics during the testrun. There are commandswhich can collect CPU,

memory, networkstatistics, DB connectionsetc.

Monitoring Tools


9/16

12/11/20



Mercury J2EE diagnostics tool canbe used to collect most of thesestatistics. It gives the responsetimes of Web Service calls, EJBcalls, query response times,breakdown of these responsetimes across different tiers etc. Italso provides info on Total memoryand CPU utilization, concurrentthreads, no. of databaseconnections etc.

Monitoring Tools


There are few performance issues which are frequentlyseen to occur during Performance testing batch process.Similar issues are found in batch processes acrossdifferent applications because the behaviour andpurpose these batch jobs are the same. Issues oftenseen are:

Slow throughput Batch taking longer time than expected

Throughput reducing with time No. of records being processed

per minute is reducing with time.

High CPU utilization

Infrequent dips in CPU utilization Thus delaying total batchprocess

SQL Queries timing out

Out of memory issues

General Performance issues


10/16

12/11/20



The approach to be followed to Performance testbatch jobs includes:

Defining scope for Performance testing

Feasibility Study

Determine test data requirements, batch executionmethodology, data dependency, methods to create data,etc.

Identify metrics to be collected.

Set up an isolated batch testing environment.

Do necessary monitoring setup.

Create required data volume and run the batch.

Analyse the throughput and other system metrics andrepeat the test if needed.

Performance Testing Approach for Batch


Reducing throughput

Batch job works uniformly throughout the run. So itshould give uniform throughput also. If the throughputis reducing with time, it can be either due to memoryissues or specific queries leading to such ambiguousthroughput.

If its a memory issue, this throughput will bestabilized after memory reaching the required limit.

Resolving such memory issues is explained in thebelow section.

In the second case where queries are the culprit,there will be few queries whose response timeincreases with time. This graph shows the inverseproportionality which holds good for query response

Performance Tuning Batch jobs


11/16

12/11/20



Reducing throughput - Throughput


Reducing throughput - Query Response

Time


12/16

12/11/20



Reducing throughput - CPU Utilization


If the SQL queries are timing out and thedatabase utilization statistics are very less, wecan easily avoid time outs by increasing the no.of connections to database (provided the job isMulti-threaded). If database utilization is alreadymaximum then the SQL queries and databasetable has to be tuned to respond faster or the

worst case, Query time out value has to beincreased to prevent transaction failures.

SQL queries timing out


13/16

12/11/20



This is one of the most often faced issues duringPerformance testing batch jobs. This can be mainlybecause of any of the three reasons:

1. Resources are not utilized effectively. CPUUtilization in this case will be lesser than limit.Increasing the no. of concurrent threads can easilyhelp this situation. Threads should be increasedconsidering current CPU and the CPU limit.

In the graph below, CPU utilized is only 20%.Throughput of the batch can be surely increased by

better utilizing the system resources.

2. Back ends are responding slow, which in turn slowsdown the whole batch process

Slow throughput


CPU Utilization

CPU utilization is high and queries are responding fast.But still if the throughput is less, then the applicationhas to be profiled to find the methods causing high CPU

with proper profiling tools like JProbe.


14/16

12/11/20



If the CPU utilization goes down infrequently during the run, it will bedue to slowly responding backend. Whenever backend responds

slowly, it makes the processor wait for the response bringing downthe CPU utilization. Data or queries taking long time have to be

analysed and tuned to respond like other queries.

Infrequent dips in CPU Utilization delayingbatch

CPU Utilization


Out of memory issues in batch jobs are similar toother applications. Whenever the application hitsout of memory, either the application will be inneed for additional memory or there will be amemory leak in the application.

If its the first case then appropriate memory asrequired by the process has to be allocated. If

its a memory leak, the leaking objects in thebatch process has to be identified and fixedusing tools like Heap dump analyser, JProbeMemory debugger etc.



15/16

12/11/20




Heap Utilization


Most of the times, applications are primarycauses of performance bottlenecks which leadsto inefficient use of the resources. Batch jobsperformance testing much depends on thehardware utilizations, so by followingperformance optimization methods will alwayshelp in utilizing resources in efficient manner and

thus make the jobs run at its peak efficiency.

Conclusion


16/16

12/11/20


www.oracle.com

www.oracledba.in

HP, LR Communities

References

2010 Wipro Ltd - Confidential32

Thank You

17_performance Testing of Batch Jobs

Documents

Transcript of 17_performance Testing of Batch Jobs