7560 stress testing your peoplesoft 8 and 8.4 online applications

7560 Stress Testing Peoplesoft

Online Applications

Sridhar Srinivasan

PeopleSoft, Inc.

Leo Cai

Technology Alliances Manager

Mercury Interactive

Agenda

Stress Testing Process FlowCapacity Planning For Your Stress Test Stress Testing TechniquesPerformance MonitoringStress Testing PeopleSoft Applications

using Load Runner: Live DemoQuestions ?

Stress Testing Process Flow

Identify Transactions /

User Breakdown

Volumetric Analysis/

Service Levels

IterativeSingle Slice Tests

Load Runner Scripts

PerformanceMonitoring

Steps 6 - 9

IterativeMulti-Server Tests

Server Sizing& Network

Sizing

CapacityPlan

PhysicalLayout

1

2

3

4

5a

6

5b

8

97

Baseline Results

Final Results

Agenda

Stress Testing Process FlowCapacity Planning For Your Stress Test Stress Testing TechniquesPerformance MonitoringStress Testing PeopleSoft Applications


Capacity Planning For Your Stress Test

Capacity Planning: Application Side

Transaction IdentificationMost Used

User Function BreakdownWho does What? How many times?

Volumetric AnalysisPeak Load Scenarios, Present and FutureResponse Time Requirements under Load



User Breakdown


Service Levels


Load Runner Scripts


Steps 6 - 9



Sizing

CapacityPlan

PhysicalLayout

1

2

3

4

5a

6

5b

8

97

Baseline Results

Final Results


Transaction Identification

It is key to stress test transactions that have the following attributes:

Critical to the business

Most Used

Resource Intensive

High Throughput Transactions

“Use transactions that are a barometer of your business, present and future”


User Workload Breakdown

Business process frequencies vary and so do the underlying transactions. The following attributes define User Breakdown

How many transactions ?

User Type Definition: Self Service Vs Manager

Which User Performs what? How Often?

“Size transaction Load for Peaks and not for Averages”


User Workload Breakdown (Contd…)

Pace your transactions for the real-world

0

20

40

60

80

100

120

1:2

0:0

0

1:2

4:0

0

1:2

8:0

0

1:3

2:0

0

1:3

6:0

0

1:4

0:0

0

1:4

4:0

0

1:4

8:0

0

1:5

2:0

0

1:5

6:0

0

2:0

0:0

0

2:0

4:0

0

2:0

8:0

0

2:1

2:0

0

2:1

6:0

0

100 User Load – No Pacing100 User Load – No Pacing

Paced Load (100 User)Paced Load (100 User)


Volumetric Analysis Based on the transactions identified and workload characteristics recorded, the following metrics will be derived:

Daily transaction VolumesAverage Workload per hourLoad FactorTransaction Growth rate for the future

“ Well-defined Metrics help build better predictive models for real-world scenarios”


Capacity Plan

Identify Transactions

/User Breakdown

Volumetric Analysis/Service Levels

1

2

“The 80/20 Rule: >80% of your volume will involve less than 20% of your transactions”



User Breakdown


Service Levels


Load Runner Scripts


Steps 6 - 9



Sizing

CapacityPlan

PhysicalLayout

1

2

3

4

5a

6

5b

8

97

Baseline Results

Final Results


Capacity Planning : Server Sizing

How many JVMs? Average JVM Supports about between 200 to

500 users based on memory footprint.

How many Application Server Domains? JOLT Handler Sizing Per PSAPPSRV supports between 20 to 60

users based on application type CRM – 50 Users (Average Memory 100 Mb) HRMS – 40 Users (Average Memory 150 Mb) FDM – 20 Users ( Average Memory 300 Mb)


Capacity Planning : Network Sizing

Define per Transaction Footprint Browser <-> Web Server Web Server <-> App Server App Server <-> DB Server

Record Transaction Duration

Average out to find the Average Network Utilization per transaction

Use Load Factor of 2.5 to 4 to define peak Network Utilization

Calculate per segment Network requirements and Size Accordingly


Capacity Planning : Size for Peaks


Capacity Planning : Defined Layout

Web Server FarmWeb Server Farm

Application ServersApplication Servers

Web Server 1

App Server 1 App Server 2

Web Server 2

Web Server n

App Server n

Database

Load Runner Driver

Volumetric Analysis/Service Levels


Sizing

3

4

Agenda

Capacity Planning For Your Stress Test Stress Testing TechniquesPerformance MonitoringStress Testing PeopleSoft Applications




User Breakdown


Service Levels


Load Runner Scripts


Steps 6 - 9



Sizing

CapacityPlan

PhysicalLayout

1

2

3

4

5a

6

5b

8

97

Baseline Results

Final Results

Stress Testing Techniques

Stress Testing: Methodology

Configure environments based on best practices

Single Slice Testing

Multiple Server Testing


Stress Testing: Configure Environments

Web Server Tuning

App Server Tuning

Caching

Network Testing for Performance

Stress Testing: Web Server Tuning

Web Server

JOLTJOLT

ApplicationServer

TUXEDO

RDBMSServer

SQLSQL

ODBC

ServicesServices

App MessagingApp MessagingProcessorProcessor


QueryQueryProcessorProcessor


ComponentComponentProcessorProcessor


Bus InterlinkBus InterlinkProcessorProcessor


User InterfaceUser InterfaceGeneratorGenerator


ProcessProcessschedulerschedulerProcessProcess

schedulerscheduler

PortalPortalProcessorProcessor


SecuritySecurityManagerManagerSecuritySecurityManagerManager

ApplicationApplicationEngineEngine


RDBMSInstance

RDBMSInstance

ApplicationApplicationData TablesData TablesApplicationApplicationData TablesData Tables

ApplicationApplicationMeta dataMeta data


PortalPortalRegistryRegistryPortalPortal

RegistryRegistry

ServletEngine

ServletEngine

PeopleSoft PeopleSoft ServletsServlets



ThreadCount TuningN = (½ * concurrent PIA users/pagelets requests) + (½ *

concurrent Portal users) + (½ * concurrent dynamic pagelet requests).

WebLogic

Change the entry in %WLS_HOME%/<peoplesoft web domain>/config.xml as

<Server ListenPort="80" Name="PIA" Notes="The PIA server is the default server for PeopleSoft 8.40 Internet Architecture."TransactionLogFilePrefix="C:\Apps\bea\wlserver6.1/config/peoplesoft/logs/" XMLEntityCache="XMLCacheMBean" XMLRegistry="PeopleSoft IG XMLRegistry"><ExecuteQueue Name="default" ThreadCount="N" />...</Server>

Change the entry in %WLS_HOME%/<peoplesoft web domain>/config.xml as

<Server ListenPort="80" Name="PIA" Notes="The PIA server is the default server for PeopleSoft 8.40 Internet Architecture."TransactionLogFilePrefix="C:\Apps\bea\wlserver6.1/config/peoplesoft/logs/" XMLEntityCache="XMLCacheMBean" XMLRegistry="PeopleSoft IG XMLRegistry"><ExecuteQueue Name="default" ThreadCount="N" />...</Server>


ThreadCount TuningWebSphere

Set MinimumSize and MaximumSize to be the same

In %WAS_HOME%/config/server-cfg.xml, locate the following line:

<webContainer xmi:id="WebContainer_1" installedWebModules="WebModuleRef_1WebModuleRef_2 WebModuleRef_3 WebModuleRef_9 WebModuleRef_10 WebModuleRef_11"><threadPool xmi:id="ThreadPool_1" minimumSize="n" maximumSize="N" inactivityTimeout="100" isGrowable="false"/></webContainer>

In %WAS_HOME%/config/server-cfg.xml, locate the following line:

<webContainer xmi:id="WebContainer_1" installedWebModules="WebModuleRef_1WebModuleRef_2 WebModuleRef_3 WebModuleRef_9 WebModuleRef_10 WebModuleRef_11"><threadPool xmi:id="ThreadPool_1" minimumSize="n" maximumSize="N" inactivityTimeout="100" isGrowable="false"/></webContainer>


JVM Heap Sizing Set MinimumHeapSize and MaximumHeapSize the same

Improved Garbage CollectionLesser Resource Allocation during run-time

Set HeapSize to minimum of 256Mb

With JDK 1.3 onwards:Garbage Collection is multi-threadedHigher JVM sizes, upto 1024M are common

JDK 1.3 Hotspot has many performance knobs (Refer to http://java.sun.com/hotspot/performance.html)

Try –noclassgc – It might help

Make Sure Heap Size does not exceed Real Memory

http://java.sun.com/hotspot/performance.html


File DescriptorsSet OS FileDescriptor to 100 * ThreadCount

For Unix SystemsUlimit –n 32767For setting it as a hardlimit set it in a startup file like rc2.d script

Windows SystemsThere is not an explicit parameter for the number of file

descriptors. It is implicitly limited by hardware resources, mainly system memory.

Make Sure you do not exceed real memory


TCP Timeout Settings PeopleSoft is Socket Intensive

Once a session is completed, a Socket is marked for closure

Socket disconnects only when Socket Timeout Expires

Defaults range from 4 minutes to 11 minutes

Reduce related OS TCP Time Wait Parameter to between 30 seconds and 1 minute

NT Registry Setting TcpTimedWaitDelay

AIX Tcp_timewait

Solaris Tcp_time_wait_interval

HP Tcp_time_wait_interval


Servlet Reload Defines interval for JVM to automatically reload servlets

In production environments, Servlets do not change often

Set this parameter off, so that reload does not occur

WEBLOGICIn %WLS_HOME%/<peoplesoft web domain>/config.xml there is a parameter – ServletReloadCheckSecs – thatdictates how often WebLogic checks whether a servlet has been modified, and if so reloads it:

<WebAppComponent Name="PORTAL" ServletReloadCheckSecs="-1" Targets="PIA"URI="PORTAL" />

WEBLOGICIn %WLS_HOME%/<peoplesoft web domain>/config.xml there is a parameter – ServletReloadCheckSecs – thatdictates how often WebLogic checks whether a servlet has been modified, and if so reloads it:

<WebAppComponent Name="PORTAL" ServletReloadCheckSecs="-1" Targets="PIA"URI="PORTAL" />

WebSphere %WAS_HOME%/<peoplesoft web domain>/[PORTAL/PSIGW/PSINTERLINKS]/WEB-INF/ibm-web-ext-xmi (usually this is the first line):<webappext:WebAppExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI"xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmi:id="WebAppExtension_1" reloadInterval="0" reloadingEnabled="false"fileServingEnabled="true" directoryBrowsingEnabled="true"serveServletsByClassnameEnabled="false">

WebSphere %WAS_HOME%/<peoplesoft web domain>/[PORTAL/PSIGW/PSINTERLINKS]/WEB-INF/ibm-web-ext-xmi (usually this is the first line):<webappext:WebAppExtension xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI"xmlns:webappext="webappext.xmi" xmlns:webapplication="webapplication.xmi"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmi:id="WebAppExtension_1" reloadInterval="0" reloadingEnabled="false"fileServingEnabled="true" directoryBrowsingEnabled="true"serveServletsByClassnameEnabled="false">


Compression

CompressResponse=truePortalCompressResponse=true

PortalAcceptCompressedEncodings=true

CompressResponse=truePortalCompressResponse=true

PortalAcceptCompressedEncodings=true

Enable Compression (Gzip, compress) for communication between browser and web server

Sends compressed HTML to the compression-enabled (HTTP 1.1 enabled) browser, such as Microsoft IE 5 or above.

Modify Configuration.properties

Stress Testing: App Server Tuning

Web Server

JOLTJOLT

ApplicationServer

TUXEDO

RDBMSServer

SQLSQL

ODBC

ServicesServices











ProcessProcessschedulerschedulerProcessProcess

schedulerscheduler



SecuritySecurityManagerManagerSecuritySecurityManagerManager



RDBMSInstance

RDBMSInstance

ApplicationApplicationData TablesData TablesApplicationApplicationData TablesData Tables



PortalPortalRegistryRegistryPortalPortal

RegistryRegistry

ServletEngine

ServletEngine




Cache Setting

Set server cache initially to 2

In PSAPPSRV.CFG;-------------------------------------------------------; EnableServerCaching -; 0 Server File caching disabled; 1 Server File caching limited to most used classes; 2 Server File caching for all typesEnableServerCaching=2;--------------------------------------------------------

In PSAPPSRV.CFG;-------------------------------------------------------; EnableServerCaching -; 0 Server File caching disabled; 1 Server File caching limited to most used classes; 2 Server File caching for all typesEnableServerCaching=2;--------------------------------------------------------


Jolt Handlers Size based on number of users per domain

Optimally run 20 users per Handler

Per Handler Memory Footprint is about 2M

In PSAPPSRV.CFG, see below:

[JOLT Listener];======================================================================; Settings for JOLT Listener;======================================================================;Address Note: Can be either Machine Name or IP address.;Address Note: %PS_MACH% will be replaced with THIS machine's nameAddress=%PS_MACH%Port=9000Encryption=0Min Handlers=60Max Handlers=80Max Clients per Handler=20Client Cleanup Timeout=10Init Timeout=5Client Connection Mode=ANYJolt Compression Threshold=1000000



Recycle Count Set Recycle count between 1000 and 3000

Helps with optimal memory usage on App Server

JOLT Compression Large HTML Pages

Set Threshold down to 100000[JOLT Listener];======================================================================; Settings for JOLT Listener;======================================================================;Address Note: Can be either Machine Name or IP address.;Address Note: %PS_MACH% will be replaced with THIS machine's nameAddress=%PS_MACH%Port=9000Encryption=0Min Handlers=40Max Handlers=60Max Clients per Handler=20Client Cleanup Timeout=10Init Timeout=5Client Connection Mode=ANYJolt Compression Threshold=100000



PSAPPSRV Sizing

Use # of CPUs to size # of PSAPPSRV’sn Times # of CPUs

where “n” is 1,2,3..

Use pq and psr commands to size the domain.

Queueing on the domain is OK as long as it clears up quickly.

Do not forget PUB/SUB, PSSAMSRV, PSQRYSRV processes

Stress Testing: Caching

PeopleSoft Apps rely heavily on cache both at the web server and app server layer

Execute couple of runsThese runs will build a representative cache at

the web, app server and database layer

Ignore the first few runs as they are predominantly cache building exercises

Stress Testing: Network Tests

FTP Large files among the serversAverage speeds on fast Ethernet should be around

2000KBytes/Sec

Make sure adapters use the same Media Speed on either endThat is, 100_Auto_neg to 100_Auto_neg or

100_full_duplex to 100_full_duplex. Mixing of these causes network errors and hence

slowdown

Use Ping between servers to check for dropoutsUsage : ping –l <packet size> -n <# of times> server

nameTime on the ping output should be <10ms (windows) or 0

ms(UNIX)Use normal packet sizes of 1544 bytes for this testing

Stress Testing: Single Slice Testing

Run one representative web server

, app server and database server

Advantages Define baseline application characteristics – usage,

scalability Iterative runs quantify improvements in a smaller

manageable environment Refine Layout and Capacity plan with results from single slice

runs Identify all configuration issues in one set of servers and

easily propagate to a multi-server environment

App Server 1

Web Server 1

Database

Load Runner Driver

Stress Testing: Multiple Server Testing

Run tests on planned layout having multiple servers at the web and app server layer.

Identify Scalability Characteristics

Iterative runs leading to optimal performance

Web Server FarmWeb Server Farm

Application ServersApplication Servers

Web Server 1

App Server 1 App Server 2

Web Server 2

Web Server n

App Server n

Database

Load Runner Driver

Agenda

Capacity Planning For Your Stress Test Stress Testing TechniquesPerformance MonitoringStress Testing PeopleSoft Applications




User Breakdown


Service Levels


Load Runner Scripts


Steps 6 - 9



Sizing

CapacityPlan

PhysicalLayout

1

2

3

4

5a

6

5b

8

97

Baseline Results

Final Results

Performance Monitoring

Diagram goes here.

Infrastructure

Response Time

Network Components

Memory

Application

CPU

Context

User Time

Resource

Load

Storage


Windows 2000

Use Perflog not PerfMon – Perfmon is resource intensive

PerfLog is part of the Administrative Tools->Performance


Windows 2000

Define Counters at a high level – mainly CPU, memory, Disk & Network


Windows NT

Use Perflog not PerfMon – Perfmon is resource intensive

Perflog is available as part of the Resource Kit under I386\logtools folder


UNIX

CPU Use vmstat or Sar


UNIX

MemoryUse Posix PS , which allows definition of custom formate.g ps –A –o “user pid comm vsz time thcount”


UNIX

NetworkUse netstat at the ethernet adapter level repetitively

Netstat – I hme0

Connect Usage Summary

Performance Monitoring: Application

Use Web Server Access Logs

Extended logging provides hits, time taken per hit and bytes into and out of the web server

Overhead is minimal, as the web server performs buffered writes to the log file

Sample Output


Web Server Access Logs: WebLogic 6.1 Example


Tuxedo Monitoring

Queuing

What does this provide?

export TUXCONFIG=/pt818/appserv/domain/PSTUXCFGexport TUXDIR=/products/tuxedo/6.5-j1.2export TUXOFFSET=/pt818/appserv/domain

while [ "$i" -lt "$t2" ]dotmadmin -r < testsleep $t1i=`expr $i + 1`done

export TUXCONFIG=/pt818/appserv/domain/PSTUXCFGexport TUXDIR=/products/tuxedo/6.5-j1.2export TUXOFFSET=/pt818/appserv/domain

while [ "$i" -lt "$t2" ]dotmadmin -r < testsleep $t1i=`expr $i + 1`done

> Prog Name Queue Name # Serve Wk Queued # Queued Ave. Len Machine JSL 0095.00200 1 - 0 - bench22 PSAPPSRV APPQ 24 - 0 - bench22 WSL 00001.00020 1 - 0 - bench22 BBL 34817 1 - 0 - bench22 PSQRYSRV QRYQ 8 - 0 - bench22


Tuxedo Monitoring

Average Response Time of Application Server Requests

Output ExampleSVCNAME 9a-10a 10a-11a 11a-12n 13p-14p 14p-15p TOTALS Num/Avg Num/Avg Num/Avg Num/Avg Num/Avg Num/Avg ICScript 13798/0.18 15675/0.14 27754/0.18 60/0.15 28744/0.16 86031/0.20 GetCertificate 2335/0.37 2643/0.28 4711/0.38 16/0.19 5030/0.59 14735/0.44 ICPanel 2275/4.25 2603/2.32 4637/3.31 5/2.07 4781/4.34 14301/3.61 TOTALS 18408/0.71 20921/0.43 37102/0.60 81/0.28 38555/0.81 115067/0.66

Add this to the PSAPPSRV entry in psappsrv.ubx:

CLOPT="-r -e tux.out {$PSAPPSRV\Spawn Server} -s@..{FS}psappsrv.lst -s@..{FS}psqcksrv.lst -sICQuery -sSqlQuery:SqlRequest -- -C {CFGFILE} -D {$Domain Settings\Domain ID} -S PSAPPSRV"

This will create a tux.out in the domain dir. To generate the report:

Shell out of psadmin and execute:

Txrpt<tux.out


Response Time Monitoring

Load Runner Provides transaction boundaries

Response time samples for transactions at P&B are recorded using QuickTest

Database

Use the tools that the database providesTKPROF for OracleEvent Monitor / Snapshots for DB2Profiler for SQL Server

Do not have the database tracing on for your final stress test runs

Agenda




Stress Testing PeopleSoft Applications using Load Runner: Live Demo

Questions ?



User Breakdown


Service Levels


Load Runner Scripts


Steps 6 - 9



Sizing

CapacityPlan

PhysicalLayout

1

2

3

4

5a

6

5b

8

97

Baseline Results

Final Results

Related Sessions

Tuesday: Aug 26th 3:00 pmPeopleSoft Experience with Production Environment

Performance and SizingOnline Performance Tuning on PeopleSoft 8 and Beyond

Agenda

Capacity Planning For Your Stress Test Stress Testing TechniquesPerformance Monitoring Stress Testing PeopleSoft Applications


Questions ?

7560 stress testing your peoplesoft 8 and 8.4 online applications

Software

Transcript of 7560 stress testing your peoplesoft 8 and 8.4 online applications