OTM Performance Review and Benchmarking

OTM SIG User Conference ‘08

AgendaPerformance Analysis MethodologiesOur Holistic Bottom-Up ApproachBenchmarkingOther OptionsQ&A

MethodologiesThe “Top-Down” Approach

Great for determining the root-cause of specific performance issues.

The “Bottom-Up” ApproachGreat for doing a site review and identifying issues and bottlenecks (current and potential).

The Holistic ApproachEnsure that all components (technical, functional and external) are taken into consideration.

MavenWire’s Holistic Bottom-Up ApproachOur Approach

Hardware / PlatformsOperating SystemJava TuningApplication Server TuningOTM Thread TuningOTM Diag ServletsAgentsRatesItinerariesLogsBenchmarking

Hardware / PlatformsCPU and Hardware Platform Matters!CPU Speed – Not a Good Indicator of Performance

Other factors (cores, memory bandwidth, on-chip cache) necessitate benchmarking

OTM Requires both high multi-threading and high single-thread performance

Lots of cores and high per-core performance

Performance of Current PlatformsLinux / x86-64Windows / x86-64 (note: memory limitations)SolarisHP-UX / PA-RISC

Note: HP-UX / Itanium currently unknown

AIX / POWER

Operating System / StatsReview system performance under production load for the previous 2 weeksUtilize System Tools to Monitor

sar / kSartop / prstat / topas / etc

Utilize Tools to TrendNagios / Munin / etc

Oracle DBThe following have improved DB performance

Ensure you’re updating DB stats regularlyPatched to 10.2.0.3Partitioning is enabledCURSOR_SHARING from EXACT to SIMILAROPTIMIZER_MODE = CHOOSESTATISTICS_LEVEL = ALLIn some cases the following helped

OPTIMIZER_FEATURES_ENABLE = 9.2.0

Otherwise – Tune it like a normal Oracle DBStandard DBA skill set and best practicesUtilize tools like the Oracle DB StatspackIncrease memory (PGA / SGA / etc) allocationPin frequently used packages/procedures to memoryDecrease storage IO Wait (more spindles, etc)Separate out indexes, tablespaces, logs

Java TuningOTM is HIGHLY Dependent on JVM PerformanceJRockit Performs considerably faster than other JVMsMany Current-Generation JVMs Self-Tune (including JRockit)Platform Specific ParametersAllocate as much Memory as Possible

2-3GB depending on platformFor both Web and App server

Monitor Garbage CollectionMost frequent JVM performance issue

OTM v6.0 will utilize a 64-bit JVMNOT a silver bullet!

Application Server TuningShouldn’t need to tune Apache or TomcatWebLogic vs. OAS

OTM has been running on WL since 1999May no longer be an issue with BEA acquisition

WebLogic TuningUtilize the WebLogic Console

http://otmapp.mavenwire.com:7001/console

Number of Execute Threads$OTM_HOME/weblogic/config/gc3domain/config.xml.templateExecuteThreadsShould be 70-90, depending on load

Percentage of Socket Reader Threads$OTM_HOME/weblogic/config/gc3domain/config.xml.templateThreadPoolPercentSocketReadersMay have to increase to 75

DB Connection PoolNow Tuned within the OTM Application

OTM TuningEnsure You’re Running the Latest RUThread Tuning

Take your time – verify resultsTune iteratively – avoid contentionIf you don’t know what a thread pool does – ask firstPay attention to Queue Size and Wait Time

Temporarily Tune Threads via the EventDiagServlet

Careful about killing threads!

Permanently Set Threads via the glog.properties file

OTM Thread Tuning

No Bottleneck? Check the MediatorDiagServlet

OTM Diagnostic ServletsOTM Event Diag Servlet

http://otmweb.mavenwire.com/GC3/glog.webserver.event.EventDiagServlet

OTM Topic Queue Assignments Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.event.TopicQueueAssignmentsServlet

OTM Mediator Diag Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.mediator.MediatorDiagServlet

OTM Object Lock Diag Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.synch.object.ObjectLockDiagServlet

OTM Connection Pool Diag Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.datasource.CPDiagServlet

OTM Bean Cache Servlethttp://otmweb.mavenwire.com/GC3/glog.webserver.beancache.BeanCacheServlet

Agents

Try to model agents so only one agent fires for any given eventReview your saved conditions for optimal performanceLeverage other functionality than Agents –Auto Assignment Rules, Contact Notifications

Rates

Ensure that rate offerings that are expired are also inactiveIf you have rate offerings active ensure that rate records are associated to them.Avoid redundancy in your rate structure. If you rate offering can only handle once piece of equipment there is no need to also define it at the rate record level.

Itineraries

The more itineraries valid for a given move the longer OTM will take to plan. Test using the order planning action “Show Routing Options”Simply your itineraries where ever possible

Logs

To much logging turned on will impact performance.Turn logging on as needed to troubleshootLog features that can impact performance

SQLRatingEngineRatingEngineDetailRatingEngineDebugPersistence

MavenWire Benchmark SuiteSuite of Benchmarks compiled to allow comparative hardware / platform testingDoes not require OTM installation, decreasing setup complexity and testing timeFreely available to the OTM CommunityAll Benchmarks utilized are Open Source Software (OSS), allowing for free use and modification as necessaryFull Details, including download, installation, runtime and comparison data available at:

http://www.otmfaq.com/forums/blogs/chrisplough/

Benchmarking - VolanoMarkVolanoMark

Java-based benchmark that simulates high transactional and multi-threaded loadReflects the performance of the following OTM activities

Web UI, Agents, Integration, General Workflow, General OTM Activities (not including optimization and planning based)

Higher numbers are better

Benchmarking - DaCapoDaCapo

Java-based benchmark that simulates highly computational, algorithmic, single-threaded processingReflects the performance of the following OTM activities

Optimization and Planning / Bulk Planning

Lower numbers are better

Benchmarking – Soap StoneSoap Stone

Java-based benchmark that tests data throughput between servers and replicates application protocols, such as HTTP, RMI and RAW.Reflects the throughput and protocols utilized between the various OTM Tiers

Browser / Web: HTTPWeb / App: RMIApp / DB: RAW

Higher numbers are better

Benchmarking – HammeroraHammerora

Benchmark based on the TPC-C and TPC-H benchmarks.Reflects the performance and scalability of the DB TierLower numbers are better

Other OptionsWeb Tier

Load BalancingSSL AcceleratorWAN / Web App Accelerator

App TierOTM SCA

DB TierOracle RAC

Online ResourcesPerformance

kSarhttp://ksar.atomique.net/

Nagioshttp://www.nagios.org/

Muninhttp://munin.projects.linpro.no/

BenchmarkingFull Replication Details

http://www.otmfaq.com/forums/blogs/chrisplough/

VolanoMarkhttp://www.volano.com/benchmarks.html

DaCapohttp://dacapobench.org/

Soap Stonehttp://soap-stone.sourceforge.net/

Hammerorahttp://hammerora.sourceforge.net/

Q & A and Discussion

Questions?

Thank You!

chris.plough@mavenwire.com866.343.4870 x701

www.MavenWire.com