Www.regoconsulting.comPhone: 1-888-813-0444 Enhancing Application Performance Root Causes and Quick...

www.regoconsulting.com Phone: 1-888-813-0444

Enhancing Application Performance

Root Causes and Quick Solutions


2

Application performance always has the potential to burden a Clarity instance and discourage users. In this session, Rego’s team of Clarity experts will review the root causes of Clarity performance issues and help you identify steps to take that will improve performance within the application and supporting infrastructure.

Enhancing Application Performance:Root Causes and Quick Solutions


3

Application performance always has the potential to burden a Clarity instance and discourage users. In this session, Rego’s team of Clarity experts will review the root causes of Clarity performance issues and help you identify steps to take that will improve performance within the application and supporting infrastructure.

● Importance of solid performance● Components impacting performance● Addressing performance problems● Real-world problems

○ Application sizing○ Database tuning & I/O○ User portlets & configuration○ Reports

Enhancing Application Performance:Root Causes and Quick Solutions


4

Performance issues impact the usability, dependability, and scalability of a system

Performance is critical to Clarity deployment and usage

Usability

ScalabilityDependability


5

Items driving performance are typically resolved by adjustments to the application configuration, application server, and database environment.

Key levers we can use to improve performance

Functional & Technical Hardware Investment Process Changes

App Configuration• Job timing• Job conflicts• Page layout• Security model• Slice settings• Period settings• Code optimization• Inactivate resources• Purge investments

DB Environment• Memory• Cores• Shared usage• Storage speed• Query performance• Long sessions• Maintenance jobs• Parameters• Reporting demand

App Server• Memory• Cores• JVM Architecture• JVM Memory • JVM GC Settings• Maintennance


6

Application configuration issues typically require a mix of both functional and technical changes. The changes are often caused by suboptimal configurations.

● Job timing○ Leverage evening downtime window but coordinate with DBA○ Run database and system maintenance jobs after Clarity jobs complete○ Avoid large data changes (e.g. post timesheets regularly)

● Job conflicts○ Avoid conflicts between slicing, posting, etc.

● Security model○ Use global rights if possible○ Minimize the use of instance rights

● Page layout○ Browsers and the application have ‘real limits’ to what they can digest

● Slice settings○ Minimize slice time ranges

● Period settings○ Close time and fiscal periods.

● Code optimization○ Suboptimal SQL and Gel is the number one issue we see!

Application configuration tends to force functional settings


7

Application server issues typically an increase in the capacity of the application servers. Key indicators of an application server issue are high application cpu and heap dumps.● Memory

○ 6 GB for the app JVM is great – but 7 GB is worse○ Never less than 2 GB for the BG – typically 2.25 GB is good

● Cores○ 1 Core per JVM or BG○ 1 Core for the OS○ 1 Core for GC

● JVM Architecture○ 1 or 2 BGs○ 50 to 150 concurrent users per JVM○ Leverage a XOG / Admin / Scheduler JVM

● JVM GC Settings○ Parallel garbage collection if more than 1 core

● Maintenance○ Weekly restarts may be necessary, but can typically be avoided if architecture sized

correctly.

Application server issues are hardware related


8

Database issues are often a mix of hardware, customization, and maintenance.

● Memory○ Most common problem. Clarity LOVES memory!

● Cores○ Uncommon problem – start with tuning and memory

● Shared usage○ Applications need to ‘play nice’

● Storage speed○ Clarity demands a high speed SAN. Memory compensates for IOPs….

● Query performance & Long running session○ Second most common problem! Efficient SQL is critical.○ Sessions persist after there are not visible on the application!

● Maintenance jobs○ Do not run standard Oracle database stats job

● Parameters○ Contact Rego for environment specific database parameters

● Reporting demand○ Monitor and plan for Crystal and Webi reporting

Database server issues are typically tuning related


9

There are a variety of sources providing key information on performance.

Rego Exchange has portlets that help monitor jobs, processes, and database (Oracle) performance.

Performance data exists in a variety of data sources

Log & Config Files• App Logs• BG Logs• App Access Logs• Process engine

Database Reports• Oracle advantages• Deadlocks • Waits• Physical IO• Cache Memory• Long sessions• Top SQL by CPU• Top SQL by IO• Table fragmentation• Table configuration

Infrastructure• Diagrams• Confirm Cores• Confirm Memory• Confirm IO• Confirm Network• GC Performance


10

Addressing performance issues is an iterative process!!!

Start with the low hanging fruit. Issues often ‘mask’ the more systemic issues.

Performance is an iterative process

Critical Configuration

• JVM Config• Query Tuning• DB Health• DB Locking• Job Schedule

Hardware & Tuning

• App HW• DB HW• DB Config• Query Tuning

Systemic

• Performance Assessment

• Maintenance Program

• Configuration Assessment


11

Any questions before we continue?

Discussion & Questions


12

Symptom● Application heap dumps, out of memory warnings, and periodic slowness● Database appears to be healthy

Clues● Clarity 13 requires additional capacity (memory & cores)● JVMs heaps were 1.5 GB or the app and 1.0 GB for the BG● Application JVMs spending significant time garbage collecting● Symptoms appear during peak usage time● Database is healthy

Resolution● Increase memory for JVMs to 6 GB for the apps and 2 GB for the BB● Increase cores to 1 core per BG, 1 core per APP, 1 core for OS, and 1 core for GC. ● Add JVM for XOGs / Admin / Scheduler

Result● Stable Dev environment● Pushing to production● If cores are limited, try memory increase first and watch GC performance

Real-world Case: Application Sizing


13

Symptom● Frequent & persistent application slowness● Database CPU and application CPU are inconsistent

Clues● Long running sessions on the database (more than a few minutes)● Blocking sessions● High physical I/O

Resolution● Moved to the Clarity stats job (Oracle) nightly & disabled the Oracle cron job● Increased the size of the database cache (PGA and SGA)● Rebuilt indexes associated with long running SQ

Result● Stable and consistent production environment● Exposed additional queries that needed to be tuned● Exposed need to establish tuning and maintenance process

Real-world Case: Database I/O and Tuning


14

Symptom ● Select users long waits with select portlets and pages● Periodic system slowness● Oddly high database CPU

Clues● Long running sessions on the database (more than a few minutes)● Top SQL includes portlet queries with poor SQL

Resolution● Tuned the queries causing the issue● Changed the page configurations to render on filter only● Set default filter for the portlets

Result● Improved portlet speed● Eliminated long running sessions (led to system issues)● This can also apply to WebI and reports

Real-world Case: User Portlets and Configuration


15

Questions

Contact US888.813.0444

Email [email protected]

Web Sitewww.regoconsulting.com

Thank you.

Presenter:Steve Seaney

mailto:[email protected]

http://www.regoconsulting.com/

Www.regoconsulting.comPhone: 1-888-813-0444 Enhancing Application Performance Root Causes and Quick...

Documents

Transcript of Www.regoconsulting.comPhone: 1-888-813-0444 Enhancing Application Performance Root Causes and Quick...