Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack...
Transcript of Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack...
![Page 1: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/1.jpg)
CP-63Costpoint with Oracle Database
Steve Sisak, Scott Wilburn Deltek
![Page 2: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/2.jpg)
Oracle Topics
SI 566450 - CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 2
» Introduction
» Oracle Components for Costpoint
» Performance Settings
» Memory
» CPU
» IOPS – Input Output Operations per
Second
» Virtual Environments
» Degree of Parallelism and Partitioning
» Technical Troubleshooting
» W_App_Status_Info table
» Tracing
» The Application Workload Repository
Report (AWR)
![Page 3: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/3.jpg)
Introduction
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 3
» Technical session on Oracle database use and configuration.
» Presented from the perspective that these are some of the most common causes of performance
issues.
» Will hopefully help you understand all the other factors that can be part of performance issues.
» Best suited to:
» Application administrators interfacing with DBA’s.
» DBA staff responsible for infrastructure related to Costpoint.
» Accountants who would like to understand more about this topic.
» Separate sessions available for SQL Server and Advanced System Management (Weblogic).
![Page 4: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/4.jpg)
Oracle Components for Costpoint
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 4
» Costpoint normally tests and supports two late releases of the Oracle database. With multiple supported versions of Costpoint this can result in 3 versions of Oracle supported in some form. Costpoint has several performance optimizations that benefit from specific Oracle features. Deltek recommends considering these features for optimal performance.
» Enterprise Edition – Gives parallel execution support.
» Partitioning Option – Supported to optimize some of the critical month end Costpoint processes.
» Oracle Tuning and Diagnostics Pack – Needed for Application Workload Repository reports. These are the number one performance management tool used by Customer Support and Engineering.
» Nice to have features include:
» Real Application Clusters (RAC).
» Features in use but not regularly tested by Deltek.
» Data Guard.
» Multitenant.
» Oracle database appliances / Exadata.
![Page 5: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/5.jpg)
Performance Settings - Memory
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 5
» Memory and Caching are key to both OS and Database performance.
» Caching is the primary way to avoid IO bottlenecks.
» Set minimum and maximum database memory to be the same value.
» Set minimum values close to expected requirements for key memory pools like SGA.
» Always leave 25% of the OS memory free and not committed to the database.
» Corporate applications running on the server might require adding memory.
» Add memory if either the database or OS show signs of memory paging to disk.
» The Advisory Statistics section of the AWR has the most important memory information.
![Page 6: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/6.jpg)
Performance Settings - CPU
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 6
» Always use minimum of 2 Ghz clock speed for CPU
» Avoid hyper threading when counting your CPU cores it impacts DOP.
» For most mid sized databases 2 to 4 CPU’s is the minimum today. With modern CPU’s a 2 CPU server can have 12 to 20 physical CPU cores.
» OS issues like the Meltdown and Spectre security patches took away 20 – 30% of CPU performance.
» Size beyond the database since the OS and other system applications need CPU to operate after all database needs are met. As with memory have 25% extra capacity.
» IO performance is the first impact if the database uses all the CPU resources.
» Need to monitor peak workload time periods.
» Make use of antivirus exclusions or switch from real time to scheduled scans.
» Never use Power Saving settings in the system BIOS, a Virtual Machine guest OS, or a Host OS.
» Deltek minimum CPU recommendations are based on physical servers and virtual CPU’s can be much different than physical ones.
![Page 7: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/7.jpg)
IOPS – Input/Output Operations per Second
CONFIDENTIAL © Deltek, Inc. All Rights Reserved.7
» Standard measurement of storage system performance. One of the most common bottlenecks.
» Faster / “More” IOPS = Faster Costpoint or more concurrent users.
» Direct 1 to 1 correlation between IOPS increases and performance of key Costpoint closing processes in Deltek’s performance lab.
» Impacts of Memory and CPU tuning on a database are minimal until IO is adequate.
» Sub 10 ms latency is target for database IO.
» Deltek performance lab database servers have current IOPS ratings of :
» 17,000 – decommissioned in 2018 but was the standard for 7 years.
» 30,000 – 40,000 internal low performance SSD array.
» 70,000 – internal high performance SSD array.
» 106,000 – Nimble AF60 all flash storage array, dual 10Gb active Ethernet storage network. Highest performance.
» Deltek uses Diskspd utility to measure IOPS. (Microsoft Technet). Available through Customer Care.
» Deltek also has a database driven load test that is not Windows specific that can be used to test IO.
![Page 8: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/8.jpg)
IOPS and Hosting
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 8
» IOPS are the most important factor for a database system like Costpoint.
» IOPS are critical for performance but often overlooked selecting instance types.
» Hosted IOPS are complex due to options to dedicate or “burst” and different limits that are tied to IOPS like throughput.
» IOPS is the first sizing parameter to consider. Throughput needs to match / sustain IOPS for the duration of a long running Costpoint process.
» Sustainable IOPS are often lower than Peak IOPS or have other limits like duration. Test and understand any hosting environment used for a database.
» Example:
» C5n.2xlarge AWS instance – 20,000 IOPS (30 min limit). 3,500 Mbps throughput (30 min limit).
» Sustainable IOPS = 10,000. Sustainable throughput = 1,750 Mbps.
» Need to understand that a “Fast” Costpoint system can be a 900 lb gorilla of IO.
![Page 9: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/9.jpg)
Virtual Environments
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 9
» Best practices with Virtualization.
» Monitor and Manage to meet the database peak workload requirements.
» There are more variables which make sizing and management more complex.
» Reserve the resources in the VM required for the database server.
» Don’t recommend sharing or pooling CPU or Memory.
» Mandatory to test virtual environments with production workloads.
» Technologies that move virtual machines or their storage can cause problems for the
Weblogic cluster and database. The short system disconnects or suspension they cause
have taken systems offline.
![Page 10: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/10.jpg)
Applications Tuned for Parallelism and Partitions
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 10
» These applications were tuned for parallel
execution and partitioning and are the only jobs
tested in a parallel job queue.
» Parallelism and Partitioning work together and
have been supported and tuned together with
Oracle since Costpoint 6.1.
» Very significant performance improvements
were realized by this tuning.
![Page 11: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/11.jpg)
Degree of Parallelism
CONFIDENTIAL © Deltek, Inc. All Rights Reserved.11
» A calculation by the database based on CPU and database configuration settings.
» Used by Costpoint to optimize performance in conjunction with Partitioning.
» Controlled through the Job queue. Job queues default to serial but have a parallel attribute that can be set. Parallelism isnot used unless parallel job queues are configured.
» An undocumented parameter is available, for Oracle databases only, that allows the DOP to be set for parallel job queues through Weblogic.
» This configurability is not supported on SQL Server.
» On the Weblogic server you add the following setting to the enterprise.properties file, with a non zero value, to set the DOP of any parallel queue running on that server. Requires Sysjar 46 or later.
» “System Name”.parallelSqlLimit=8 - Sets Dop to 8 for jobs running through a parallel queue.
» Future KB on the topic KB 99594
» This allows easy management and testing of the DOP that can be supported by the CPU and IO of a Costpoint Oracle database system.
» Don’t use any other methods to attempt to control parallelism. Don’t assign parallel attribute to tables or indexes.
» In Oracle RAC it is mandatory that all parallel jobs be pinned to the same server. They will not perform well if distributed.
» Parallel_force_local = True
![Page 12: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/12.jpg)
Partitioned Tables
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 12
» The following tables can be partitioned with an optional partitioning script, supplied with the
Oracle installation, which is tested exclusively for Oracle performance.
» Most are partitioned by PD, and possibly Sub_Pd. A FY partition can be safely added but
was not a part of the partition scripts. FY and any other partition are the customer’s
responsibility to test and maintain.
![Page 13: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/13.jpg)
Technical Troubleshooting
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 13
» Information about the issue.
» Complete description.
» AWR.
» Unusual Wait Events?
» Is the Costpoint SQL the top SQL?
» Application Trace / Tkprof output.
» Is it repeatable and how to repeat?
» Has anything changed?
» A system diagram can summarize the architecture and improve troubleshooting.
» Weblogic diagnostic image.
» Check for errors that happen at the time of the incident.
» W_App_Status report for the impacted process will give the performance history.
![Page 14: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/14.jpg)
W_App_Status_Info table
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 14
» Automatically stores run time information for all reports and processes run in Costpoint
» The data can be accessed in any screen with an Actions / Report button on the menu.
» A Costpoint report interface allows you to quickly run reports for any Costpoint job like Calculate Revenue. Reports can also be run by time period and in that case you get a list of everything in Costpoint that ran at a particular time.
» Great way to track run time changes in key jobs.
» Answers the question “what else was running” also records other key data like application server used and how the job was submitted.
» Should be used to double check time periods being used for AWR reports to make sure the correct job was running.
» Supply both baselines and trends for Costpoint processes that can help evaluate tuning or hardware impacts.
![Page 15: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/15.jpg)
W_App_Status_Info data
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 15
» APP_ID, ACTION_ID and RPT_ID – Identify the work.
» COMPLETE_FL – Completion status.
» EXEC_METHOD_CD – How job was submitted ( UI, API, Async, PrcSrv).
» USER_ID – Submitter.
» SYS_TEXT – Job messages usually completion or error.
» JOB_START_DT, JOB_END_DT or ACTION_START_DT, ACTION_END_DT – Start and
stop times and dates to allow calculation and tracking of elapsed time.
» COMPANY_ID – Company run for.
» SERVER_ID – Application server doing the work.
![Page 16: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/16.jpg)
Weblogic Trace Options
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 16
» traceSlowStatements.
» Maintenance screen only trace for special situations.
» enableSQLTrace.
» Weblogic based application text trace. No DBA involvement but only elapsed time from each statement. No database impact.
» enableOracleTrace.
» Oracle specific trace details – DBA required uses Oracle Trace / Tkprof facilities. Traces the specific application by starting the database trace from Weblogic. Will trace every time the application is run. Critical for performance analysis.
» Use Trcsess utility to combine multiple trace files from parallel applications before Tkprof. (No KB yet).
» For full details see KB’s 96692, 85245, 75574.
» Most universal for database performance is enableSQLTrace.
» Traces are for 1 application per entry in the enterprise.properties file.
» In special situations for maintenance screens use traceSlowStatements.
![Page 17: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/17.jpg)
enableSQLTrace Output
CONFIDENTIAL © Deltek, Inc. All Rights Reserved.17
» Comparison of the same statement in two time periods.» 71. Statement:
» MERGE INTO D_585PJPCOMPR_POP POP USING ( SELECT POP2.PROJ_ID AS PROJ_ID, ? AS nMaxLvlNo,
SUM(isnull(POP1.PROJ_V_FEE_AMT,'NUM') + isnull(POP1.PROJ_V_CST_AMT,'NUM')) AS PROJ_V_TOT_AMT, SUM(isnull(POP1.PROJ_F_FEE_AMT,'NUM') /*
C:1 R:All */
» First Executed on: Sat Jun 01 07:06:35 CDT 2019
» Parsed: 8
» Executed : 8
» Has literals: YES
» Total Exec Time: 12485139 milliseconds (200 minutes approximately)
» Invoked by: application.pj.pjpcompr.Pjpcompr.funcGetContractAndFundedValues(Pjpcompr.java:11187)
» 71. Statement:
» MERGE INTO D_585PJPCOMPR_POP POP USING ( SELECT POP2.PROJ_ID AS PROJ_ID, ? AS nMaxLvlNo,
SUM(isnull(POP1.PROJ_V_FEE_AMT,'NUM') + isnull(POP1.PROJ_V_CST_AMT,'NUM')) AS PROJ_V_TOT_AMT, SUM(isnull(POP1.PROJ_F_FEE_AMT,'NUM') /*
C:1 R:All */
» First Executed on: Tue Jun 04 07:06:48 CDT 2019
» Parsed: 8
» Executed : 8
» Has literals: YES
» Total Exec Time: 226934 milliseconds (4 minutes approximately)
» Invoked by: application.pj.pjpcompr.Pjpcompr.funcGetContractAndFundedValues(Pjpcompr.java:11187)
![Page 18: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/18.jpg)
enableOracleTrace Output
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 18
» Trace output must be run through Tkprof utility before sending to Deltek.
![Page 19: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/19.jpg)
AWR - Overview data
CONFIDENTIAL © Deltek, Inc. All Rights Reserved.19
» Always confirm the date and time. Remember W_App_Status can help. Make sure the report includes the time period that an issue occurred.
» Most reports are run for one or two hours at most.
» Make sure to have a long enough retention period to allow going back to get this data.
» Keep snapshots of all month end and peak periods for reference (Baselines).
» By reviewing these reports you will know what’s normal for your system.
![Page 20: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/20.jpg)
AWR - Top Wait Events
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 20
» Wait events are normal and consistent. This section helps spot unusual or abnormal delays in the system.
» New top wait events should always be investigated especially after upgrades or if performance issues are occurring.
![Page 21: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/21.jpg)
AWR - SQL ordered by Elapsed Time
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 21
» Key for identifying problem SQL and making sure there are not other SQL interferences.
» Costpoint applications being reviewed should be the longest running SQL.
![Page 22: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/22.jpg)
AWR - SQL ordered by IO Wait Time
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 22
» Helps identify problem SQL and determining if IO is part of the issue.
» Used with Tablespace IO statistics.
![Page 23: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/23.jpg)
AWR - Log Switches
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 23
» Good for this metric is less than 6 per hour. It usually indicates the log files are too small.
» There are a couple of other important metrics like:
» Sorts (disk) – Instance Activity – Ideally 0.
» PGA Cache Hit % - Ideally mid to high 90’s
» From your baseline look for anything abnormal. The AWR includes a report comparing a current period to a baseline automatically.
![Page 24: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/24.jpg)
AWR - Tablespace IO
CONFIDENTIAL © Deltek, Inc. All Rights Reserved.24
» IO performance on reads with less than 10ms is crucial for performance
» If you don’t have this section in Oracle 12.2 you need to update your statistics settings. Oracle KB article available.
![Page 25: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/25.jpg)
AWR - Memory Pool Data
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 25
» Buffer Pool, Shared Pool and PGA Advisories are very important to gauge database
memory.
![Page 26: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/26.jpg)
AWR - Oracle initialization parameters that are not default values.
CONFIDENTIAL © Deltek, Inc. All Rights Reserved.26
» Undocumented parameters are normally directions from Oracle to address issues and may be asked about.
![Page 27: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/27.jpg)
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 27
AWR - Baselines» In coordination with accounting and using the W_App_Status report, run an AWR that covers
the month end processing for each period of the year.
» Save January and December AWR’s as baselines as a minimum.
» Use them to review performance during the year
» Evaluate intermittent performance issues.
» Do capacity planning.
» Evaluate performance tests and configuration or hardware changes.
» For your longest running process supplement the AWR with a trace. Having a known picture
of good performance is extremely valuable in troubleshooting.
![Page 28: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/28.jpg)
CONFIDENTIAL © Deltek, Inc. All Rights Reserved. 28
Q & A
![Page 29: Costpoint with Oracle Database - Microsoft · 2019-11-21 · » Oracle Tuning and Diagnostics Pack –Needed for Application Workload Repository reports. These are the number one](https://reader030.fdocuments.us/reader030/viewer/2022041003/5ea50964e2d964597d618867/html5/thumbnails/29.jpg)