Dm perf tuning2 PDF

© Copyright IBM Corporation 2010 TrademarksMonitoring and tuning InfoSphere Master Data ManagementServer, Part 2: Monitor the DB2 layer and learn about differentmonitoring tools

of 19

Monitoring and tuning InfoSphere Master DataManagement Server, Part 2: Monitor the DB2 layer andlearn about different monitoring toolsJesse Chen ([email protected])Senior Performance EngineerIBM

Yongli An ([email protected])MDM Performance ManagerIBM

Adam Muise ([email protected])IBM InfoSphere EvangelistIBM

25 October 2010

This article series provides guidelines on how to effectively monitor and properly tune eachlayer of the infrastructure in IBM InfoSphere™ Master Data Management Server by focusing onan IBM "Blue stack." The blue stack includes MDM Server, WebSphere® Application Server,DB2® on AIX®, and AIX operating-system-level resources. Part 1 described for each layer alist of key parameters and variables that affect performance. It also offers detailed instructionson how to monitor, and it provides general recommendations on how to tune each layer.This Part 2 of the series includes practical examples of monitoring MDM Server, along with asummary of all the recommended tools for your reference.

View more content in this series

Introduction

This is Part 2 of an article series on how to effectively monitor and tune the IBM InfoSphere MasterData Management Server (MDM Server) stack for overall optimal MDM Server performance. Part1 covered important guidelines on how to effectively monitor, tune, and optimize MDM Server andthe WebSphere Application Server layer for optimal performance, with brief discussions aboutbest practices for a list of key areas and features that affect performance in these two layers. ThisPart 2 moves on to the lower parts of the stack to cover guidelines and recommendations on howto effectively monitor and tune the DB2 layer. This article also describes tools that are commonlyused to monitor the operating system level resources.

http://www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/developerworks/ibm/trademarks/

mailto:[email protected]



http://www.ibm.com/developerworks/data/library/techarticle/dm-1010mdmservertuning1/index.html

http://www.ibm.com/developerworks/views/data/libraryview.jsp?sort_by=&show_abstract=true&show_all=&search_flag=&product_by=-1&topic_by=-1&type_by=Articles&industry_by=-1&search_by=+InfoSphere+Master+Data+Management+Server%2C+Part&Go.x=50&Go.y=10



developerWorks® ibm.com/developerWorks/

Monitoring and tuning InfoSphere Master Data ManagementServer, Part 2: Monitor the DB2 layer and learn about differentmonitoring tools

of 19

Monitoring MDM Server database and tuning DB2 on AIXJust as important as monitoring and tuning other layers in the MDM Server stack, monitoringand tuning the database layer is critical for optimal performance of the MDM system. In general,common DB2 best practices apply. There is not really anything specific or special about tuningthe MDM database. The most important practice is to monitor regularly and to plan tuning andoptimization proactively to avoid a negative impact to your production systems. This section coverssome basic recommendations on what to monitor and how to monitor the DB2 database in yourMDM system. This section also describes several known tuning tips that are specific to the MDMServer database.

There are a variety of tools you can use to monitor and analyze DB2 performance. For example,you might traditionally rely on snapshot and event monitors for performance monitoring andevaluation. This approach is still supported and working in recent DB2 versions. However, startingin DB2 Version 9.7, table functions are available as a light-weight alternative to the traditionalsystem monitor. These new table functions are planned to be updated and enhanced with newreleases of DB2, while the snapshot interface will not be. For details, see Resources for links tothe product manual and the DB2 information center. There are also various commercial tools thatleverage the same underlying monitoring technology in DB2 for end users to achieve the samemonitoring purpose.

This section describes the approach that works for old and new DB2 versions. You can eitherfollow the steps to do exactly the same, or learn the techniques and ideas and apply them whenusing different tools. Alternatively, the key information above is also available from the reports suchas DBSUMMARY, CURRENTSQ, CURRENTAPPS, CONNECTION, and LOCKWAIT, which aregenerated by the MONREPORT module in DB2 Version 9.7 or later.

Database monitoring

Although this section focuses on monitoring the database itself, that is, what's going on insidethe database, it's also very important to monitor the status of the MDM database server in termsof memory, CPU, and input/output activities. In particular, it's critical to have a close eye on theutilization of the input/output system that supports the MDM database.

You need to understand exactly how well the database performs and what's going on insidethe MDM Server database as result of the MDM workload running in the system. There are twoprimary tools, each serving a different purpose, with which you can access database monitorinformation: the snapshot monitor and event monitors. The snapshot monitor enables you tocapture a picture of the state of the database activity at a specific moment in time, while eventmonitors log data only as specified database events occur.

The snapshot monitor tool is a powerful but simple monitoring tool that incurs very little overhead.Snapshots provide important information not only about the database overall, but also about theapplications connected to the database at a specific time. To effectively monitor the system, youcan take regular snapshot data at different times of a day and over an extended period of time(covering days and months), or you can target specific workload intervals that are known to be

ibm.com/developerWorks/ developerWorks®


of 19

performance sensitive. A well-conceived monitoring plan, especially during performance testing orduring important production workloads, can help implementers foresee and plan for many potentialdatabase performance challenges.

Most of the monitor counters are cumulative, reflecting the total count of a particular activity sincethe last snapshot monitor reset within your session from which you are issuing your db2 command.If no snapshot monitor reset is issued, the data reflects a running count since the first db2connection was established. Typically the data collected without a reset is not very useful becausethe counters span a longer period of time, including drastically different workloads along peaksand valleys in load. Focusing in on a particular workload or time period with snapshot collectionreveals the unique characteristics of the workload and enables more decisive tuning decisions. Fora deeper understanding of a database workload during optimization or problem resolution, try thelow-impact investigative technique of taking snapshots in intervals, with monitor resets betweeneach capture. For example, to monitor a one-hour period for a production workload, a script can beeasily constructed to do the logic of thread 1 and thread 2 as shown in the thread in Listing 1.

Thread 1 in Listing 1 is for taking multiple interval snapshots of 60 seconds each with a resetmonitor at the beginning of each interval snapshot.

Listing 1. Thread 1 pseudo codedb2 -v update monitor switches using bufferpool on sort on lock on statement on \table on uow ondb2 -v get monitor switchescount = 0while (count < 10) sleep 300 ## not being monitoreddb2 -v reset monitor all ## clear the counters sleep 60 ## collect data for only 60-second interval db2 -v get snapshot for all on <db_name> 2>&1 > snapshot_${count}_withReset.outend whiledb2 -v update monitor switches using bufferpool off sort off lock off \statement off table off uow offdb2 -v terminate

Thread 2 in Listing 2 is for taking multiple snapshots over the period without issuing the resetmonitor command after the initial reset monitor command.

Listing 2. Thread 2 pseudo codedb2 -v update monitor switches using bufferpool on sort on lock on statement on table \on uow ondb2 -v get monitor switchescount = 0while (count < 10) loop sleep 360 ## taking a snapshot every 6 minutes db2 -v get snapshot for all on <db_name> 2>&1 > snapshot_${count}_NoReset.outend whiledb2 -v update monitor switches using bufferpool off sort off lock off \statement off table off uow offdb2 -v terminate

When analyzing the snapshot data, there are certain metrics that require special attention. Fromthe database snapshot, note the counters in Listing 3 and monitor the ratio among these counters.



of 19

Buffer pool hit ratio is very important to monitor. It can be calculated based on the followingcounters or reported with the latest new table functions.

Listing 3. Buffer pool hit ratio countersBuffer pool data logical reads =Buffer pool data physical reads =Buffer pool index logical reads =Buffer pool index physical reads =

Obviously, with availability of physical memory on the database server, a high buffer pool hit ratiois always better for your database performance. The counters in Listing 4 can give you a good ideaof the SQL distribution from your workload. In particular, the ratio between select statement andupdate/inserts/delete statements is useful to tell the split between data inquiries and updates.

Listing 4. SQL distributionCommit statements attempted =Rollback statements attempted =Dynamic statements attempted =Static statements attempted =Failed statement operations =Select SQL statements executed =Xquery statements executed =Update/Insert/Delete statements executed =

The counters in Listing 5 are important to tell on average, how many rows are deleted, inserted,updated, selected, and read. The ratios among them are key characteristics of your workload.Any dramatic changes in the ratios require some explanation and could indicate some potentialissues. In particular, the ratio between rows read and selected can tell if all SQL statements areusing indexes efficiently. If this ratio is extremely high, then it may cause performance issues. Moreinvestigation is needed to isolate the SQL statement(s), which causes the high read/select ratio.Consequently some optimization might be needed. For example, a better index might be provided.

Listing 5. SQL distributionRows deleted =Rows inserted =Rows updated =Rows selected =Rows read =

From dynamic SQL snapshot data as the initial step based on the counters in Listing 6, you shouldbe able to list out the top SQL statements in terms of total cost, frequency, and cost per invocation.If some of the top SQL statements are considered unexpectedly expensive, further action isrequired to analyze the SQL access plan to see if anything can be improved.

Listing 6. SQL distributionNumber of executions =Total execution time (sec.microsec)=Statement text =



of 19

From the table space snapshot data, the used page objects can be used to tell the total size ofthe index, tables, and the total database size. It's always good to know the growth trend of eachobject and the overall database to help you plan ahead for disk space requirements and databasetuning. This is where you can also tell which tables are the most active ones based on Rows Readand Rows Written, as well as the size of each active table and their individual growth patterns, asshown in Listing 7.

Listing 7. SQL distributionTable Schema =Table Name =Data Object Pages =Index Object Pages =Rows Read =Rows Written =

Event monitors are used to collect information about the database and about any connectedapplications when specified events occur. Events that the event monitor can capture includedatabase activity, such as connections, deadlocks, statements, or transactions. You can define anevent monitor by the type of event or events you want it to monitor. For example, a deadlock eventmonitor can be used to monitor deadlocks. When a deadlock occurs, it collects information aboutthe applications involved, the SQL statements participating in the deadlock, and the locks causinghigh contention.

One common usage of event monitor in the MDM Server environment is deadlock eventmonitoring to understand and resolve excessive deadlocks that sometimes happen in anMDM Server database. By default, all databases have an event monitor defined namedDB2DETAILDEADLOCK, which records detailed information about deadlock events. TheDB2DETAILDEADLOCK event monitor starts automatically when the database starts.

After formatting the event monitor files, typically you should be able to find all the SQL statementsparticipating in the deadlocks. By correlating the SQL statements back to the MDM Serverapplication layer, you should be able to understand the application flows leading to the deadlocks.It could be due to the customized logic, a particular use case, or the data itself (quality anddistribution). Once you know the cause, make a decision accordingly to resolve the issue. Anothermajor factor that affects database performance is the input/output capacity. Similarly, closelymonitor the disks utilization where the database containers reside. If monitoring shows unbalancedinput/output usage across disks with a bottleneck associated with a set of hot disks, make adatabase layout adjustment to maximize the input/output and to reduce the impact of the limitinghot disks.

Database tuning

Most of the areas where data tuning happens correspond to the key areas mentioned earlier formonitoring purposes. This section discusses specific tuning recommendations.

DB2 buffer pool tuning is one of the easiest tuning steps that potentially can give a quickperformance boost. With the buffer pool runtime statistics gathered from the monitoring tasks, you



of 19

can see whether any performance gain can be achieved from buffer pool tuning. In general, it'sdesirable and recommended to have a high data hit ratio (80% or higher) and to have very highindex hit ratio (90% or higher).

For a large database, it's possible that increasing the buffer pool size can have minimal effect onthe buffer pool hit ratio. The number of data pages can be so large that the statistical chances of ahit are not improved by increasing its size. Instead, you might find that tuning the index buffer poolhit ratio achieves the desired result. This can be achieved using either of two methods:

• Split the data and indices into two different buffer pools and tune them separately.• Use one buffer pool, but increase its size until the index hit ratio stops increasing.

The first method is often more effective, but because it requires indices and data to reside indifferent table spaces, it may not be an option for existing databases. It also requires tuning twobuffer pools instead of one, which can mean slightly more work, particularly when memory isconstrained.

When multiple buffer pools are in use, the buffer pool snapshot data can be used to calculate thebuffer pool hit ratio of each pool to indicate the effectiveness of each buffer pool. If needed, reducethe size of the buffer pool that cannot get help from the extra memory so that the available memorycan be allocated to other low-hit-ratio buffer pools that can get noticeable improvement from someextra memory allocation.

Another important factor that affects SQL statement performance is database statistics. Having up-to-date statistics is always necessary to ensure optimal SQL performance. Recent DB2 releasessupport automatic runstats as part of the autonomic features. When enabled, automatic statisticscollection helps improve database performance by ensuring that you have up-to-date tablestatistics. If this is not the desired option in a given environment, you can instead use DB2 runstatsor reorgchk commands explicitly in a scheduled maintenance window.

The DB2 runstats command updates statistics that profile the physical characteristics of adatabase table, along with its associated indexes. Dynamic SQL statements automatically usethese updated statistics. Use runstats on a regular basis as part of regular database maintenance.You can use runstats either by using a cron job or by manually using a script. Along with runstats,the DB2 reorgchk command should also be run to confirm whether a reorganization of certaintables or indexes is warranted.

Listing 8 shows commands that dynamically determine MDM Server database tables that runstatscan run on. A ksh file actually runs the runstats commands.

Listing 8. runstats commandsdb2 connect to MDMDB user <uid> using <password>db2 -x select tabname from syscat.tables where type='T' and tabschema='SCHEMA1' \| sed -e "s/ *.$//" | sed -e "s/^/db2 -v runstats on table SCHEMA1./" \| sed -e "s/$/ with distribution and detailed indexes all/" > .\my_runstats.ksh

The my_runstats.ksh file should contain lines similar to those shown in Listing 9.



of 19

Listing 9. runstats scriptdb2 -v runstats on table SCHEMA1.CONTACT with distribution and detailed indexes alldb2 -v runstats on table SCHEMA1.CONTRACT with distribution and detailed indexes alldb2 -v runstats on table SCHEMA1.ADDRESS with distribution and detailed indexes alldb2 -v runstats on table SCHEMA1.IDENTIFIER with distribution and detailed indexes alldb2 -v runstats on table SCHEMA1.ADDRESSGROUP with distribution and detailed indexes all

Each line is a runstats command on a table or index. By running the my_runstats.ksh program,performance statistics on all tables of the MDMDB will be profiled. To shorten the duration forupdating statistics, it is recommended that you split the ksh file into several files so that you canrun multiple scripts concurrently to update different sets of tables in parallel to utilize the CPUpower on the DB2 server. To use the DB2 reorgchk command to check the statistics of all thetables, and to determine if reorganization of certain tables or indexes is required, study the outputfile that result from issuing the commands in Listing 10.

Listing 10. reorgchk commandsdb2 -v connect to MDMDB user <uid> using <password>db2 -v reorgchk current statistics on table all

In some cases, the reorgchk command can also be used to update the database statistics withone simple command, as shown in Listing 11. However, it does not easily offer the flexibility ofrunning multiple concurrent scripts to update statistics against different sets of tables in the samedatabase.

Listing 11. reorgchk commandsdb2 -v connect to MDMDB user <uid> using <password>db2 -v reorgchk current statistics on table all ##old copy of the statisticsdb2 -v reorgchk update statistics on table all ## update statistics

For slow transaction issues, the generic steps include:

• Determining the specific MDM Server transaction• Running the particular transaction to further gather SQL statements involved• Confirming the most poorly performing SQL statements

The identified problematic SQL statements should be explained to show the access plan for anyneeded, specific tuning. Tuning can include adding extra indexes to yield a more efficient accessplan, which makes tables volatile to encourage index usage and avoid table scans if possible. Insome cases, the out-of-the-box SQL should replaced by a customized SQL based on the specificbusiness requirements. Using the pluggable inquiry feature satisfies the performance requirement.All of these actions are typical performance-tuning exercises completed during the performance-testing phase of an MDM implementation project. The performance-testing phase is a critical partof the MDM project to validate whether predefined performance goals are met with everythingspecific to the implementation. Plan this phase with sufficient time and resources.

One of the most important steps to optimizing SQL statements is to explain the statementsof interest and to study their access plans. The first objective is to eliminate any costly table



of 19

scan operations. The next objective is to further reduce the SQL cost by optimizing the accessplan. The EXFMT format for the explain output file is recommended, because it provides theneeded information to help such SQL optimization. If the OOTB SQL cannot provide the neededperformance because of the combination of business customization and data, you can rewrite theSQL to meet your specific business needs using the pluggable inquiry feature in MDM Server. SeeResources for tips on how to write customized SQL statements for optimal performance.

A few deadlocks once in a while typically are not concerning. However, it can be a problem whenthe frequency or percentage of deadlocks is high. See Resources for a Technote about applyingsome lock-avoidance registry settings in DB2. Applying the registry variables or performingother database tuning that leads to overall better SQL performance will help reduce or eliminatedeadlocks.

Note that for DB2 for z/OS®, it's not always acceptable to apply the lock-avoidance settings,because the settings affect all the applications that share the same DB2 system.

For more DB2 tuning best practices, see Resources.

Optimizing the default isolation level in JDBC

Out of the box, MDM Version 8.0 and later defaults to the JDBC isolation level ofTRANSACTION_REPEATABLE_READ or, in DB2 terms: read stability (RS). While this isolationlevel is fine from a functional standpoint, lock contention in an actively used database can bereduced by using the TRANSACTION_READ_COMMITTED isolation level, which is also knownas cursor stability (CS). If you are migrating to MDM, this point is especially important, becausethis step was not previously required. MDM defaults to the more optimistic isolation level of CS.One simple and effective way to change the default isolation level in MDM to the more optimisticCS is to add a custom directive on the jdbc/DWLCustomer data source in WebSphere ApplicationServer. Complete the following steps in the WebSphere Application Server administrative console.

1. Click Resources > JDBC provider.2. Click Configured_JDBC_Provider > DataSource.3. Click the name of the data source (DWLCustomer) for which you want to customize the

default isolation level.4. Under Additional properties, click Custom properties > New.5. Specify webSphereDefaultIsolationLevel in the Name field, and a value of 2 in the Value

field. The default value is 4. See Table 1 for isolation levels.6. Click OK, and save the configuration.7. Synchronize your changes across nodes.8. Stop and restart the application server.

Table 1. Isolation levels

Possible values JDBC isolation level DB2 isolation level

8 TRANSACTION_SERIALIZABLE Repeatable read (RR)

4 TRANSACTION_REPEATABLE_READ Read stability (RS)



of 19

2 TRANSACTION_READ_COMMITTED Cursor stability (CS)

1 TRANSACTION_READ_UNCOMMITTED Uncommitted read (UR)

Exploring recommended monitoring toolsMDM Server supports multiple operating system platforms, databases, and application servers.This section describes performance monitoring tools that work on a subset of these platforms.Used effectively, these tools will help you narrow down performance issues and help pinpoint rootcauses.

UNIX CPU performance monitoring tools

Listings 12 and 13 show CPU utilization using different tools for UNIX systems.

Listing 12. AIX CPU utilization# sar -u 5 100AIX othello 1 5 000C30AD4C00 06/17/0512:38:09 %usr %sys %wio %idle12:38:24 12 1 1 8512:38:39 26 2 0 7212:38:54 9 2 1 89

Listing 13. Solaris CPU utilization# sar 5 100

SunOS cage 5.9 Generic_118558-05 sun4u 07/12/2005

15:31:07 %usr %sys %wio %idle15:31:12 0 0 0 10015:31:17 1 0 0 9915:31:22 0 0 0 10015:31:27 0 0 0 100

The mpstat command is another tool to monitor CPU utilization on Solaris. The mpstat commandreports CPU utilization on a per-CPU basis, and it reports extended statistics, including contextswitches, interrupts, and so on. The mpstat command is shown in Listing 14.

Listing 14. mpstat commandmpstat 5 5 will take 5 samples every 5 seconds.mpstat 5 100CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 2 14 0 26 5 1 60 2 3 9 0 170 1 1 0 99 3 15 0 22 28 24 62 1 2 10 0 142 1 1 0 99 34 19 0 24 193 14 65 1 2 11 0 153 1 1 0 99 35 17 0 26 5 1 67 2 3 11 0 152 1 1 0 99 66 17 0 25 89 1 69 2 2 10 0 154 1 1 0 99 67 15 0 23 5 1 68 2 2 10 0 149 1 1 0 99 98 14 0 23 5 1 66 2 2 10 0 148 1 1 0 99 99 14 0 23 5 1 101 2 2 10 0 149 1 0 0 99130 14 0 26 5 1 66 2 2 11 0 137 1 0 0 99131 14 0 22 5 1 67 2 2 11 0 140 1 0 0 99162 15 0 22 5 1 67 2 2 10 0 144 1 0 0 99



of 19

UNIX input/output performance monitoring toolsListings 15 and 16 are sample outputs of the iostat command on AIX and Solaris. The reports aredifferent for each operating system. On AIX, the iostat command with the -a option gives input/output statistics on a per adapter basis, which can be very useful.

Listing 15. AIX iostat output AIX# iostat -a

System configuration: lcpu=4 disk=12

tty: tin tout avg-cpu: % user % sys % idle % iowait 0.0 0.0 0.2 0.5 99.2 0.1

Adapter: Kbps tps Kb_read Kb_wrtnscsi0 5.2 1.2 196495 2154064

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtnhdisk1_Path0 0.0 0.0 0.0 0 0hdisk2_Path0 0.0 0.0 0.0 7397 4hdisk3_Path0 0.0 0.0 0.0 2199 28hdisk0_Path0 0.7 5.2 1.2 186899 2154032

Adapter: Kbps tps Kb_read Kb_wrtnfcs0 11.8 1.3 110301 5166084

Paths/Disk: % tm_act Kbps tps Kb_read Kb_wrtnvpath1_hdisk17 0.0 0.7 0.2 22729 282512vpath1_hdisk13 0.0 0.7 0.2 17239 281396vpath0_hdisk16 0.0 2.3 0.2 7217 1006120vpath0_hdisk12 0.0 2.3 0.2 8758 1010056vpath3_hdisk19 0.0 2.3 0.2 22024 1006064vpath2_hdisk14 0.0 0.6 0.2 8980 281792vpath3_hdisk15 0.0 2.3 0.2 14352 1016480vpath2_hdisk18 0.0 0.6 0.2 9002 281664

The Solaris iostat command has many options that will help monitoring input/output devices.Listing 16 shows the -x option, which gives extended statistics but there are many more argumentsavailable.

Listing 16. Solaris iostat outputSolaris# iostat -x

extended device statisticsdevice r/s w/s kr/s kw/s wait actv svc_t %w %bsd15 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0 0sd16 0.0 0.0 0.0 0.0 0.0 0.0 2.4 0 0sd17 3.7 0.6 9.2 5.1 0.0 0.0 1.3 0 0sd45 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0sd46 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0sd47 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0sd75 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0 0sd76 0.0 0.0 0.0 0.0 0.0 0.0 1.6 0 0sd105 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0 0sd106 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0 0sd107 0.0 0.0 0.0 0.0 0.0 0.0 1.3 0 0sd136 0.0 3.1 0.0 12.6 0.0 0.0 0.6 0 0sd137 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0sd139 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0sd140 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0



of 19

sd2315 0.0 0.0 0.0 0.0 0.0 0.0 0.4 0 0sd2316 0.0 0.0 2.9 0.0 0.0 0.0 13.8 0 0sd2317 0.3 3.7 2.1 26.5 0.0 0.0 1.4 0 0sd2442 0.0 0.0 0.0 0.0 0.0 0.0 0.2 0 0sd2443 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 0

UNIX network performance monitoring toolsListings 17 and 18 are sample outputs of the netstat command on AIX and Solaris.

Listing 17. AIX netstat outputAIX (snapshot every 5 seconds on the en0 adapter)# netstat -I en0 5input (en0) output input (Total) outputpackets errs packets errs colls packets errs packets errs colls260984 0 128140 0 0 514023 0 405638 0 0342 0 331 0 0 342 0 331 0 0837 0 828 0 0 837 0 828 0 0671 0 670 0 0 673 0 672 0 0

Listing 18. Solaris netstat outputSolaris (snapshot every 5 seconds on the ce8 adapter)# netstat -I ce8 5 input ce8 output input (Total) outputpackets errs packets errs colls packets errs packets errs colls39248977 0 39931746 0 0 39739399 0 40422168 0 07 0 0 0 0 7 0 0 0 06 0 0 0 0 6 0 0 0 07 0 31 0 0 7 0 31 0 08 0 0 0 0 8 0 0 0 09 0 0 0 0 9 0 0 0 05 0 0 0 0 5 0 0 0 0

UNIX memory usage monitoring toolsListings 19 and 20 are sample outputs of the vmstat command on AIX and Solaris.

Listing 19. AIX memory usage(snapshot every 5 seconds)# vmstat 5System Configuration: lcpu=4 mem=14336MBkthr memory page faults cpu----- ----------- ------------------------ ------------ ----------- r b avm fre re pi po fr sr cy in sy cs us sy id wa 1 1 772314 2075422 0 0 0 0 0 0 575 810 242 0 1 99 0 0 0 772156 2075580 0 0 0 0 0 0 579 579 214 0 1 99 0 0 0 772414 2075321 0 0 0 0 0 0 571 1034 198 0 0 99 0 0 0 772006 2075730 0 0 0 0 0 0 560 1060 163 0 0 99 0 0 0 772006 2075730 0 0 0 0 0 0 562 521 165 0 1 99 0

Listing 20. Solaris memory usage(snapshot every 5 seconds)# vmstat 5 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s1 s1 s1 s4 in sy cs us sy id 0 0 0 39372952 28946544 127 272 9 2 2 0 0 0 0 4 0 803 2498 1152 1 1 99 0 0 0 43181064 29508744 0 1 0 0 0 0 0 0 0 0 0 384 559 328 0 0 100 0 0 0 43179664 29508112 232 536 16 0 0 0 0 0 0 8 0 616 2286 361 1 1 99 0 0 0 43181064 29508744 0 0 0 0 0 0 0 0 0 0 0 379 774 326 0 0 100 0 0 0 43179648 29508096 232 536 16 0 0 0 0 0 0 8 0 618 2304 366 1 1 99



of 19

Automating resource monitoring on UNIX using nmon

It's useful to automate the resource utilization collection during a performance test or in aproduction environment. IBM offers a free tool called nmon that enables you to do just that. Thenmon tool runs on the AIX and Linux platforms. The tool is designed to collect and analyze (withthe nmon analyzer Excel spreadsheet) AIX performance data, including the following:

• CPU utilization• Memory use• Kernel statistics• Run queue information• Disks input/output rates• Transfers• Read/write ratios• Free space on file systems• Disk adapters• Network input/output rates• Paging space• Paging rates• CPU and AIX specification• Top processors• IBM HTTP web cache• User-defined disk groups• Machine details and resources• Asynchronous input/output• Workload manager• ESS disks• NFS• Dynamic LPAR changes

To start the nmon process and log entries to a file, enter nmon -f -s 15 -c 120. This command logsperformance statistics every 15 seconds for 120 intervals, or (15*120)/60 = 30 minutes. The logis in a file called <hostname>_date_time.nmon in the running directory. You might want to run thiswith the nohup option if you need to log out of the system.

When the tool finishes, run the sort command sort -A mymachine_311201_1030.nmon >res_usage_test1.csv to sort and generate a comma-delimited file for the Excel nmon analyzer touse. Open the Excel spreadsheet called nmon analyzer v301.xls, and load the .csv file, as shownin Figure 1.



of 19

Figure 1. Excel nmon processing tool

After analysis, save the output file as a regular Excel file. In the output file, resource utilization islisted under each tab. The first sheet is the summary sheet. Figure 2 shows an example of thegenerated output.

Figure 2. Sample of processed nmon output

(View a larger version of Figure 2.)

WebSphere JVM monitoring tools: IBM Pattern Modeling and Analysis Tool forJava Garbage Collector

IBM Pattern Modeling and Analysis Tool for Java Garbage Collector is a tool that parses verboseGC trace, analyzes Java heap usage, and recommends key configurations based on patternmodeling of Java heap usage. See Resources for more details. Figure 3 shows a samplescreenshot of the tool in action.

http://www.ibm.com/developerworks/data/library/techarticle/dm-1010mdmservertuning2/sidefile-figure2.html



of 19

Figure 3. IBM Java Pattern and Analysis Tool for GC

WebSphere JVM HeapAnalyzer

HeapAnalyzer is a graphical tool to discover possible Java heap leaks. See Resources for moredetails. Figure 4 shows a screenshot of the tool.

Figure 4. IBM HeapAnalyzer



of 19

MDM Server built-in traces

MDM Server enables logging at different levels. You can also turn on tracing for a particularcomponent (functional wise). To troubleshoot performance issues, one of the most-used tracefacilities is the PerformanceTracking parameter. Complete the following steps to use thePerformanceTracking parameter.

1. Enable performance monitor from the configelement table, as shown in Listing 21.

Listing 21. Enabling performance monitordb2 -v update configelement set value='-1' where name in \('/IBM/DWLCommonServices/PerformanceTracking/level')db2 -v update configelement set value='true' where name like \'/IBM/DWLCommonServices/PerformanceTracking/%enabled'

2. Configure log4J in properties.jar for the performancemonitor.log, as shown in Listing 22.

Listing 22. Configuring log4Jlog4j.appender.performanceLog=org.apache.log4j.RollingFileAppenderlog4j.appender.performanceLog.Encoding=UTF-8log4j.appender.performanceLog.Threshold=ALLlog4j.appender.performanceLog.layout.ConversionPattern=%d : %m%nlog4j.appender.performanceLog.layout=org.apache.log4j.PatternLayoutlog4j.appender.performanceLog.File=../../../logs/MDM80IT6/performancemonitor.loglog4j.logger.com.dwl.base.performance.internal.PerformanceMonitorLog=INFO,\performanceLoglog4j.appender.performanceLog.MaxBackupIndex=2log4j.appender.performanceLog.MaxFileSize=100MB

3. Restart WebSphere Application Server.4. Run the warmup of the transactions.5. Clean the performance monitor files, by entering code such as bash-3.00# cat /dev/null >

performancemonitor.log.6. Run the transaction being debugged several times (usually 100 times).7. Stop the test.8. Verify that the file starts with code as shown in Listing 23.

Listing 23. Success message for file"21194878586562 processTx : processTx(String,Object) @ DWLRequestHandler : :\1961194878586562 : 24785000 : : SUCCESS"

ConclusionIBM InfoSphere Master Data Management Server helps companies gain control over businessinformation by enabling them to manage and maintain a complete and accurate view of theirmaster data. InfoSphere MDM Server enables companies to extract maximum value from masterdata by centralizing multiple data domains and providing a comprehensive set of pre-built businessservices that support a full range of master data management functionality. Highly flexible in



of 19

nature, MDM Server offers a wide range of customization capabilities that enable companies todevelop their own services that meet their customers' demands. Due to the highly customizablenature, you need to understand the impact that each tier has on overall performance. This articleseries contains conceptual and practical information to help you understand InfoSphere MDMServer and the ways that you can monitor and tune it to meet your performance goals andbusiness needs. It provides valuable reference information that you can refer to when you develop,analyze, and troubleshoot performance issues in your MDM system.

Acknowledgments

We would like to thank David Borean, Lena Woolf, Bill Xu, Steve Reese, and Berni Schiefer fortheir input and suggestions for this article series.



of 19

Resources

Learn

• Go to Part 1 of the series for details about the goals and how to begin your approach tooptimization.

• Explore DB2 performance tuning scenarios (developerWorks May 2008).• Refer to the IBM Redbook™New Mainframe: z/OS Basics for an introduction to z/OS basics.• Continue tuning and monitoring with Best practices: Tuning and monitoring database system

performance (developerWorks, various dates).• Get the IBM Pattern Modeling and Analysis Tool for Java Garbage Collector for a tool that

parses verbose GC trace, analyzes Java heap usage, and recommends key configurationsbased on pattern modeling of Java heap usage.

• See HeapAnalyzer for a graphical tool to discover possible Java heap leaks.• Review the Redbook WebSphere Application Server V6 Scalability and Performance

Handbook for more details.• Check out DB2 UDB for z/OS Version 8 Performance Topics for more about performance.• Browse the MDM Server Infocenter for background information.• Read the IBM Redpaper WebSphere Customer Center: Understanding Performance for more

details.• Learn to load a large volume of Master Data Management data quickly using RDP MDM

Server maintenance services batch (developerWorks Aug 2009) .• Look at MDM Server Support Technote: MDM Server Performance Must-Gather information.• Walk through DB2 tuning tips for OLTP applications (developerWorks Jan 2007) for more

tips.• Learn more about Information Management at the developerWorks Information Management

zone. Find technical documentation, how-to articles, education, downloads, productinformation, and more.

• Stay current with developerWorks technical events and webcasts.• Follow developerWorks on Twitter.

Get products and technologies

• Build your next development project with IBM trial software, available for download directlyfrom developerWorks.

Discuss

• Participate in the discussion forum for this content.• Check out the developerWorks blogs and get involved in the developerWorks community.


http://www.ibm.com/developerworks/data/library/techarticle/dm-0508chong/

http://www.redbooks.ibm.com/abstracts/sg246366.html?Open

http://www.ibm.com/developerworks/data/bestpractices/systemperformance/

http://www.ibm.com/developerworks/data/bestpractices/systemperformance/

http://www.alphaworks.ibm.com/tech/pmat

http://www.alphaworks.ibm.com/tech/heapanalyzer

http://www.redbooks.ibm.com/abstracts/sg246392.html



http://publib.boulder.ibm.com/infocenter/mdm/v8r0m0/index.jsp?Open

http://www.redbooks.ibm.com/abstracts/redp4499.html

http://www.ibm.com/developerworks/data/library/techarticle/dm-0908mdmdataload/index.html

https://www-304.ibm.com/support/entdocview.wss?uid=swg21370106

http://www.ibm.com/developerworks/data/library/techarticle/anshum/0107anshum.html

http://www.ibm.com/developerworks/data/

http://www.ibm.com/developerworks/data/

http://www.ibm.com/developerworks/offers/techbriefings/

http://www.twitter.com/developerworks/

http://www.ibm.com/developerworks/downloads/

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1214

http://www.ibm.com/developerworks/blogs/

http://www.ibm.com/developerworks/community



of 19

About the authors

Jesse Chen

Jesse F Chen is a senior software engineer in the Information Managementgroup with a focus on software performance and scalability. He has 12 years of ITexperience in software development and services. His recent projects include FileNetand DB2 integration and optimization, Master Data Management, and WebSphereProduct Center. Jesse is also a frequent contributor and speaker at multiple ECMconferences on topics of performance, scalability, and capacity planning. He earneda Bachelor's degree in Electrical Engineering and Computer Science from theUniversity of California, at Berkeley, and a M.S. degree in Management in Informationand Systems from New York University.

Yongli An

Yongli An is an experienced performance engineer focusing on Master DataManagement products and solutions. He is also experienced in DB2 database serverand WebSphere performance tuning and benchmarking. He is an IBM CertifiedApplication Developer and Database Administrator - DB2 for Linux, UNIX, andWindows. He joined IBM in 1998. He holds a bachelor degree in Computer Scienceand Engineering and a Masters degree in Computer Science. Currently Yongli is themanager of the MDM performance and benchmarks team, focusing on Master DataManagement Server performance and benchmarks, and helping customers achieveoptimal performance for their MDM systems.

Adam Muise

Adam Muise is a Senior Consultant for IBM InfoSphere products in the InformationManagement Technology Ecosystem. Here he enables IBM Business Partners tosell and implement InfoSphere software, with a focus on Master Data Managementsolutions. In his previous role at IBM, he was a founding member of the MDMPerformance and Benchmarks team. Before joining IBM in 2006, Adam worked as aSenior Consultant for Performance and High Availability at Hewlett Packard and as aPerformance Specialist at CIBC. Adam holds an honors degree from York Universityin Information Technology and Digital Humanities.

© Copyright IBM Corporation 2010(www.ibm.com/legal/copytrade.shtml)Trademarks(www.ibm.com/developerworks/ibm/trademarks/)

http://www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/developerworks/ibm/trademarks/



of 19

Dm perf tuning2 PDF

Documents

Transcript of Dm perf tuning2 PDF