GE Centricity EMR / CPSPerformance and Scalability Troubleshooting – Steven Oubre & Chris Hays, GEHC

Centricity EMR/CPS - Performance & Scalability TroubleshootingObjectives:• Understand the process needed for

identifying performance and scalability issues.

• Understand application / tools support for troubleshooting issues.

• Understand the GEHC process for resolving performance and scalability issues.

Process for identifying performance and scalability (PNS) issuesQuestions to ask yourself (or your staff):• Can you identify the problematic workflow?• Is it affecting more than one user (global vs.

local problem)?• Is the problem reproducible?

– Can you reproduce the problem on a different client (Terminal Server vs. fat client)?

• Do you have our support number handy?– Call support before GE management, the support engineers

are trained troubleshooters! Support #: 888.436.8491– Email:

– Centricity PM: CentricityPMServices@ge.com– Centricity EMR: CentricityEMRServices@ge.com

Application / Tools Support for Troubleshooting IssuesApplication – based tools• Trace capture

– SQL, GUI, Workflow (PNS instrumentation), Full, MEL

Platform – based tools• OS (Windows, HPUX, AIX)

– Perfmon, glance, nmon, Topas, vmstat• DBMS (SQL Server, Oracle)

– Oracle AWR/ASH, SQL Server Dashboard / DMV views

IP Performance Core Team Process

Application Tracing MethodsEMR / CPS-Chart Module hot-keys:• SQL trace: ctrl-alt-s, logfile: c:\tracesql.txt

– Used for tracking down long running sql queries, etc. • Full DB trace: ctrl-alt-t, logfile: c:\trace.txt

– SQL trace on steroids, records formulas firing, Causes client to slow down.• UI trace: ctrl-alt-u, logfile: c:\traceui.txt

– Used for tracking UI clicks, can help in tracing a problematic workflow / application crash

• MEL trace: ctrl-alt-m, logfile: c:\tracemel.txt– Used for troubleshooting complex forms

• For EMR 6.1 / 9.2: Go to Options menu and select Tracing…, full diagnostic dialog for configuring tracing options. Very useful in Citrix / WTS installations. Will be available in CPS 9.x in the near future.

Performance CountersCommon OS/Server performance

measurements• CPU

– processor utilization (Is the server busy)– CPU queue length (Ideally should = # processors or less on average)

• Disk– Queue length (looking for i/o bottlenecks)– Disk utilization (looking for hot spots)

• Memory– Used vs. Available (possible memory pressure)– Pagefile usage (possible undersized RAM config)

• Network– Retries– Utilization

Resolution of PNS Issues90% of all server-side PNS issues

resolved thru db tuning / hw optimizations

Issues are resolved through 1 of the following:• Server side fix: Indexing (new/modified), PL/SQL

stored procedure, initialization parameters, splitting of db files onto separate media

• Next service pack / software release– In the interim, these are available through support

• Client side fix: Custom DLL / defs file change (if PNS issue is client-side critical). Will usually get rolled into next sp release.

How to run DB Performance ReportsOracle• AWR: In the [Oracle_Home]\rdbms\admin folder

– Cmd prompt: sqlplus ml/ml@(dbname) @awrrpt.sql– Follow the prompts (select html format – easier to read/navigate)– 1 hour snapshots are good for tracking down db performance issues

• ASH: similar to AWR but for individual session– Sqlplus ml/ml@(dbname) @ashrpt.sql– Follow the prompts (select html format – easier to read/navigate)

SQL Server• Performance Dashboard reports

– Run out of SQL Server Mgmt Studio, select Reports from DB popup menu

– can download current versions from MS website.• Call CPS Support for custom script

PNS Instrumentation

Used to measure internally various workflow-related timings at the client level.

Can be enabled system-wide or for individual clients.

Instrumentation data is stored in the database.

PNS Instrumentation – How to enableCPS (start a query window in SQL Server

Mgmt Studio):• select * from medlists where tablename =

'PNS_Instrumentation‘  (Make sure the Code column value is ‘On’)

• Create the Instrumentation.Config.xml file in client installation folder or

• Update the Default Configuration in the PNS_Configuration table– select * from PNS_Configuration– update PNS_Configuration set config=‘{put xml here}’

where configurationid=1

PNS Instrumentation - Instrumentation.Config.xml file Contents<Instrumentation MasterSwitch ="On“ FlushInterval=“10”> <!-- Measure workflow timing.--> <WorkflowTiming OnOff="On" /> <ResourceSnapshot OnOff="On"/> <PerformanceCounters OnOff=“Off" Interval="10"> <!-- Specify which performance counters to trace. Here is an example.

Remember, not all counters have an instance. --> <PerformanceCounter Category ="Process" Name="% Processor

Time" Instance="CPOPM06"/> </PerformanceCounters></Instrumentation>

PNS Instrumentation – How to enable (continued)EMR (using sqlplus/w or other Oracle db

query tool):• All workstations:

– Update ml.symbol set value=1 where object=‘Clinic’ and name=‘EnableWrkflwLog’;

– commit; – restart client application(s)

• Single workstation: – In client installation folder, set “_EnableWrkflowLog” to “1”

in mlsetvar.txt and restart application.

PNS Instrumentation – Workflow events we currently capture - CPSCPS/EMR:Load EMR hubOpen ChartJoin UpdateOpen DocumentEnd UpdateHold documentSign UpdateSign documentSign OrderSend OrderRx RefillPrintSend FlagRoute DocumentNew MedicationChange MedicationSave and Continue Medication

CPS/PM:Login TimeOpen ScheduleCreate or Reschedule Appointment Cancel Appointment Review Patient Demographic InfoReview Patient Insurance InfoCreate PatientSave PatientNew visitSave VisitPost single chargePost payment

PNS Instrumentation – Workflow events we currently capture – EMR/OracleChart OpenLoginNew/Join UpdateOpen DocumentEnd UpdateSign OrderSend OrderRxRefillPrint – this one logs a context field that shows what type of object was printed (48 types currently)

Send FlagRoute DocumentSign DocumentSign UpdateHold UpdateAdd new medChange medSave and Continue med

PNS Instrumentation – Useful Reporting QueriesCPS:select * from vTimingEvent where sessionid > ##; -- grouped one:select timingeventDescription,count(*) total ,max(waittime) theMax, min(waittime)

theMin, avg(waittime) theAvg, round(stdev(waittime),2) theStdev from vTimingevent where sessionid > ## group by timingeventdescription;

 select * from vresourceusage where sessionid > ##; -- grouped oneselect description,count(*) total, max(procgdihandlecount)

maxgdi,max(procuserhandlecount) maxusr from vresourceusage where sessionid > ## group by description;

EMR:select * from vtimingevent where sessionid > ##;-- grouped one:select timingeventdescription,count(*) total ,max(waittime) theMax, min(waittime)

theMin, avg(waittime) theAvg, round(stddev(waittime),2) theStdev from vTimingevent where sessionid > ## group by timingeventdescription;

Issues Historically affecting Performance and Scalability• Data set size

– databases keep growing, requiring client app / db tuning changes to handle larger data sets (obs, large user tables, unsigned documents, etc)

• Hardware Issues– Older Cisco switches / routers had auto-negotiation issues, would drop

connections to 10Mb Half– Undersized workstation clients w.r.t. 3rd party add ons (CCC for example)– Undersized server hardware– Current hardware specifications available at:

http://centricitypractice.gehealthcare.com (password-enabled)• RDBMS issues

– Oracle: 8.0 -> 8i -> 9i -> 10gR1 –> 10gR2 Each version required adjustments to OS / SW / HW configurations in order to make them work

– SQL Server: 2000 -> 2005 (sp1 -> sp2 -> sp3) Same thing

Questions?