SQL Server & Siebel In About One Day

By Frank McBath

frankmcb@microsoft.comwww.siebelonmicrosoft.com

All Alliance Materials

• Posted at:

www.siebelonmicrosoft.com

• All materials referenced in this presentation are posted on the website, too.

Frank McBath

Americas Technical Specialist

Siebel Microsoft Global Alliance

18 Years in IT

With MS since 1999

Author of SQL Server Backup & Recovery

Co-Author of SQL Server High Availability

McBath’s IT Life

• Roles & Technologies:• Programmer

– 4 years

• Networking – 3 years

• Database – 10+ years

• Enterprise Applications – 7 years

• Consulting – 8+ years

• Production Support – 2 years as mgr of MS’s

1.5TB SAP database - Basis

• Types of Companies:• Small, Medium, Huge• Non-profit (American Cancer

Society)• Education (Michigan State

University)• High Tech (SAP, Microsoft)• Outsourcing (Perot)• Fortune 100 (Exxon, United

Technologies, Cooper Industries)

What I do now…

• Make you successful using Microsoft technologies and Siebel.

• Think of me as a broker…

Format of this Day…

• Up to you… I have much more material than I can cover in a day…

• Heavy Q&A• Go over Tech Notes in detail• White board…• Demo’s• Hands On • Use PowerPoint for some structure

Topics for the Day

• Marketecture Slides• Joint Support Queue• 64 bit• Performance &

Monitoring• Query Repro• DB Best: Lab• Tools

– Enterprise Manager– SQL Agent– Query Analyzer– Profiler– Perfmon

• Memory: SQL & Siebel– /3GB AWE /PAE– 64 Bit

• Disk– Architecting a scalable

solution• EIM

– Everything you wanted to know

• Cursors– What they are– How they work – Implicit Cursor

Conversion

Marketecture

Siebel 7.7 Benchmark Results

• 10/04/2004 13,000 Concurrent User Siebel 7.7 Performance and Scalability Benchmark on HP-UX servers

• 10/01/2004 8,000 Concurrent User Siebel 7.7 Performance and Scalability Benchmark on HP BladeSystem infrastructure and HP Integrity servers running Microsoft SQL Server 2000 (64-bit)

• 10/01/2004 3,000 Concurrent User Siebel 7.7 Performance and Scalability Benchmark on a single HP BladeSystem Infrastructure solution running Microsoft SQL Server 2000

• 08/27/2004 12,500 Concurrent User Siebel 7.7 Industry Applications Performance and Scalability Benchmark on IBM eServer p690s / IBM DB2 UDB on eServer p5 570

• 02/17/2004 2,500 Concurrent User Siebel 7 Performance and Scalability Benchmark on IBM eServer pSeries® 650 and IBM DB2 UDB

Source: Siebel’s Website http://www.siebel.com/crm/performance-benchmark.shtm

30,000 concurrent Users30,000 concurrent Users

World Class Performance

Siebel 7.7 8,000 User PSPP User Benchmark on HP/Windows/SQL64

Node Functional UseAverage CPU (%)

UtilizationAverage Memory

Utilization (MB)

3x Proliant BL20pG2 Web Server – Application Requests 28% 1,100

1x Proliant BL20pG2 Web Server – HTTP Adapter 5% 2,336

1x Proliant BL20pG2 Siebel Gatew ay Server 3% 400

9x Proliant BL20pG2 Siebel Application Server- End Users 61% 4,068

1x Proliant BL20pG2 Siebel Application Server - EAI HTTP Adapter 25% 5,778

1 x Integrity rx5670 Microsoft SQL Server 2000 (64-bit) 95% 27,853

Latest 7.7 8,000 User Benchmark

Oracle supported 10K users on rp8400 with 16x CPU 875Mhz with Oracle 9.x/Hp-ux posting 35% CPU and 18.2GB memory.

MSSQL supported 12K users on rx5670 with 4x CPU 1.5Ghz with sql2K/windows2003 posting 47% CPU and 13.3GB memory. Result:17.95% less CPU** 26% less memory60% less cost20% more users (12K vs 10K)

SQL Server 2000 64-bit did more with less.

*HP rx5670 is around $50K on HP web if you pay the full price.*HP rp8400 base price $124K** rp8400 16 CPU SpecInt 98.2, rx5600 4 CPU SpecInt 60** (cont) (98.2 * 35%) = 34.37 , (60 * 47%) = 28.2** (cont) 1 – (28.2/34.37) = 17.95%** www.spec.org** Scott Hall Slide

Oracle 10K vs. SQL Server 12K

Memory, CPU & Clustering LimitsWindows Server 2003 and SQL Server 2000

MaxResource

Supported

Windows Server 2003, Standard Edition

Windows Server 2003, Enterprise Edition

Windows Server 2003, Datacenter Edition

32-bit 32-bit 64-bit 32-bit 64-bit

SQL Server 2000,StandardEdition

RAM (GB) 2 2 2

CPU 4 4 4

ClusterNodes

SQL Server 2000,Enterprise Edition

RAM (GB) 3 321 641

CPU 4 8 32

ClusterNodes

4 4

SQL Server 2000,64-bit Edition

RAM (GB) 64 512

CPU 8 64

ClusterNodes

8 8

1 1 With Address Windowing Extensions (AWE) in SQL Server 2000

Cost Savings

Scalability

The highly scalable database platform for memory intensive, performance-critical business applications

Optimized for Windows Server 2003 and ItaniumOptimized for Windows Server 2003 and Itanium

Great performanceGreat performance Large memory addressability (up to 32 TB) Large memory addressability (up to 32 TB) Nearly unlimited virtual memory (up to 8 TB)Nearly unlimited virtual memory (up to 8 TB) I/O savings due to larger memory buffer poolsI/O savings due to larger memory buffer pools

T-SQL code-compatibility with SQL Server 2000T-SQL code-compatibility with SQL Server 2000

8 node clustering support8 node clustering support

Same on-disk format as 32-bit for easy migrationSame on-disk format as 32-bit for easy migration One setup for database & OLAP based on Windows One setup for database & OLAP based on Windows

Installer technologyInstaller technology Compelling alternative to expensive Unix solutionsCompelling alternative to expensive Unix solutions

Manageability

SQL Server 64 Bit (I)

SQL Server 64 Bit (II)

• Memory problems are gone…

• No more headaches with 3GB for proc cache

• But… Have to use client side tools

Joint Support Queue

Joint Support Queue

• What is it?

• How can it help you?

• How do you engage it?

• Smitty’s presentation on our website:– http://www.siebelonmicrosoft.com/

mssiebel/Technology/WorkShopsContent.aspx#One

PSS Service Offering

• Health Check– sql_health.doc

• Sample of the report– C:\tmp\workshop\companyx_report.doc

• Great idea for Pre-Go Live Check

Performance, Monitoring & Life

The Whole Picture• Everyone wants 5 9’s…• Architecting a scalable solution…• Any of the below are just a part… You need a combination of all of

them for a truly highly available system.• Disk design• Clustering• Log Shipping• Backups• DBCCs• Restores• Rotations & Offsites

– Ex. Are you on the call list?• Monitoring• Processes

– Sytematized– How are things escalated?– Documented? Phone Numbers?

Disk Architecture & Siebel

• Tech Note 9

• Let’s walk through it…

Throughput (I)• SQLIOStress.exe

• Simulates SQL Server Activity.• http://support.microsoft.com/default.aspx?kbid=231619

• This article outlines the SQLIOStress utility. You can use this utility to perform stress tests on disk subsystems to simulate Microsoft SQL Server 2000 and Microsoft SQL Server 7.0 read, write, checkpoint, backup, sort, and read ahead activities.

The utility was formerly named SQL70IOStress, but it has been upgraded to handle both SQL Server 7.0 and SQL Server 2000 I/O patterns. It has also been renamed SQLIOStress.

Throughput Numbers (II)

backup database siebeldb to disk='NUL:'go

Processed 87976 pages for database 'siebeldb', file 'sieb752_Data' on file 1.Processed 1 pages for database 'siebeldb', file 'sieb752_Log' on file 1.BACKUP DATABASE successfully processed 87977 pages in 28.853 seconds (24.978 MB/sec).

Throughput (II) Cont.

• Note the issue with a differential backup and backing up to a NUL: device.

• Measure throughput with:– SAN vendor’s tools– Perfmon

• Windows System Monitor – PERFMON.EXE– System Monitor in the Performance console, started from

Administrative Tools• Query Analyzer – ISQLW.EXE

– Graphical showplan – Statistics profile– Statistics IO

• Profiler- PROFILER.EXE– Spot problematic queries– Use the Tuning or Duration templates– Monitor the overhead carefully on your system

• Index Tuning Wizard & Siebel: OLTP & EIM usage• Throughput Analysis

– Backup to NUL: – SQLIOStress.exe

Tool Box

Perfmon• Make sure you set DISKPERF –Y on command line to get counters dumped

• Performance Object: Physical Disk• Counters:

– %Disk Time • Number for entire logical drive should be less than 100%

– Avg Disk Queue Length (system requests on avg. waiting for disk access)• Want 0• If it is above 2 (def. above 3), look into it• See if it’s sustained queueing or temporary

– Avg. Disk Read/Write /sec (diff counters, and remember Logical vs. Physical)• Nice to have: 5 – 7 ms (might be optimistic)• Realistic (today’s technology): 20 – 25 ms on a moderately loaded system• Log device service write times should be below 20 ms• Technology dependent …

• See BOL (index “Bottlenecks” then “Monitoring Disk Activity”) for some more tips

Know Your Server’s Limits

• Avoid Thrashing: CPU & IO.• You typically don’t want CPU more than 80%• Monitor the disk sub system• Watch for disk queueing, but not useful if you don’t

know the physical layout of the SAN LUNs.• The following Perfmon counters work no matter…• Logical Disk -> Avg Disk/Sec [Read | Write | Transfer]• Counters are in milliseconds.

– .010 = 10 milliseconds (ms)

• What the numbers mean:– 10ms Good– 10ms to 20ms Reasonable– 20ms to 50ms Busy– more than 50ms Bad

File System Hot Spots

• Shows stall time per I/O request on a per file basis in milliseconds.

• Run and show the output.• Cumulative… not for “spot” inspection.

select DbId, FileId,IoStallMS/(NumberReads+NumberWrites)

‘ms/io’ from ::fn_virtualfilestats(-1,-1)

System Monitor• Useful Counters

– Processor - % Processor Time– Physical Disk - %Disk Time, Avg. Disk Queue Length– Memory – Available MBytes– System – Context Switches / sec– SQL Server Locks – Lock Waits/sec, Number of Deadlocks/sec– SQLServer: Access Methods

• Full Scans/sec, Page Splits/sec, Table Lock Escalation/sec– SQLServer: Buffer Manager

• Buffer Cache Hit Ratio, Lazy Writes/sec, Page Reads/sec, Page Writes/sec, ReadAhead Pages/sec

– SQLServer: Databases - Transactions/sec– SQLServer: General Statistics - User Connections

• Q150934 – How to Create a Performance Monitor Log

Troubleshooting SQL Server• Tech Note 5: Materials for Troubleshooting SQL Server Performance• Supplemental Reading:• 283696INF: Job to Monitor SQL Server 2000 Performance and Activity• 283784 INF: How to View SQL Server 2000 Activity Data• 283725 INF: How to View SQL Server 2000 Blocking Data• 283886 HOW TO: View SQL Server 2000 Performance Data• 283786 INF: How to Monitor SQL Server 2000 Traces• 324692Support WebCast: How to Collect and Analyze Performance Data in

Microsoft SQL Server• 271509INF: How to Monitor SQL Server 2000 Blocking• 224587HOW TO: Troubleshoot Application Performance with SQL Server• 224453INF: Understanding and Resolving SQL Server 7.0 or 2000 Blocking

Problems• 243589HOW TO: Troubleshoot Slow-Running Queries on SQL Server 7.0 or

Later• 243588HOW TO: Troubleshoot the Performance of Ad-Hoc Queries• 251004INF: How to Monitor SQL Server 7.0 Blocking• 244455INF: Definition of Sysprocesses Waittype and Lastwaittype Fields for

SQL Server 7.0

Poor Index Proc

• Tech Note 27: Poor Indexes• This stored procedure analyzes tables for

“poor” index. Looks for cardinality of “1” which means 1 unique value for all existing rows.

• (Same slide as a bit later!)• Bring up on QA and show

C:\poor_index_proc.sql

Look at SYSINDEXES• Query sysindexes and look for very uniform values. • What are the chances that all indexes will be the same size…

– 100% NULL or 100% ‘Y’, etc…

• select name, dpages, reserved, • reserved * 1024. * 8. 'bytes used'• from sysindexes (nolock)• where id = object_id('S_OPTY')

• sp_helpindex S_OPTY– S_OPTY_F10 CAMP_CON_ID

• select count(distinct(CAMP_CON_ID)) • from S_OPTY (nolock)• 1 Distinct values for over 4 million rows• (1 row(s) affected)

• 33 Indexes (at least) that are poor and take up 3.7G of disk.

name dpage reserved bytes used

S_OPTY_P1 446628 521967 4,275,953,664

S_OPTY_U1 26615 53675 439,705,600

S_OPTY_II 18272 36766 301,187,072

S_OPTY_U2 26612 26816 219,676,672

S_OPTY_M56 21670 21784 178,454,528

S_OPTY_M57 21670 21784 178,454,528

S_OPTY_M52 20849 20952 171,638,784

S_OPTY_M53 19554 19648 160,956,416

S_OPTY_M55 19135 19224 157,483,008

S_OPTY_M50 18923 19016 155,779,072

S_OPTY_M51 18923 19016 155,779,072

S_OPTY_M54 18923 19016 155,779,072

S_OPTY_F5 18491 18576 152,174,592

S_OPTY_F4 17955 18040 147,783,680

S_OPTY_M1 17825 17904 146,669,568

S_OPTY_M8 15918 15984 130,940,928

S_OPTY_V2 19020 14128 115,736,576

S_OPTY_F1 14039 14096 115,474,432

S_OPTY_F10 14039 14096 115,474,432

S_OPTY_F3 14039 14096 115,474,432

S_OPTY_F50 14039 14096 115,474,432

S_OPTY_F51 14039 14096 115,474,432

S_OPTY_F52 14039 14096 115,474,432

S_OPTY_F53 14039 14096 115,474,432

S_OPTY_F54 14039 14096 115,474,432

S_OPTY_F57 14039 14096 115,474,432

S_OPTY_F58 14039 14096 115,474,432

S_OPTY_F59 14039 14096 115,474,432

S_OPTY_F6 14039 14096 115,474,432

S_OPTY_F60 14039 14096 115,474,432

S_OPTY_F7 14039 14096 115,474,432

S_OPTY_F8 14039 14096 115,474,432

S_OPTY_F9 14039 14096 115,474,432

S_OPTY_M2 14039 14096 115,474,432

S_OPTY_M3 14039 14096 115,474,432

S_OPTY_M4 14039 14096 115,474,432

S_OPTY_M9 21369 14096 115,474,432

S_OPTY_V1 14039 14096 115,474,432

S_OPTY_V3 14039 14096 115,474,432

S_OPTY_V4 14039 14096 115,474,432

S_OPTY_V5 14039 14096 115,474,432

S_OPTY_V51 14039 14096 115,474,432

S_OPTY_V52 18980 14096 115,474,432

S_OPTY_V53 14039 14096 115,474,432

S_OPTY_M61_X 14039 14096 115,474,432

S_OPTY_F61_X 14039 14096 115,474,432

S_OPTY_M6 13377 13424 109,969,408

SQL Server Profiler

Profiler Terminology• Template

– Defines criteria for what to monitor– Saved in .tdf file

• Trace– Captures data based upon selected events, data

columns, and filters • Filter

– Limits the results (Equal, Not like, etc…)• Event Category

– Defines the way events are grouped• Event

– Action generated within SQL engine

Profiler

• Use Built-in templates

• Find the worst-performing queries– Filter by duration

• Identify the cause of a deadlock

• Monitor stored procedure performance

• Audit activity – C2 audits

• Reorder your output columns by Duration, CPU, read, writes, textdata, etc.

Profiler in Production• Can be very CPU intensive

– My experience in production: 8x at 100%• Filter! Filter! Filter!• Let the computer tell you what’s going foul• To proactively find out what is going wrong

– Filter on Duration > 30,000ms– Run for 24 hours– This will show you all the poor running queries on your

system.

McBath’s Oil Argument

• Do you notice your car running better when you change your oil? No. But your car sure does.

• Make your database server run as efficient as possible. Makes it more scalable. More with less.

• Can’t make it run faster? – Look at IO’s in loops or running all the time. – Repack to higher fill factor. – 5 to 4 IO’s is a 20% increase.

How to Use Profiler

• Which events to Watch

• Filter:– Real Time– Sorting out Problems– What happens if you don’t…

• Why am I capturing so much garbage in Profiler?

Capturing Performance Data

TechNote 5: Materials for Troubleshooting Microsoft SQL Server Performance

More Uses for Profiler

• Capture and filter for 24 hours

• Profiler as stress testing tool– Capture and play back– Will only hammer the database, not the

network or the Siebel application

Lab: Profiler

• EIM– Find the statements that are running more than,

for example, more than 10000 Reads– C:\temp\eim.trc

• Analytics– Find the Perf Problems:– C:\temp\analytics000.trc– C:\temp\analytics001.trc

• Siebel App– C:\temp\app.trc– Tech Note 25: Query Perf Impact

Managing Memory

• Tech Note 21: Memory• boot.ini• 3GB• AWE• PAE• SQL Server BOL:

– mk:@MSITStore:C:\Program%20Files\Microsoft%20SQL%20Server\80\Tools\Books\adminsql.chm::/ad_1_server_1fnd.htm

SQL Parallelism & Siebel• set statistics io off• go

• sp_configure 'max degree of parallelism', 1 • go• • reconfigure• go

• With Parallelism:• • Table 'EIM_CON_DTL'. Scan count 1, logical reads 10146, physical reads 0,

read-ahead reads 0.

• Without Parallelism:• • Table 'EIM_CON_DTL'. Scan count 1, logical reads 552, physical reads 0, read-

ahead reads 0.

Indexes: Siebel & SQL Server

Siebel Database

• Over 2,300 Tables• Many with over 120 columns per table• 2,200 Clustered Indexes (ROW_ID)• 10,500 Non-Clustered Indexes

– Over Indexed, many NULL indexes• 10,000+ Default Constraints• 25,000+ Check Constraints• Very few Stored Procedures• Some Triggers

– used by workflow / assignment manager

Siebel Logical Schema vs. Physical Schema

• Siebel database schema is designed as a cross platform logical schema, which is managed by Siebel Tools

• The logical schema is translated to a physical schema by using Siebel database utilities, such as DDLIMP

• The logical schema may be altered and mapped to a compatible physical data type depending on the DB platform and code page

Don’t Be Afraid to Add/Change Indexes…

• You almost always have to.• Work closely with Siebel Expert Services • Make sure your Siebel meta data is in sync

with SQL Server meta data… or bad things can happen next time you DDLSYNC…

• See examples in the next slides

Reality (I)• Out of Box…

• sp_helpindex EIM_CONTACT3

• EIM_CONTACT3_T01• EIM_CONTACT3_T02• EIM_CONTACT3_T03• EIM_CONTACT3_T04• EIM_CONTACT3_T05• EIM_CONTACT3_T06• EIM_CONTACT3_T07• EIM_CONTACT3_T08• EIM_CONTACT3_T09• EIM_CONTACT3_T10• EIM_CONTACT3_T11• EIM_CONTACT3_T12• EIM_CONTACT3_T13• EIM_CONTACT3_T14• EIM_CONTACT3_T15• EIM_CONTACT3_T16• EIM_CONTACT3_T17• EIM_CONTACT3_T18• EIM_CONTACT3_T19• EIM_CONTACT3_T20• EIM_CONTACT3_U1

• A Large Customer…

• sp_helpindex EIM_CONTACT3

• EIM_CONTACT3_T01 • EIM_CONTACT3_U1

Reality (II)• S_CONTACT_EI• S_CONTACT_F10• S_CONTACT_F11• S_CONTACT_F12• S_CONTACT_F13• S_CONTACT_F15• S_CONTACT_F2• S_CONTACT_F3• S_CONTACT_F4• S_CONTACT_F5• S_CONTACT_F6• S_CONTACT_F7• S_CONTACT_F8• S_CONTACT_II• S_CONTACT_M1• S_CONTACT_M11• S_CONTACT_M12• S_CONTACT_M13• S_CONTACT_M14• S_CONTACT_M15• S_CONTACT_M16• S_CONTACT_M17• S_CONTACT_M18• S_CONTACT_M19• S_CONTACT_M2• S_CONTACT_M20• S_CONTACT_M21• S_CONTACT_M22• S_CONTACT_M3• S_CONTACT_M4• S_CONTACT_M6• S_CONTACT_M8• S_CONTACT_M9• S_CONTACT_P1• S_CONTACT_U1• S_CONTACT_U2• S_CONTACT_V1• S_CONTACT_V2• S_CONTACT_V3• S_CONTACT_V5

• S_CONTACT_EI • S_CONTACT_F6_X • S_CONTACT_II • S_CONTACT_M1 • S_CONTACT_M50 • S_CONTACT_M8 • S_CONTACT_ML1_X • S_CONTACT_ML2_X • S_CONTACT_ML3_X • S_CONTACT_ML4_X • S_CONTACT_ML5_X • S_CONTACT_ML6_X • S_CONTACT_P1 • S_CONTACT_PREM01_X • S_CONTACT_PREM02_X • S_CONTACT_U1 • S_CONTACT_U2 • S_CONTACT_V3

• Indexes in RED were custom.

Poor Indexes

• Get rid of 100% NULL indexes• Cost a lot on INSERTS/UPDATES/DELETES, ex. EIM• Cost disk space• Cost tape space when you back them up• Only time one might be used: on an aggregate. It’s

cheaper than a full scan. Rare, though.• Stored Procedure below will examine the indexes on

the top 100 tables for indexes that probably will not be used.

C:\tmp\poor_index_proc.sql

Query Repro & Cursors

Query Repro

• Covers about 90% of cases• Query Repro: 1 hour• How to capture the commands• RPC:Starting v. RPC:Completed• Tieing beginning to end via process,

spid, ...• How to read the query plan• Statistics IO, Profile

Siebel OM Query Execution and Fetching Mechanism

• 20 tables in a join… • Siebel OM uses Server side API cursors

– For List applet functionality i.e. to maintain user state and support pending result sets

– To support multiple active statements per connections– Fast Forward cursor with auto-fetch or Dynamic cursor

when accessing text columns– Tech Note 25– Sometimes there is an implicit conversions to Keyset

(order by not covered by index )– Average fetch size is 3 or 4 rows – this is computed by

dividing the ODBC buffer size by row size• Siebel uses ODBC Prepare/Execute

– Example: Select * from table where x = ?• What it looks like…

SQL Fetch Mechanism

c1 c2 c3 c4 c5

a b c d e

f g h i j

a b c d e

f g h i j

a b c d e

f g h i j

Siebel OM

3 rows

sp_cursorprepex (….., 3)

sp_cursorfetch (…,3)

3 rows

declare @P1 intset @P1=-1declare @P2 intset @P2=0declare @P3 intset @P3=28688 Fast Forward, Parameterized, Auto Fetch, Auto Close (undocumented and subject to change)declare @P4 intset @P4=8193declare @P5 intset @P5=10 exec sp_cursorprepexec @P1 output, @P2 output, N'',N'SELECT T1.LAST_UPD_BY, T1.ROW_ID, T18.PRTNR_TYPE, T13.CREATED_BY, T2.ASGN_USR_EXCLD_FLG,

.

.

. T2.PAR_OU_ID FROM dbo.S_PARTY T1 INNER JOIN dbo.S_ORG_EXT T2 ON T1.ROW_ID = T2.PAR_ROW_ID INNER JOIN dbo.S_ACCNT_POSTN T3 ON T2.PR_POSTN_ID = T3.POSITION_ID AND T2.ROW_ID = T3.OU_EXT_ID INNER JOIN dbo.S_PARTY T4 ON T3.POSITION_ID = T4.ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T5 ON T2.PAR_OU_ID = T5.PAR_ROW_ID LEFT OUTER JOIN dbo.S_PRI_LST T6 ON T2.CURR_PRI_LST_ID = T6.ROW_ID LEFT OUTER JOIN dbo.S_POSTN T7 ON T2.PR_MGR_POSTN_ID = T7.ROW_ID LEFT OUTER JOIN dbo.S_USER T8 ON T7.PR_EMP_ID = T8.ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T9 ON T2.PAR_BU_ID = T9.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T10 ON T1.PAR_PARTY_ID = T10.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_PRTNR T11 ON T1.ROW_ID = T11.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT_SS T12 ON T1.ROW_ID = T12.PAR_ROW_ID LEFT OUTER JOIN dbo.S_BU T13 ON T1.ROW_ID = T13.PAR_ROW_ID LEFT OUTER JOIN dbo.S_OU_PRTNR_TIER T14 ON T2.PR_PRTNR_TIER_ID = T14.ROW_ID LEFT OUTER JOIN dbo.S_ASGN_GRP T15 ON T2.PR_TERR_ID = T15.ROW_ID LEFT OUTER JOIN dbo.S_INDUST T16 ON T2.PR_INDUST_ID = T16.ROW_ID LEFT OUTER JOIN dbo.S_ADDR_ORG T17 ON T2.PR_ADDR_ID = T17.ROW_ID LEFT OUTER JOIN dbo.S_OU_PRTNR_TYPE T18 ON T2.PR_PRTNR_TYPE_ID = T18.ROW_ID LEFT OUTER JOIN dbo.S_POSTN T19 ON T3.POSITION_ID = T19.PAR_ROW_ID LEFT OUTER JOIN dbo.S_USER T20 ON T19.PR_EMP_ID = T20.PAR_ROW_ID LEFT OUTER JOIN dbo.S_ORG_SYN T21 ON T2.PR_SYN_ID = T21.ROW_ID LEFT OUTER JOIN dbo.S_ORG_BU T22 ON T2.BU_ID = T22.BU_ID AND T2.ROW_ID = T22.ORG_ID LEFT OUTER JOIN dbo.S_PARTY T23 ON T22.BU_ID = T23.ROW_ID LEFT OUTER JOIN dbo.S_ORG_EXT T24 ON T22.BU_ID = T24.PAR_ROW_ID WHERE ((T2.PRTNR_FLG != ''N'') AND ((T13.BU_FLG = ''N'' OR T13.BU_FLG IS NULL) AND T2.PRTNR_FLG = ''Y''))OPTION(FAST 40)', @P3 output, @P4 output, @P5 output

Typical Siebel OM Query

Build plan and return 40 rows ASAP

sp_cursor* calls

• What is all this stuff?• Undocumented calls• To repro query capture RPC:Starting in profiler

which will have the correct parameters for the sp_cursor calls

• Do not compare performance of queries run directly with those run by the app through the cursor model.

• C:\temp\cursor.xls

sp_cursor* Dox• sp_cursorprepexec procedure• Prepares a SQL statement for cursorization and returns an opened cursor as

well as a prepared handle that can be subsequently passed to sp_cursorexecute.

• Syntax: • sp_cursorprepexec ( <prepared handle> , <cursor> , <formal parameter dfn>

,<statement> [ ,<scrollopt> [,<ccopt> [ ,<rowcount> [ ,<bound param>] [,...n]]]] )

• Arguments:• <prepared handle>• Required• In/out• integer• Server generated prepared handle identifier.• • <cursor>• Required• Out• Integer• Server generated cursor identifier.• • <formal parameter defn>• Required• In• String• Null if the stmt is not a parameterized sql stmt else defn of variables

substituted for parameter markers in the stmt.• • <stmt>• Required• In• String• This defines the result set of the cursor.•

• <scrollopt>• Optional• in/out• Integer• Scroll option: • 1: KEYSET• 2: DYNAMIC • 4: FORWARD_ONLY• 8: INSENSITIVE• 16: FAST_FORWARD• 4096: PARAMETRIZED_STMT• 8192: AUTO_FETCH• 16384: AUTO_CLOSE• 32768: CHECK_ACCEPTED_TYPES• This parameter is both input and output because it is possible that the requested

option is not appropriate for the cursor defined by <statement>.

• <ccopt>• Optional• in/out• Integer• Concurrency control option:• 1: READ_ONLY• 2: SCROLL_LOCKS (previously known as LOCKCC)• 4: OPTIMISTIC (previously known as OPTCC)• 8: OPTIMISTIC (previously known as OPTCCVAL)• 8192: ALLOW_DIRECT• 16384: UPDT_IN_PLACE• 32768: CHECK_ACCEPTED_OPTS• • <rowcount>• Optional• out• Integer• Result set rows• • <bound_param>• Optional• in• any type

Why do I get a different query plan in the Query Analyzer ?

• Bind value (Prepare Execute model)– Hard Coding Values instead of binding at Run Time

• Cursor (SQL Server API cursor)– Not putting the “ODBC Wrapper”

• SQL hint (Fast 40)– Not including compiler options

• Text column• Implicit Cursor Conversion

– Table Spools in one plan, but not the other– Capture on Implicit Cursor Event in Profiler

• Also capture on “integer data” column

(N)TEXT Columns

• (N)TEXT column may cause performance problems

• In the Siebel database schema– A logical TEXT data type is always translated to a

physical (N)TEXT column– A VARCHAR data type can be translated to either a

(N)VARCHAR column or a (N)TEXT columnVARCHAR(2000+) is translated to a (N)TEXT column

One size fits all: Implicit Cursor Conversions

• One type of cursor is requested, but it cannot be fulfilled in it’s native call.

• Rather than fail, SQL Server converts internally.• Performance problems.• For example, Siebel uses an “option fast 40”• Fast forward, read only requested with an ORDER

BY, yet no index on the WHERE clause that is ordered. SQL Server converts to a KEYSET cursor which spools off to TEMPDB for the sort.

• Fix: make an index that matches the ORDER BY.• KEYSET conversion goes away.• SQL Profiler: Event -> Cursors ->

CursorsImplicitConversion

Profiler Properties

ODBC implicit cursor conversions: BOL

Query Repro: Quick and Dirty

• Capture on RPC: Starting Event on Profiler• Cut and paste it into Query Analyzer• 99% of the time it will give you the same plan that is coming

out of Siebel.• DON’T: Spool Siebel out at the client, hard code the values

and put into Query Analyzer. – Probably won’t work– For example, it won’t have the OPTION FAST 40– Don’t hard code variables

How to make the Query in QAprint 'declaring variables'declare @P1 intdeclare @P5 intdeclare @P6 intset @P1=NULL-- set @P5=28688-- SCROLLOPT 28676 = 16384 (AutoClose) + 8192 (AutoFetch) + 4096 (Parameterized) + 4 (Forward Only)set @P5=28676set @P6=8193print 'running sp_cursorprepare'exec sp_cursorprepare @P1 output

, N'@P1 varchar(30)'-- , N'SELECT * FROM authors WHERE au_lname like @P1 OPTION (FAST 1)'

, N'SELECT * FROM authors WHERE au_lname like @P1', 1, @P5 output, @P6 output

print 'declaring more variables'declare @P2 intdeclare @P3 intdeclare @P4 intset @P3 = 1set @P2=NULLset @P3=24592-- SCROLLOPT - 24592 = 16384 (AutoClose) + 8192 (AutoFetch) + 4 (Forward Only)set @P4=8193set @P5=15print 'executing cursor'exec sp_cursorexecute @P1, @P2 output, @P3 output, @P4 output, @P5 output, 'R%'print 'select the results from the cusor execute'select @P1,@P2,@P3,@P4, @P5, @P6

Which SPID Eating up CPU

Locating the Query Using the Most CPU

• Perfmon– Counters:

• Threads: – % Processor Time– ID Process (NT PID)– ID Thread (KPID)

» select spid » from master..sysprocesses » where kpid = <ID Thread>

• Select all SQLSERVR/0 to SQLSERVR/99• DBCC INPUTBUFFER(spid)

Quick Estimate for Long Running Queries

select datediff(mi, last_batch,getdate())'minutes',spid, waittype, cpu, physical_io, convert(char(15),hostname), convert(char(15),program_name), convert(char(20),getdate()),spid, last_batch, cmdfrom master..sysprocesseswhere spid > 50 andcmd not like '%WAIT%' anddatediff(mi, last_batch,getdate()) > 1order by last_batch

Reading Query Plans, IO, & Statistics

Set Statistics Profile/IO

• What are they?• How to run: one at a time• Believe this over everything else.• How to Read them

– 1 read = 8K page– Logical v. Physical– Read aheads

• Kevin’s e-mail:– Customer is going crazy. Somewhere in Siebel the

SQL they spit out included an UPPER on a ROWID which caused a scan in an inner loop.  560GB read for the query, 140 seconds.

Estimated vs. Actual

• Query Analyzer gives you and *ESTIMATE*

• Statistics profile is the *ACTUAL* plan

• Can be different based upon several factors, ex. memory pressure and cached information

Statistics IO

use siebeldbgo

set statistics io ongo

select count(*) from S_CONTACTgo

set statistics io offgo

----------- 6

(1 row(s) affected)

Table 'S_CONTACT'. Scan count 1, logical reads 1, physical reads 0, read-ahead reads 0.

Quick Demo

• Set statistics IO

• Set statistics profile

Siebel and Update Statistics

• Problem:

– Siebel queries join a lot of tables

– A lot of the tables joined might be smaller than the threshold to execute an automatic update statistics

– Result: Bad plans for the join

• Stale statistics and EIM

– A Plan that tips over. 98 jobs run in 5 minutes. 2 run in 1 hour or… a lot longer.

– EIM running faster than Auto Update Stats can kick off.

• Solution run a manual update statistics on those tables

• Check rowmodctr in sysindexes for indid=1

• See Appendix for code on auto update stats

Example of Stale Statistics• set statistics io on • Table ‘S_CONTACT'. Scan count 0, logical reads 0,

physical reads 0, read-ahead reads 0.• Table ‘S_OPPTY'. Scan count 4382, logical reads 21439,

physical reads 650, read-ahead reads 877.• Table ‘S_PARTY'. Scan count 2, logical reads 17573,

physical reads 199, read-ahead reads 0.• After UPDATE STATISTICS with sample of 10%:• Table ‘S_CONTACT'. Scan count 0, logical reads 0,

physical reads 0, read-ahead reads 0.• Table ‘S_OPPTY'. Scan count 192, logical reads 1440,

physical reads 0, read-ahead reads 11.• Table ‘S_PARTY'. Scan count 4, logical reads 1507, physical

reads 3, read-ahead reads 59.

Waittypes: What they are, why they are

important.

Why Wait Types?

• Wait types will help define where your bottleneck is.• They are seen in the master..sysprocesses table in a

column called “waittype”.• select waittype, * from master..sysprocesses• There are all kinds of waittypes. For example,

blocking due to database locks, network io, disk queueing, etc…

• The key to solving throughput is understanding what’s damming up the river.

Wait Types

• If a thread goes into a sleep status, a wait state is set. • The wait state is contained in master..sysprocesses in

the columns waittype, and lastwaittype.• Lastwaittype is a character description of the last

wait state for this thread. It is not reset until another wait state occurs.

• Waittype is a varbinary wait state that is the current wait state. A wait time of 0 means the thread is currently running.

• See SQL_PERF.DOC for detailed information:

Microsoft Word Document

Track_waitstats stored procedure

• Track_waitstats is a stored procedure that will capture waitstats from DBCC SQLPERF, and provide a ranking of descending order based on percentage.

• This is useful in identifying the greatest opportunites for performance improvements.

• See Appendix for Stored Procedure on Perf• See the sample output…

Sample Output

Commentary• The above sample shows the majority share of wait time, 48%,

being due to network IO waits. Improving network IO is the single largest opportunity for improving application performance.

• Other lesser opportunities in the above example include LCK_M_X (exlusive locks) and WRITELOG (transaction log). Exclusive lock waits account for almost 13% of total wait time. An examination of transaction management may offer clues as to whether improvements can be made here.

• WRITELOG means threads are waiting for physical writes to complete to the transaction log. Given the 11% writelog waits, a further analysis of PERFMON disk queues for the transaction log will confirm whether the IO capacity of the transaction log drives have trouble keeping up with write requests as shown by steady and high disk queues.

EIM:Siebel & SQL Server

The Methodology

• Attached is a PDF of the whole tuning methodology. (33” x 44” Poster).

• Includes SQL Server & Siebel steps.

• Also on Alliance website.

• Siebel Note 409: EIM

C:\tmp\eim_perf_flowchart_final.pdf

Why This Presentation?

• Everyone does EIM.

• All databases have problems doing it.

• Goal: Show you how to optimize SQL Server for EIM.

Symptoms of EIM?

• The first several thousand will always go fast.• Performance deteriorates over time in a logarithmic pattern.

– Why? Because the b-trees grow and more levels to traverse.– The curve grows logarithmically, *NOT* arithmetically. – Ex. The first 5K rows go in at a rate of 1 minute. After 2

weeks of loading that same 5K rows takes 1 hour.

0102030405060708090

Rows

Min

ute

s

Series1

Series2

Phases in Data Loading

• Get flat file • Scrub the data• Load the EIM_* tables• Run EIM• Check results• Archive and restart• What everyone wants to do… load

100M rows and leave…

Best First Strategy

• Drop all but P (cluster) and U (unique) indexes

• Remove Hints (IFB File)• sp_indexoption• Smaller EIM table• 20K batch• Try and run 2 or more jobs in parallel until

disk or CPU the bottleneck.• Remember to put indexes back on after load.

Case Study• Activities: EIM_ACTIVITY• Average Row Size: 2,703 bytes• Batch Size: 20K, Siebel Out of the Box• Baseline: 1 min 36 seconds. 12,500 rows/min, 750K rows/hr = 2G/hr

• Tuning Done: – Clustered and Unique index only on both Base & EIM tables– sp_indexoption for row locks on EIM_ACTIVITY table.

• After Tuning: Start: 9:05PM, End: 11:23PM, 138 minutes.• 8 EIM threads in parallel against EIM_ACTIVITY• Batches: 726 (179 to 905 = 14,520,000 records)• 14,520,000 rows / 138 min = 105,217 rows/min, 6.3M rows/hr = 17G /hr

• HDS / Hitachi 9500 Series SAN with 2G Cache• 8 CPU x 8G RAM • 50% to 60% CPU• No degradation over load run time. Linear.

Efficiently Loading Data (I)• Load into pre-staging tables

– Scrub in tempdb• Minimal logging• Recovery model for EIM during the load for the siebeldb.

– ALTER DATABASE siebeldb SET RECOVERY SIMPLE– Implications. Be very aware of them. Read BOL.

• Scrub in SQL Server:– Efficiencies of SQL Server caching, memory management,

cpu usage, and a big database backend server.– Use set wise processing, not cursors– If have to use cursors, use Fast Forward/Read Only

• PL/SQL – Consider use of “NOLOCK” for better performance. Dirty

Reads. Works well if you are the only one on the server, etc…

• See Appendix for performance samples on cursors

Efficiently Loading Data (II)

• BULK INSERT vs BCP– BULK INSERT is “in memory”– BCP is more configurable– Both are single threaded– Only run on one CPU

• Run multiple BULK INSERTS at once across multiple CPUs.

• Use Bulk Insert When Possible• Perf is Linear: 5M/sec/cpu on 500Mhz

Bulk Insert vs. BCP

Siebel Database Server

Siebel

a11

a22

3a3

4a4

b1

b2

b3

b4

5

6

7

8

Vcc1

0

GND

0

SQL Server Instance Memory

Data File

Mainframe

Bul

k In

sert a1

1

a22

3a3

4a4

b1

b2

b3

b4

5

6

7

8

Vcc1

0

GND

0

BCP NT Process Memory

Siebel Database Server

Siebel

a11

a22

3a3

4a4

b1

b2

b3

b4

5

6

7

8

Vcc1

0

GND

0

SQL Server Instance Memory

Data File

Mainframe

BC

P

Mem

ory

to

M

emo

ry

Co

py

Data Read DirectlyTo SQL Server Memory

Data Read Into BCPMemory…

Then Copied To SQL Server Memory.

Better Design

More On Bulk Loading

• Gert Draper’s Presentation: – 39 Slides of the best information on – High Performance Data Loading

Microsoft PowerPoint Presentation

Efficiently Loading Data (III)

• Rules of thumb for bulk operations: – One bulk op per CPU.– Limit the batch Commit Size (batch size) to about

2,000 rows per batch.– Adjust up or down based on your testing. – Remember, if loading in clustered index in

sequence, only use one thread. Else, 2 bulk operations would step on each other and block.

• Bulk operations are very high performance. – Conditions in BOL (ex. TABLOCK)

• Try and load the data pre sorted

Efficiently Loading Data (IV)

• Run a Siebel server on the database server so all EIM related traffic is contained on one physical machine.

• No network if possible.

• Big Company: Running EIM over network. Put all on database server, cut runtime 30%. Broke up after the load.

• Presize DB. No autogrow!

Examples of DTS/ETL Designs

Siebel Database Server

Siebel App ServerRunning EIMMainframe Data Source

DTS Scrubbing ServerSiebel Database ServerDTS Scrubbing

Siebel Application

Mainframe Data Source

Good DesignPoor Design

Too Much Network Traffic

Separate Siebel When Loading

Done

Efficiently Loading Data (V)

• Disable any triggers on the databases (such as workflow triggers) and then re-apply/enable them after the process is done.

• If this is not for an initial data load, this means that Workflow Manager or Assignment Manager will not function during the load for the new or updated data.

• Disable docking replication. Will see in SQL Profiler (ex. INSERT into S_DOCK_TXN_LOG)

Running in Parallel

• Blocking & Lock escalation

• sp_indexoption

• 1 row trick

sp_indexoption

• Use sp_indexoption to control locking behavior on a more granular level.

• Key ideas:– Enable row level locking – Disable page level locking – On clustered index of EIM table– Base table, less so because clustered index is on

ROW_ID which is more random. EIM table clustered index is on BATCH NUMBER

• What about using more resources?– Chances are, single user.

Table Locks and EIM

• Three types of locks: row, page, table.

• SQL Server does dynamic lock escalation. (See with SP_LOCK)

• If enabled row, disabled page… you still have table locks.

• This can cause severe concurrency issues.

• The fix…

The “1 Row Trick”

• Insert a dummy row into the EIM table.

• Begin a transaction, update the dummy row, don’t commit it.

• This will hold 1 lock on the table and prevent lock escalation to a table level lock.

• Now only row level locking!

1

1

1

1

1

2

2

2

2

2

3

3

3

3

3

4

4

4

4

4

99

Database Pages

Note how 2 different batches are on same database page.

EIM_CONTACT TABLE

BATCH_NUM

Note the dummy row. Batch “99”.

Batches in Parallel

sp_indexoption Usage:

EXEC sp_indexoption 'EIM_CONTACT.EIM_CONTACT_U1', 'disallowpagelocks', TRUEgo

EXEC sp_indexoption 'EIM_CONTACT.EIM_CONTACT_U1', 'allowrowlocks', TRUEgo

One Row Trick:

begin tran update EIM_CONTACT set ROW_ID = '' where IF_ROW_BATCH_NUM = ‘99’

Note: NO COMMIT!

1211

• If still having problems…• Trace Flag 1211 turns off lock escalation at the

database server level.• Row locks ONLY!• Brute force.• Have to turn it on via command line “-T1211”

– sqlservr –d"C:\Program Files\Microsoft SQL Server\MSSQL\Data\master.mdf" –T122

• May have to use the “-G” switch. See next slide.• Can cause significant memory pressure in an OLTP

environment due to the 3GB space.• Last resort• Don’t forget to turn it off!

-G• mk:@MSITStore:C:\Program%20Files\Microsoft%20SQL%20Server\80\Tools\Books\

adminsql.chm::/ad_1_start_8m43.htm• Specifies the amount of virtual address space (in megabytes) SQL Server will leave

available for memory allocations within the SQL Server process, but outside the SQL Server memory pool. This is the area used by SQL Server for loading items such as extended procedure .dll files, the OLE DB providers referenced by distributed queries, and automation objects referenced in Transact-SQL statements. The default is 128 megabytes (MB). Use of this option may help tune memory allocation, but only when physical memory exceeds 2 gigabytes (GB) for the SQL Server 2000 Personal Edition or SQL Server 2000 Standard Edition, or 3 GB for SQL Server 2000 Enterprise Edition. Configurations with less physical memory will not benefit from using this option. Use of this option may be appropriate in large memory configurations in which the memory usage requirements of SQL Server are atypical and the virtual address space of the SQL Server process is totally in use. Incorrect use of this option can lead to conditions under which an instance of SQL Server may not start or may encounter run-time errors.

• Use the default for the -g parameter unless you see the following warning in the SQL Server error log:WARNING: Clearing procedure cache to free contiguous memoryThis message may indicate that SQL Server is trying to free parts of the SQL Server memory pool in order to find space for items such as extended stored procedure .dll files or automation objects. In this case, consider increasing the amount of memory reserved by the -g switch. Using a value lower than the default will increase the amount of memory available to the buffer pool and thread stacks; this may, in turn, provide some performance benefit to memory-intensive workloads in systems that do not use many extended stored procedures, distributed queries, or automation objects.

SQL Parallelism & EIM (I)set statistics io offgo

sp_configure 'max degree of parallelism', 1 go reconfigurego

With Parallelism: Table 'EIM_CON_DTL'. Scan count 1, logical reads 10146, physical reads 0, read-ahead reads 0.

Without Parallelism: Table 'EIM_CON_DTL'. Scan count 1, logical reads 552, physical reads 0, read-ahead reads 0.

SQL Parallelism & EIM (II)

• May want to turn back on for parallel index creation when putting indexes back on.

• Then turn it off when Siebel running.

One Huge EIM Table… Same query, more logical/physical reads, also hurts cache… Why? B-Tree is deeper.

UPDATE EIM_CON_DTLSET CXP_X_START_DT = DATEADD(hh, - DATEPART(hh, CXP_X_START_DT), DATEADD(mi, - DATEPART(mi, CXP_X_START_DT), DATEADD(ss, -DATEPART(ss, CXP_X_START_DT), DATEADD(ms, - DATEPART(ms, CXP_X_START_DT), CXP_X_START_DT)))) WHERE (IF_ROW_BATCH_NUM = 1 AND CXP_X_START_DT IS NOT NULL)

select count(*) from EIM_CON_DTL-- 25,226,736 Table 'EIM_CON_DTL'. Scan count 8, logical reads 13492, physical reads 106, read-ahead reads 12546.

select * into EIM_CON_DTL from EIM_CON_DTL_full where IF_ROW_BATCH_NUM between 1 and 6-- 240,765 Table 'EIM_CON_DTL'. Scan count 1, logical reads 10032, physical reads 13, read-ahead reads 96.

Depth of B-Tree & EIM Table

EIM_ Table

More data in EIM table means more hops and IO.

IF_BATCH_NUM =1EIM_ Table

IF_BATCH_NUM =1

Less data in EIM table means less hops and IO.

Better Design

Pro’s: Less MaintCon’s: More IO

Pro’s: More MaintCon’s: Less IO

Covered Index ExampleCovering indexes are not always the answer… IO was significantly less but the run time

was the same due to superior hardware. More problems caused with extra indexes than solve.

create index EIM_COVER_IDX on EIM_CON_DTL(IF_ROW_BATCH_NUM, T_PARTY__STA, IF_ROW_STAT_NUM, PARTY_UID, PARTY_TYPE_CD, CXP_NAME, CXP_TYPE, T_CX_CONPHONE__STA,T_PARTY__RID, T_CX_CON_NAME__STA)

Before:

Table 'EIM_CON_DTL'. Scan count 1, logical reads 10032, physical reads 0, read-ahead reads 0.

Table 'S_LST_OF_VAL'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0.Table 'CX_CON_NAME'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0.(0 row(s) affected)

After:

Table 'EIM_CON_DTL'. Scan count 1, logical reads 489, physical reads 0, read-ahead reads 0.Table 'S_LST_OF_VAL'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0.Table 'CX_CON_NAME'. Scan count 0, logical reads 0, physical reads 0, read-ahead reads 0.(0 row(s) affected)

Profiler & EIM Traces

• Siebel EIM Level Trace Level– Maximum Verbose

• Capture on events: – RPC: COMPLETED – SQL: BATCHCOMPLETED

• Save profiler traces and filter on Reads > 5000. These tend to be your problems.

• Correlate these to the Siebel EIM Trace files.

Hints & EIM (I)/*

UPDATE BT SET BT.END_DT = IT.AS_END_DT,

BT.NAME = IT.AS_NAME,BT.START_DT = IT.AS_START_DT,BT.X_CHANGE_CODE = IT.X_CHANGE_CODE,BT.X_CHANGE_DATE = IT.X_CHANGE_DATE,BT.X_CHANGE_TYPE = IT.X_CHANGE_TYPE,BT.X_POLICY_TYPE = IT.X_POLICY_TYPE,BT.X_PREMIUM = IT.X_PREMIUM,BT.X_PRINTED_FLG = IT.X_PRINTED_FLG,BT.X_PRODUCT_DESC = IT.X_PRODUCT_DESC,BT.X_PRODUCT_TYPE = IT.X_PRODUCT_TYPE,BT.X_RATE_PLAN_CD = IT.X_RATE_PLAN_CD,BT.X_SOURCE_SYSTEM = IT.X_SOURCE_SYSTEM,BT.LAST_UPD = @P1,BT.LAST_UPD_BY = @P2,BT.MODIFICATION_NUM = BT.MODIFICATION_NUM + 1

FROM dbo.S_ASSET BT (INDEX = S_ASSET_P1),dbo.EIM_ASSET IT (INDEX = EIM_ASSET_M1)

WHERE (BT.ROW_ID = IT.T_ASSET__RID ANDIT.IF_ROW_BATCH_NUM = 10410001 ANDIT.IF_ROW_STAT_NUM = 0 ANDIT.T_ASSET__EXS = 'Y' ANDIT.T_ASSET__UNQ = 'Y' ANDIT.T_ASSET__DUP = 'N' ANDIT.T_ASSET__STA = 0)

*//*

WITH HINTS:Table 'S_ASSET'. Scan count 1273, logical reads 4038, physical reads 0, read-ahead reads 0.Table ‘EIM_ASSET'. Scan count 1, logical reads 5875, physical reads 0, read-ahead reads 0.WITHOUT HINTS:Table 'S_ASSET'. Scan count 1273, logical reads 4038, physical reads 0, read-ahead reads 0.Table ‘EIM_ASSET'. Scan count 1, logical reads 1774, physical reads 0, read-ahead reads 0.*/

Hints & EIM (II)• WITH HINT:• Table 'S_CONTACT'. Scan count 1142, logical reads 8008, physical reads 0, read-ahead reads

0.• Table ‘EIM_CONTACT'. Scan count 1, logical reads 3162, physical reads 0, read-ahead reads

0.• WITHOUT HINT:• Table 'S_CONTACT'. Scan count 1142, logical reads 8008, physical reads 0, read-ahead reads

0.• Table ‘EIM_CONTACT'. Scan count 1, logical reads 231, physical reads 0, read-ahead reads 0.

• WITH HINT:• Table 'S_APPLD_CVRG'. Scan count 1, logical reads 394774, physical reads 0, read-ahead

reads 280810.• Table ‘EIM_ASSET5_FN'. Scan count 1, logical reads 366, physical reads 0, read-ahead reads

0.• WITHOUT HINT:• Table 'S_APPLD_CVRG'. Scan count 1268, logical reads 10203, physical reads 697, read-

ahead reads 0.• Table ‘EIM_ASSET5_FN'. Scan count 1, logical reads 366, physical reads 0, read-ahead reads

0.

IFB File (I)

IFB File (II)• [Siebel Interface Manager]• USER NAME = "SADMIN"• PASSWORD = "SADMIN"• PROCESS = Import ISS

• ; FIXES• USE INDEX HINTS = FALSE• USE ESSENTIAL HINTS = FALSE

• ;• ; This group of processes provides samples for import data through• ; all the interface tables, broken up into logical groups. Note• ; that the order of import is often significant.

• [IMPORT ISS]• TYPE = IMPORT• TABLE = EIM_ISS• BATCH = 1• ONLY BASE TABLES = S_ISS• ONLY BASE COLUMNS = S_ISS.NAME,S_ISS.LANG_ID,S_ISS.BU_ID

EIM, Hints & Errors2003-10-20 17:52:45 UPDATE dbo.EIM_ISS SET T_ISS__RID = '1-239-' + CONVERT(varchar, MS_IDENT - 1),

T_ISS__UNQ = 'Y' FROM dbo.EIM_ISS T1 WHERE (T_ISS__EXS = 'N' AND

ROW_ID =(SELECT MIN(ROW_ID) FROM dbo.EIM_ISS T2 (INDEX (EIM_ISS_T05)) WHERE ((T2.ISS_LANG_ID = T1.ISS_LANG_ID OR

(ISS_LANG_ID IS NULL AND T1.ISS_LANG_ID IS NULL)) ANDT2.T_ISS_BU_ID = T1.T_ISS_BU_ID ANDT2.ISS_NAME = T1.ISS_NAME ANDT2.IF_ROW_BATCH_NUM = 1 ANDT2.IF_ROW_STAT_NUM = 0 )) AND

IF_ROW_BATCH_NUM = 1 ANDIF_ROW_STAT_NUM = 0 ANDT_ISS__STA = 0) GenericLog GenericError 1 2003-10-20 17:52:45[Microsoft][ODBC SQL Server Driver][SQL Server]Index 'EIM_ISS_T05' on table 'dbo.EIM_ISS' (specified in the FROM clause) does not exist. GenericLog GenericError 1 2003-10-20 17:52:45 (compmain.cpp 16(1555) err=100101 sys=0) SBL-EIM-00101: Invalid arguments to function. GenericLog GenericError 1 2003-10-20 17:52:45 (smisched.cpp 17(821) err=100101 sys=0) SBL-EIM-00101: Invalid arguments to function.

Server Level Disabling of Hints

• Sometimes hints are hard coded and you cannot take them out.

• As a last resort, if putting commands in IFB File do not work, the DBA can turn on Trace Flag 8602 at the database level. (SQL Server Magazine Nov 2002)

• This causes SQL Server to ignore ALL hints passed to the optimizer even if they are hard coded.

8602

• Can turn on from Query Analyzer…– DBCC TRACEON(-1, 8602)– DBCC TRACEOFF(-1,8602)

• Or… at the command line level…– sqlservr –d"C:\Program Files\Microsoft

SQL Server\MSSQL\Data\master.mdf" -T8602

IFB File (III)

• Limit the Siebel metadata read• Configure IFB using: • ONLY BASE TABLES, IGNORE BASE

TABLES, ONLY BASE COLUMNS, IGNORE BASE COLUMNS.

• Will see time consumed in EIM log file• Read Siebel Technote #409.

Stale Statistics• What happens when stats get old…

– Bad plans. A query that normally runs in a few seconds can take 30 minutes.

• How do the stats get stale?– EIM updates every row in the EIM_* table. – The process that auto updates stats doesn’t “wake up”

in time between runs.• Correct this by running UPDATE STATISTICS between

runs or a SQL AGENT job that wakes up and runs.• Consider turning off auto update stats for the data load.• It’s all about getting predictable performance.

– 100 runs… 97 run in 5 minutes… 3 run in 4 hours… each… at midnight when no one is around to kill it.

• Strategy: Hard to pick a bad plan with only a clustered index.

Fragmentation Happens…• During big data loads• Run DBCC REINDEX to correct• Think about:

– Fill Factor– Pad Index– Check with Perfmon:

• SQL Server: Access Methods -> Page Splits / sec• Trend over time

• Defrag & update stats between large runs• DBCC INDEXDEFRAG, REINDEX, and drop recreate

– Samples and run times– Pro’s and con’s

• Possibly use file groups for initial load. For example, put the EIM tables in their own FG… or put the indexes on their own FG.

Recovery Models in EIM

• Run a full or differential backup frequently during the day.

• Coalesce database during backup. Known point to restart jobs if needed.

• Weigh the issues of recovery vs. lost data.• Note: Switching from FULL to SIMPLE will

break the log chain and have recovery consequences. Always make a full backup after switching to SIMPLE.

Indexes & EIM• In general, indexes degrade performance

on data loading.• Some indexes are needed: Clustered and

Unique.• Some are needed for child table references.• Add indexes back on after loads done.

Remember to re-enable parallelism for the index create, then turn off for production.

• Work with ES/TAM to prune off 100% NULL indexes

Reality (I)• Out of Box…

• sp_helpindex EIM_CONTACT3

• EIM_CONTACT3_T01• EIM_CONTACT3_T02• EIM_CONTACT3_T03• EIM_CONTACT3_T04• EIM_CONTACT3_T05• EIM_CONTACT3_T06• EIM_CONTACT3_T07• EIM_CONTACT3_T08• EIM_CONTACT3_T09• EIM_CONTACT3_T10• EIM_CONTACT3_T11• EIM_CONTACT3_T12• EIM_CONTACT3_T13• EIM_CONTACT3_T14• EIM_CONTACT3_T15• EIM_CONTACT3_T16• EIM_CONTACT3_T17• EIM_CONTACT3_T18• EIM_CONTACT3_T19• EIM_CONTACT3_T20• EIM_CONTACT3_U1

• A Large Customer…

• sp_helpindex EIM_CONTACT3

• EIM_CONTACT3_T01 • EIM_CONTACT3_U1

Reality (II)• S_CONTACT_EI• S_CONTACT_F10• S_CONTACT_F11• S_CONTACT_F12• S_CONTACT_F13• S_CONTACT_F15• S_CONTACT_F2• S_CONTACT_F3• S_CONTACT_F4• S_CONTACT_F5• S_CONTACT_F6• S_CONTACT_F7• S_CONTACT_F8• S_CONTACT_II• S_CONTACT_M1• S_CONTACT_M11• S_CONTACT_M12• S_CONTACT_M13• S_CONTACT_M14• S_CONTACT_M15• S_CONTACT_M16• S_CONTACT_M17• S_CONTACT_M18• S_CONTACT_M19• S_CONTACT_M2• S_CONTACT_M20• S_CONTACT_M21• S_CONTACT_M22• S_CONTACT_M3• S_CONTACT_M4• S_CONTACT_M6• S_CONTACT_M8• S_CONTACT_M9• S_CONTACT_P1• S_CONTACT_U1• S_CONTACT_U2• S_CONTACT_V1• S_CONTACT_V2• S_CONTACT_V3• S_CONTACT_V5

• S_CONTACT_EI • S_CONTACT_F6_X • S_CONTACT_II • S_CONTACT_M1 • S_CONTACT_M50 • S_CONTACT_M8 • S_CONTACT_ML1_X • S_CONTACT_ML2_X • S_CONTACT_ML3_X • S_CONTACT_ML4_X • S_CONTACT_ML5_X • S_CONTACT_ML6_X • S_CONTACT_P1 • S_CONTACT_PREM01_X • S_CONTACT_PREM02_X • S_CONTACT_U1 • S_CONTACT_U2 • S_CONTACT_V3

• Indexes in RED were custom.

Look at SYSINDEXES• Query sysindexes and look for very uniform values. • What are the chances that all indexes will be the same size…

– 100% NULL or 100% ‘Y’, etc…

• select name, dpages, reserved, • reserved * 1024. * 8. 'bytes used'• from sysindexes (nolock)• where id = object_id('S_OPTY')

• sp_helpindex S_OPTY– S_OPTY_F10 CAMP_CON_ID

• select count(distinct(CAMP_CON_ID)) • from S_OPTY (nolock)• 1 Distinct values for over 4 million rows• (1 row(s) affected)

• 33 Indexes (at least) that are poor and take up 3.7G of disk.

name dpage reserved bytes used

S_OPTY_P1 446628 521967 4,275,953,664

S_OPTY_U1 26615 53675 439,705,600

S_OPTY_II 18272 36766 301,187,072

S_OPTY_U2 26612 26816 219,676,672

S_OPTY_M56 21670 21784 178,454,528

S_OPTY_M57 21670 21784 178,454,528

S_OPTY_M52 20849 20952 171,638,784

S_OPTY_M53 19554 19648 160,956,416

S_OPTY_M55 19135 19224 157,483,008

S_OPTY_M50 18923 19016 155,779,072

S_OPTY_M51 18923 19016 155,779,072

S_OPTY_M54 18923 19016 155,779,072

S_OPTY_F5 18491 18576 152,174,592

S_OPTY_F4 17955 18040 147,783,680

S_OPTY_M1 17825 17904 146,669,568

S_OPTY_M8 15918 15984 130,940,928

S_OPTY_V2 19020 14128 115,736,576

S_OPTY_F1 14039 14096 115,474,432

S_OPTY_F10 14039 14096 115,474,432

S_OPTY_F3 14039 14096 115,474,432

S_OPTY_F50 14039 14096 115,474,432

S_OPTY_F51 14039 14096 115,474,432

S_OPTY_F52 14039 14096 115,474,432

S_OPTY_F53 14039 14096 115,474,432

S_OPTY_F54 14039 14096 115,474,432

S_OPTY_F57 14039 14096 115,474,432

S_OPTY_F58 14039 14096 115,474,432

S_OPTY_F59 14039 14096 115,474,432

S_OPTY_F6 14039 14096 115,474,432

S_OPTY_F60 14039 14096 115,474,432

S_OPTY_F7 14039 14096 115,474,432

S_OPTY_F8 14039 14096 115,474,432

S_OPTY_F9 14039 14096 115,474,432

S_OPTY_M2 14039 14096 115,474,432

S_OPTY_M3 14039 14096 115,474,432

S_OPTY_M4 14039 14096 115,474,432

S_OPTY_M9 21369 14096 115,474,432

S_OPTY_V1 14039 14096 115,474,432

S_OPTY_V3 14039 14096 115,474,432

S_OPTY_V4 14039 14096 115,474,432

S_OPTY_V5 14039 14096 115,474,432

S_OPTY_V51 14039 14096 115,474,432

S_OPTY_V52 18980 14096 115,474,432

S_OPTY_V53 14039 14096 115,474,432

S_OPTY_M61_X 14039 14096 115,474,432

S_OPTY_F61_X 14039 14096 115,474,432

S_OPTY_M6 13377 13424 109,969,408

Summary (I)

• Optimize your EIM– Batch Size– Hint Removal: Siebel and SQL Server– Turn off docking replication– Get rid of workflow triggers– Only load up the tables needed from the Siebel

meta data. Loading the whole catalog can represent 25% the time of your whole batch run.

– Run batches in parallel– Exclude validation of non-used data. If you know

something is never NULL (ex. EmpID), then don’t check for it.

Summary (II)• Update Stats

– Stale stats can cause problems– Investigate turning autostats off

• Defrag between large runs– Defrag both EIM_ and base tables– On large initial loads, put fill factor and pad index to 50%. Cut

down on page splits. Default is 100% full.• Use minimally logged operations to load and scrub.

– Bulk Insert, SELECT/INTO– Recovery Models– Run all data loading locally– Scrub data inside SQL Server. No cursors.

• Make the right indexes– Try monster clustered index only– Get rid of unused indexes– Add them back after runs– Work with ES to resolve support issues.

Questions?

Reference (I)

• SQL Server 2000 Failover Clustering• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/deploy/confeat/failclus.asp• Products Designed for the Microsoft Windows Operating Systems• http://www.microsoft.com/whdc/hcl/default.mspx• Microsoft Windows Clustering: Storage Area Networks• http://www.microsoft.com/windowsserver2003/techinfo/overview/san.mspx• Recommended Private "Heartbeat" Configuration on a Cluster Serverhttp://support.microsoft.com/default.aspx?scid=kb;en-us;258750• Cluster Network Role Changes Automatically• http://support.microsoft.com/default.aspx?scid=kb;EN-US;254651• Network Failure Detection and Recovery in a Two-Node Server Cluster• http://support.microsoft.com/default.aspx?scid=kb;EN-US;242600• Cluster Resources Quorum Log Size Defaults to 64 KB• http://support.microsoft.com/default.aspx?scid=kb;en-us;225081• Removing the HBA Cable on a Server Cluster• http://support.microsoft.com/default.aspx?scid=kb;en-us;294173• Support for Multiple Clusters Attached to the Same SAN Device• http://support.microsoft.com/default.aspx?scid=kb;en-us;304415• How the Cluster Service Takes Ownership of a Disk on the Shared Bus• http://support.microsoft.com/default.aspx?scid=kb;en-us;309186• The Microsoft support policy for server clusters, the Hardware Compatibility List, and the Windows Server Catalog• http://support.microsoft.com/default.aspx?scid=kb;en-us;309395• Microsoft Support for Server Clusters with 3rd Party System Components• http://support.microsoft.com/default.aspx?scid=kb;en-us;814607

Reference (II)• Supported Fibre Channel Configurations• http://support.microsoft.com/default.aspx?scid=kb;en-us;317162• Troubleshooting Multiple Cluster Symptoms on the Same SAN• http://support.microsoft.com/default.aspx?scid=kb;en-us;311081• Explanation of Why Server Clusters Do Not Verify that Resources will Work

Properly on All Nodes• http://support.microsoft.com/default.aspx?scid=kb;en-us;303431• HOWTO: Rebuild or Move MSDTC Used with a SQL Failover Cluster

http://support.microsoft.com/default.aspx?scid=kb;en-us;294209• Clusrest.exe: Cluster Quorum Restore Utility• http://www.microsoft.com/windows2000/techinfo/reskit/tools/existing/

clusrest-o.asp• Recovering from a Lost or Corrupted Quorum Log• http://support.microsoft.com/default.aspx?scid=kb;en-us;245762• How to Properly Restore Cluster Information• http://support.microsoft.com/default.aspx?scid=kb;DE;248998

Reference (III)• Related articles – SQL Server Configuration• HOW TO: Configure memory for more than 2 GB in SQL Server• http://support.microsoft.com/default.aspx?scid=kb;en-us;274750• Related articles – SQL Server Security• SQL Server Security Center• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/

security/prodtech/dbsql/default.asp• Microsoft Security Bulletin Search• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/

security/current.asp• Implementing Security• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/

prodtechnol/sql/reskit/sql2000/part3/c1061.asp• SQL Server Security How-to Guide• http://support.microsoft.com/common/canned.aspx?R=d&H=SQL%20Server

%20Security%20How-to%20Guide&LL=kbSQLServSearch&Sz=kbsecurity&sd=gn

Reference (IV)• SQL Server 2000 Auditing• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/prodtech/dbsql/sql2kaud.asp• Securing SQL Server 2000 Resource Guide• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/chklist/sql2ksrg.asp• Securing Windows 2000 Terminal Services• http://www.microsoft.com/technet/prodtechnol/win2kts/maintain/optimize/secw2kts.asp• Related articles – SQL Server Disaster Recovery• Benchmark: High Transaction Throughput During Online Database Backup• http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/sql/evaluate/featfunc/bnchmark.asp• Benchmark: High Performance Online Database Backup for Very Large Databases• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/evaluate/featfunc/bnchmrk2.asp• Database Backup and Recovery• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/maintain/monitor/c11ppcsq.asp• Backing Up and Restoring Databases• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/proddocs/admincmp/75517c03.asp• Restoring System and User Databases• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/training/c16_rest.asp• Restoring a Database• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/maintain/monitor/11ppcsqc.asp• Related articles – SQL Server Log Shipping• Log Shipping in SQL Server 2000 - Part 1• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/deploy/prodspecs/logship1.asp• Log Shipping in SQL Server 2000 - Part 2• http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/sql/deploy/prodspecs/logship2.asp• HOW TO: Transfer Logins and Passwords Between Instances of SQL Server• http://support.microsoft.com/default.aspx?scid=kb;en-us;246133

Reference (V)• Related articles – SQL Server Performance• HOW TO: Troubleshoot Application Performance with SQL Server• http://support.microsoft.com/default.aspx?scid=kb;en-us;224587• INF: How to Monitor SQL Server 2000 Blocking• http://support.microsoft.com/default.aspx?scid=kb;EN-US;271509• INF: How to Monitor SQL Server 7.0 Blocking• http://support.microsoft.com/default.aspx?scid=kb;EN-US;251004• INF: Definition of Sysprocesses Waittype and Lastwaittype Fields for SQL Server 7.0• http://support.microsoft.com/default.aspx?scid=kb;EN-US;244455• INF: Troubleshooting Stored Procedure Recompilation• http://support.microsoft.com/default.aspx?scid=kb;EN-US;243586• HOW TO: Identify the Cause of Recompilation in an SP:Recompile Event• http://support.microsoft.com/default.aspx?scid=kb;EN-US;308737• HOW TO: Identify the Cause of Recompilation in an SP:Recompile Event• http://support.microsoft.com/default.aspx?scid=kb;EN-US;308737• FIX: Concurrency Enhancements for the Tempdb Database• http://support.microsoft.com/default.aspx?scid=kb;en-us;328551• HOW TO: Troubleshoot Slow-Running Queries on SQL Server 7.0 or Later• http://support.microsoft.com/default.aspx?scid=kb;EN-US;243589• HOW TO: Use the SQLIOStress Utility to Stress a Disk Subsystem Such As SQL Server• http://support.microsoft.com/default.aspx?kbid=231619• IOMETER: Intel's I/O Performance Analysis Tool• http://sourceforge.net/projects/iometer

Case Study (I)• Problem: EIM initial processing 30,000 – 40,000 rows per hour• Final Results: EIM initial processing 1,300,000 rows per hour• Process: Standard Siebel Expert Services methodology• Tools:

– Siebel SQL Tracing for EIM to show 10 most expensive queries– Window System Monitor to measure I/O, CPU and memory– SQL Server Profiler to capture database events

• Steps:– Spot problematic queries – long running insert– Memory, SQL settings all fine, I/O was poor due to HW problems – Use Query Analyzer (Show Execution Plan) to determine whether additional

index will help – in this case: No– Minimize I/O by dropping indexes on Base Tables (S_CONTACT) – 400,000+– Separate data into batches by key ranges to avoid deadlocks and run 4 EIM

processes in parallel – 1.3 Million rows / hour– Recreate indexes, update stats, defrag table and indexes– Investigate I/O subsystem Hardware / SAN

Reference

• Siebel Note 409

• www.siebelonmicrosoft.com

Appendix (II)Configuration Specs Real World

• sp_configure • go •

• name minimum maximum config_value run_value • ----------------------------------- ----------- ----------- ------------ ----------- • affinity mask -2147483648 2147483647 0 0 • affinity64 mask -2147483648 2147483647 0 0 • allow updates 0 1 0 0 • awe enabled 0 1 1 1 • c2 audit mode 0 1 0 0 • cost threshold for parallelism 0 32767 5 5 • Cross DB Ownership Chaining 0 1 0 0 • cursor threshold -1 2147483647 -1 -1 • default full-text language 0 2147483647 1033 1033 • default language 0 9999 0 0 • fill factor (%) 0 100 0 0 • index create memory (KB) 704 2147483647 0 0 • lightweight pooling 0 1 0 0 • locks 5000 2147483647 0 0 • max degree of parallelism 0 32 1 1 • max server memory (MB) 4 2147483647 2147483647 2147483647 • max text repl size (B) 0 2147483647 65536 65536 • max worker threads 32 32767 255 255 • media retention 0 365 0 0 • min memory per query (KB) 512 2147483647 1024 1024 • min server memory (MB) 0 2147483647 0 0 • nested triggers 0 1 1 1 • network packet size (B) 512 65536 4096 4096 • open objects 0 2147483647 0 0 • priority boost 0 1 0 0 • query governor cost limit 0 2147483647 0 0 • query wait (s) -1 2147483647 -1 -1 • recovery interval (min) 0 32767 0 0 • remote access 0 1 1 1 • remote login timeout (s) 0 2147483647 20 20 • remote proc trans 0 1 0 0 • remote query timeout (s) 0 2147483647 600 600 • scan for startup procs 0 1 1 1 • set working set size 0 1 0 0 • show advanced options 0 1 1 1 • two digit year cutoff 1753 9999 2049 2049 • user connections 0 32767 0 0 • user options 0 32767 0 0 •

• exec master..xp_msver

• Go

• Index Name Internal_Value Character_Value

• ------ -------------------------------- -------------- ---------------------------------

• 1 ProductName NULL Microsoft SQL Server

• 2 ProductVersion 524288 8.00.789

• 3 Language 1033 English (United States)

• 4 Platform NULL NT INTEL IA64

• 5 Comments NULL NT INTEL IA64

• 6 CompanyName NULL Microsoft Corporation

• 7 FileDescription NULL SQL Server Windows NT

• 8 FileVersion NULL 2000.080.0789.00

• 9 InternalName NULL SQLSERVR

• 10 LegalCopyright NULL © 1988-2003 Microsoft Corp. All r

• 11 LegalTrademarks NULL Microsoft® is a registered tradem

• 12 OriginalFilename NULL SQLSERVR.EXE

• 13 PrivateBuild NULL NULL

• 14 SpecialBuild 51707904 NULL

• 15 WindowsVersion 248381957 5.2 (3790)

• 16 ProcessorCount 8 8

• 17 ProcessorActiveMask 8 ff

• 18 ProcessorType 2200 PROCESSOR_INTEL_IA64

• 19 PhysicalMemory 32759 32759 (34350374912)

• 20 Product ID NULL NULL

•

• (20 row(s) affected)

•

Cursor vs. Set Wise• The following is based on a 9.5 million row table. 2 CPU. 2G RAM.

• DECLARE scrub_cursor CURSOR FOR• SELECT y• FROM table_x• order by y

• OPEN scrub_cursor

• FETCH NEXT FROM scrub_cursor • INTO @y• WHILE @@FETCH_STATUS = 0• BEGIN• … then update table_x

Cursor• DECLARE scrub_cursor CURSOR <FAST_FORWARD> FOR• SELECT y• FROM table_x <nolock>• order by y

• -- 4:23 minutes• -- normal:• • ------------------------------------------------------ • 2003-08-06 09:48:26.560 • ------------------------------------------------------ • 2003-08-06 09:52:49.230

• -- 3:50 minutes• -- fast forward:• • ------------------------------------------------------ • 2003-08-06 09:56:16.140 • ------------------------------------------------------ • 2003-08-06 10:00:05.780

Set Wise• Just as much “work” for SQL Server to crunch 1 row as 10K

rows…

– select getdate()– go

– update table_x– set ROW_ID = '0'– go

– select getdate()– go– -- 26 seconds – ------------------------------------------------------ – 2003-08-06 09:44:14.263

– (943093 row(s) affected) – ------------------------------------------------------ – 2003-08-06 09:44:40.030

Another Strategy

• Drop all indexes• Let the Index Tuning Wizard tell you what you

need.– On both the EIM_* and Base Tables

• Use SQL Server Profiler to find scans (ex. > 10000 Reads). Add those indexes back on.

• Use SP_INDEXOPTION and enable row level ONLY locking (disable page level locking). This facilitates scaling through multiple parallel jobs.– Explain how IF_BATCH_NUM impacts parallel jobs.

• Know your data!

Log Shipping

• Log Shipping

• Jobs

• How to setup

• Failing over and syncing users