SQL Server Technical Article

Writer: Jonathan Foster

Technical Reviewer: Robert L. Davis

Special Acknowledgement: Jimmy May, Andre Ciabattari, John Yi, and Ken Hughes.

Published: April 2011

Applies to: SQL Server 2008 R2 – Hosting TempDB on Fusion-IO ioDrive Duo 1280GB MLC

Summary: This paper compares the TempDB performance between direct-attached physical disks and the Fusion-IO MLC DIMMS.

Copyright

This document is provided “as-is”. Information and views expressed in this document, including URL and other Internet Web site references, may change without notice. You bear the risk of using it.

This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.

© 2011 Microsoft. All rights reserved.

2

Introduction

The Product Quality & Online group (PQO) at Microsoft was asked to design a centralized data store hosting copies of all the OLTP and data warehouses managed by their parent group, Customer Service & Support (CSS). While the long-term plan is a scale-out approach using Parallel Data Warehousing (PDW), the short-term plan is a scale-up approach based on the familiar star schema where several individual servers running ETL’s and replication publications feed one very powerful server (VPS) that presents all the data to users. Due to the ad hoc nature of the queries expected to be run on the VPS, the monthly increase of end-users, and the need to actively manage maintenance tasks across 200+ databases, the designers were concerned about the disk i/o performance of TempDB.

The main driving factor for a scale-up design was driven by end-user needs to perform cross-database queries. Linked servers queries and distributed queries would not provide acceptable performance to end users. A scale-up design was implemented in 2009 using an HPDL5-580 G5 w/ 24 CPU cores and 256GB RAM. The server was attached to an EMC CX4-240 SAN with a mixture of Solid State (SSD), Fiber Channel (FC), and SATA drives. Performance analysis revealed that TempDB accounted for approximately 30% of all the I/O on the server on any given day. For this reason, the TempDB was hosted on 2 LUNs spread across 48 72GB SAS 15k drives that were direct-attached to the server. At an estimate of 180 IOPS per disk this setup provided approximately 5750 IOPS.

The size of the data being hosted and the number of users grew significantly in 2009/2010 which forced another upgrade in 2011. The to-be presentation server is a HP5-DL580 G7 w/ 48 cores and 1TB of RAM. The server was attached to an EMC CX4-960 SAN having a mixture of Solid State (SSD), Fiber Channel (FC), and SATA drives. The new G7’s attached storage for hosting TempDB consists of 32 300GB SAS 10k disks. An estimate of 140 IOPS per disk provides approximately 2770 IOPS for the entire LUN resulting in a 50% throughput reduction from the G5 attached storage.

TempDB storage usage from 2009-2010 was typically 400GB per day with a maximum of 800GB. Performance of TempDB on the G5 was very good with disk response times predominately in the 5ms-8ms range and an average queue depth less than 1. The average daily load on TempDB physical disks by 80% of the users is approximately 400 writes and 480 reads. The remaining 20% of the users produced an average daily load of approximately 55400 writes and 85050 reads, and these users are the primary stakeholders of the data. The number of primary stakeholders is expected to increase dramatically in the next 12 months as will the amount of data they will be querying. For this reason it is imperative that the performance of TempDB be maintained at least to current levels which eliminate the use of the G7 attached storage. Use of SAN storage to host TempDB is not desired because the number of spindles required to provide the needed throughput results in a significant amount of unused storage. Since the TempDB database accounts for over 20% of the I/O, segregating that I/O off of the SAN heads helps reduce the overall load on the SAN as well.

Test EnvironmentHardware

Platform HP ProLiant SE326M1 2U SAS ModelCPU (Clock Speed, Cache, Max TDP) 2 x 8 Intel Xeon L5520 2.26GHz, 8MB L3 Cache, 60WMemory 12 x 4GB PC3-10600R (48GB total)HDDs (Capacity, Rotational Speed, Interface, Form Factor)

20 x 146GB 10K SAS 2.5” (T$) , 5 x 300GB 10K SAS 2.5”(H$)

ioDIMMS 2 x Fusion 640 ioDrive Duo cards (U$)LUN Configuration LUN1 & LUN2 = 2x146GB RAID1 (OS & SQL)

LUN3 = 7x300GB RAID5 (SQL data and log files)LUN4 = 16x146GB RAID10 (TempDB)LUN5 = 2x640 MCL ioDimms RAID1 (TempDB)

3

Controller HP P410/256 Smart Array Controller with BatteryPower Supply 2 x 750W Gold N+1 Redundant PowerNetwork Single Dual Port Embedded Intel NIC

Storage ConfigurationHDDs Phase 1 ~ T$ = RAID1+0; H$ = RAID5

Phase 2 ~ T$ = RAID 0; H$ = RAID5ioDimms U$ = RAID 0

SoftwareOS Windows Server 2008 R2 EnterpriseSQL Microsoft SQL Server 2008 R2 (RTM) - 10.50.1600.1

(X64)

Test The test will involve a variety of TempDB intensive queries executed concurrently via SSIS while capturing disk performance counters to measure I/O throughput. The test will be executed 6 times against the same server: 3 runs with TempDB hosted on SAS drives and 3 runs with TempDB hosted on the Fusion-IO drives.

Test Details1) DDL Tasks

a. Analyze index fragmentation in a 1.25TB database having 1243 indexes in 436 tables (max index size = 3530024kb and min index size = 16kb).

b. Reorganize all indexes found to have more than 10% and less than 30% fragmentation (est. 130 indexes at 5840KB)i. Rebuild all indexes using the Sort in TempDB option that are found to have more than

29% fragmentation (est. 688 indexes at 56380144KB)ii. Run DBCC CHECKTABLE on 11242382KB table

2) DML Tasks – (executed simultaneously with the above-mentioned DDL Tasks)a. Create and populate 3 temp tables with 1247328KB each of sorted data including LOB data

types (all subsequent items are performed in 3 streams synchronously the only difference being the WHERE qualifiers in the final queries)

i. Regular SELECTs and INSERTs:

4

1. Perform sorted distinct select of 2469264KB worth of data into user table from each temp table.

2. Insert 4938528KB of data to new temp table using UNION ALL3. Insert 4938528KB of data to new temp table4. Insert 555375 KB of data to new temp table using sorted SUBSTRING

SELECT with WHERE qualifier on LOB column.ii. Cursors:

1. Insert 4938528KB of data to new temp table using UNION ALL2. Insert 4938528KB of data to new temp table by combining 4

NVARCHAR(MAX) columns into 1 NVARCHAR(MAX) column3. Create 7 record metadata table having 1 NVARCHAR(MAX) column4. Cursor through 4938528 rows in table ii looking for data elements found in

the 7 record meta table. INSERT found records to new temp table.b. Create and populate temp table with 28GB of data including LOB and uniqueidentifier data

types.i. Use FOR WHILE loop to populate 2nd temp table with all rows from 1st temp table

while scrambling the LOB data. Create clustered index on 1st temp table.ii. SELECT INTO 3rd temp table of all columns and rows from temp tables 1 & 2. Create

clustered index on 3rd temp table.iii. Using the 3rd temp table, perform 100 sorted selects using a WHERE qualifier on the

LOB columnsiv. Using the 1st & 2nd temp tables via JOIN, perform 100 sorted selects using a WHERE

qualifier on the LOB column and uniqueidentifier columns.v. Using the 1st & 2nd temp tables via JOIN, perform 200 sorted selects using a WHERE

qualifier on the LOB column and uniqueidentifier columns.c. Create and populate temp table with 327680KB of LOB data.

i. Use FOR WHILE loop to populate 2nd temp table with all rows from 1st temp table while scrambling the LOB data.

ii. SELECT INTO 3rd temp table of all columns and rows from temp tables 1 & 2.iii. Using the 3rd temp table, perform 100 sorted selects using a WHERE qualifieriv. Using the 1st & 2nd temp tables via JOIN, perform 100 sorted selects using a WHERE

qualifier.v. Using the 1st & 2nd temp tables via JOIN, perform 200 sorted selects using a WHERE

qualifierd. Using a stored procedure, create and populate a 100 record sampling of source data. Use a

FOR WHILE loop to select 1 unique row based upon specific column value 100 separate times.

5

Screen shot of test SSIS package

Success CriteriaIf the following benchmarks are met or surpassed while hosting TempDB on Fusion-IO cards, we will consider this configuration as superior to the conventional attached storage:

1) 20% less time to run the SSIS package end-to-end2) 30% less disk queuing 3) Disk queue length will not exceed 32 (2 x # of physical disks) for longer than a cumulative 120

seconds

Test Results Fusion-IO vs. SAS RAID 1+0Test Fusion-IO SAS Performance

DeltaEntire test package end-to-end

22562 seconds 24096 seconds 6.37%

DDL Task Fusion-IO SAS Performance Delta

All DDL Tasks End-to-End

22562 seconds 24096 seconds 6.37%

Analyze 1243 indexes

731 seconds 1360 seconds 46.25%

Rebuild 688 indexes 7680 seconds 7920 seconds 3.04%Reorganize 130 indexes

14100 seconds 14700 seconds 4.09%

DBCC Checktable 51 seconds 116 seconds 56.04%

DML Task Fusion-IO SAS Performance Delta

All DML Tasks End-to-End

5461 seconds 9921 seconds 44.96%

6

Generate RandomIO in TempDB using 100 Individual and Distinct Queries

5 seconds 9 seconds 45.45%

Build 320MB Temp Tables filled with bogus LOB data then run a variety of serialized SELECTs

491 seconds 518 seconds 17.90%

Create 28GB Temp Table w/ indexes and 380MB table then perform a JOIN to SELECT

1902 seconds 6074 seconds 68.69%

Create 200MB temp table containing LOB data and cursor through all rows for a string value

32 seconds 50 seconds 36%

Create 300MB temp table containing LOB data and cursor through all rows for a string value

91 seconds 81 seconds -10.09%

Create 500MB temp table containing LOB data and cursor through all rows for a string value

1171 seconds 1250 seconds 6.33%

Create a populate 3 separate 1200MB Temp Tables

706 seconds 815 seconds 13.38%

Create 200MB table and use to populate two 225MB LOB tables via JOIN then SELECT

53 seconds 44 seconds -16.99%

Create 256MB table and use to populate two 200MB LOB tables via JOIN then SELECT

462 seconds 542 seconds 14.77%

Create 256MB table and use to populate two 350MB LOB tables via JOIN then SELECT

548 seconds 538 seconds 1.83%

Disk Metric Fusion-IO SAS Performance Delta

Avg. Logical Disk Queue Length (Total)

.09 26 99.66%

Avg. Logical Disk Queue Length (initial 1000 seconds)

7.31 212.29 96.56%

Avg. Logical Disk Queue Length

0.01 0.29 96.56%

7

(remaining seconds)Cumulative seconds where avg disk queue length exceeded 32

90 380 76.32%

Test Results Fusion-IO vs. SAS RAID 0Test Fusion-IO SAS Performance

DeltaEntire test package end-to-end

22562 seconds 25390 seconds 11.14%

DDL Task Fusion-IO SAS Performance Delta

All DDL Tasks End-to-End

22562 seconds 25390 seconds 11.14%

Analyze 1243 indexes

731 seconds 897 seconds 18.51%

Rebuild 688 indexes 7680 seconds 8234 seconds 6.73%Reorganize 130 indexes

14100 seconds 16128 seconds 12.58%

DBCC Checktable 51 seconds 131 seconds 61.07%

DML Task Fusion-IO SAS Performance Delta

All DML Tasks End-to-End

5461 seconds 8556 seconds 36.18%

Generate RandomIO in TempDB using 100 Individual and Distinct Queries

5 seconds 7 seconds 28.58%

Build 320MB Temp Tables filled with bogus LOB data then run a variety of serialized SELECTs

491 seconds 530 seconds 7.36%

Create 28GB Temp Table and 380MB table then perform a JOIN to SELECT

1902 seconds 4156 seconds 54.24%

Create 200MB temp table containing LOB data and cursor through all rows for a string value

32 seconds 63 seconds 49.21%

Create 300MB temp table containing LOB data and cursor through all rows for a string value

91 seconds 106 seconds 14.16%

Create 500MB temp table containing LOB

1171 seconds 1544 seconds 24.16%

8

data and cursor through all rows for a string valueCreate a populate 3 separate 1200MB Temp Tables

706 seconds 896 seconds 21.21%

Create 200MB table and use to populate two 225MB LOB tables via JOIN then SELECT

53 seconds 103 seconds 48.55%

Create 256MB table and use to populate two 200MB LOB tables via JOIN then SELECT

462 seconds 575 seconds 19.66%

Create 256MB table and use to populate two 350MB LOB tables via JOIN then SELECT

548 seconds 576 seconds 4.87%

Disk Metric Fusion-IO SAS Performance Delta

Avg. Logical Disk Queue Length (Total)

.09 12.36 99.28%

Avg. Logical Disk Queue Length (initial 1000 seconds)

7.31 276.13 97.36%

Avg. Logical Disk Queue Length (remaining seconds)

0.01 1.66 99.40%

Cumulative seconds where avg disk queue length exceeded 32

90 300 70.00%

9

The first phase of the test was the most disk intensive and specifically the metrics captured in the first 900 seconds caused the running averages to be abnormally high. The first graph below illustrates the queue depth difference between Fusion-IO and SAS due the first 1000 seconds of the test:

1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 960

1000

2000

3000

4000

5000

6000

Fusion_RAID0_QLengthSAS_RAID10_QLengthSAS_RAID0_Qlength

Eliminating these first 1000 seconds from the test results helps illustrate more accurately the difference between Fusion-IO and SAS over the bulk of the test:

2 162 322 482 642 802 962 11221282144216021762192220820

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Fusion_RAID0_QLengthSAS_RAID10_QLengthSAS_RAID0_Qlength

10

The following graph compares the WRITE throughput between the 3 disk configurations for the first 5000 seconds of the test. Notice that the Fusion-IO cards are able to perform a disproportionate amount of WRITE’s which reduces queue (seen above) thereby finishing the operation more quickly:

1 39 77 1151531912292673053433814194574950

100000000

200000000

300000000

400000000

500000000

600000000

700000000

800000000

Fusion_Disk Write Bytes/secSAS_RAID10_Disk Write Bytes/secSAS_RAID0_Disk Write Bytes/sec

Performance monitor Summary Report for Fusion-IO

11

Performance Monitor Summary Report for SAS @ RAID10:

Performance Monitor Summary Report for SAS @ RAID 0

12

Summary & ConclusionWhen the results are strictly measured against the success criteria, the Fusion-IO card surpassed expectations for the the two queuing criteria in both SAS RAID configurations. The time-to-completion criteria delta was not as pronounced since the package ran 11% faster compared to the SAS RAID 0 configuration, and completed only 6% faster compared to the SAS RAID 10 configuration. The following caveat should be considered though, before issuing a “failure” judgment for the third success criteria.

The DDL tasks that showed the least amount of improvement were also the most disk intensive on the RAID 5 data drive (H$). Because most of that index rebuild and reorganize activity was occurring on H$ drive, holding the Fusion-IO drives accountable for that activity is not a reasonable expectation. Those DDL tasks that were TempDB dependent definitely met the success criteria. More importantly, the DML task metrics alone demonstrate that having TempDB hosted on Fusion-IO resulted in surpassing the success criteria by 50% to 100%.

Careful interpretation of the results certainly reveals that hosting TempDB on Fusion-IO cards will result in a considerable performance improvement for queries that are dependent on TempDB. Similar throughput could be achieved with conventional disks. However, the number of disks required to achieve the same throughput would result in several hundred gigabytes of “wasted space” since TempDB files must be the sole occupants of the physical disks allocated to them. Server OEM’s are no longer offering smaller 15k SAS drives in their standard SKU’s and SAN manufacturers are phasing out the offering of smaller drives. Futhermore, reducing the load on the SAN by eliminating the 10% to 20% of total I/O from tempDB activity will help improve query processing performance.

For more information:

http://www.microsoft.com/sqlserver/: SQL Server Web site

http://technet.microsoft.com/en-us/sqlserver/: SQL Server TechCenter

http://msdn.microsoft.com/en-us/sqlserver/: SQL Server DevCenter

http://www.Fusion-IO.com/: Fusion-IO

Addendum

The below table shows the performance timings for the DML package tasks on the HP5-DL580 G7 with TempDB hosted on the 30 SAS drives. The Fusion-IO cards were not installed on this server, therefore, the performance delta is not entirely valid given that the test environment was not the same. However, its worth noting that the Fusion-IO timings were still better even when compared to SAS setup with 24 additional processors and 976 additional gigabytes of RAM.

13

DML Task Fusion-IO HP5-DL580 G7 Performance Delta

All DML Tasks End-to-End

5461 seconds 8970 seconds 35.75%

Generate RandomIO in TempDB using 100 Individual and Distinct Queries

5 seconds 4 seconds -20.00%

Build 320MB Temp Tables filled with bogus LOB data then run a variety of serialized SELECTs

491 seconds 724 seconds 52.55%

Create 28GB Temp Table and 380MB table then perform a JOIN to SELECT

1902 seconds 4227 seconds 55.01%

Create 200MB temp table containing LOB data and cursor through all rows for a string value

32 seconds 74 seconds 56.76%

Create 300MB temp table containing LOB data and cursor through all rows for a string value

91 seconds 124 seconds 26.62%

Create 500MB temp table containing LOB data and cursor through all rows for a string value

1171 seconds 785 seconds -32.97%

Create a populate 3 separate 1200MB Temp Tables

706 seconds 1626 seconds 56.59%

Create 200MB table and use to populate two 225MB LOB tables via JOIN then SELECT

53 seconds 53 seconds 0%

Create 256MB table and use to populate two 200MB LOB tables via JOIN then SELECT

462 seconds 568 seconds 18.67%

Create 256MB table and use to populate two 350MB LOB tables via JOIN then SELECT

548 seconds 785 seconds 30.20%

14