Optimizing OLTP Oracle Database Performance using PCIe SSDs

Kai Yu

Dell Oracle Solutions Engineering

OUG Norway

Vårseminar 2013

HPC Engineering

About Author

• Kai Yu, Senior Architect, Dell Oracle Solutions Lab

18 years Oracle DBA and Solutions Engineering

Specializing in Oracle RAC, Oracle VM and Oracle EBS

Oracle ACE Director, Oracle papers author/presenter

Winner of 2011 OAUG Innovator of Year Award

Winner of 2012 Oracle Excellence Award: Technologist

of the Year: Cloud Architect

Oracle Blog: http://kyuoracleblog.wordpress.com/

HPC Engineering

3

Dell Global Solutions Engineering Lab

HPC Engineering

• Dell | Oracle Solutions Engineering : www.dell.com/oracle Dell | Oracle Partnership offers customers complete a solution Solutions Deliverables:

Tested and Validated Configuration Deployment Guild Oracle Advisor Detailed Oracle Solutions Deliverables List(SDL)

Dell Oracle Solutions Reference Configuration

Oracle Virtualization Technology Oracle Enterprise Manager & Cloud Oracle Applications Proof of Concepts

Dell Global Solutions Engineering Lab

HPC Engineering

Agenda

• Introduction to Express Flash PCIe SSD

• Configuring PCIe SSDs devices

• Use Cases of PCIe SSDs for Oracle Database

• Performance Study 1: PCIe to store Database objects

• Performance Study 2: PCIe as Oracle Smart Cache

• QAs

HPC Engineering

Introduction to Express Flash PCIe SSD • Introduction to Solid State Drive (SSD) Technology

Solid-State Drives (SSDs) use solid-state NAND flash memory to store data No rotating magnetic recording media, No mechanical moving parts of HDDs. No limited by mechanical constraints Two Type of NAND Flash: Single-level cell (SLC) NAND: one bit per Cell for high performance Multilevel cell (MLC) NAND: two bit per Cell for high capacity Different Types of SSD Formats and OS Driver Stacks: SATA Use SATA driver SAS Use SAS driver PCIe Connects to PCIe Bus:

• Express Cache PCIe SSDs Built with SLC NAND flash Used as an internal storage of Dell PowerEdge servers

6

HPC Engineering

Introduction to Express Flash PCIe SSD

7

Media Bay & Control

Panel SA

S 0

SA

S 1

SA

S 2

SA

S 3

SA

S 4

SA

S 5

SA

S 6

SA

S 7

PC

Ie S

SS

0

PC

Ie S

SS

1

PC

Ie S

SS

2

PC

Ie S

SS

3

Directly connected to PCIe bus Up to four PCIe SSDs per server 2.5 inch plugged into a device bay Capacity: 175GB/350GB Hot Swappable: add or remove a device without halting or rebooting the system

HPC Engineering

PCIe SSDs and Oracle Database Performance

• Dell Express Flash PCIe SSDs Built with SLC NAND flash Removes the performance bottleneck by the

mechanical parts of HDDs Eliminates the latency/bottleneck between server & external storage Designed for applications with low latency and heavy small random r/w IOs Performance: 1.8/1.2 GB/s for sequential r/w, 200k/320K IOPs Low Latency: < 50us, Capacity: 175GB/350GB NAND flash management by firmware on PCIe SSD, no burden for server CPU Durability:12.5 and 25 petabytes of drive life (175GB/350GB capacity,

respectively). 74898 times of the driver capacity.

8

HPC Engineering

Configuring PCIe SSD Devices

• Steps to configure Express Flash PCIe SSD drivers: Download: Express_Flash_PCIe-SSD_DRVR_RHEL6.2_1NY9K_A00_1.2.32-1.tar.gz

Unzip the driver: gunzip and tar –xvf Load it: $rpm -ivh kmod-mtip32xx-1.2.32-1.el6.x86_64_rhel6u2.rpm PCIe device partitions:

# more /proc/partitions major minor #blocks name 252 0 341873784 rssda PCIe1 252 256 341873784 rssdb PCIe2 252 256 341873784 rssdc PCIe 256 256 341873784 rssdd PCIe4

Partition these devices using fdisk /dev/rssda /dev/rssda1 Create Linux RAID configuration: mdadm --create --verbose /dev/md0 --level=1 --raid-devices=2 /dev/rssda1 /dev/rssdb1

mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/rssdc1 /dev/rssdd1

Set the proper ownership and permissions for devices in /etc/udev/rules.d

KERNEL=="md0",OWNER="grid", GROUP="asmadmin", MODE="0775"

KERNEL=="md1",OWNER="grid", GROUP="asmadmin", MODE="0775"

9

HPC Engineering

Use Cases of PCIe SSDs for Oracle Database • Use PCIe SSDs stores the entire database schema.

• PCIe SSDs stores portions of a database schema four PCIe SSDs combining with MD3220 external storage with 24 HDDs

• PCIe SSD used as Oracle Smart flash cache in a 11g R2 RAC Database

two PCIe SSDs combining with MD3220 external storage with 24 HDDs

10

PCIE SSDSAS/SATADell Power Edge R720Dell Power Edge R720

PCI-e SSD

Dell PowerVault MD3220 SAS

PCIE SSDSAS/SATA

PCI-e SSD

HPC Engineering

Use PCIe SSDs to store database objects • Single Node Database: use PCEI SSDs to store database objects

• MD3220 SAN to store the rest of the database: 24 Hard Drive Disks • Base line configuration: all the database objects are store in MD 3220 SAN

11

HPC Engineering

Use PCIe SSDs to store database objects • Create ASM diskgroup on PCIe SSDs: Two ASM disks: /dev/md01 and /dev/md02 each is 350GB, total PCIED diskgroup capacity is 700GB /dev/md01: mirroring of rssda1 and rsddb1 /dev/md02: mirroring of rssdc1 and rssdd1 • Possible Configurations: User Case 1: PCIe SSDs store the entire database User Case 2: PCIe SSDs store various portions of the database • Performance Test Tool: Quest Benchmark Factory: TPCC type workload testing. Generates concurrent workloads by incremental: 50-3000 users Measure the maximal OLTP throughput TPS (Transactions per Second) Measure the average responsible time at given # of the concurrent users

12

HPC Engineering

User Case 1: PCIe SSDs stores the Entire database • Performance Comparison of two configurations:

Baseline: the entire database schema stored in Hard Disk Drives (HDDS) Compared group: the entire database schema stored in PCIe SSDs TPS Comparison: 569% increase Response time (MS) Comparison: 250% increase of # of users for 1 second time 99 times reduce of query response time for 600 users

13

HPC Engineering

• Performance Improvement vs. Portions of database in PCIe SSDs: – Config1 : all the database objects stored in HDDs. (Base Line) – Config2: all indexes stored in PCI-e SSD Drives , – Config3: all indexes plus one most active table stored in PCI-e SSDs – Config4: all indexes plus four active tables stored in PCI-e SSD Response time Comparisons: TPS Comparisons: 12%-200% increase of # of users in 1 sec mark 14%-310% TPS Increases 2-16 times decrease of response time for 800 users

14

User Case 2: PC stores a portion of database

HPC Engineering

• Use PCIe as the smart flash cache: an extension of database buffer cache

• Configure PCEI SSD as Oracle Smart cache:

Set udev rule in /etc/udev/rules.d KERNEL=="rssda1",OWNER="oracle", GROUP="dba", MODE="0775"

15

PCIE SSDSAS/SATADell Power Edge R720Dell Power Edge R720

PCI-e SSD

Dell PowerVault MD3220 SAS

PCIE SSDSAS/SATA

PCI-e SSD

Database

Buffer Cache

Database

Buffer Cache

Smart Flash

Cache

Smart Flash

Cache

3. :Read Block from

Smart Flash Cache to

Buffer Cache

1: Read a data

block from disk

to Database

buffer

2: The block is

written to flash

cache

PCIe SSDs as Smart Cache for RAC

HPC Engineering

• Configure PCEI SSD as Oracle Smart cache: Set smart cache: alter system set db_flash_cache_file='/dev/rssda1' sid='*' scope=spfile alter system set db_flash_cache_size=20G sid='*' scope=spfile; Solve Oracle Bug with 11.2.03 on Linux: SQL> startup ORA-00439: feature not enabled: Server Flash Cache need to apply patch Patch 12949806; Oracle metalink note [ID 1473149.1] ORA-00439 during Startup of Database with Oracle Flash Cache

• Performance comparisons with baseline (no flash cache) Baseline: no flash Cache Compared groups: 20GB, 40GB, 60GB and 80GB flash cache Oracle buffer cache Size: 8GB Oracle database size: 500GB

• Monitor the smart cache usage: SELECT name, value FROM v$sysstat WHERE name IN ('physical read flash cache hits', 'physical reads', 'consistent gets', 'db block gets', 'flash cache inserts');

16

PCIe SSDs as Smart Cache for RAC

HPC Engineering

PCIe SSDs as Smart cache for RAC SELECT name, value FROM v$sysstat

WHERE name IN ('physical read flash cache hits', 'physical reads', 'consistent gets', 'db block gets', 'flash cache inserts'); NAME VALUE ---------------------------------------------------------------- ---------- db block gets 10912661 consistent gets 194736483 physical reads 4514407 physical read flash cache hits 2047180 flash cache inserts 246469

• 44.8% increase of TPS (Transactions Per Second)

17

HPC Engineering

PCIe SSD as Smart cache for RAC • 5.8 times decrease of Query Average Response Time

18

HPC Engineering

PCIe SSD as Smart cache for RAC • Observed different database wait patterns

19

HPC Engineering

PCIe SSD as Smart Cache for RAC • Free buffer wait event

– Process A: Fast data read from smart flash cache – Process B: Slow process to free a slot in memory due to the slow writing a dirty block to disk – Fast process A has to wait for slow process B Free buffer wait

20

HPC Engineering

Summary • Three user cases of PCIe SSDs for Oracle Database

1) Use PCIe SSDs to store the entire application database schema;

2) Use PCIe SSDs to store portion of application database schema.

3) Use PCIe SSDs as Oracle database smart flash cache • Performance Results: 1) Use Case1: 569% TPS increase and 99 times response time reduction, database size is limited as 700GB. 2) User Case2: have some different level of performance gains: 14% to 410% TPS increases and 41.3% to 20 times query response time reductions depending on how much is stored in PCIe SSDs. 3) Use case 3: achieved 44.5% TPS increase and 5.8 times response time reduction. Size of Smart cache increase doesn’t translate the further improvement due to the “free buffer wait”

21

HPC Engineering

A Customer Case Study of SSDs on Oracle • Optimizing Oracle Database Performance using SSDs Case study completed in Oslo, Norway The databases were serving an application from software vendor DIPS that is used at several hospitals in Norway. 20,000 concurrent users connecting to the five databases Four node RAC databases on Oracle Linux 6.3 Tested performance by replaying the real applications workloads using Oracle Real Application Testing (RAT)

• Test Results: transactions/queries continued with the same response time even after an enormous increase of load The response time was better in our PoC than the actual production system at the hospital. I/O wait was very low, and reserve CPU capacity was very high even with only three nodes in cluster. In other words the solution had a lot of spare capacity

22

HPC Engineering

A Customer Case Study of SSDs on Oracle

23

HPC Engineering

Questions?

Comments?

HPC Engineering

Contact me at kai_yu@dell.com or visit my Oracle Blog at http://kyuoracleblog.wordpress.com/