Exadata Deep Dive · Oracle Exadata Deep Dive: Architecture and Internals Kothanda Umamageswaran...

Copyright © 2015, Oracle and/or its affiliates. All rights reserved.

Oracle Exadata Deep Dive: Architecture and Internals

Kothanda Umamageswaran Vice President, Exadata Development Gurmeet Goindi Technical Product Strategist

X5-2 X5-8


Exadata Vision

Dramatically Better Databases on Smart Full-Stack Platforms

• Smart Hardware - Scale-out, Database optimized compute, networking, and storage for fastest performance and lowest costs

• Smart Software – Unique full-stack algorithms for fastest and most efficient OLTP, Analytics, Consolidation, and In-Memory DB

• Smart Integration - Full-stack optimization, automation, and support to reduce operations costs

2


Exadata Momentum: Thousands of Deployments

• Petabyte Warehouses

• Business Applications – SAP, Oracle, Siebel, PSFT, …

• Online Financial Trading

• E-Commerce Sites • Massive DB Consolidation • Public SaaS Clouds

– Oracle Fusion Apps, NetSuite, Salesforce, Intuit, SAS …

3

World’s #1 Selling Database Machine

9 of the top 10 Banks, 7 of the top 10 Retailers, 10 of the top 10 Telcos use Exadata


Exadata Hardware


Smart Hardware Architecture for Database Highest Performance - Always Available - Starts Small, Scales Huge

5

Smart Server Architecture Scale-Out Servers, Fastest CPUs

Smart Network Architecture Unified Ultra-fast InfiniBand Smart Storage Architecture Fully Scale-Out Storage

Smart Flash Architecture Ultra-fast PCIe Flash Cards Smart Tiering Architecture Tier Flash & Ultra-Capacity SAS Disks

Smart Cost Architecture Industry Standard CPUs, Flash, Disks

Smart Offload Architecture Application Co-Processors in Storage

Eliminates Legacy Anchors: Fibre Channel, Flash behind controllers and networks, Dumb Disk Trays Slow Cluster Interconnect, Low Volume Components, Proprietary Storage OS

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. 6

New Hardware Introduced This Year

Fastest Storage • 1.6TB PCIe Flash Cards in High

Capacity Servers • All-Flash Exadata Storage Server • 4 Million OLTP IOPs to SQL • 263 GB/sec Analytics bandwidth

Fastest Database Servers • X5-2 with 18-core Intel E5 Haswell 2-

socket • X5-8 with 18-core Intel E7 Haswell 8-

socket

Elastic Configurations • Start Small • Add compute or storage

servers as needed • Optimize for OLTP, DW,

In-Memory, Consolidation

Database Server

Extreme Flash Storage

High-Capacity Storage

Largest Capacity • 8 TB SAS Disks – Helium • 1.3 PETABYTES per rack

High-Capacity Storage Servers


• Scale-Out Database Servers

– 2 socket x86 processors – 36 cores – 256-768 GB DRAM

• Fastest Internal Fabric – 40 Gb/s InfiniBand – Ethernet external connectivity

• Scale-Out Intelligent Storage

– High-Capacity Storage Server

– Extreme Flash Storage Server

Exadata X5-2: Jan 2015 Release | Oct 2015 8TB Disks

7

New

New


Exadata X5-2 Database Server

• Fastest Intel Haswell Xeon Processor – 2 Eighteen-Core Intel® Xeon® E5-2699 v3 Processors

(2.3GHz) – 256 GB DDR4 (8 x 32GB) – Expandable to 768GB (24 X

32GB)

• On-board 4 x 1GbE/10GbE Base-T Ethernet, 2 x 10GbE SFP+ PCIe Ethernet card

• Dual-port InfiniBand QDR (40 Gb/s) HCA • 4 x 2.5” 600GB 10K RPM Disk Drives • Disk Controller HBA with 1GB Cache – No More

Batteries

• Redundant Power Supplies, Fan, ILOM 8

New 18-core “Haswell-EP” CPUs, DDR4 Memory expandable to 768 GB


Exadata X5-2 High Capacity Storage Server Hardware

• 2 Eight-Core Intel® Xeon® E5-2630 v3 2.4 GHz – 96 GB DRAM (4 x 8 GB + 4 x 16 GB)

• Fastest PCIe Flash – 4 x 1.6 TB Flash Accelerator F160 PCIe Card – Non Volatile Memory Express (NVMe) protocol

• 12 x 3.5” 8TB SAS drives – 96 TB Raw Capacity • Dual-port InfiniBand QDR (40Gb/s) HCA – all

ports active

• Disk Controller HBA with 1 GB Cache and Super Capacitors – No More Batteries

• Redundant Power Supplies, Fans, ILOM

9

New 8-core “Haswell-EP” CPUs, Larger NVMe Flash Cards, 8TB Disk Drives


New: Exadata X5-2 Extreme Flash Storage Server Hardware

• 2 Eight-Core Intel® Xeon® E5-2630 v3 2.4 GHz – 64 GB DRAM (8 x 8 GB)

• Fastest PCIe Flash Drives – 8 x 1.6 TB 2.5-inch Flash Accelerator F160 PCIe

Drives – Non Volatile Memory Express (NVMe) protocol – Hot Swappable

• Dual-port InfiniBand QDR (40Gb/s) HCA – all ports active

• Redundant Power Supplies, Fans, ILOM

10

New 8-core “Haswell-EP” CPUs, NVMe Flash Drive, No Spinning Drive


• Scale-Out Database Servers – 8 socket x86

processors – 144 cores – 2-6 TB DRAM

• Fastest Internal Fabric – 40 Gb/s InfiniBand – Ethernet external connectivity

• Scale-Out Intelligent Storage

– High-Capacity Storage Server

– Extreme Flash Storage Server

Announcing: Exadata X5-8: Oct 2015 Release

11

Same Networking and Storage as X5-2

New


Announcing: Exadata X5-8 Database Server

• 8 Eighteen-Core Intel® Xeon® E7-8895 v3 Processors (2.6GHz)

• 2 TB (64 x 32GB) - Expandable to 6TB (192 X 32GB)

• 4 x Dual-port 10GbE PCIe ports • 4 x Dual-port InfiniBand QDR (40 Gb/s) Express

modules

• 8 x 2.5” 600GB Disk Drives

• Disk Controller HBA with 1 GB Cache – No More Batteries

• Redundant Power Supplies, Fans, ILOM 12

18-core “Haswell-EX” CPUs, Up to 3X More Memory


X5-2 Elastic Configurations

13

Optimize Exadata for any Workload

Start with 2 Database Servers 3 Storage Servers

Add Servers Any Kind

Any Quantity

Full Rack

Database Server



Qtr Rack

576 DB Cores 13.3 TB RAM 480 TB Disk

396 DB Cores 8 TB RAM

140 TB Flash

512 Cores 90 TB Flash Cache 1,344 TB Disk

DB In-Memory Machine

Extreme Flash OLTP Machine

Data Warehousing Machine

Configuration Examples

15 DB Servers 5 Storage Servers




X5-8 Elastic Configurations

14

Very Large Scale Compute and Memory

Start with 2 Database Servers 3 Storage Servers

Add Servers Any Kind

Any Quantity

Full Rack Half Rack

576 DB Cores 24 TB RAM 672 TB Disk

288 DB Cores 12 TB RAM

179 TB Flash

Consolidation Machine

Extreme Flash OLTP Machine

Configuration Examples


2 DB Servers 14 EF Storage Servers

Database Server




Performance Extreme Flash High-Capacity

Analytic Scans 263 GB/s 140 GB/s

OLTP Reads (8K) 4.14 M IOPS 4.14 M IOPS

OLTP Writes (8K) 4.14 M IOPS 2.69 M IOPS

Flash Latency 0.25 ms @ 2M IOPS

0.25 ms @ 1M IOPS

Exadata X5 Fastest Database Machine

* Full Rack : 8 DB servers, 14 storage servers

Capacity Extreme Flash High-Capacity

Cores (for SQL offload) 16 16

Disk (per server) - 96 TB

Flash (per server) 12.8 TB 6.4 TB

Disk (full rack)* - 1,344 TB

Flash (full rack)* 179.2 TB 89.6 TB

State-of-the-art NVMe PCIe flash Consistently Low Response Times

Optimized InfiniBand I/O Protocols

Exadata Storage Server Software Smart Scan (SQL Offload)

Smart Flash Cache I/O Resource Management

Hybrid Columnar Compression

Extreme Flash Storage Server

All-Flash

High-Capacity Storage Server

Disk + Flash Cache

15


Exadata Cloud Service


Exadata Cloud Service Single Cloud Service for ALL Database Workloads

• Deploy Mission Critical Enterprise Databases with a few clicks

• 100% Database Compatibility enables easy migration and hybrid deployments

• Oracle Experts Manage Infrastructure

• Pay per Use • Complete Isolation of tenants with no

overprovisioning

17

OLTP

Data Warehousing

In-Memory Analytics

DB Consolidation

Disaster Recovery

App Certification


Decades of Database Innovation - Proven at Millions of Mission Critical Deployments Exadata Cloud Service: Scalable, Available, Secure, Compatible

Advanced Security, Label Security, Database Vault

Multitenant

Active Data Guard

Partitioning

Advanced Compression

Real Application Testing

OLAP, Analytics, Spatial and Graph

Management Packs

Real Application Clusters

In Memory Database InfiniBand Fabric

Columnar Flash Cache

HCC

10:1

I/O I/O I/O

PCI Flash Smart Flash Cache, Smart Flash Log

Hybrid Columnar Compression

I/O Resource Management

Exafusion Direct-to-Wire Protocol

Offload SQL to Storage

Network Resource Management

In-Memory Fault Tolerance

All Exadata Platform Innovations

All Oracle Database

Innovations


Smart Exadata Software


Exadata – Unique Smart Platform Software

Smart Storage • Hybrid Columnar Compression

reduces space usage by 10x • Database aware Flash Caching gives

speed of flash with capacity of disk

Smart OLTP • Special InfiniBand protocol for best

OLTP latency : 250us IOs @ 2MM IOPs • Ultra-fast transactions using DB

optimized flash logging algorithms

Smart Consolidation • Workload prioritization from CPU to

Network to Storage ensures QOS • 4X more Databases in same hardware

Smart Analytics • Move queries to storage, not storage

to queries • Automatically parallelizes queries

across all storage servers • 100x Faster Analytics


Dozens of Additional Smart Platform Capabilities

21

Smart Analytics • Storage Index data skipping • Storage offload for min/max

operations • Data Mining Storage Offload • Storage offload for LOBs and

CLOBs • Auto Flash Caching for Table

Scans • Reverse Offload to DB servers • Offload Index Fast Full Scans • Offloads Scans on Encrypted

Data, with FIPS compliance • Active Bonding of InfiniBand • Instant data file creation

Smart OLTP • Smart network packet

prioritization • I/O Prioritization by DB, User, or

workload to ensure QOS • Active AWR includes storage

stats for end to end monitoring • Write-back Flash Cache • Cell-to-Cell Rebalance preserving

Flash Cache • Secure disk and flash erase • Database scoped security • Full-stack security scanning • Exachk full-stack validation • NVMe flash interface for lowest

latency IO

Smart Availability • In-Memory Fault Tolerance • Offload backups to storage servers • Prioritize rebalance of critical files • Elimination of false drive failures • Flash and disk life cycle

management alert • Avoid reading Predictive failed

disks • Cell software transparent restart • I/O hang hardening • Prevent shutdown if mirror server

is down • Confinement of poor performing

drives


Smart Platform Capabilities Introduced This Year

Smart OLTP • 3X faster OLTP messaging - DB talks

directly to InfiniBand, bypassing OS

• Instant detection of Node Failure

• Sub-second Capping of I/O Latency

Smart Licensing • Capacity-on-Demand reduces

license cost by disabling cores • Trusted Partitions limit license

scope of specialized options

Smart Analytics • 5X faster using Columnar Flash Cache • 3X faster JSON/XML using storage

offload

Smart Consolidation • Zero Overhead Xen VMs • Snapshots for Test/Dev • InfiniBand Partitioning

FIN VM

SALES VM

DB Vault

Option

Spatial & Graph Option

VM

VM


Smart Availability: Fast Mirror Resynchronization

• With 12.1.0.2 Grid Infrastructure, resync of an offline disk is treated just like a rebalance – Increased power level can cut down

resync time by over an hour – Improvement multiplies by #cells for

rolling upgrades

• Recommend upgrade to 12.1.0.2 GI

23

81

39

21

11 7 6 6

18 10

6 4 4 4 4 0

10

20

30

40

50

60

70

80

90

1 2 4 8 16 32 64

Min

utes

to R

esyn

c Ce

ll

ASM Power Limit

Resync Time Across Power Levels

HC Cell

EF Cell

One Hour Savings


Smart Management: RESTful Service and Exacli • RESTful Management of storage servers

– No need to log onto storage servers and run cellcli – Attribute level security

– Use exacli from the compute node

– Use curl or web browser to send commands to the storage server

• Lockdown the storage server and disable ssh access

• Upcoming Exadata Cloud Service will allow customers to manage and monitor storage

24

[root@compute08 ~]# exacli cell11 cell11/Username:scott Password:************** cell11> list griddisk CELL-06015: Current user does not have privileges to run this command. cell11> list griddisk attributes name, size data_CD_00_scam07celadm11 2.6954193115234375T data_CD_01_scam07celadm11 2.6954193115234375T ...


Additional Features Enabled • X5-8 server with up to 25% improvement in performance

• InfiniBand Partitioning for security and isolation in both virtual and bare metal environments

• IPv6 support for management, ILOM, client access ethernet networks

• Fixed Allocations for databases in the flash cache – Databases cannot over run their flash cache size and starve other

databases in a consolidated environment

• Up to 15% elapsed time speedup due to faster reverse offload

• Cell to Cell offload of resilver, resync, rebalance preserves flash cache

• Exadata Snapshot Management for pluggable and non-pluggable databases in Enterprise Manager 12c Release 5

25


Conclusion: Smart Platforms are the Future Dramatic Performance and Cost Advantages

26

• Smart Scan • InfiniBand Scale-Out

• Database Aware Flash Cache • Storage Indexes • Columnar Compression

• IO Priorities • Data Mining Offload • Offload Decrypt on Scans

• In-Memory Fault Tolerance • Direct-to-wire Protocol • Columnar Flash Cache • JSON and XML offload • I/O latency capping • Instant failure detection

• Network Resource Management • Multitenant Aware Resource Mgmt • Prioritized File Recovery

• Unified InfiniBand

• Scale-Out Servers

• Scale-Out Storage

• Application Co-Processors

• PCIe Flash

• Tiered Disk/ Flash

• Software-in- Silicon

Oracle Database Platforms Smart by Design Smarter Every Year

Smart Phone or

Basic Phone

Exadata Deep Dive · Oracle Exadata Deep Dive: Architecture and Internals Kothanda Umamageswaran...

Documents

Transcript of Exadata Deep Dive · Oracle Exadata Deep Dive: Architecture and Internals Kothanda Umamageswaran...