Accelerate your Exadata

deployment With the skills you already know

Marc Fielding

September 2013

About me • Senior Consultant with Pythian’s

Advanced Technology Group

• 12+ years Oracle production

systems experience starting with

Oracle 7

• Blogger and conference

presenter

pythian.com/news/author/fielding

• Occasionally on twitter: @mfild

2 © 2013 Pythian

About Pythian • Recognized Leader:

– Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and Microsoft SQL Server

– Work with over 250 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments

• Expertise:

– Pythian’s data experts are the elite in their field. We have the highest concentration of Oracle ACEs on staff—8 including 2 ACE Directors—and 2 Microsoft MVPs.

– Pythian holds 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC

• Global Reach & Scalability:

– Around the clock global remote support for DBA and consulting, systems administration, special projects or emergency response

© 2013 Pythian 3

© 2013 Pythian 4

Why this presentation

So you have Exadata. What next? • Underpinning Exadata are well-known Oracle

technologies

• SQL*Plus works just like on any other Oracle system

• RAC and Grid Infrastructure, 11gR2 (or 12c!)

• ASM is there

• Services, SCAN, etc no different than other Oracle RAC

environments

© 2013 Pythian 5

So what is different? • It’s an engineered system: you run the same

hardware as other Exadata customers

• Oracle development has tested on the exact same configuration

• It’s a good thing: much fewer obscure OS issues, driver issues etc

• Single point of contact at Oracle support – hardware and software

• And yeah, there are are a few new concepts…

© 2013 Pythian 6

© 2013 Pythian 7

Storage servers

Management switch

Compute servers

Empty space

InfiniBand switches

KVM

The bits you know already • It’s running Oracle database software

• Database servers: Xeon-based servers running Oracle Linux 5

• 2, 4, 8- or more-node 11gR2 RAC cluster

• Clustered ASM instances

• Ordinary Ethernet connections to the outside

© 2013 Pythian 8

The secret sauce • InfiniBand networking

– Gives amazing latency: forget 1ms, try 1µs

– Great capacity too: 40gbps

– Used for RAC interconnect and storage access – replaces fiber channel

• Storage servers: also Linux servers – But they run Exadata-specific software

– Take I/O requests from database servers and send to disk

– Also have understanding of database concepts

© 2013 Pythian 9

Differences from SAN/NAS • No single “storage head” – storage management is

handled by the ASM cluster instead

• Easy scaling: when you add storage capacity, you get processing capacity too

• Switched network increases network capacity as you scale

• Usually dedicated database storage, so often managed by DBAs too

• Hot/cold storage: tier outer and inner tracks on a disk

© 2013 Pythian 10

Smart Scans (1) • Somewhat analogous to parallel query

• Multiple parallel processes

• Communicate with shadow processes with SQL-type constructs – rows, columns etc

• Save communication capacity by only sending the data you need, not entire data blocks

• select col1 from big_table;

• Sends only col1 data, for example

• Also more complex actions like predicate evaluation, decryption, bloom filter joins…

© 2013 Pythian 11

Smart Scans (2) • Put processing closer to the storage

• Inherently parallel: processing happens on all storage servers that might have matching data

• Only apply to bulk, direct path operations – usually full table scans

• Excludes – Index range scans

– Table access by index rowid

– Blocks with uncommitted changes

– LOBs

© 2013 Pythian 12

• Used in conjunction with smart scans

• Can avoid doing I/O entirely

• Split storage up into 1m “storage regions”

• Track high/low values

• Work best with pre-sorted data

• Example: select * from tab where col1 < 20;

© 2013 Pythian 13

Storage Indexes

2-99 4-57 33-60 90-120

Hybrid Columnar Compression (1) • New storage type with superb compression ratios: 10x or more

• Group blocks into compression units

• Store columns together instead of rows

• Typical databases have repeated values: the infamous “Y”/”N” column for example

• Then apply well-known compression like bzip2 or gzip on top

© 2013 Pythian 14

Hybrid Columnar Compression (2)

• Much like smart scans, data must be loaded in bulk:

CREATE TABLE AS SELECT, INSERT

/*+APPEND*/, SQL*Loader direct path, etc

• Otherwise data will get OLTP compression instead

• CPU overhead to compress/decompress (though

smart scans can offload decompression)

• Locks affect entire compression units rather than

single rows

© 2013 Pythian 15

Recommendations: HCC • Best for data that is inserted in large batches (data

warehouses for example)

• Also useful with time-based partitions: re-compress old, no-longer-modified data

• There are 4 levels; ARCHIVE HIGH has significant CPU overhead so use for data that’s rarely accessed

• Use DBMS_COMPRESSION to predict compression ratios, and see results afterwards

© 2013 Pythian 16

Flash cache • Designed for OLTP

• Flash memory has 5x better read latency than hard drives

• Transparent to applications

• Smart enough not to cache RMAN backup I/O, ASM secondary extents, and other non-repeatable reads

• DBAs can use resource manager and SQL to prioritize cache objects

© 2013 Pythian 17

Flash cache: recommendations • With new Exadata X3, you may fit entire DB in

flash. Enjoy!

• Entirely different from the similarly-named Database Smart Flash Cache in 11gR2

• You can put ASM disks on flash, but ASM redundancy reduces usable space

• You can use back-end commands to actually see what’s being cached

© 2013 Pythian 18

I/O Resource Management

• You can use DBRM in Oracle 11g to manage

CPU time, and manage runaway queries

• But what if your system is bottlenecked on I/O

• In Exadata, just turn on IORM and your CPU

priorities apply to I/O too

• Can also allocate I/O capacity across databases

© 2013 Pythian 19

Management tools

• OEM 12c: put a lot of work into managing Exadata

• Can now manage

all Exadata

components (incl

switches etc)

• Consolidation

planning tools to

evaluate workloads

© 2013 Pythian 20

Management tools (2)

• Command-line tools all there: SQL*Plus etc

• For storage servers, a new tool: cellcli to

manage storage servers

• cellcli can be combined with dcli to run a

command on multiple machines.

• cellcli changes to one storage server aren’t

visible to others

© 2013 Pythian 21

Hardware management • Like other enterprise hardware, Exadata has out-of-

band management

• ILOM lets you access hung servers without driving to the datacenter

• Very useful when doing patches and reboots

• Monitors system health and can send alerts when metrics are out of range

• Most configuration changes made thorough Exadata cellcli

© 2013 Pythian 22

Backups • Backups run through RMAN – no snapshot support

• Similar RMAN configuration to non-Exadata

• Use high-speed InfiniBand network if you can

• Oracle ZFS appliance can be configured for InfiniBand for example

• Use IORM to manage backup I/O to avoid performance degradation at backup time

• Use multiple channels to keep the disks busy

© 2013 Pythian 23

Disaster recovery • No storage-level replication like SRDF

• Data guard is the overwhelming favourite

• Active data guard works well with Exadata

• Non-Exadata standby: watch HCC

• Most common approach: two datacenters with identically configured Exadata machines

© 2013 Pythian 24

Data migration • Like backups: no storage-level replication

• Physical copies – Data guard physical standby

– Transportable tablespaces

– Cross-platform incrementals

• Logical transfers – Data Pump

– Direct path over DB Link

– GoldenGate

– Data guard logical standby

– External tables / SQL*Loader

© 2013 Pythian 25

Patching • Exadata has many components

• All of which have periodic patches

• Major work has been done to improve patching

• Recommended quarterly full-stack patches include patches for all equipment

• Do not apply CPU/PSU patches – Exadata bundle patches include security fixed, plus high-priority bug fixes

• Different patching options involve different levels of downtime, data-loss risk, and execution time

• Review the options

© 2013 Pythian 26

Diagnostic tools • exachk is developed by Oracle development to

compare your configuration with Oracle-recommended values

• Large library of tests, and regularly being added

• Good idea to run before and after system changes like patches or configuration updates

• Download from My Oracle Support

• Required root access for full information

© 2013 Pythian 27

New Exadata install – what to expect

• PDF configuration questionnaire – How you want your system built out

• Hardware requirements – Exadata is very dense – look at power, weight, cooling etc

• Site survey

• Equipment delivered

• Hardware configuration

• Software configuration

• Installation report

© 2013 Pythian 29

Decisions to make

• High performance / high capacity

• ASM normal / high redundancy

• Data/FRA storage split

• Hot/cold diskgroups?

© 2013 Pythian 30

Working with Oracle Support • Oracle Support manages all Exadata components:

database, hardware, OS, network

• Even hard disk failures must go through Oracle Support for triage and troubleshooting

• Dedicated engineered systems team handles front-line support

• Much better experience than ordinary support

• Ordinary support-management skills are doubly important though: creating SRs properly, escalations, etc

© 2013 Pythian 31

Organizational support models • Exadata blurs the lines

• Oracle support procedures are new for admins used to dealing with other vendors

• Generally most Exadata work is DBAs

• Most common support model is to create a team with mostly DBAs to manage entire Exadata stack

• Organizational reasons may require a more traditional model though

• For you as a DBA: reach out

© 2013 Pythian 33

But overall

• As a DBA, you’ll still be doing much the same

work as you’re used to on Exadata

• Performance tuning, backups, code

deployments, disaster recovery all exist in

Exadata land

© 2013 Pythian 34

Gaining Exadata skills • Not many of us have datacenters in our

basements to try out new technologies

• Learn technologies like RAC, and “real” ASM with redundancy, direct-attach disk

• But there are plenty of resources: – Exadata courses: Oracle university workshop,

and others

– Exadata books, particularly Expert Oracle Exadata

– Conference presentations – even if you can’t attend, try and download the slides

© 2013 Pythian 35

A word about the IOUG • The IOUG is bring you today’s user group day

• By Oracle users, for Oracle users

• Collaborate 2014: April 7-11, Las Vegas

• Master class events this fall: Milwaukee, Rochester, Chicago,

Atlanta

• IOUG Kiosk: Moscone West, 2nd floor

© 2013 Pythian 36

Thank you and Q&A

© 2013 Pythian 37

fielding@pythian.com

1-877-PYTHIAN

pythian.com/blog

http://is.gd/PythianFacebook

@mfild @pythian

http://linkedin.com/company/pythian

My upcoming sessions:

Database Private Clouds

with Oracle 12c

Tuesday, high noon

Moscone South 236

It’s a solid state world:

How Exadata X3 leverages

flash storage

Wednesday 3:30pm

Westin SF Metropolitan I