Database Virtualizationand instant cloning

Kyle Haileyhttp://dboptimizer.com

Who is Kyle Hailey

1990 Oracle– 90 support– 92 Ported v6– 93 France– 95 Benchmarking – 98 ST Real World Performance

2000 Dot.Com 2001 Quest 2002 Oracle OEM 10gSuccess!

First successful OEM design

Who is Kyle Hailey

1990 Oracle– 90 support– 92 Ported v6– 93 France– 95 Benchmarking – 98 ST Real World Performance

2000 Dot.Com 2001 Quest a 2002 Oracle OEM 10g 2005 Embarcadero

– DB Optimizer

Who is Kyle Hailey

• 1990 Oracle 90 support 92 Ported v6 93 France 95 Benchmarking 98 ST Real World Performance

• 2000 Dot.Com• 2001 Quest • 2002 Oracle OEM 10g• 2005 Embarcadero

DB Optimizer

• 2010 Delphix

When not being a Geek- Have a little 4 year old boy who takes up all my time NoCOUG board

IOUG Liaison

Database Cloning Challenge

If you can’t satisfy the business demands then your process is broken.

Two Parts

I. Cloning TechnologyII. Accelerate your business

Part I : Cloning Technology

3. Virtual2. Thin Clone1. Physical

=

database

1. Physical Cloning

Problem

Developers

QA and UAT

Reports

First copy

Production

• CERN -  European Organization for Nuclear Research

• 145 TB database• 75 TB growth each year• Dozens of developers want copies.

workaroundsDevelopers

QA and UAT

ReportsShared

Sub set copy

Production

Many copies

Physical Clones

Database SubsetsShared Databases

Subsets

ProductionThe Production ‘Wall’

Classic problem is that queries that run fast on subsets hit the wall in production.

Developers are unable to test against all data

Shared Full

Shared access = Poor Productivity

Developerfrustration

Databases become old and unrepresentative.

Never enough environments

Average customer makes 12 copies of production- Charles Garry, Database Product Manager Oracle

Physical CopiesTime consuming

Copy RMAN backup, archive logs, copy data over, recover

Coordinate System, Storage ,Database ,Network, manager

Space consuming 20 report DBs x 40 TB = 800TB

=> bottlenecks

“Culture of NO”

Setup Develop

Setup Develop

QA

$40M

$75M

$850M

$27,000M

Storage

IT

Develop

Business

“Culture of NO”

ERP Project Failures 2011

• NYC CityTime : delays $63 M => $760 M • Montclair Uni: delays sues PeopleSoft• Idaho : delays ERP cost millions

Standish : IT Project Failure Rate

1994 1996 1998 2000 2002 2004 2009

31% 40% 28% 23% 15% 18% 24%

★http://www.galorath.com/wp/software-project-failure-costs-billions-better-estimation-planning-can-help.php*http://www.pcworld.com/article/246647/10_biggest_erp_software_failures_of_2011.html

NoDBAhttp://martinfowler.com/bliki/NoDBA.html

Clone 1 Clone 3

99% of blocks are Identical

Clone 2

2. Thin Cloning

Clone 1 Clone 2 Clone 3

Thin Provision

2. Thin Cloning

Cornerstone Technology: File System Snapshots

Thin Cloning

1. Put database in hot backup2. Snapshot DB Files @ point in time3. Take database out of hot backup

Copy to non-production Snapshot again Export over NFS or FC to Host

4. Recovery Database

I. ClonedbII. Copy on Write Snapshots

a) EMC (COFW)b) EMC w/ SRDF,Recover Point

III. Allocate on Writea) ZFSb) EMC VNX (ROW)c) Netapp (WAFL)

2. Thin Provision Cloning

RMAN backup

dNFSsparse file

I. clonedb

RMAN backup

dNFSsparse file

I. clonedb

I. CloneDB1. dNFS 11.2.0.2+

– cd $ORACLE_HOME/rdbms/lib– make -f ins_rdbms.mk dnfs_on

2. Clonedb.pl initSOURCE.ora output.sql– MASTER_COPY_DIR="/rman_backup”– CLONE_FILE_CREATE_DEST="/nfs_mount”– CLONEDB_NAME="clone"

3. sqlplus / as sysdba @output.sql– startup nomount PFILE=initclone.ora – Create control file backup location– dbms_dnfs.clonedb_renamefile ('/backup/file.dbf' , '/clone/file.dbf');– alter database open resetlogs;Tim Hall

www.oracle-base.com/articles/11g/clonedb-11gr2.php

I. CloneDB

RMAN

physical

Target A NFS Server

Clone 1

Clone 2

Read only Clone 3

Target B

Clone 1

Clone 2

Clone 3

I. clonedbII. Copy on First Write (COWF)

a) EMC BCVb) EMC w/ SRDF,Recover Point

III. Allocate on Writea) ZFSb) EMC VNX (ROW)c) Netapp (WAFL) ZFS

2. Thin Provision Cloning

D

ActiveFile

SystemSnapshot

CBA

File System constant - EMC

II. Copy on Write a) EMC snapshots

D

ActiveFile

SystemSnapshot

DCBA

Changes written in place - EMCChanges to new area - VMware

II. Copy on Write a) EMC snapshots

Non-Prod FilerEMC Filer

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

SnapshotsMax 16No BranchingMultiple Copies

a) EMC BCVIII. Allocate on Write

BCV Day 1

snapshot

Snapshotor break

Day 2

Timefinder – snapshots multiple LUNs

SnapshotsWrite Penalty

Non-Prod FilerEMC Filer

ProductionDatabase

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

File system level

SRDF Symetrix

Snapshot day 1

Snapshot day 1

Snapshot day 2

Snapshot day 2

II. Copy on Write b) EMC SRDF

Recover Point Appliance

SnapshotsMax 16No Branching

I. clonedbII. Copy on Write (COFW)

a) EMC BCVb) EMC SRDF or Recover Point

III. Allocate on Writea) ZFSb) EMC VNX (ROW)c) Netapp (WAFL)

2. Thin Provision Cloning

Data Blocks

root

a. ZFSb.EMC VNX (ROW)c. Netapp(WAFL)

III. Allocate on Write

Jonathan Lewis © 2013

ZFS

a b c d e f g h i

Data

Metadata128 entries per block

a b c d e f g h i

"Updates"

b' c'

Data only becomes visible when root block is written

Jonathan Lewis © 2013

Aging data

Blocks that can be freed when we don’t want the older snapshot any more

a d e f g h i

b' c'

b c

Jonathan Lewis © 2013

a d e f g h i

New State

b' c'

Jonathan Lewis © 2013

Cloning

b' c'a d e f g h i

My clone(filesystem)

Your clone(filesystem)

Jonathan Lewis © 2013

a) ZFS

• 1 disk = 1 filesystem• ~1990: N disks = 1 FS• 2001: ZFS starts• 2005: ZFS ships• 2008: ZFS storage appliance ships• 2010: Delphix heads ZFS open source

• Dephix also produces own file system DxFS originally based on ZFS

III. Allocate on Write

FS vs. ZFS• FS per Volume

• FS limited bandwidth

• Storage stranded

• Many FS in a pool

• Grow automatically

• All bandwidth

Storage PoolVolume

FS

Volume

FS

Volume

FS ZFS ZFS ZFS

3a) ZFSIII. Allocate on Write

a) ZFS

Snapshot rootLive root

Delphix Proprietary and Confidential

ZilIntent Log

III. Allocate on Write

ZFS Appliance + RMAN

1. ZFS Appliance– Project db_master– Project db_clone – 4 file systems: datafile, redo, archive, alerts

2. Source Database– NFS Mount ZFS Appliance – Backup with RMAN

3. ZFS Appliance• Select db_master• Snapshots• Then each filesystem on db_master clone it onto db_clone

4. Target Host– NFS Mount db_clone– recover clone

cloning-solution-353626.pdf

a) ZFS III. Allocate on Write

Oracle ZFS Appliance + RMAN

1. physical

ZFS Storage Appliance

RMAN Copyto NFS mount

Target A

Clone 1

Clone 1

Snapshot

NFS

a) ZFS III. Allocate on Write

cloning-solution-353626.pdf

RMANCopytoday

RMANCopy

Yesterday

NetApp FilerNetApp Filer

Database Luns

snapshot

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

snapshot

clones

b) NetappIII. Allocate on Write

NetApp FilerNetApp Filer

ProductionDatabase

Database Luns

snapshot

Target A

Target B

Clone 1

Clone 2

snapshot

snapshot

snapshot

b) NetappIII. Allocate on Write

NetApp FilerNetApp Filer

Physical Database

Database Luns

snapshot

Target A

Target B

Clone 1

Clone 2

snapshot

snapshot

snapshot

b) NetappIII. Allocate on Write

Netapp tr-3761.pdf

Netapp

Snap Manager

SnapManagerRepository

Protection Manager

Snap Drive

Snap Manager

Snap Mirror

Flex Clone

RMANRepository

Production

Development

DBA

Storage Admin

NetApp Filer - DevelopmentNetApp Filer - Production

Database Luns

Target A

Target B

Target C

Clone 1

Clone 2

Clone 3

Clone 4

Snap mirror

Snapshot Manager for Oracle

Flexclone

Repository Database

Netapp

SnapDrive

Protection Manage

Production

Development

I. clonedbII. Copy on First Write (COFW)

a) EMC BCV b) EMC SRDF or Recover Point

III. Allocate on Writea) ZFS b) EMC VNX (ROW)c) Netapp (WAFL)

2. Thin Provision Cloning

• Clonedb • EMC BCV• ZFS

• EMC SRDF• EMC VNX• Netapp*

2. Review: Thin Provision Cloning

*Netapp is best solution and complicated

2. Thin Cloning

2. Thin Provision Cloning

3. Database Virtualization

Virtualization Layer

Virtualization

SMU

DatabaseVirtualizationAppliance(DVA)

3 Physical Copies 3 Virtual Databases

One time backup of source database

Database

Production

Instance

File system

RMAN APIs

ZFS (Oracle) and DxFS (Delphix) Compress Data

Database

Production

Instance

File system

Data is compressed typically 1/3 size

Incremental forever change collection

Database

Production

Instance

File system

Changes are collected automatically foreverData older than retention widow freed

Typical Architecture

Database

File system

Production

Instance

Database

File system

Development

Instance

Database

File system

QA

Instance

Database

UAT

Instance

File system

Database

File system

Production

Instance

Database

Development

Instance

Database

QA

Instance

Database

UAT

Instance

Virtualization Layer

x86 hardware

Allocate StorageAny type

SMU

Choose your virtualization Layer:• Delphix and Oracle SMU automated out of box• Netapp and EMC SRDF/Recover Point require massive scripting• Delphix share blocks in memory

ZFS Storage Appliance

a) Oracle 12c SMUOracle Snap Management Utility for ZFS Appliance

• Requires ZFS Appliance• Supports Linux , Solaris 10+, Windows 2008+• GUI

– snapshot source databases – provision virtual databases

3. Database Virtualization

Use Cases

1. Development2. Backup3. Reporting

1. Development Acceleration

1: Development Acceleration

a) Developer each get a copy– Fast, fresh, full, frequent– Self service

b) Branching *c) Federated *

* Delphix only

Source

FastSource Database

Target HostVirtual

Database

NFS

Fiber

Fiber

RMAN over TCP

No Data Movement

Source

Fresh

Virtual Database

Fiber

Source

Frequent

Virtual Database

Virtual Database

Target Hosts

Virtual Database

Virtual Database

Fiber

Full clones

Self Service

1 b) Rapid QA via Branching

dSource

1 b) Branching

Developer VDB

QA VDB

Devv2.6 v2.6v2.6

QA UAT

v2.6

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

Devv2.6 v2.6v2.6

QA UAT

v2.6Production

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

Source Control for the database data

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

DevProd

2.6

Dev

QA

Prod

2.6

Dev

QA

UAT

Prod

2.6

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Data Control = Source Control for the Database

Dev

QA

UAT

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

2.8

Data Control = Source Control for the Database

Production Time Flow

1 c) Federated Cloning

Source2

Source3

Source1

1 c) Federated sources

Virtual Database

Virtual Database

Virtual Database

Virtual Database

“I looked like a hero”Tony Young, CIO Informatica

1. Review Development Use Cases

a) Developer each get a copy

b) QA (Branching) *

c) Federated *

* Delphix only

2. Quality

a) Forensicsb) A/B testingc) Recovery

Source

2 a) Forensic Analysis

Virtual Database

Source

2 b) Upgrades, Patches, RAT, A/B

Virtual Database

• Production vs Virtual– invisible index on Prod– Creating index on virtual

• Flashback vs Virtual• Keep tests for compare

2 b) Upgrades, Patches, RAT, A/B

2 c) Recovery

Source

2 c) Logical Recovery Production

Virtual Database

Source

2 c) Logical Recovery Development

Virtual Database

VDB rolled back

Source

Recovery

VDB

V2P

2. Review Quality

a) Forensics

b) A/B testing

c) Recovery : Logical and physical

3: Analyticsa) Fast refreshes

b) Temporal queries

c) Confidence testing

Fast Refreshes• Refresh in minutes• Without data movement• Faster , cheaper

Physical Virtual

Temporal Data

3: reportinga) Fast refreshes

b) Temporal queries

c) Confidence testing

Review: Use Cases

1. Developmenta) Full, Fresh, Fast , Self Serveb) QA Branchingc) Federated

2. Qualitya) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical

3. Reportinga) Fast refreshb) Temporal Datac) Confidence testing

Typical Architecture

Database

File system

Production

Instance

Database

File system

Development

Instance

Database

File system

QA

Instance

Database

UAT

Instance

File system

Database

File system

Production

Instance

Database

Development

Instance

Database

QA

Instance

Database

UAT

Instance

Delphix & Oracle SMU

EM 12c: Snap Clone DBaaS

Handles Clone Provisioning

Netapp can handle source database change synchronization with development storage array

EM 12c clone provisioning not currently compatible with Netapp synchronization technology

Netapp EM 12c w/ Netapp DVA

Thin clone limited (1) limited (1) yes (1) limited # snapshots

Source linking limited (2) no yes (2) no continuous pull

Provision VDB no limited (3) yes (3) limited to snapshots

User ready no limited (4) yes (4) no branching

Cloud ready no no yes

DVA = database virtualization appliance

over 10 times

"perhaps the single largest storage consolidation opportunity history“

Oracle 12c

80MB buffer cache ?

200GBCache

5000

Tnxs

/ m

inLa

tenc

y

300 ms

1 5 10 20 30 60 100 200

with

1 5 10 20 30 60 100 200Users

8000

Tnxs

/ m

inLa

tenc

y

600 ms

1 5 10 20 30 60 100 200Users

1 5 10 20 30 60 100 200

Memory Prices

• EMC sells $1000/GB• X86 memory $30/1GB

• TB RAM on a x86 costs around $32,000 • TB RAM on a VMAX 40K costs around $1,000,000

$1,000,000

$32,000

$6,000

Prices to cache 200GB database in memory either on• Target hosts• EMC array• Delphix Appliance

Database Virtualization