Instant Cloning: Boosting Application Development

Kyle Haileykyle@delphix.com

http://kylehailey.com

60% Projects Over Schedule

85% delayed waiting for data

CIO Magazine Survey:

Data is the problem

Get the right dataTo the right peopleAt the right time

Problem in IT

The Phoenix Project

One of the most powerful things that IT can do is get environments to development and QA when they need it

• IT bottlenecks• Setting Priorities• Company Goals• Defining Metrics• Fast Iterations

IT version of

“The Goal”

by E. Goldratt

Production

Development

Business Intelligence

BackupData Guard

QA

UAT

Tape

Sandbox

Forensics

Analytics

Reporting

Data Flood

Part II : Solutions: Thin cloning to Databases Virtualization

Part I : Problem: Copying Databases

Part III : Project Acceleration with Virtualization

Database Cloning Challenge

If you can’t satisfy the business demands then your process is broken.

Problem

Developers

QA and UAT

Reports

First copy

Production

• CERN -  European Organization for Nuclear Research

• 145 TB database• 75 TB growth each year• Dozens of developers want copies.

Tradeoff: Speed, Quality, Cost

What We’ve Seen

1. Inefficient QA: Higher costs of QA2. QA Delays : Greater re-work of code3. Sharing DB Environments : Bottlenecks4. Using DB Subsets: More bugs in Prod5. Slow Environment Builds: Delays

“if you can't measure it you can’t manage it”

1. Inefficient QA: Long Build times

Build Time

QA Test

96% of QA time was building environment$.04/$1.00 actual testing vs. setup

Build

2. QA Delays: bugs found late require more code re-work

Build QA Env QA Build QA Env QA

Sprint 1 Sprint 2 Sprint 3

Bug CodeX

1 2 3 4 5 6 70

10203040506070

Delay in Fixing the bug

Cost ToCorrect

Software Engineering Economics – Barry Boehm (1981)

3. Full Copy Shared : Bottlenecks

Frustration Waiting

Old Unrepresentative Data

http://martinfowler.com/bliki/NoDBA.html

4. Subsets : cause bugs

Production4. Subsets : cause bugs

Classic problem is that queries that run fast on subsets hit the wall in production.

Developers are unable to test against all data

The Production ‘Wall’

5. Slow Environment Builds: 3-6 Months to Deliver Data

Management

DBA

System Admin

Storage Admin

Developers Submit Request

Disk Capacity?

Approve Request $$ (2 Weeks)

Approve Request $$

(1 Week)

RequestAdditional Storage?

ProvisionCapacity

File SystemConfigured?

Configure LUNS & Build File System

Coordinate Replication w/ Infrastructure

Re-Parameterize & Configure DB

Mount Recovery DB to

Specific PIT

Begin Work

Approve Request $$ (2 Weeks)

(3 Days)

(3 Days)

(2 Days)

(3 Days)

(3 Days)

…….1-2 Weeks of Approvals, Delays, and Provisioning……

16

5. Slow Environment Builds: culture of no

DBA Developer

Never enough environments

bottlenecks

What We’ve Seen

1. Inefficient QA: Higher costs2. QA Delays : Increased re-work3. Sharing DB : Bottlenecks4. Subset DB : Bugs5. Slow Environment Builds: Delays

Clone 1 Clone 3Clone 2

99% of blocks are identical

Clone 1 Clone 2 Clone 3

Thin Clone

I. EMC II. Netapp III. ZFS

2. Thin Cloning

Database Luns

Production FilerTarget A

Target B

Target C

snapshotclones

1. Put database in hot backup 2. Take Snapshot (limits EMC 16, Netapp ,VNX 255, ZFS unlimted)3. Take database out of hot backup4. Clone Snapshot (Netapp , ZFS, EMC VNX)5. Export Clone, Mount on target host

InstanceInstance

InstanceInstance

InstanceInstance

InstanceInstance

Instance

Source

Database LUNs

snapshotclonesProduction Filer

Development Filer

Problem: How do you get data off Production?

Instance

Target A

Target B

Target C

InstanceInstance

InstanceInstance

InstanceInstance

Instance

Three Core Parts

Production

File System

Instance

DevelopmentStorage

21 3

Copy & sync

Snapshot & Clone

Mount, recover, rename

Instance

Netapp EM snap clone

Three Core Parts

Production

File System

Instance

DevelopmentStorage

1

Copy & sync

Instance

Snap Manager

SnapManagerRepository

Protection Manager

Snap Drive

Snap Manager

Snap Mirror

Flex Clone

RMANRepository

Production

Development

DBA

Storage Admin

1 tr-3761.pdf

Netapp

NetApp Filer - DevelopmentNetApp Filer - Production

Database Luns

Snap mirror

Snapshot Manager for Oracle

Flexclone

Repository Database

SnapDrive

Protection Manage

Production

Development

1NetappTarget A

Target B

Target C

InstanceInstance

InstanceInstance

InstanceInstance

Instance

Three Core Parts

Production

File System

Instance

DevelopmentStorage

Instance

3

Mount, recover, rename

3

Oracle EM 12c Snap Clone3

EM 12c

Agents instance

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned• Set up a profile – a source database that can be used for thin cloning• Set up a service template – init.ora values

Test Master

Instance

Source

? instance

CloneLinux

ZFS orNetApp

PoolProfile

ZoneTemplate

Production Development QA UAT

Instance Instance Instance InstanceInstance Instance Instance Instance

Database

File systemFile system

Database

File systemFile system

Database

File system

Database

File systemFile systemFile systemFile system

Where we are

Where we want to be

Database

File system

Production

Instance

Database

Development

Instance

Database

QA

Instance

Database

UAT

Instance

Snapshots

Instance Instance Instance Instance

EM 12c: Snap Clone

Production Development

Flexclone Flexclone

Netapp Snap Manager for Oracle

Thin Cloning

3. Database Virtualization

Three Physical CopiesThree Virtual Copies

Data Virtualization Appliance

Install Delphix on x86 hardware

Intel hardware

Allocate Any Storage to Delphix

Allocate StorageAny type

Pure Storage + DelphixBetter Performance for 1/10 the cost

One time backup of source database

Database

Production

File system

File systemUpcoming

Supports

InstanceInstanceInstance

DxFS (Delphix) Compress Data

Database

Production

Data is compressed typically 1/3 size

File system

InstanceInstanceInstance

Incremental forever change collection

Database

Production

File system

Changes

• Collected incrementally forever• Old data purged

File system Time Window

Production

InstanceInstanceInstance

Source Full Copy Source backup from SCN 1

Snapshot 1Snapshot 2

Snapshot 1Snapshot 2

Backup from SCN

Snapshot 1Snapshot 2

Snapshot 3

DropSnapshot 1

Snapshot 2Snapshot 3

Cloning

Database

Production

Instance

File systemFile system Time Window

Database

InstanceInstance

InstanceInstance

Typical Architecture

Production Development QA UAT

Instance Instance Instance InstanceInstance Instance Instance Instance

Database

File systemFile system

Database

File systemFile system

Database

File system

Database

File systemFile systemFile systemFile system

With Delphix

Database

Production

Instance

Database

Development

Instance

Database

QA

Instance

Database

UAT

InstanceInstance Instance Instance Instance

File system

Three Core Parts

Production

File System

Instance

DevelopmentStorage

21 3

Copy Sync SnapshotsTime FlowPurge

Clone (snapshot)CompressShare CacheStorage Agnostic

Mount, recover, renameSelf Service, Roles & Security Rollback & Refresh Branch & Tag

Instance

Overview of Database Virtualization

1. Fast, Fresh, Full2. Free3. Branching4. Federated5. Self Serve

Fast, Fresh, Full

Instance

Time Window

Instance

Development VDB

Source

Free

gif by Steve Karam

Instance

Instance

Instance

Instance

Source

Branching

Instance Instance

Instance

Source

Source Dev

QA branched from Dev

Federated Cloning

Federated

Instance

Instance

Instance

Instance

Source1

Source2

Source1

“I looked like a hero”Tony Young, CIO Informatica

Self Service

Use Cases

1. Development Acceleration

2. Quality

3. BI

DevOps

DevOps With Delphix

1. Efficient QA: Low cost, high utilization2. Quick QA : Fast Bug Fix3. Every Dev gets DB: Parallelized Dev4. Full DB : Less Bugs5. Fast Builds: Culture of Yes

1. Efficient QA: Lower cost

Build Time

QA Test

1% of QA time was building environment$.99/$1.00 actual testing vs. setup

Build Time

QA Test

Build

Rapid QA via Branching

2. QA Immediate: bugs found fast and fixed

Sprint 1 Sprint 2 Sprint 3

Bug CodeX

QA QA

Build QA Env QA Build QA Env Q

A

Sprint 1 Sprint 2 Sprint 3

Bug CodeX

3. Private Copies: Parallelize

gif by Steve Karam

4. Full Size DB : Eliminate bugs

Production

5. Self Service: Fast, Efficient. Culture of Yes!

Management

DBA

System Admin

Storage Admin

Developers Submit Request

Disk Capacity?

Approve Request $$ (2 Weeks)

Approve Request $$

(1 Week)

RequestAdditional Storage?

ProvisionCapacity

File SystemConfigured?

Configure LUNS & Build File System

Coordinate Replication w/ Infrastructure

Re-Parameterize & Configure DB

Mount Recovery DB to

Specific PIT

Begin Work

Approve Request $$ (2 Weeks)

(3 Days)

(3 Days)

(2 Days)

(3 Days)

(3 Days)

…….1-2 Weeks of Approvals, Delays, and Provisioning……

Quality

1. Forensics2. Testing 3. Recovery

1. Forensics: Investigate Production Bugs

Instance

Time Window

Instance

Development

Anomaly on ProdPossible code bugAt noon yesterday

Spin up VDB of Prod as it was during anomaly

2. Testing : Rewind for patch and QA testing

Instance

Time Window

Instance

Development

Time Window

Prod

2. Testing: A/B

Instance

Time Window

Instance

Instance

• Keep tests for compare• Production vs Virtual

– invisible index on Prod– Creating index on virtual

• Flashback vs Virtual

Test A with Index 1

Test B with Index 2

3. Recovery: Surgical recover of Production

Instance Instance

Development

Time Window

Spin VDB up Before drop

Problem on ProdDropped Table Accidently

Source

Time Window

3. Recovery Surgical or Full Recovery on VDB

Instance

Instance

Dev1 VDB

Time Window

Dev1 VDB

InstanceSource

Source

Dev2 VDB Branched

3. Recovery: Virtual to Physical

Instance Instance

VDB

Source

Time Window

Spin VDB up Before drop

Corruption

3. Recovery

Business Intelligence

ETL and Refresh Windows

1pm 10pm 8am noon

ETL and DW refreshes taking longer

1pm 10pm 8am noon20112012201320142015

ETL and Refresh Windows

20112012201320142015

1pm 10pm 8am noon

10pm 8am noon 9pm

6am 8am 10pm

ETL and DW Refreshes

Instance

Prod

Instance

DW & BI

Data Guard – requires full refresh if usedActive Data Guard – read only, most reports don’t work

Fast Refreshes

• Collect only Changes• Refresh in minutes

Instance Instance

DW

Prod

Instance

BI

ETL24x7

Temporal Data

BI

a) Fast refreshes

b) Temporal queries

c) Confidence testing

Review: Use Cases

1. Development Accelerationa) Full, Fresh, Fast , Self Serveb) QA Branchingc) Federated

2. Qualitya) Forensicsb) Testing : A/B, upgrade, patchc) Recovery: logical, physical

3. BIa) Fast refreshb) Temporal Datac) Confidence testing

Oracle 12c

80MB buffer cache ?

200GBCache

5000

Tnxs

/ m

inLa

tenc

y

300 ms

1 5 10 20 30 60 100 200

with

1 5 10 20 30 60 100 200Users

8000

Tnxs

/ m

inLa

tenc

y

600 ms

1 5 10 20 30 60 100 200Users

1 5 10 20 30 60 100 200

$1,000,000

$6,000

About Delphix• Founded in 2008, launched in 2010• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))• Based in Silicon Valley, Global Operations

Fortune #1 Walmart #1 social: Facebook #1 US Bank: Wells Fargo #1 Networking : Cisco#1 cable provider: Comcast #1 auction site: Ebay #1 insurance: New York Life #1 chip manufacture: Intel

Devv2.6 v2.6v2.6

QA UAT

v2.6

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

Devv2.6 v2.6v2.6

QA UAT

v2.6Production

v2.6 v2.6v2.6v2.7

v2.6 v2.6v2.6v2.8

Source Control for the database data

v2.6v2.6 v2.6v2.6

v2.6v2.7 v2.6v2.7

v2.6v2.8 v2.6v2.8

DevProd

2.6

Dev

QA

Prod

2.6

Dev

QA

UAT

Prod

2.6

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Dev

QA

UAT

Prod

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT2.8

Data Control = Source Control for the Database

Dev

QA

UAT

Dev

QA

UAT

2.6

2.7

Dev

QA

UAT

2.8

Data Control = Source Control for the Database

Production Time Flow

Database Virtualization

$40M

$75M

$850M

$27,000M

Storage

IT

Develop

Business

Good, Cheap, Fast : choose two

Fast

GoodCheap

FS vs. ZFS

• FS per Volume

• FS limited bandwidth

• Storage stranded

• Many FS in a pool

• Grow automatically

• All bandwidth

Storage PoolVolume

FS

Volume

FS

Volume

FS ZFS ZFS ZFS

# of database copies

IncreasedRevenueFor Company

Three Core Parts

Production

Instance

Time Window

Instance

Self Service

Development

21 3

Source Syncing Storage (DxFS)

Database going Global

Globalization Reduces Windows

20112012201320142015

1pm 10pm 8am noon

10pm 8am noon 9pm

6am 8am 10pm

I. Clonedb Oracle II. EMC

• Copy on first write (COFW)III. Netapp

• write anywhere file system (WAFL)• & EMC VNX redirect on write (ROW)

IV. ZFS

2. Thin Cloning

RMAN backup

dNFSsparse fileI. clonedb

RMAN backup

dNFSsparse fileI. clonedb

CloneDB

1. dNFS 11.2.0.2+– cd $ORACLE_HOME/rdbms/lib– make -f ins_rdbms.mk dnfs_on

2. Clonedb.pl initSOURCE.ora output.sql– MASTER_COPY_DIR="/rman_backup”– CLONE_FILE_CREATE_DEST="/nfs_mount”– CLONEDB_NAME="clone"

3. sqlplus / as sysdba @output.sql– startup nomount PFILE=initclone.ora – Create control file backup location– dbms_dnfs.clonedb_renamefile ('/backup/file.dbf' , '/clone/file.dbf');– alter database open resetlogs;

Tim Hallwww.oracle-base.com/articles/11g/clonedb-11gr2.php

I. Clonedb Oracle II. EMC

• Copy on first write (COFW)III. Netapp

• write anywhere file system (WAFL)• & EMC VNX redirect on write (ROW)

IV. ZFS

Thin Cloning

D

ActiveFile

SystemSnapshot

CBA

II. EMC Copy on Write

D

ActiveFile

SystemSnapshot

DCBA

Write penalty (read and two writes)Limit 16 snapshotsNo Branching (snapshots of snapshots)

II. EMC Copy on Write

I. Clonedb Oracle II. EMC

• Copy on first write (COFW)III. Netapp

• write anywhere file system (WAFL)• & EMC VNX redirect on write (ROW)

IV. ZFS

Data Blocks

root

III. Netapp (WAFL) and EMC (ROW)

• 255 snapshots• Branching possible

I. Clonedb Oracle II. EMC

• Copy on first write (COFW)III. Netapp

• write anywhere file system (WAFL)• EMC redirect on write (ROW)

IV. ZFS

Snapshot rootLive root

ZilIntent Log

IV. ZFS Allocate on Write

Unlimited Instantaneous SnapshotsUnlimited Instantaneous ClonesBranching easy and unlimited

Oracle EM 12c Snap Clone3

Test Master

Instance

Source

? instance

Create a “Test Master” databaseOn ZFSSA or Netapp

ZFSSA or Netapp

Oracle EM 12c Snap Clone3

EM 12c

• Register Netapp or ZFS with Storage Credentials

Test Master

Instance

Source

? instance

ZFS orNetApp

Install EM 12cInstall Snap Clone option

Oracle EM 12c Snap Clone3

EM 12c

Agents

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage.

Test Master

Instance

Source

? instance

Linux

ZFS orNetApp

Oracle EM 12c Snap Clone3

EM 12c

Agents

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage• Register test master database• .

Test Master

Instance

Source

? instance

Linux

ZFS orNetApp

Oracle EM 12c Snap Clone3

EM 12c

Agents

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage• Register test master database• Enable Snap Clone for the test master database

• .

Test Master

Instance

Source

? instance

Linux

ZFS orNetApp

Enable Snap Clone

Oracle EM 12c Snap Clone3

EM 12c

Agents

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones

Test Master

Instance

Source

? instance

Linux

ZFS orNetApp

Zone

CPU limitsMemory limits

Oracle EM 12c Snap Clone3

EM 12c

Agents instance

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned

Test Master

Instance

Source

? instance

CloneLinux

ZFS orNetApp

Pool

Zone

Oracle EM 12c Snap Clone3

EM 12c

Agents instance

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned• Set up a profile – a source database that can be used for thin cloning

Test Master

Instance

Source

? instance

CloneLinux

ZFS orNetApp

PoolProfile

Zone

Oracle EM 12c Snap Clone3

EM 12c

Agents instance

• Register Netapp or ZFS with Storage Credentials• Install agents on a LINUX machine to manage the Netapp or ZFS storage. • Register test master database• Enable Snap Clone for the test master database• Set up a zone – set max CPU and Memory and the roles that can see these zones• Set up a pool – a pool is a set of machines where databases can be provisioned• Set up a profile – a source database that can be used for thin cloning• Set up a service template – init.ora values

Test Master

Instance

Source

? instance

CloneLinux

ZFS orNetApp

PoolProfile

ZoneTemplate

Choose your virtualization Layer:• Delphix and Oracle SMU

SMU

ZFS Storage Appliance

Oracle 12c SMUOracle Snap Management Utility for ZFS Appliance

• Requires ZFS Appliance• Supports Linux , Solaris 10+, Windows

2008+• GUI

– snapshot source databases – provision virtual databases

over 10 times

perhaps the single largest storage consolidation opportunity history“

Original Snapshot Snapshot Day 2