Thank you for joining us, the webinar will start at:10:00 Pacific / 11:00 Central / 13:00 East / 18:00 UK Time

Taking Database Development to the 21st Century

Presenters

Kyle Hailey @kylehhailey• Technical Evangelist at Delphix• Oracle ACE, member of the OakTable Network

Uri Margalit @UriMargalit • Director, Product Management• Presenter at world-wide conferences: ODTUG, ilOUG,

Perforce Merge, etc…

Before we start

• You will be on mute for the duration of the event

• We are now talking so please type a message in the Questions box in the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)

• There will be a Q+A session at the end but please feel free to type your questions in the Questions box in the Control Panel in advance

• A recording of the full webinar will be put up online

About Delphix

• Founded in 2008, launched in 2010• CEO Jedidiah Yueh (founder of Avamar: >$1B revenue))• Based in Silicon Valley, Global Operations• 20% of Fortune 500

About DBmaestro

• Founded in 2008• Headed by Yariv Tabac and Yaniv Yehuda• Headquartered in Israel, Global Operations

Taking Database Development to the 21st Century

Kyle Hailey & Uri Margalit

Risks

80% of unplanned downtime is due to Change

50% More than

of this, half is due to human errors

Technology required

Reducing Risk

Smaller changes Agile Automation no human error, more with less DevOps synergy

Source Control : Code

• GitHub• SVN• Perforce• TFS• RTC• VSS

How do you source control the database?

Database Challenge

• Crucial • Must be in sync with application changes• Objects not file system

– Schema Structure– PL/SQL Code– Central resource

• Lacks Version Control• Relevant data for the task

Database Requirements

• How can we – Enforced version control changes?– Manage content?– Branch versions?– Merge branches?– Have right data for the task?

Data is the constraint

60% Projects Over Schedule

85% delayed waiting for data

Data is the Constraint

CIO Magazine Survey:

only getting worse

Why is Data the Constraint ?

1. Development: Bottlenecks & bugs2. QA: Expensive, single threaded, slow 3. Provisioning: Delays

Frustration Waiting

1. Development: Bottlenecks & bugs

1. Development: Bottlenecks & bugs

1. Development: Bottlenecks & bugs: subsets cause bugs

96% of QA time was building environment$.04/$1.00 actual testing vs. setup

QA Build QAQA Build QA

BugX

1 2 3 4 5 6 70

10203040506070

Delay in Fixing the bug

Cost ToCorrect

Software Engineering Economics – Barry Boehm (1981)

2. QA: slow, expensive, single threaded

Management

DBA

System Admin

Storage Admin

Developers Submit Request

Disk Capacity?

Approve Request $$ (2 Weeks)

Approve Request $$

(1 Week)

RequestAdditional Storage?

ProvisionCapacity

File SystemConfigured?

Configure LUNS & Build File System

Coordinate Replication w/ Infrastructure

Re-Parameterize & Configure DB

Mount Recovery DB to

Specific PIT

Begin Work

Approve Request $$ (2 Weeks)

(3 Days)

(3 Days)

(2 Days)

(3 Days)

(3 Days)

…….1-2 Weeks of Approvals, Delays, and Provisioning……

Developer Asks for DB Get Access

Manager approves

DBA Request system

Setup DB

System Admin

Requeststorage

Setup machine

Storage Admin

Allocate storage (take snapshot)

3. Delays

3. Delays : culture of no

Three Physical Copies Three Virtual Copies

Delphix - is software

• Installs as VM• Any Storage• Supports

- - - - - Application Data

With Virtual Data

Production

Instance

Database

Dev & QA

Instance

Database

Reporting

Instance

Database

Backup

Instance Instance Instance

Database

InstanceInstance

Database

InstanceInstance

File system

Database

What We’ve Seen With Delphix

1. Development: Parallelized, less bugs2. QA: Low cost, fast bug detection 3. Provisioning: Fast, Culture of Yes

Parallel environments

Instance

Instance

Instance

Instance

Source

1. Development : Parallelize

gif by Steve Karam

1. Development: Eliminate bugs

1. Provisioning: Fast, Efficient, Self Service, Culture of Yes!

1. Development: Version Control

Dev

QA

2.1

Dev

QA

2.2

2.1 2.2

Instance

Prod

DVA

Dev

QA

Instance

Prod

DVATime Flow

• Low Resource• Find bugs Fast

2. QA: Fast and Efficient

What We’ve Seen With Delphix

1. Development: Parallelized, less bugs2. QA: Low cost, fast bug detection 3. Provisioning: Fast, Culture of Yes

But …

How do we manage the changes and make sure the version we deploy is the correct one?

Version Control is a constraint

“

“

It was difficult to track who made a change to a database object and what change they made.(working-around file based version control)Sr. DBA @ Large USA Bank

It took hours to get releases working. some changes were not documented and left out. we actually preferred crashes in integration. It is much worse when something works, but works wrong. in production…(manual and error prone releases)Sr. R&D Manager @ Credit Card company

Version Control is a constraint

We recently had a disaster - the script in the version control was not updated and when executed in production, ran the wrong revision. That cost tens of thousands of $(an out-of-process update to QA that was not properly tracked)Developer @ Algo Trading company

We had multiple releases to production every day. That is one release a week with multiple follow up fixes, and yet more fixes(code overrides, partial versions, wrong versions – all pushed to production)CTO @ Credit Card company

“

“

Version Control is a constraint

We had an incident where a trigger was not correctly implemented during a code release. a trigger from a previous build was used instead which was only detected on Tuesday morning on the first business day after our code release. this was a customer-facing application and made our team look, and feel, bad about the release.

we realized that we needed to bring more discipline and rigor to our database changes.(manual process are hard to repeat over and over without errors)Sr. DBA @ Large USA Bank

“

Root Causes

• Manual script based version control process• Deployment tools unaware of version control• No red-flags…

Two Isolated Processes

Version Control Process (file based)

Development Process

Check-Out Script

Modify Script

Get updated Script from DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

?

??

?

A

A’

Challenges…

Un-coordinated Process

Silos in developmentDeployment delays…

Out-of-Process Changes

Errors in production

Lack of confidence in automation

Code overrides

Different methodologies

Lack of history Missing changes

wrong versions

Scripts & Version Control• Challenges…

– Code-overrides– Working on the wrong revisions – Scripts do not always find their way to the version control

solution– Out of process updates go unnoticed– Hard to locate outdated update scripts

• Playing safe? what we really need: – The actual code of the object– The upgrade script– A roll-back script

Testing Scripts • Single object based scripts

– Hard to test in their entirely– Hard to test due to colliding dependencies– Need to run in a specific order…

• Large multi object based script– Represents the entire update - can deal with

dependencies– Much harder to deal with project scope changes– Hard to mange – a big list of commands

1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.1

X1.11.1.1

Scripts… Build Once Deploy Many

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of Process Change

X

X

X

X

X

?

X

Scripts are Static…• Scripts, unless super sophisticated:

– Unaware of changes made in the target environment– Time passed from their coding to the time they are run– Potentially overriding production hot-fixes & work done in

parallel by another team• Content changes are very hard to manage

– Metadata & lookup content does not practically fit into the VC

– In most cases they are simply not managed

Poll

What level of management you have for you DB development?

1. Enforced version control; Baseline aware automation

2. Version control scripts; Simple automation

3. Version control scripts; Simple Comp & Sync

4. None

Dealing With Challenges…

Coordinated Process Traceability

Start in the Beginning

No Out-of-Process Changes

Impact Analysis

Automation

Task Based Development

Well Defined Processes

Version Control – 1 Enforced Process

Dealing With Challenges… • Integrated Version Control process

– Leverage proven version control best practices • Forcing check in & out for changes• Labels• etc..

– No code-overrides– Always working with the correct revision– All changes are documented

• Integrated Version Control process– Always know who did what, when, why and from where– No out-of-process changes– Supporting structure, code and content

• No time spent on manual coding of the change scripts

1.11.21.31.41.51.61.7 1.11.11.41.7 1.1.11.11.1.1

Build & Deploy On Demand

Int QA Stage Prod

Database Deploy Script

Environment

Re-Base (due to defects)

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.4

1.4 1.7

1.1.1 1.7

Out of Process Change

1.7

*

1.7

Source vs. Target

Action

= No Action

≠ ?

Source vs. Baseline

Target vs. Baseline

Action

= = No Action

≠ = Deploy Changes

= ≠ Protect Target

≠ ≠ Merge Changes

You do not have all of the information

With baselines aware analysis the unknown is now known

Simple Compare & Sync Baseline aware Analysis

Deployment Automation - Safety Net

An Index exists in Target (Prod) and not in Source (QA), what should we do? Drop the Index?

Bonus Points – Task Base Development• Correlate each database change with a change

request – Task ID– Work Item– Trouble Ticket– CR– etc…

• Partial deployments (a feature, a collection of bugs, etc…)

• Scope changes easily synced between code and database

Live Demo

• Clone 2 virtual copies of the Trunk1. Dev12. Dev2

• Make changes & merge them into the Trunk:– Developer1 modifies Dev1– Developer1 merges changes into the Trunk– Developer2 modifies Dev2– Developer2 merges changes into the Trunk

• Rely on enforced changes & automation

Data Version Control

Dev1

Instance

Prod

Dev2

Trunk clones

Trunk

Merge Dev1 to ForkMerge to dev2

Dev2

Dev1Merge to dev1

Merge Dev2 to Fork

Trunk

Merge Dev1 to Fork

Merge Dev2 to Fork

Fork

Fork

Fork

Fork

Q&AKyle Hailey @kylehhaileyDelphix: delphix.com

Uri Margalit @UriMargalit DBmaestro: dbmaestro.com

Thanks!Kyle Hailey @kylehhaileyDelphix: delphix.com

Uri Margalit @UriMargalit DBmaestro: dbmaestro.com