Challenges and Best Practices of Database Continuous Delivery

50

Transcript of Challenges and Best Practices of Database Continuous Delivery

Page 1: Challenges and Best Practices of Database Continuous Delivery
Page 2: Challenges and Best Practices of Database Continuous Delivery

▪You will be on mute for the duration of the event▪Please type a message in the Questions box in

the Control Panel if you can’t hear us (please check your speakers and GoToWebinar audio settings first)

▪If you have questions during the session, please submit them on the Q&A bar on your GoToWebinar dashboard and we will address them at the end

▪A recording of the full webinar will be put up online

Before we Begin

Page 3: Challenges and Best Practices of Database Continuous Delivery

Yaniv YehudaCo-Founder & CTO at DBmaestro

Spent the last years raising awareness about the challenges around database development and deployment, and how to support database Continuous Delivery.

Joined NessPRO as a product manager for the line of SAP complementary products and DevOps solutions for the DB.

Shachar FurmanProduct Manager, Central Systems, NessPRO

About the presenters

Page 4: Challenges and Best Practices of Database Continuous Delivery

Outline

▪ The Adoption of Continuous Delivery▪ The Reasons why the Database is Often Left Behind▪ Single Source of Truth for your Database▪ Best Practices for Including the Database in CD▪ The Bigger Picture – Automation and Stopping the

Line▪ Q&A

Page 5: Challenges and Best Practices of Database Continuous Delivery

NessPRO is Ness Technologies’ products group and the sole representative in

Israel of more than 30 international and Israeli organizations which develop

software products designated for the enterprise market.

About NessPRO

Page 6: Challenges and Best Practices of Database Continuous Delivery

About DBmaestroThe leading provider of DevOps for DatabaseDatabase development and deployment automation

Page 7: Challenges and Best Practices of Database Continuous Delivery

Media Coverage

Page 8: Challenges and Best Practices of Database Continuous Delivery

OperationsDevelopment

Smoother EffortLess Risk

Effort PeaksHigh Risk

Agile & DevOps

Page 9: Challenges and Best Practices of Database Continuous Delivery

The pain–Fortune 1000 by IDC

Application Downtime Cost

$2B/YDeployments/

month 2x Growth

Accelerate Delivery by 20%

Compliance & Audit

Enforcement

Infra Failure Hourly Cost

$100K

IDC DevOps Best Practices metrics: Fortune 1000 Survey, December 2014

Loss of Reputation

Page 10: Challenges and Best Practices of Database Continuous Delivery

▪Recently Conducted Survey

Continuous Delivery moving ahead!

Page 11: Challenges and Best Practices of Database Continuous Delivery

▪Every business is an IT business▪Customers demand that you deliver new features faster

− Agile Development− Process Automation− DevOps

▪Can’t wait 6 months for that next waterfall release…▪If you don’t, your competitor probably will

DevOps & CD: a must for every company

Page 12: Challenges and Best Practices of Database Continuous Delivery

Continuous Integration

Continuous Delivery

Continuous Deployment

Continuous Processes

Page 13: Challenges and Best Practices of Database Continuous Delivery

▪ More rapid changes▪ Fewer changes backed out▪ Better collaboration▪ Fewer defects

▪ Ultimately better service ▪ Happy customers ▪ Profitability

How Do I Measure Success?

Page 14: Challenges and Best Practices of Database Continuous Delivery

Why Continuous Delivery?

Page 15: Challenges and Best Practices of Database Continuous Delivery

15

But…what about the

database?

Page 16: Challenges and Best Practices of Database Continuous Delivery

Only 13% are actually performing basic CD practices for the DB!!!

Manual work: cant scale, cant match CD frequency

not repeatable, prone to error

Continuous Delivery is big and getting bigger, but...

Page 17: Challenges and Best Practices of Database Continuous Delivery

Manual steps lead to Chaos in Emergency Times

90%Rate this as a risk factor, but

53%Break the process and test urgent hot-fixes in pre-

production

Page 18: Challenges and Best Practices of Database Continuous Delivery

Old adage but true– The database is often neglected and

therefore can become the weakest link– Manual processes

Database/Code Silos exist…– Don’t always communicate effectively– Need to follow same procedures & best

practices Essential from a compliance and business point of view

Should be the strongest link

Is the database the Weakest link in a chain ???

Page 19: Challenges and Best Practices of Database Continuous Delivery

The Database is a constraint

Page 20: Challenges and Best Practices of Database Continuous Delivery

Only 13% automate… the rest do manual steps…

Page 21: Challenges and Best Practices of Database Continuous Delivery

What is the problem?

▪Root Causes for issues:− Challenging manual source control process− Static deployments code overrides and configuration drift− Dynamic deployments tools unaware of version control− No release automation red-flags – don’t know when to stop

the line…

Page 22: Challenges and Best Practices of Database Continuous Delivery

File Version Control Process

Today: Two isolated processes DB Development Process

Check-Out Script

Modify Script

Get updated Script from DB

Check-In Script

Compile Scriptin DB

Debug Scriptin DB

?

??

?

A

A’

Version control is out of sync from the database and cannot act as a Single Source of

Truth

Page 23: Challenges and Best Practices of Database Continuous Delivery

90%Rate this as a risk factor, yet

72%Admit database may not be in sync with the source repository

Page 24: Challenges and Best Practices of Database Continuous Delivery

Challenges:Code-overridesWorking on the wrong revisions Scripts do not always find their way to the version control solutionOut of process updates go unnoticedHard to locate outdated update scripts

Playing safe? What we really need: The upgrade scriptThe actual code of the objectA roll-back script

Scripts & version control

Page 25: Challenges and Best Practices of Database Continuous Delivery

X1.11.1.11.11.21.31.41.51.61.7

Int QA Stage Prod

Database Deploy Script

DevDev

DevModel

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.11.11.41.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

1.1 1.2

1.2 1.3

1.3 1.4

1.4 1.5

1.5 1.6

1.6 1.7

Out of Process Change

XX

XXX

? 1.1.1

X

Challenge with static scripts…

Configuration drift…

Page 26: Challenges and Best Practices of Database Continuous Delivery

60%Of those manually building scripts have to fix or tweak them regularly as part of a

deployment process

Page 27: Challenges and Best Practices of Database Continuous Delivery

80%Rate this as a risk factor, yet

71%Follow manual processes to create their database

deploy scripts

Page 28: Challenges and Best Practices of Database Continuous Delivery

▪Scripts− Hard to test in their entirely (holistically)− Hard to test due to colliding dependencies− Need to run in a specific order…− Much harder to deal with project scope changes

▪Scripts, unless super sophisticated:− Unaware of changes made in the target environment− Time passed from their coding to the time they are run− Potentially overriding production hot-fixes or work done in parallel by

another team

Scripts are static…

Page 29: Challenges and Best Practices of Database Continuous Delivery

Test cases using compare & sync tools:

An index exists in source (QA) but not in target (Production)What should we do? Add the index or not?

Using tools

Page 30: Challenges and Best Practices of Database Continuous Delivery

Safe to automate?Sure… (?)

Challenge with ‘Compare & Sync’

Page 31: Challenges and Best Practices of Database Continuous Delivery

Safe to automate?No. Requires manual inspection…

Challenge with ‘Compare & Sync’

Page 32: Challenges and Best Practices of Database Continuous Delivery

70%of those using compare & sync tools have to review and fix the results as they can't always

trust them to automatically deploy correctly

Page 33: Challenges and Best Practices of Database Continuous Delivery

A compare & sync tool:Is unaware of any changes that occurred before the time it ranHas no knowledge of changes that took place at the target environment

Does not leverage version control for more informationUnable to deal with conflicts & merges between different teams

Requires manual inspection Requires detailed knowledge regarding each change as part of the process

So…no automation… We fear for automating problems into

production and a major risk!!!

Challenges

Page 34: Challenges and Best Practices of Database Continuous Delivery

The Solution to the challenges

Page 35: Challenges and Best Practices of Database Continuous Delivery

Modern VC integrated DB process

Page 36: Challenges and Best Practices of Database Continuous Delivery

Revision history

Actions

Standard IDE

Change management

Enforced and integrated to existing process

Page 37: Challenges and Best Practices of Database Continuous Delivery

LeverageVersion control knowledge

intoDeployment decisions

Page 38: Challenges and Best Practices of Database Continuous Delivery

1.11.21.31.41.51.61.7

*

Int QA Stage ProdDev

DevDev

Model

1.1 1.2

1.2 1.3

1.3 1.4

1.1 1.7

1.1.1 1.7

1.1 1.1 1.11.41.7

Out of Process Change

1.1.11.7 1.1.11.7

Validate

1.4 1.5

1.5 1.6

1.6 1.7

Configuration Drift prevention / conflict identification and Validated execution

1.4 1.5

1.5 1.6

1.6 1.7

OR

Baseline aware analysis

Validated execution / Build & deploy on demand

Validate

Page 39: Challenges and Best Practices of Database Continuous Delivery

Understand the nature of the changesRaise red flags on conflicts Support out-of-process changesUtilize baseline aware analysis

Safety Net Deployment Automation

Page 40: Challenges and Best Practices of Database Continuous Delivery

If we had the index in the baseline (previous version) and no longer in Dev (i.e. - removed) => we should take it down from production…

(Deploy Change)

Deploying changes if needed

Development BaselinePrevious Label /

Production Golden Copy

Production

Page 41: Challenges and Best Practices of Database Continuous Delivery

Development BaselinePrevious Label /

Production Golden Copy

Production

BUT… If no index in baseline => someone else added it to Production…we should protect the NEW index on production!!!

(Protect Target)

Or protecting target environment…

Page 42: Challenges and Best Practices of Database Continuous Delivery

Dealing with conflicts => merging changes

Page 43: Challenges and Best Practices of Database Continuous Delivery

Conflict Resolving – Meta Data/Content

Page 44: Challenges and Best Practices of Database Continuous Delivery

Continuous Delivery Pipeline Builder

• Define a process• Automate the process• Prevent/Alert out of process changes

Page 45: Challenges and Best Practices of Database Continuous Delivery

Raise red flags to stop the line…if requires human intervention

Impact Analysis! Not Damage Control…

Page 46: Challenges and Best Practices of Database Continuous Delivery

Automate “everything”– Package the deployment of database changes along with all your

other application components to give a unified picture Move the process upstream

– Easily promote the same package (including database changes!) from one environment to the next, handling environment-specific differences automatically

Create the deployment pipeline

For successful CD:

Page 47: Challenges and Best Practices of Database Continuous Delivery

Requited integrations into entire continuous delivery pipeline:

IDEs (Oracle, Microsoft, MySQL in dev)

Source control (MsTFS, Git, Subversion etc.)

Task based development and ticketing systems (Jira, MsTFS, IBM RTC etc.)

Build tools (Jenkins, bamboo etc.)

ARA tools (IBM uDeploy, CA release automation, Chef etc.)

Integrated CD world…

Page 48: Challenges and Best Practices of Database Continuous Delivery

▪ Focusing on changes rather than managing changes and dealing with re-work, boosted overall productivity of 250 developers. We estimate we were able to do 15% more with the same resources.

▪We went from several fix-centric deployments a day, to one feature-centric deployment a week.▪The amount of incidents in production has declined as well. We had 20% less incidents. ▪CIO @ Credit Card company

“Testimonials - Efficiency

Page 49: Challenges and Best Practices of Database Continuous Delivery

▪ Regulation requirement - SOX and derived 357▪Auditing - change management approach with change management auditing. You have to do it anyway, do it automatically and efficiently. ▪Change management in production itself is regulatory required (ITIL). But you cannot ensure it without managing the whole process starting at Dev.▪We have to comply with regulation - but the business benefits from it. ▪CIO @ Credit Card company

“Testimonials - Regulation