Creating a complete recovery solution

Adam Backman adam@wss.comPartner, White Star Software

Why backup?

No reason to backup – Our stuff never fails Just takes up resources We already have redundancy I hate changing the tapes I’m tired I’m hungry I don’t feel good Are we there yet?

Cost of Lost Data

Loss of data is expensive− Cost to rekey− Cost of lost business

Can you recover the data− Not everyone has paper trail

Focus on recovery

Building a Good Recovery Strategy

Know your business− Components of business

how people do business with you− Components of systems

Tools (applications and physical) Know your risks (fire, flood, hurricane, …) Be inclusive

− Technical people (network, phones, facilities, …)− Business people (people who own the application functionality)

Build an execution plan with contingencies

Why be inclusive?

Business people know what they need− What they use in the application− What they need to do their job (phone, fax, internet, …)

Stops the finger pointing when a problem occurs Business people generally control the budget Need decision makers

− Authorize spending − Data loss vs. recovery time decisions

Focus on Recovery

Loss of dataRogue program, hacked, poor testing

Loss of diskDisk failure

Loss of machineMachine failure

Loss of machine roomFire, flood,…

Loss of buildingNatural disaster

Creating a plan

Goals (Event-based goals)− Data corruption (Data)− If we lose a disk (DB)− If we have a fire (Machine)− If we have a natural disaster (Facility)

Hardware Software Other stuff

Main Goal

The main goal for any recovery plan is to minimize or eliminate the impact to the customer

Creating a plan - Goals

Acceptable downtime (Generally cost based)Everyone wants zero but it is generally cost prohibitive

Planned outages− Hardware install and maintenance − Software upgrade− O/S upgrade or patch

Notifications (Both before and during outage)− Who− When− What do they do?

Creating a Plan – Other Considerations

What makes your business run?− Phones− Faxes− Business to Business (EDI, XML Feed, …)

Can people work from home? Do you have/need another location? Contact lists in case of major catastrophe

− Kept up-to-date− Kept online and printed in an accessible location

Cover All Sides

Everyone should be running after image journaling Need removable backup periodically (daily)

− Wide scale events (fire, flood, …)− To recover from after image journals

Replication is the new normal− OE Replication− Log-based replication− Hardware replication

Portions of a OpenEdge Database

Database table of contents (.db) The data files (.d*) Before image journals (.b*) After image journals (.a) Modified buffers in memory

How probkup -online works

1. Lock the DB2. Do a pseudo-checkpoint3. Switch AI files (if necessary)4. Backup the BI file5. Unlock the DB6. Backup the database

Other Items to Consider

Application files Data outside the database (GIS, photos, …) User files (reports, output, …) External systems (EDI, Warehouse, …)

Backup’s Impact to Production

Backing up production− Pause during backup of before image journal− Uses I/O capacity of production

Split mirror backup− Use of quiet point keeps pause to a minimum− Pause is non-zero

After Image file backup− Still needs a backup to begin the process− Little impact for backup process− Potentially long recovery time

After Image Management

Use OpenEdge after image management− Automated− Supported− Nice features

Use script-based after image management− Custom− Requires support and testing

Replication options

OpenEdge Replication− Supported by Progress− Real time replication− Allows for reporting against target database− Additional license cost

Log-based replication− Free− Requires self-support− Not real time (greater potential data loss)− No reporting option

Hardware replication

OpenEdge Replication – Single Target

Production (source) Reporting (target)

Shared Memory

Shared Memory

Replication Server

Replication Agent

Source DB

Target DB

OpenEdge Replication

Production (source)

Target 1

Shared Memory

Shared Memor

y

Replication Server

Rep Agent

Source DB

Target DB

Target 2

Shared Memor

y

Rep Agent

Target DB

Multiple Targets

Log-Based Replication

Log-based replication has been used for years Log-based replication provides a vehicle for replication without the licensing costs of OE Replication

Not real-time Code for this type of replication must be

maintained by the user and there is no official support from the vendor

Hardware-Based Replication

Hardware-based replication is a function of the hardware vendors and thus supported directly by them

This method is NOT supported by Progress ALL write operations must be guarantied across

the source and target disk systems

Archiving – Why is it important

Needs to be accessible Needs to be secure Needs to be in a different location Need to have policies

− Physical access− Number of saved archives − Security

Archiving

Who does your archiving (Iron Mountain, third-party, someone’s house, …)

What do you keep− Two weeks of dailies− 5 weeks of weeklies− 1 year of monthlies

How to label you backups− Who did the backup− Command to restore− Date and Time

Pre-archiving

After image files − Move to remote system

Database backups− Move to remote system before moving to removable

media Support files User data

Archiving (continued)

Data Archiving – What data archiving?− Archive/Delete?− Archive/Save historical− Archive/Save aggregates

After Image file archiving− At least 2 backups worth− I recommend a week or more if possible

How about if I am a SAAS user

Who is your provider Verify their recovery plan Run dry run of at least one recovery scenario Have specific service level agreements

− Time to recover− Maximum loss of data− Penalties for missing times (need to cover your costs)

How about if I am a SAAS provider

Build regular recovery plan Unique concerns

− Security− Compliance (HIPAA, SOX, …)

Build achievable SLAs for your users that at a minimum include:− Data loss− Uptime− Maximum recovery time− Maintenance time

Implementing Your Plan

First implementation should be a totally manual process to insure the steps work and allow for documentation

Document the process as you go− Who are you logged in as?− Exactly what you typed− Where you were (console, remote, …)− Can things be done in parallel or sequentially− Where are the logs − What to look for in the logs (good and bad)

Documentation

All recovery documentation should be VERY specific as the person doing the recovery will not be familiar with you or your environment

Create documents for normal maintenance− Backups− Database growth− Modification of OS, Application, printers, …

Create scenario based recovery plans− Lose a disk (or disk pair)− Fire − Flood

Testing Your Plan

Who does the test?− Not the person who wrote it− The backup person for the implementation− Someone who is “always” there regardless of technical

ability How often to test?

− Material data change− Any change in database configuration

New directories New areas

− Do you have a second site or redundant hardware?− Do you have enough disk capacity (space and throughput)

How to test your plan

Fail over to your backup system− You may lose the use of the backup system but that is a good

trade in the vast majority of cases− Maybe not live but at least a restore and test

Fail back to your primary system− After a live failover

Contingency planning for:− Personnel− Physical plant − Equipment − Lead time for resources (hours or days)

Summary: Recovery Planning

You still need to backup Backup your backup not production if possible Be inclusive when building your team Create and maintain a comprehensive plan

Include everything needed to use the application: Hardware Applications Data

Create and maintain physical and online contact lists and store it with your archived backups

Conclusion: Recover planning

Always backup what you have now, however little, before starting to recover

Test your plan periodically (At least annually)

Still have questions?

Please feel free to contact me directly.

Adam BackmanWhite Star Software(603)897-1010adam@wss.com

THANK YOU

Thank you for your time