Disaster And Business Continuity by Tom Canavan

Post on 08-May-2015

668 views 1 download

description

Disaster And Business Continuity by Tom Canavan presentation given at CMS Expo in Denver, December 2008.

Transcript of Disaster And Business Continuity by Tom Canavan

5/16/08 ~ Joomla|Expo

Disaster and Business Continuity

Tom Canavan

Chief Information Officer

eWomenNetwork, Inc

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Disasters DO happen

Disaster preparedness is what you do before, not after, a disaster hits.

Crackers/hackers are only part of your concern.

Disaster—the meaning of the word has its root in early Italian, from the word disastro (Meaning away from star).

It was thought that an unfavorable position of a star or planet was the cause of mishaps and calamities1

1 Robert K. Barnhart, “The Barnhart concise Dictionary of Etymology – The origins of American English words”,

(New York: HarperCollins books, 1995) 208

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

What do you consider a disaster?

4-19-1995Murrah Bldg

Okla City

9-11-2001Ground Zero

8/28/2005Hurricane Katrina

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

But what about

One of my staff overwriting a critical directory

on one of my servers while I was here

at CMS Expo?

Is that a disaster?

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

I’ll take Disaster Recovery Planning for $500.00

-QUIZ-

Who has a working DR Plan?

If your site was offline for 7 to 10 days,

would your company go bankrupt?

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

404: Page Not found

A 1978 Study by the University of Minnesota showed that if a business could not recover their systems within a week,

will be out of business in a year.

That’s only four to six days

of interruption of services in 1978

Aasgaard, D.O. et al., “An evaluation of Data processing ‘Machine room’ Loss and Selected Recovery Strategies,” MISRC Working Papers

(Minneapolis, MN: University of Minnesota, 1978)

1

1-

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Disaster Planning Life Cycle

DetermineRisks

DocumentYour

Business

BuildYourplan

Test & document

1

2

3

4

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Worst Practices for DR/DP

• Failure to get management support

• No risk assessment

• No written plan

• Lack up ‘good’ backup’s

• You put the tapes where??

• NOT CONSIDERING the human element

• NOT testing your plan

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Today’s agenda

Planning

Determine risks

Fortify

Test/Document

The elements, issues and challenges with planning

Hackers are only one concern – there’s more

Chances are GOOD you are exposed somewhere to attack

Test and Documentation is vital to a healthy plan

Communications Who needs to be informed, how to inform, Media/Press

Ω

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Determine Risks

• What ‘could’ go wrong?– Hardware/Software Failure, DNS, Hackers

• What can you do to mitigate it?– Hot site, backups, planned recovery

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Determine Risks

• People– Safety (of staff)– Where will they work?– Do they KNOW procedures (fire drill much?)

• Telephones, Pagers, Cell Phones, Email

• Hosting– Co-Location (shared, dedicated, VPS)– Workstations

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Determine Risk• Restoration costs BY host ($$$)• Backups, Yes but..

– License keys– Copies of source/apps – do they exist?– Safe place to keep digital media

• Identify ‘stakeholders’• Insurance – Do you have any?• Your own computers – virus free?• What about your ‘backup server’ itself?

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Affordability of a Risk• Elements to consider

– How much $$$ are you willing to spend– Does management buy into your plan?– Are they willing to commit to it financially?– Does your site “justify” a DR plan

• Determine if risks JUSTIFY cost

At the end of the day, if you have a blog site,

then perhaps its not worth it. If you have an

ecommerce site, then it WILL be.

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Key Points

• Know your risks

• Know your what the costs are– Cost of experiencing the risk– Cost of restoration from downtime

• Have a plan to mitigate and recover

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Why do need a plan?Recognize that trouble WILL come

– Mr. Murphy on line one for you…

Your plan should be : SMART basedSpecific, Measurable, Attainable,

Realistic, and Time-sensitive

"A good plan, violently executed now, is better than a perfect plan next week.“

General George Patton

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Preparing to Plan

Recognize the following– A hard to execute plan will likely fail– Avoid ‘confirming to multiple opinions– Staff members will fight the plan– A plan untested is no good– Plans take time to build– A solid “one-page” plan is better than none

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Planning Elements

• RTO/RPO – what is yours?– Recovery Time Objective– Recovery Point Objective

• Who is in charge?– Who else is in charge

• Moving parts of your plan• Where to store media, labeling, media type

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Planning Elements

• Do you have a ‘fall-back’

• When will you ‘activate’ you plan?

• Define a communications strategy

• Which ‘systems’ have priority?

• Develop a schedule to plan

• Can you afford your plan?

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Key Points

• Keep your planning team small

• Involve Sr. Mgmt, CAREFULLY

• Keep strong focus, for short bursts– Planning takes ‘time’ – and comfort

• Your Plan WILL fail the first time you use it

• Your staff will not buy in at first

• Setup a start, middle and end for plan

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortification• Preparation of your site is key – check:

– Extensions, hosting, root kits, open ports

• Set permissions correctly– Files and directories (644 / 755)

• Latest version of Joomla (1.x and 1.5)

• Check your HOST’s setup– Ports, Versions of apache, etc.

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify at risk code

Can you find the problem?

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Vulnerable Code

It’s missing the critical code:

// no direct access

defined( '_VALID_MOS' ) or die( 'Restricted access‘);

While this problem is less prevalent - It still exists and can trip you up

Note: the previous code snip was purposely modified for demonstration purposes only !

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify - .htaccess

.htaccess – your first line of defense

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify - Permissions

• Permissions– Very common problem– Check files and Dirs– FILES: 644– DIR : 755

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify – PHP.INI• Safe Mode: OFF

• Open basedir: none

• Display Errors: ON

• Short Open Tags: ON

• File Uploads: ON

• Magic Quotes: ON

• Register Globals: OFF

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify - Versions

• Using 1.0.xx– Make sure you are at least at 1.0.15

• Using 1.5 – Make sure you are at least at 1.5.3

Older versions are exploitable

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify – Common Trip Ups

• Common issues• Admin still named ADMIN• Easy to guess passwords like P@ssw0rd• Permissions set wrong• Lack of .htaccess or php.ini• Vulnerable components• Hosts not setup properly

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortify - Poor Host SecurityExample: Ports open that need not be

Real case from JoomlaRescue.com Client– The host had 1,700 ports open. – Port 53 – Allows for Zone Transfers– Port 23 – Telnet – Allowed “Banner Grabbing”– Port 21 – Allowed me (shouldn’t have) to FTP in– Port 6667 (note BackOrfice) – Cult of the Dead Cow– And 1,677 more – (HUN???)

Host told client:

“That’s ok you have a Virtual private Server (VPS) setup”

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Fortification Tools

• Tools to check host out:– NMAP (only with host’s permission)– Tools from http://centralops.net/co

• Domain Dossier

– Joomla Health Check (available from J!)

• Google – Google Hacks (again permission please)

• Hire JoomlaRescue.com

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Documentation

Documentation is a product of your risk assessment, goals, planning and

fortification.

It’s the chief cornerstone of your DR plan.

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Documentation

• Documentation considerations• First recognize its not the Holy Bible

– It CAN be changed as needed to fit

• Establish a review process• It will change from time to time • Make sure the Date is on it

• Keep it in a safe place• Key DR team members must have it• Don’t let it fall into competitors hands

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Maintaining your plan

• Test your plan • Accomplished through drills• Document the results• Change documentation as needed• Collect old docs, distribute new

• Tracking changes• Why did you change it?

Always ask WHY changes

will increase survivability

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Drill for results

• Establish a ‘failure’ test

• Purpose:– To shake down your documentation– To train your staff– To learn where your plan works and fails

• Establish a ‘regular’ drill time– Key members should be present at each test

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Some things your plan should have

• Team member contact information– Plan initiation instructions

• ‘when’ we activate the plan

– Location of backup media– Passwords and other security information– Contact for host

• Technical support, escalation procedures

– Instructions on HOW to restore

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Documentation Example

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

A few words on drilling

Conducting a live test helps increase your sites survivability by proving your plan works,

and ensuring your staff knows their job

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

About your plan

"No plan survives first engagement with the enemy"

Von Clausewitz.—Prussian Military Thinker

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Key Points

• Your Plan/Docs is a living document– Care and feed for it

• Test it once you develop– Conduct regular drills

• Change it if is not working

• Establish a process for distribution

• Keep it safe

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Communications

• Understanding what crisis communication means• Preparing media communications in advance• Communicating with your team and externally

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Crisis Communication

• Internal with team• Coordinates efforts for recovery

• Internal with employees other staff• Helps to control rumors

• Communications with media / customers• Prepare plan in advance• This helps you control the message• Helps retain customer base

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Media Communications

• Media contact– Baseline communication regarding the event.– Reestablishes trust and ensure facts not conjecture. – The message should drive the behavior you want – Accomplish this through advanced preparation

• Talking points for employees.

• A template for developing a news release.

• A list of reporters, media outlets or blog sites you want your message directed to.

• A fact sheet for media, both downloadable PDF and paper based.

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Staff Communications

• Establish a communications tree

• Assign a Communications person or team

• Make sure you do two things– Communicate openly and often with DR team– Carefully distribute information to rest of staff

Keep in mind what you say, may end up

on a blog or in the paper.

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Tools for communication

• www.freeconferencecall.com

• Establish a media checklist

• Establish a Priority system

• Be as ‘open’ as you can– If your hacked and had credit card data

stolen, it may not be the best time to discuss it DURING the crisis

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Key Points

• Be sure you have a plan to communicate

• Keep in mind nothing is “off the record”

• Internal/External communications is vital – Keeps speculation down

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Dodging The Bullets - Book

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

A Rabbit? My men are not afraid of a Rabbit!