Disaster And Business Continuity by Tom Canavan

5/16/08 ~ Joomla|Expo

Disaster and Business Continuity

Tom Canavan

Chief Information Officer

eWomenNetwork, Inc

5/16/08 ~ Joomla|Expo © Tom Canavan – JoomlaRescue.comv1

Disasters DO happen

Disaster preparedness is what you do before, not after, a disaster hits.

Crackers/hackers are only part of your concern.

Disaster—the meaning of the word has its root in early Italian, from the word disastro (Meaning away from star).

It was thought that an unfavorable position of a star or planet was the cause of mishaps and calamities1

1 Robert K. Barnhart, “The Barnhart concise Dictionary of Etymology – The origins of American English words”,

(New York: HarperCollins books, 1995) 208


What do you consider a disaster?

4-19-1995Murrah Bldg

Okla City

9-11-2001Ground Zero

8/28/2005Hurricane Katrina


But what about

One of my staff overwriting a critical directory

on one of my servers while I was here

at CMS Expo?

Is that a disaster?


I’ll take Disaster Recovery Planning for $500.00

-QUIZ-

Who has a working DR Plan?

If your site was offline for 7 to 10 days,

would your company go bankrupt?


404: Page Not found

A 1978 Study by the University of Minnesota showed that if a business could not recover their systems within a week,

will be out of business in a year.

That’s only four to six days

of interruption of services in 1978

Aasgaard, D.O. et al., “An evaluation of Data processing ‘Machine room’ Loss and Selected Recovery Strategies,” MISRC Working Papers

(Minneapolis, MN: University of Minnesota, 1978)

1

1-


Disaster Planning Life Cycle

DetermineRisks

DocumentYour

Business

BuildYourplan

Test & document

1

2

3

4


Worst Practices for DR/DP

• Failure to get management support

• No risk assessment

• No written plan

• Lack up ‘good’ backup’s

• You put the tapes where??

• NOT CONSIDERING the human element

• NOT testing your plan


Today’s agenda

Planning

Determine risks

Fortify

Test/Document

The elements, issues and challenges with planning

Hackers are only one concern – there’s more

Chances are GOOD you are exposed somewhere to attack

Test and Documentation is vital to a healthy plan

Communications Who needs to be informed, how to inform, Media/Press

Ω


Determine Risks

• What ‘could’ go wrong?– Hardware/Software Failure, DNS, Hackers

• What can you do to mitigate it?– Hot site, backups, planned recovery


Determine Risks

• People– Safety (of staff)– Where will they work?– Do they KNOW procedures (fire drill much?)

• Telephones, Pagers, Cell Phones, Email

• Hosting– Co-Location (shared, dedicated, VPS)– Workstations


Determine Risk• Restoration costs BY host ($$$)• Backups, Yes but..

– License keys– Copies of source/apps – do they exist?– Safe place to keep digital media

• Identify ‘stakeholders’• Insurance – Do you have any?• Your own computers – virus free?• What about your ‘backup server’ itself?


Affordability of a Risk• Elements to consider

– How much $$$ are you willing to spend– Does management buy into your plan?– Are they willing to commit to it financially?– Does your site “justify” a DR plan

• Determine if risks JUSTIFY cost

At the end of the day, if you have a blog site,

then perhaps its not worth it. If you have an

ecommerce site, then it WILL be.


Key Points

• Know your risks

• Know your what the costs are– Cost of experiencing the risk– Cost of restoration from downtime

• Have a plan to mitigate and recover


Why do need a plan?Recognize that trouble WILL come

– Mr. Murphy on line one for you…

Your plan should be : SMART basedSpecific, Measurable, Attainable,

Realistic, and Time-sensitive

"A good plan, violently executed now, is better than a perfect plan next week.“

General George Patton


Preparing to Plan

Recognize the following– A hard to execute plan will likely fail– Avoid ‘confirming to multiple opinions– Staff members will fight the plan– A plan untested is no good– Plans take time to build– A solid “one-page” plan is better than none


Planning Elements

• RTO/RPO – what is yours?– Recovery Time Objective– Recovery Point Objective

• Who is in charge?– Who else is in charge

• Moving parts of your plan• Where to store media, labeling, media type


Planning Elements

• Do you have a ‘fall-back’

• When will you ‘activate’ you plan?

• Define a communications strategy

• Which ‘systems’ have priority?

• Develop a schedule to plan

• Can you afford your plan?


Key Points

• Keep your planning team small

• Involve Sr. Mgmt, CAREFULLY

• Keep strong focus, for short bursts– Planning takes ‘time’ – and comfort

• Your Plan WILL fail the first time you use it

• Your staff will not buy in at first

• Setup a start, middle and end for plan


Fortification• Preparation of your site is key – check:

– Extensions, hosting, root kits, open ports

• Set permissions correctly– Files and directories (644 / 755)

• Latest version of Joomla (1.x and 1.5)

• Check your HOST’s setup– Ports, Versions of apache, etc.


Fortify at risk code

Can you find the problem?


Vulnerable Code

It’s missing the critical code:

// no direct access

defined( '_VALID_MOS' ) or die( 'Restricted access‘);

While this problem is less prevalent - It still exists and can trip you up

Note: the previous code snip was purposely modified for demonstration purposes only !


Fortify - .htaccess

.htaccess – your first line of defense


Fortify - Permissions

• Permissions– Very common problem– Check files and Dirs– FILES: 644– DIR : 755


Fortify – PHP.INI• Safe Mode: OFF

• Open basedir: none

• Display Errors: ON

• Short Open Tags: ON

• File Uploads: ON

• Magic Quotes: ON

• Register Globals: OFF


Fortify - Versions

• Using 1.0.xx– Make sure you are at least at 1.0.15

• Using 1.5 – Make sure you are at least at 1.5.3

Older versions are exploitable


Fortify – Common Trip Ups

• Common issues• Admin still named ADMIN• Easy to guess passwords like P@ssw0rd• Permissions set wrong• Lack of .htaccess or php.ini• Vulnerable components• Hosts not setup properly


Fortify - Poor Host SecurityExample: Ports open that need not be

Real case from JoomlaRescue.com Client– The host had 1,700 ports open. – Port 53 – Allows for Zone Transfers– Port 23 – Telnet – Allowed “Banner Grabbing”– Port 21 – Allowed me (shouldn’t have) to FTP in– Port 6667 (note BackOrfice) – Cult of the Dead Cow– And 1,677 more – (HUN???)

Host told client:

“That’s ok you have a Virtual private Server (VPS) setup”


Fortification Tools

• Tools to check host out:– NMAP (only with host’s permission)– Tools from http://centralops.net/co

• Domain Dossier

– Joomla Health Check (available from J!)

• Google – Google Hacks (again permission please)

• Hire JoomlaRescue.com


Documentation

Documentation is a product of your risk assessment, goals, planning and

fortification.

It’s the chief cornerstone of your DR plan.


Documentation

• Documentation considerations• First recognize its not the Holy Bible

– It CAN be changed as needed to fit

• Establish a review process• It will change from time to time • Make sure the Date is on it

• Keep it in a safe place• Key DR team members must have it• Don’t let it fall into competitors hands


Maintaining your plan

• Test your plan • Accomplished through drills• Document the results• Change documentation as needed• Collect old docs, distribute new

• Tracking changes• Why did you change it?

Always ask WHY changes

will increase survivability


Drill for results

• Establish a ‘failure’ test

• Purpose:– To shake down your documentation– To train your staff– To learn where your plan works and fails

• Establish a ‘regular’ drill time– Key members should be present at each test


Some things your plan should have

• Team member contact information– Plan initiation instructions

• ‘when’ we activate the plan

– Location of backup media– Passwords and other security information– Contact for host

• Technical support, escalation procedures

– Instructions on HOW to restore


Documentation Example


A few words on drilling

Conducting a live test helps increase your sites survivability by proving your plan works,

and ensuring your staff knows their job


About your plan

"No plan survives first engagement with the enemy"

Von Clausewitz.—Prussian Military Thinker


Key Points

• Your Plan/Docs is a living document– Care and feed for it

• Test it once you develop– Conduct regular drills

• Change it if is not working

• Establish a process for distribution

• Keep it safe


Communications

• Understanding what crisis communication means• Preparing media communications in advance• Communicating with your team and externally


Crisis Communication

• Internal with team• Coordinates efforts for recovery

• Internal with employees other staff• Helps to control rumors

• Communications with media / customers• Prepare plan in advance• This helps you control the message• Helps retain customer base


Media Communications

• Media contact– Baseline communication regarding the event.– Reestablishes trust and ensure facts not conjecture. – The message should drive the behavior you want – Accomplish this through advanced preparation

• Talking points for employees.

• A template for developing a news release.

• A list of reporters, media outlets or blog sites you want your message directed to.

• A fact sheet for media, both downloadable PDF and paper based.


Staff Communications

• Establish a communications tree

• Assign a Communications person or team

• Make sure you do two things– Communicate openly and often with DR team– Carefully distribute information to rest of staff

Keep in mind what you say, may end up

on a blog or in the paper.


Tools for communication

• www.freeconferencecall.com

• Establish a media checklist

• Establish a Priority system

• Be as ‘open’ as you can– If your hacked and had credit card data

stolen, it may not be the best time to discuss it DURING the crisis


Key Points

• Be sure you have a plan to communicate

• Keep in mind nothing is “off the record”

• Internal/External communications is vital – Keeps speculation down


Dodging The Bullets - Book


A Rabbit? My men are not afraid of a Rabbit!

Disaster And Business Continuity by Tom Canavan

Technology

Transcript of Disaster And Business Continuity by Tom Canavan