Techniques  for  Restoring  Telecom   after  Disasters  &  Outages

45
MARK REYNOLDS, UNIVERSITY OF NEW MEXICO TIM RUFF, TELECOM RECOVERY  Techniques for Restoring Telecom after Disasters & Outages

Transcript of Techniques  for  Restoring  Telecom   after  Disasters  &  Outages

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 1/45

M A R K R E Y N O L D S , U N I V E R S I T Y O F N E W M E X I C O

T I M R U F F , T E L E C O M R E C O V E R Y  

Techniques for Restoring Telecomafter Disasters & Outages

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 2/45

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 3/45

3

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 4/45

The Problem4

Incoming local DIDs

Incoming toll-free numbers Outbound dial tone

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 5/45

1970 1975 1980 1985 1990 1995 2000 2004

Droughts

Floods

Earthquakes

Epidemics

Slides

Volcanic eruptions

Wild fires

Wind Storms

Insect infestation

Source: EM-DAT : The OFDA/CRED International Disaster Database

http://www.em-dat.net, UCL - Brussels, Belgium

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 6/45

Uh-

6

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 7/45

Common Causes of Phone Outages

Equipment failure Power outage

New Installation

Moving

 Water leak 

 Animals

Cable cut  Wiring short

Technician mistake

 Winter storm

 Accidents

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 8/45

October 17, 1989 4:00 PM

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 9/45

October 17, 1989 6:00 PM

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 10/45

Silicon Valley, April 9, 2009

1:30 AM  Vandals sever major fiber backbone withhacksaws

 AT&T, Verizon, Sprint down

Landlines, cellular, internet down No 9-1-1 service, no credit cards

Lasted 14 hours

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 11/45

Pandemic Quarantine

United States H1N1 stats as of 10/17/09:

98,000 hospitalizations

3,900 deaths

 What if the next strain is more severe? Are you prepared to have some/most/all employees work from home?

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 12/45

The 6 Vulnerabilities of Telecommunications

LocalCarrier

Long Dist.Carrier

12

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 13/45

 Vulnerability 1: Equipment Failure

LocalCarrier

Long Dist.Carrier

13

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 14/45

 Vulnerability 2: Power

LocalCarrier

Long Dist.Carrier

14

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 15/45

 Vulnerability 3: Last Mile (cable cut)

LocalCarrier

Long Dist.Carrier

15

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 16/45

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 17/45

 Vulnerability 5: Long Distance Carrier

Long Dist.Carrier

LocalCarrier

17

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 18/45

 Vulnerability 6: Evacuation / Quarantine

LocalCarrier

Long Dist.Carrier

18

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 19/45

The 6 Vulnerabilities of Telecommunications19

1. Equipment failure

2. Power outage

3. Last mile (cable cut)

4. Local carrier failure/congestion5. Long distance carrier failure/congestion

6. Evacuation/pandemic quarantine

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 20/45

20

QwestBCR 

Qwest

(505) 272-XXXX Hospital

Dept.Cell

Phones

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 21/45

21

Good:

Can divert calls after any failures except carrierfailures and congestion

Not so good: Calls go direct to cell phones, no transfer, no

greeting, no centralized voicemail, no attendant, etc.

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 22/45

Qwest

UNM  New Recovery Plan22

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 23/45

Qwest

UNM  New Recovery Plan23

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 24/45

Recovery server

Qwest

UNM  New Recovery Plan24

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 25/45

Recovery Server

Qwest

Internet

UNM  New Recovery Plan25

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 26/45

Recovery Server

Qwest

Internet

UNM  New Recovery Plan26

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 27/45

Recovery Server

Qwest

Internet

UNM  New Recovery Plan27

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 28/45

UNM  New Recovery Plan28

Recovery Server

Qwest

Internet

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 29/45

UNM  New Recovery Plan29

Recovery Server

Qwest

BCR 

Qwest

Hospital(505) 272-XXXX 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 30/45

UNM New Recovery Plan30

Good:

Incoming calls get back into Hospital PBX

If PBX fails, calls get to same employees on cell, sat,

home, other locations, recordings, etc. Real-time control, unlimited flexibility 

Backup dial tone using any broadband connection

Not so good:

Local numbers down if Qwest is down (toll-freenumbers recoverable regardless of Qwest failure)

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 31/45

The 13 Most Critical Mistakes

1. Underestimating the pain of outages

2.

3. Confusing outbound with inbound

4. Confusing disasters with outages5. Neglecting vulnerabilities

6. Inadequate mass communication

7. Inadequate control

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 32/45

The 13 Most Critical Mistakes

8. No satellite phone

9. Insufficient capacity 

10. Reliance on hot site

11. Putting backup equipment on-premise12. Lack of testing

13. Spending too much

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 33/45

10 Low-Cost Improvements

1. Twitter/Facebook 

2. Skype

3. MagicJacks

4. Create voluntary reimbursement agreements withemployees

5. Gather alternate phone #s, email

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 34/45

10 Low-Cost Improvements

6. Create Emergency section on website

7. Publish a toll-free hotline

8. Retain some POTS (copper) lines, wired outside the

phone system9.

carrier

10. Obtain GETS/WPS cards

(www.ncs.gov/services.html)

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 35/45

FAQs35

 What if we already have a backup PBX?

 What if we already have a SONET ring, SHARPS,SHINS, etc.?

 What if we already have a mass notification system?  What if we already have cell phones, radios, satellite

phones, etc.?

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 36/45

FAQs36

calls to you?

1. Neustar/Port DR 2.  We need your help in pressuring carriers to adopt

Port DR!

3.  Advantages of toll-free over local

4. Still need numbers loaded in a backup carrier

5. Still need a place to send the calls

6. Backup satellite T1 (for delivery & outbound)

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 37/45

Backup Satellite T1

Off-siteControl system

Public Phone

Network 

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 38/45

Off-siteControl system

Public Phone

Network 

Backup Satellite T1

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 39/45

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 40/45

Off-siteControl system

Public PhoneNetwork 

Backup Satellite T1

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 41/45

Off-siteControl system

Public PhoneNetwork 

Backup Satellite T1

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 42/45

Off-siteControl system

Public PhoneNetwork 

Backup Satellite T1

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 43/45

FAQs43

calls to you?

How much does this cost?

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 44/45

 What Next?

Determine what next steps to take.

Identify people involved in making and approving aplan.

Establish a project to implement the plan. Communicate and test the plan.

44

8/6/2019 Techniques for Restoring Telecom after Disasters & Outages

http://slidepdf.com/reader/full/techniques-for-restoring-telecom-after-disasters-outages 45/45

 What are the questions?

 What are the important numbers at the institutionthat should function during a disaster?

How will calls to the numbers identified above behandled?

 Who would authorize and activate the emergency routing of numbers?

 Who would administer and maintain the

information?

45