LU - ACUTA 2009 Conference Presentation
-
Upload
dr-michael-dobe-sr -
Category
Education
-
view
101 -
download
0
description
Transcript of LU - ACUTA 2009 Conference Presentation
![Page 1: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/1.jpg)
Lessons Learned From the Texas Gulf Coast
ACUTA 2009Palm Springs, California
![Page 2: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/2.jpg)
About Your SpeakersTexas State University System
Dr. Michael E. Dobe, Sr., Associate Vice President, Chief Information Officer & Adjunct Professor at Lamar University (Beaumont, Texas)
http://www.linkedin.com/in/mdobe Texas A & M System
Mr. Steven Conway, Director of Computing and Info Services at Texas A&M University at Galveston (Galveston, Texas)
http://www.linkedin.com/pub/5/b7/24a
![Page 3: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/3.jpg)
About the Checklist
Steven Conway
Texas A&M University at Galveston (TAMUG)
![Page 4: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/4.jpg)
ACUTA/ATIS Hurricane Preparedness Project
Formed Committee May 2007Reviewed and Modified ATIS documentPublished ACUTA Hurricane Checklist Oct
2007
![Page 5: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/5.jpg)
http://www.acuta.org/?2026
Hurricane Preparedness
Checklist
ACUTA - The Association for Information Communications Technology Professionals in Higher Education
Developed in cooperation withATIS – The Alliance for Telecommunications Industry Solutions
Hurricane Katrina, August 2005, Terra MODIS True Color Image
Courtesy of Louisiana State University
![Page 6: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/6.jpg)
Committee Amy Phillips, Louisiana State University Health Sciences Center Frank O'Quinn, Louisiana State University Sandra Thompson, University of West Florida James Shea, Boston University Jeanne Spinosa, Johnson & Wales University Garret Yoshimi, University of Hawaii George Denbow, University of Texas at Austin Michele Narcavage, University of Pennsylvania Steve Ellis, University of Notre Dame Steve Conway, Texas A&M University at Galveston Walt Magnussen, Texas A&M University - Task Force Chair Veronica Lancaster, ATIS Brad Ruark, ATIS /AT&T Jeri Semer, ACUTA Executive Director Aaron Fuehrer, ACUTA Staff Liaison
![Page 7: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/7.jpg)
ACUTA Hurricane Preparedness Checklist
![Page 8: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/8.jpg)
Using the Checklist As An After-Action Review Tool
Perspectives From Beaumont
Dr. Michael Dobe, Sr.Lamar University
![Page 9: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/9.jpg)
About Lamar UniversityLocated in Beaumont, TX (Approx 40 Minutes
from Louisiana Border)Central Data Center in Beaumont Serves
Three Campuses:Lamar UniversityLamar Institute of TechnologyLamar State College- Orange
Total Students Served – Approximately 16,000
![Page 10: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/10.jpg)
Proximity of Beaumont and Galveston
119 mi – about 1 hour 57 mins
![Page 11: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/11.jpg)
Rita, Edouard, Gustav, IkeHurricane Activity in SE Texas
Rita – September 2005 (CAT 4 With Major Damage to Beaumont)
2006 and 2007 Atlantic Hurricane Seasons Quiet in SE Texas
Edouard – August 2008 (Missed SE Texas)Gustav – September 1, 2008 (Minor Impact in
SE Texas)Ike Hit SE Texas Less Than Two Weeks After
Gustav…
![Page 12: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/12.jpg)
Hurricane Ike’s Path
http://en.wikipedia.org/wiki/File:Ike_2008_track.png
![Page 13: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/13.jpg)
Hurricane Ike Timeline Lamar evacuated campus 10 Sept 2008Hurricane Ike landfall Evening of 12-13 Sept 2008Low Lying Areas of Beaumont Flooded, Extensive
Flooding in Orange County, Bridge City Hardest HitLamar University Campus in Beaumont Did Not FloodLibrary, Family and Consumer Sciences, and
Montaigne Center Buildings Had Roof Damage24 Sept 2008 Essential Staff Returned to Campus25 Sept 2008 Classes Resumed at Lamar University
![Page 14: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/14.jpg)
Assessing Services During IkeLU Lessons LearnedStorm FactsContact ListsEmergency Sites and
LodgingAdministration,
Communications, and Manpower
Safety, Security and Training
InventoryPhysical AssetsPower and FuelNetwork ServiceMain Computer Center
and Key Telecommunications Hubs
Long-Term PreparationsPost-Storm ActivitiesDamage Assessment and
Post-Storm Activities
![Page 15: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/15.jpg)
Storm Facts NOAA WebsiteMajor News Media: CNN, Weather Channel
and Storm Pulse (http://www.stormpulse.com/)
Importance of Local News Media: KFDM and Entergy
Government Contacts: Jefferson County and FEMA
Local Contacts: State Police and Beaumont Police
![Page 16: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/16.jpg)
![Page 17: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/17.jpg)
![Page 18: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/18.jpg)
![Page 19: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/19.jpg)
Contact Lists Updated and Distributed Internal Contacts
For GustavLeadership Team Phone ListPocket Card for Satellite Phone NumbersITS Contact List
External Contact Lists Maintained By Staff Sections
![Page 20: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/20.jpg)
![Page 21: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/21.jpg)
Emergency Sites and Lodging Emergency Control Site Remained on
Campus in BeaumontTemporary Lodging and Shelter Arranged by
IndividualsDisaster Recovery/Business Continuity Site at
San Marcos State University
![Page 22: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/22.jpg)
![Page 23: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/23.jpg)
Texas State University Systemhttp://www.tsus.edu/
TSUS Institutions (East Texas Locations in Bold)Lamar University Sam Houston State University Sul Ross State University Sul Ross State University Rio Grande
College Texas State University-San Marcos Lamar Institute of Technology Lamar State College-Orange Lamar State College-Port Arthur
![Page 24: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/24.jpg)
Sam Houston State in Hunstville
153 mi – about 2 hours 29 mins
TOO CLOSE FOR COMFORT
![Page 25: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/25.jpg)
Stephen F. Austin State
128 mi – about 2 hours 36 mins
NOT MEMBER OF TSUS
![Page 26: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/26.jpg)
San Marcos State University
251 mi – about 4 hours 15 mins
During Ike – Approx. 16 hrs
LAMAR UNIVERSITY BC Site
![Page 27: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/27.jpg)
Administration, Communications and Manpower Importance of Clear Chain of Command
and Clear Delineation of ResponsibilitiesITS Set Up Two Teams: Offsite B/C Team
and Onsite Command Center TeamCommunication with Employees via LU
Website and ConnectED (Now Blackboard Connect) to E-Mail and Phones.
Communications Methods: Cell Phones, Satellite Phones, Land Lines and SIP Phones for Key Employees
![Page 28: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/28.jpg)
ACUTA NOTE: Effective Leadership Was Key to Success
![Page 29: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/29.jpg)
ACUTA Note: Importance of Coordination of Operations Functions
![Page 30: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/30.jpg)
ACUTA NOTE: Importance of Coordination of the Academic Response
![Page 31: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/31.jpg)
Administration, Communications and Manpower ITS Hunkered Down With LUPD in Beaumont
Daily Meetings for Coordination in Beaumont on Campus
Reporting and Notification Lead By President and Leadership Team
Conference Bridges For Coordination With Remote Staff Managed by Network Services
Public Coordination Handled by LU PDManaged Through Office of VP for Finance and Amin
Supplemental PersonnelAccountability and Time ReportingCosts
![Page 32: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/32.jpg)
LU IT Services Staff DispositionLocal Core Team Stayed on Campus
ThroughoutRemote Core Team Relocated to San Marcos
Area – Worked on Ordering Replacement Systems
Remaining Staff Dispersed With FamiliesAT&T Conference Bridge to Maintain Contact
With Dispersed Staff
![Page 33: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/33.jpg)
Onsite IT Service Lead: John Genuardi, Customer Support Director
![Page 34: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/34.jpg)
Onsite Network Support: Patrick Stewart, Manager of Network Services
![Page 35: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/35.jpg)
Onsite Data Center Lead: Srinivas “Sri” Vandaraj, Senior Security Analyst
![Page 36: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/36.jpg)
Safety, Security and Training President’s Weekly Security Meeting
Initiated as Outcome of RitaSystem Access Managed By Onsite Core ITS
Team (Issues With Passwords: Need One-Time Password System)
Physical Re-Entry Credentials Provided by Chief of Police
ITS and PD Coordinated Security Cameras Shutdown and Start-upTraining for ConnectedED
![Page 37: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/37.jpg)
Inventory and Physical AssetsImpact of Key Staff ResignationPlanning: AVP (IT Infrastructure and Security)
Responsible for Availability of Emergency Supplies and Work Supplies
AVP Had Resigned Between Gustav and Ike!Single Point of Staff Failure – Transitioned
Responsibilities Quickly …Equipment Inventory, To Include Network and
Data Center Backup and Recovery Plans Needed to Be Transitioned ON THE FLY! (New Role of Onsite Lead)
Note on Vehicle Planning: Golf Carts Proved Critical After Storm
![Page 38: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/38.jpg)
Power and Fuel: Handled By Facilities ManagementStaff Continuity Maintained - Gustav to Ike Communication with Local Fuel VendorsCommunication and Coordination With Power
CompaniesEngine/Genset (portable and permanent)Fuel TanksBatteriesRectifiersStaging and Deployment
![Page 39: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/39.jpg)
Network ServicePreventative Measures:
No Redundant Routes for Network (Big Exposure) LEARN Connectivity Initiative Q 1 of 2009Half-Completed VoIP Migration (Half Cisco/Half
Avaya)Priority Areas:
Network Services Spread Between Windows Services Group (DHCP) and Unix Group (DNS).
Failure to Standardize Led to Confusion and LAN Service Outages During the Storm (Premature AD Shutdown).
Failure to Document All Services Creates Risk
![Page 40: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/40.jpg)
Significant Telecomm EventsGOT LUCKY -- LU OC-3 Circuit to Houston Stayed Up
Throughout Event – NO INTERNET CONNECTIVITY OUTAGE (Gigaman to Orange Also Stayed Up)
Onsite Team Kept Campus Services Running Until Night of Landfall (Sept. 12-13)
Throughout Storm VoIP access via WAN using SIP Phones (Ordered More Broadband Cards After Storm)
Cell Service Unreliable Beaumont to Remote SitesResidential lines out for over several weeksSatellite Phones Proved Difficult to Use
![Page 41: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/41.jpg)
Main Computer Center and Key Telecommunications Hubs Success Story: No Systems Shutdown
Sequence in Place For Data Centers. Planning Initiated in Gustav Continued Into Ike Lead Up
Downside to Not Running Your Own Power:Power Failure Exposure Due to Unmet Power
Requirements in Data CenterConcerns re: Lack of Fuel Led Local Team to Bring
Down Systems After EventTape Backups: Contract for Off Site Had Just
Lapsed! Can You Sing “Do the Hustle”?Wind and High Water: Failure to Stock Pile
Sandbags
![Page 42: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/42.jpg)
![Page 43: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/43.jpg)
Long Term Preparations Maintain Year-Round
Example of Successful Planning: Replication of Website and Content Management Service
Criticality of ERP Disaster Recovery/BC Site at San Marcos
Hardware In Place in San Marcos Software to Be Configured With Data Replication (Tested)
By Hurricane Season 2009Long-Term Plans/Prevention
Move to the Cloud: Portal and Mail Outsourcing Initiatives Importance of Service Continuity: Redesign of Network
Services (Standardization Under Network Services Group)
![Page 44: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/44.jpg)
![Page 45: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/45.jpg)
![Page 46: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/46.jpg)
Post-Storm Activities Led By Onsite Team Lead Employee SafetyContactsConference BridgesCostsTemporary Lodging/ShelterExtra Work ForceSafetyEmergency Supply AvailabilityPower and Fuel
![Page 47: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/47.jpg)
![Page 48: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/48.jpg)
Flooding in Low-Lying Areas
![Page 49: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/49.jpg)
![Page 50: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/50.jpg)
Damage to People’s Homes
![Page 51: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/51.jpg)
Damage to Local Businesses
![Page 52: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/52.jpg)
Damage Assessment and Service Restoration Announcement Via ConnectED on Return of
Staff to Campus (September 24th), Return of Students to Campus (September 25th)
Class Schedule Readjusted by Office of the Provost, With Resumption of Classes on September 25th
President’s Faculty/Staff Convocation on September 25th
![Page 53: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/53.jpg)
![Page 54: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/54.jpg)
Perspectives From Galveston
Steven Conway
Texas A&M University at Galveston (TAMUG)
![Page 55: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/55.jpg)
Hurricane Ike TAMUG evacuated campus 10 Sept 2008Hurricane Ike landfall 12 Sept 200875% of Galveston was floodedTAMUG did not flood, 2 buildings had roof
damageTAMUG restarted classes at TAMU in College
Station TX Wed 24 Sept91% of the students returned20 Jan 2008 Classes resume in Galveston
![Page 56: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/56.jpg)
Significant Telecomm EventsCell service stayed up throughout storm-Cat 2UTMB Gigaman Circuit to Houston stayed upTAMUG-UTMB Gigaman available 24 Sept
when power returned on campusT1’s out for over a monthResidential lines out for over 6 weeksVoIP access to College Station via WAN
![Page 57: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/57.jpg)
Hurricane Ike Flooding Galveston
A house burns uncontrolled in a flooded neighborhood as Hurricane Ike approaches the Texas coast, Friday, Sept. 12, 2008, in Galveston, Texas. (AP Photo/Matt Slocum)
![Page 58: LU - ACUTA 2009 Conference Presentation](https://reader037.fdocuments.us/reader037/viewer/2022110115/548aa4a2b47959cb4c8b4580/html5/thumbnails/58.jpg)
TAMUG Boat on Causeway