Document Change Control - Harvard University Information...

HUIT MAJOR INCIDENT PROCESS

Incident Commander Manual

Version 3.1

November 2014

Matthew Wollman

This page left intentionally blank.

HUIT Incident Commander Manual

Document Change ControlVersion # Date of Issue Author(s) Brief Description

1.0 03/25/2013 Matthew WollmanJanet Crystal

Initial Draft of Document

2.0 09/09/2013 Matthew Wollman Updated instructions for:

RingCentral ServiceNow HUIT Status Dashboard

3.0 Chris Nadeau Updated Instructions for:

Documenting a Major Incident Ticket In ServiceNow

NetContacts External Communication Updating a Ticket in ServiceNow Update Communications – Internal Update Communications – External Resolve a Major Incident Ticket Resolve Communications – Internal Resolve Communications – External

3.1 11/02/2014 Matthew Wollman Updated:

Incident Commander Responsibilities and Escalation

Service Status Dashboard Screenshots and instructions

Appendices Grasshopper instructions


Table of ContentsDocument Change Control.......................................................................................................................................3

Incident Management and Major Incidents.............................................................................................................9

Incident Commander Responsibilities and Escalation..............................................................................................9

Incident Commander Responsibilities..................................................................................................................9

Incident Commander Escalation.........................................................................................................................10

Incident Communicator..........................................................................................................................................10

Coordinating Resources......................................................................................................................................10

Major Incident Initial Procedures...........................................................................................................................13

Major Incident Activities.....................................................................................................................................13

Suspected Major Incident Identification.........................................................................................................13

Initial Communication and Escalation............................................................................................................13

Incident Documentation.........................................................................................................................................14

Documenting a Major Incident Ticket in ServiceNow.........................................................................................14

Create a Major Incident Ticket.......................................................................................................................14

Populate a Major Incident Tab.......................................................................................................................15

Required Fields for a Major Incident Form.....................................................................................................15

Communications....................................................................................................................................................17

Internal and External Communication....................................................................................................................17

Communicating a Major Incident.......................................................................................................................17

Internal Communications–INITIAL..........................................................................................................................18

Initial Internal E-mail Communication–ALL MAJOR INCIDENTS..........................................................................18

Initial Internal Communication–CONFERENCE BRIDGE......................................................................................19

External Communications–INITIAL.........................................................................................................................20

Initial Communication via Netcontacts e-mail–FAS NETWORK IMPACT ONLY...................................................20

Initial Communication via the HUIT Website–EXTERNAL IMPACT ONLY............................................................22

Initial Communication via Social Media–EXTERNAL IMPACT ONLY....................................................................26

ServiceNow Ticket Update.....................................................................................................................................29

Update a Major Incident Ticket..........................................................................................................................29

Update Communications–INTERNAL......................................................................................................................30


Internal Update Communication via E-mail–ALL MAJOR INCIDENTS..................................................................30

Update Communications–EXTERNAL.....................................................................................................................31

Update Communication via the FAS Netcontacts email–EXTERNAL IMPACT ONLY............................................31

Update Communication via the HUIT Website –EXTERNAL IMPACT ONLY........................................................32

ServiceNow Ticket Resolution................................................................................................................................35

Resolve a Major Incident Ticket.........................................................................................................................35

Resolution Notification (Update an Existing Ticket)........................................................................................35

Resolved Upon Notification (Create a New Ticket).........................................................................................35

Complete a Major Incident Form.......................................................................................................................36

Resolution Communications–INTERNAL.................................................................................................................37

Resolution Internal Email Communication–ALL MAJOR INCIDENTS...................................................................37

Resolution Communications–EXTERNAL................................................................................................................38

Resolution Communication via the FAS Netcontacts Email–FAS NETWORK IMPACT ONLY...............................38

Resolution Communication via Social Media–EXTERNAL IMPACT ONLY............................................................39

Resolved Upon Notification................................................................................................................................40

Resolution Communication via the HUIT Website–EXTERNAL IMPACT ONLY....................................................40

Resolution Notification...................................................................................................................................40

Resolved Upon Notification................................................................................................................................42

After Hours Responsibilities...............................................................................................................................45

On-call Equipment..............................................................................................................................................45

Compensation....................................................................................................................................................45

Grasshopper Call Forwarding.............................................................................................................................46

Configure On-call Forwarding.............................................................................................................................46

Manual Notifications..........................................................................................................................................49

Type: NEW INCIDENT.........................................................................................................................................49

Template: NEW INCIDENT..................................................................................................................................49

Type: UPDATE.....................................................................................................................................................50

Template: UPDATE.............................................................................................................................................50

Type: RESOLUTION OR RESOLVED UPON NOTIFICATION...................................................................................51

Template: RESOLUTION......................................................................................................................................51

External Notification Templates.........................................................................................................................52



Type: NEW INCIDENT.........................................................................................................................................52

Template: NEW INCIDENT..................................................................................................................................52

Type: RESOLUTION.............................................................................................................................................53

Template: RESOLUTION......................................................................................................................................53

Type: RESOLVED UPON NOTIFICATION.............................................................................................................54

Template: RESOLVED UPON NOTIFICATION.......................................................................................................54

Emergency Changes...........................................................................................................................................55

Emergency Change Notification–EXTERNAL IMPACT.........................................................................................55

Emergency Change Notification–NO EXTERNAL IMPACT...................................................................................55

After-Hours Emergency Change Template.........................................................................................................56


Template: AFTER-HOURS EMERGENCY CHANGE................................................................................................56


Incident Management and Major IncidentsThe purpose of the Harvard University Information Technology (HUIT) Major Incident Process is to provide a unified system for resolving Major Incidents as quickly as possible through proper identification, predefined escalation paths, and prompt communication procedures across all of its services.

A Major Incident is defined as the interruption or degradation of a core production service (any centralized HUIT-provided service that serves multiple customers and users) that results in the disruption of its customers’ ability to carry out University teaching, learning, research, and/or administration.

The goals of a Major Incident process are to:

Minimize the impact of service interruptions to the HUIT user community and its business operations Ensure that service interruptions receive immediate attention and resources for service restoration Ensure that customers are well-informed of service interruptions, degradations, and resolutions

Incident Commander Responsibilities and Escalation

Incident Commander ResponsibilitiesThe Incident Commander has the highest level of responsibility during a Major Incident and is accountable for its lifecycle through coordination, documentation, and communication. The roles of the HUIT Incident Commander and the HUIT Incident Communicator may be combined for incidents that are of short duration or which are deemed less critical. For incidents of longer duration or those with greater impact, the responsibility is escalated to Manager or Director.

The Incident Commander is responsible for the following:

Participating in and facilitating in a Conference Bridge Maintaining communication with Technical Resources and Service Owners for status updates and additional

information Coordinating resources needed to troubleshoot, communicate, and/or make decisions to resolve a Major

Incident Ensuring that internal and external communications about a Major Incident are completed in a timely

manner Creating a Problem record.


Incident Commander EscalationDepending on the scale of the event, the Incident Commander role may require further escalation to a HUIT Manager or Director. While the Incident Communicator role always remains with the original Incident Commander, the criteria for escalating above the Incident Commander Role are guided by any individual or combination of the following conditions:

1. A Major Incident is one of the Critical Services listed in Appendix D in this document.2. A Major Incident is significant enough to affect over 1000 users of one or more services.3. A Major Incident is not or cannot be resolved within four hours.

Incident CommunicatorThe Incident Communicator is in charge of documentation and communication during a Major Incident, both internally to HUIT and externally to customers and end-users. The roles of the HUIT Incident Commander and the HUIT Incident Communicator may be combined for incidents that are of short duration or which are less critical. For incidents of longer duration or those with greater impact, the responsibility should be escalated to Manager or Director.

The Incident Communicator is responsible for the following:

Participating in a Conference Bridge Communicating internally to HUIT and externally to the website and subscribed users on Twitter and

Facebook. Maintaining a record of events throughout a Major Incident Notifying HUIT and any external parties of the resolution of the Major Incident.

Coordinating ResourcesThe Incident Commander is responsible for escalating incidents to Technical Resources and Service Owners / Practice Managers. To ensure prompt response, escalations are initially handled by phone calls to each group’s on-call number. Leave a message, if the on-call staff does not answer the phone. Your call should be returned within 30 minutes. If not, the Incident Commander should escalate to their manager.

NOTE: The current on-call phone list is kept at https://harvard.service-now.com/nav_to.do?uri=/cms/Contact+Info.do and is available on the wallet card.


https://harvard.service-now.com/nav_to.do?uri=/cms/Contact+Info.do

Major Incident Initial Procedures


Major Incident Initial Procedures

Major Incident ActivitiesThe Incident Commander has the overall responsibility for the following functions:

Major Incident Identification Initial Communication and Escalation Incident Coordination Conference Bridge Communication External Communication Internal Communication Investigation Resolution Documentation

Suspected Major Incident IdentificationAnyone on the HUIT staff can notify the Incident Commander of a suspected Major Incident. Survey the environment for indications that a Major Incident has occurred, such as an increase in calls reporting similar symptoms or the ability to duplicate the symptoms. Contact Technical Resources for additional indicators. Prioritize the event based on the following questions:

How many users, groups, departments, or schools are affected? Is the application or service critical at this time of year? What is the business impact of the outage? Can people still work? What can’t they do?

Initial Communication and EscalationEscalate to the on-call contact for the group responsible for the service or application affected by the Major Incident:

Determine and confirm the scope of a Major Incident For network interruptions, determine and confirm which buildings and regions are affected Ensure that appropriate technical resources are allocated Establish the method and schedule for receiving updates from the technical teamEstablish a Conference Bridge, if necessary – page 19


Incident Documentation

Documenting a Major Incident Ticket in ServiceNowDocumenting a Major Incident Ticket in ServiceNow involves two steps:

Creating a Major Incident Ticket Populating a Major Incident Form

Create a Major Incident Ticket1. Open ServiceNow and create a new Incident from the Incident menu.2. Enter information into the required fields.3. The required fields will have a red indicator next to them and turn green after being filled out.

The following table shows the required fields in the Main Window:

Field Name Usage Notes

Customer Enter Your Name as the customer of the

Service The HUIT service or Application most affected by the Major Incident, for instance “ASPerIN”, Choose the most appropriate one

Category Should be troubleshooting, or interruption of service if available

Assignment Group Assign to your primary group

Assigned To Assign to your self

Short Description Should be <name of service> Interruption or Degradation

Description Description of Initial Major Incident Report, who reported, what time, details, etc

4. The Customer should always be the current Incident Commander. Enter the Incident Commander HUID and hit enter for ServiceNow to populate the rest of the customer Fields.

5. Enter the Service affected by the Major Incident. If there are multiple choose the most appropriate one.6. Enter the Category; it should be troubleshooting, or interruption of service if available7. In the Assignment Field, assign the ticket to yourself as the current Incident Commander.


8. Enter the Short Description only as the name of the service and Interruption or Degradation, i.e. <name of service> Interruption or degradation

9. Enter details abou the Major incident in the Description field.10. Save the ServiceNow ticket.

NOTE: Keep a ServiceNow ticket open as long as a Major Incident remains unresolved.

Populate a Major Incident Tab1. The Major Incident tab is located next to the “Notes” and “Resolution” tabs.NOTE: The Major Incident Tab will only show up once the Impact and Urgency equals a Major Incident Priority.

Required Fields for a Major Incident FormFor a Major Incident, the Incident Commander is responsible for ensuring that all required fields in the ServiceNow ticket and a Major Incident Form are populated correctly. At the time of creation, only fields in the Summary Tab are necessary.


MI State State of the Major Incident (Alert “New”, Update, Resolved)

Services Impacted The HUIT services or Applications affected by the Major Incident, for instance “ASPerIN”

Groups Involved Technical teams involved in the investigation and resolution of a Major Incident

Owner Group Technical team that owns a Major Incident and is responsible for its resolution

External Vendor

Locations Impacted List of locations affected by the Major Incident

Customer / Business Impact

Detailed description of impact to known partners and business processes

Current Actions The actions taken by the technical resources to investigate / resolve the incident.

Bridge Code on New/Update

Bridge Code will auto populate and be displayed in any New or Update alert sent

Time Reported Date and time that a Major Incident is reported to the Incident Manager

Service Outage Time Confirmed state date and time

Service Restoration Time Confirmed end date and time


Major Incident Duration Length of time for a Major Incident in hours and minutes

2. Fill out the required fields in the Summary section.

3. Click the Save button in the upper right corner.


CommunicationsAfter the initial creation of the ServiceNow ticket, as the Incident Commander, you are responsible for immediately communicating a Major Incident.

Internal and External Communication

Communicating a Major IncidentFor an internal Major Incident, use email to communicate with HUIT staff. These emails require a significant level of technical detail. Enter all communications activity in the Work Info entry field of the ServiceNow ticket.

For an external Major Incident–one that affects outside services and/or organizations–use the HUIT website e-mail external distribution lists to communicate with HUIT staff. These emails require less technical detail than required by the internal notification.

You may also need to create an ACD posting. If a Major Incident does not have any user-facing impact (i.e., Jira) or if a Major Incident is resolved upon notification, posting to the web site and sending external email is not required.

Communications include both external and internal:

Initial communications Update communications Resolution communications


Internal Communications–INITIAL

Initial Internal E-mail Communication–ALL MAJOR INCIDENTSIn your Major Incident Ticket in ServiceNow use the “Email the customer” function in ServiceNow to communicate all Major Incident to HUIT staff and Student Computing Technicians regardless of the impact.

1. From the Major Incident Tab, select “Alert” for the MI State. 2. Open the “Email the Customer” function in ServiceNow. The necessary data from the Major Incident Tab will

be automatically entered into the form. The To address will also automatically have the correct address ([email protected]).

3. Review the text and make any necessary corrections in the Major Incident Summary section.4. Select Send in the top right of the ServiceNow email window.


Initial Internal Communication–CONFERENCE BRIDGEDistribute the participant code using a HUIT alert

Call (866) 890-3820 and enter one of the leader codes listed below:

Primary Bridge: Leader Code: 71284835# Participant: 38793366#

Secondary Bridge: Leader code 58775889# Participant code: 86065154#

The agenda should include at a minimum:

Confirming that the correct personnel are involved and escalating to additional personnel as necessary Briefing all participants on the impact, symptoms, and current actions in progress to resolve the incident Discussing the update schedule with bridge attendees

Additional phone functions available during a conference call are listed in the following table:

Conference Bridge Options

Who Key Description

Anyone *0 Operator Assistance

Leader *2 Begin/End conference recording

Anyone *6 Mute / Unmute your line

Leader *7 Lock/Unlock your conference

Leader 72# Roll Call

Leader 81# Mute all Lines

Leader 80# Unmute all lines


External Communications–INITIAL

Initial Communication via Netcontacts e-mail–FAS NETWORK IMPACT ONLYCommunicate all Major Incidents that affect FAS network services and/or customers using FAS Webmail.

1. Log into webmail.fas.harvard.edu using “cts” as a username and the cts password.2. Once logged in select “Folders” then “Incident Management Templates”.3. Select one of the following templates depending on the current Major Incident State.

a. SERVICE Interrupted Degraded – This template can be used for either new communications or updates.

b. SERVICE Restored – Resolution email to be sent at the end of a Major Incident.c. RESOLVED UPON NOTIFICATION [Service Name] Restored – Resolution email to be used to

report a Major Incident that was resolved upon notification to Incident Management. 4. Fill in the required fields of the notification email.5. In the To field, enter all the appropriate e-mail distribution list addresses in the To field. Separate

multiple e-mail addresses using commas. Always include [email protected] in the blind copy (bcc) field. See below for available addresses. ( See addresses in the table below)

The following fields that appear in green are required fields:

External Communications

Field Name Description

Subject

This is the short description that appears on the HUIT website and as the Subject of the external e-mail. Use the help text in the email templates for information on the correct format for filling in this field. It must correctly reflect whether it is an "Interruption" or “Degradation” (initial notification) or that service has been "Restored" (resolution notification).

Beginning Use the correct date and time format (military time).

Expected Resolution

Fill this in only if the information is available.

Ending If the end date/time is known. Will only be used for Resolution emails.

Services or Regions Affected

The format is displayed in the email templates. Include the Service description from SNOW (in the Extended Service Info screen) if available.

Summary Information in this field is displayed on the website for the event detail. Use the sample template located below the field when creating your message.

Contact Information

Pre-populated in the templates.


Major Incident E-mail Distribution Lists (External Impact Only)

List Purpose

[email protected] Services interruptions that are not location-specific and for any campus-wide network issues

[email protected] Network interruptions in the North Precinct region

[email protected] Network interruptions in the Northwest region

[email protected] Network interruptions in the Yard region

[email protected] Network interruptions in the River region


https://www.fas-it.fas.harvard.edu/admin

mailto:[email protected]






Initial Communication via the HUIT Website–EXTERNAL IMPACT ONLY1. Log in to the HUIT website administration page using http://services.huit.harvard.edu/hugate.

2. In the upper left corner, look for a wrench icon. Click on the icon to access the Administration Menu.


http://services.huit.harvard.edu/hugate

3. In the Administration Menu, click on the Add content button.

4. Select Service Status Updates.


The following table lists the fields needed to complete a Service Alert including a brief description. Required fields are highlighted in green:

External Communications

Field Name Description

Alert Name This is the overall description of the incident for which an alert is being posted, such as “ICEMail Interruption”

Update Title This is the title of the posting. For a Yellow posting this should be a factual description of what users or the service desk are reporting, such as “Users report

Category Corresponds to the Parent Service and Service as seen in ServiceNow. Click the drop-down and choose the Parent service. In this case it will be Exchange

Active Is the incident currently happening? Accept the default of Yes.Status Set this to Red if there is a current Major Incident. Set this to Yellow if we are

unsure and are hearing reports of an outage, or we want to place a cautionary What This is an expanded description of the incident. For a Yellow incident this will be a

factual statement of what users are reporting. ExampleWho Describe which population of users is affected. You may not be able to determine

this at the time and you may leave it blank.When Enter the time when the first report was received.

Where If a particular area is affected, describe it here. You may not be able to determine this at the time and you may leave it blank.

Why If you have an easy way to explain why in user-facing language you may enter it here; it is ok to leave blank if we don’t know why.

Status Describe any steps being taken towards resolving the issue. This would include “HUIT Staff are investigating the issue” or “HUIT Staff are working to restore

Further details Any additional information you may have. Example


5. Click Save.


Initial Communication via Social Media–EXTERNAL IMPACT ONLY1. Go to http://twitter.com and login using the Compose New Tweet box or the Compose New Tweet

button in the upper right corner. Ask a member of the ITSM team for the password.

2. Cut and paste the subject from the HUIT web posting and the URL directly into the web posting.

3. Click the Tweet button. The Tweet will automatically post to the HUIT Facebook Page (https://www.facebook.com/HarvardUniversityIT)

4.


https://www.facebook.com/HarvardUniversityIT

Major Incident Update Procedures


ServiceNow Ticket UpdateIncident Commanders should be updating the ServiceNow ticket in real-time, with who is on the conference bridge, what is discussed, decisions made, troubleshooting conducted, etc.

Update a Major Incident TicketUpdate a Major Incident Ticket as often as necessary to reflect changes in the progress of a Major Incident.

1. On the Notes tab, document all activities relative to resolving a Major Incident.


Update Communications–INTERNAL

Internal Update Communication via E-mail–ALL MAJOR INCIDENTSSend an internal update by email to huit-alerts in the following two situations:

A significant change has been identified in the status, or impact of a Major Incident. A Major Incident remains unresolved at the end a business day.

1. Send updates using the ServiceNow Major Incident Tab.

2. In the Update Text field, fill in the update information. Make changes to any fields on the Summary tab, as necessary.

3. In the MI State Field, select the Update the drop-down list.4. Open the “Email the Customer” function in ServiceNow. The necessary data from the Major Incident Tab will

be automatically entered into the form. The To address will also automatically have the correct address ([email protected]).

5. Review the text and make any necessary corrections in the Major Incident Summary section.6. Select Send in the top right of the ServiceNow emailer.7.


Update Communications–EXTERNAL

Update Communication via the FAS Netcontacts email–EXTERNAL IMPACT ONLYExternal updates to a Major Incident communication are provided only when a significant change has been identified in the status, or impact of a Major Incident.

1. Log into webmail.fas.harvard.edu using “cts” as a username and the cts password.2. Once logged in select “Folders” then “Incident Management Templates”.3. Select one of the “SERVICE Interrupted Degraded” templates. 4. Update the required fields of the notification email.5. In the To field, enter all the appropriate e-mail distribution list addresses in the To field. Separate

multiple e-mail addresses using commas. Always include [email protected] in the blind copy (bcc) field. See below for

available addresses. (See addresses in the table below)



List Purpose













Update Communication via the HUIT Website –EXTERNAL IMPACT ONLYUpdate the HUIT System Status Dashboard, frequently during the event, preferably once every hour.

1. Log in to the HUIT Administration screen using https://services.huit.harvard.edu/hugate.2. Go to the HUIT system status dashboard Home Page using

https://services.huit.harvard.edu/status-dashboard.3. Select the correct event in the Service Impacted section.4. Select the Add Alert Update link in the left hand menu

5. Update the information as needed.6. Click on Save.


https://services.huit.harvard.edu/status-dashboard

https://services.huit.harvard.edu/hugate

Major Incident Resolution Procedures


ServiceNow Ticket Resolution

Resolve a Major Incident TicketResolve a Major Incident Ticket only after the Incident Commander has confirmed its resolution with On-call Service Owner.

A Major Incident is resolved in one of two ways:

Resolution Notification: Updating an existing ticket when a service interruption is resolved after an initial notification is sent

Resolved Upon Notification: Creating a new ticket when a service interruption is resolved before an initial notification is sent

Resolution Notification (Update an Existing Ticket)1. Open the ServiceNow ticket2. Change the Status to Resolved and populate the remaining required fields.

Resolved Upon Notification (Create a New Ticket)Create a new ticket using the instructions under ServiceNow Ticket Creation on page 13, and then follow the instructions under Resolve a Major Incident Ticket.


Complete a Major Incident FormUpdate a Major Incident Form to reflect that the Major Incident is resolved.

1. Open the existing Major Incident Form from the Major Incident tab.2. Populate resolution fields on the Update/Resolution Details tab.

NOTE: After the resolution of a Major Incident, the Incident Commander must populate all required fields.

The following table shows the required fields:


Resolution Actions Actions performed by the technical group to resolve a Major Incident

Root Cause Analysis Technical group’s current understanding of the cause of a Major Incident

Root Cause Select the closest option in the dropdown for Root Cause

Next Steps Next steps will auto populate.

3. Click the Save button in the upper right corner.


Resolution Communications–INTERNAL

Resolution Internal Email Communication–ALL MAJOR INCIDENTSAfter updating the fields, go to the Email a Customer function.

1. Select Resolved for the MI State

2. Open the “Email the Customer” function in ServiceNow. The necessary data from the Major Incident Tab will be automatically entered into the form. The To address will also automatically have the correct address ([email protected]).

3. Review the text and make any necessary corrections in the Major Incident Summary section.4. Select Send in the top right of the ServiceNow emailer.5.


Resolution Communications–EXTERNAL

Resolution Communication via the FAS Netcontacts Email–FAS NETWORK IMPACT ONLYThe following fields are required:

1. Log into webmail.fas.harvard.edu using “cts” as a username and the cts password.6. Once logged in select “Folders” then “Incident Management Templates”.7. Select one of the “SERVICE Restored” template. 8. Update the required fields of the notification email.9. In the To field, enter all the appropriate e-mail distribution list addresses in the To field. Separate

multiple e-mail addresses using commas. Always include [email protected] in the blind copy (bcc) field. See below for available addresses. (See addresses in the table below)



List Purpose













Resolution Communication via Social Media–EXTERNAL IMPACT ONLY1. Go to http://twitter.com and login using the Compose New Tweet box or the Compose New Tweet button in

the upper right corner. Ask a member of the ITSM team for the password.

2. Cut and paste the subject from the HUIT web posting and the URL directly into the web posting.

3. Click the Tweet button. The Tweet will automatically post to the HUIT Facebook Page (https://www.facebook.com/HarvardUniversityIT).


Resolved Upon NotificationFollow the instructions on page 18 for posting an Initial Notification. Select the correct helper text to reflect that the incident has been resolved.

Resolution Communication via the HUIT Website–EXTERNAL IMPACT ONLY

Resolution Notification1. Log in to the HUIT administration screen using https://services.huit.harvard.edu/hugate.2. Go to the HUIT home page using https://services.huit.harvard.edu/status-dashboard3. Select the correct event in the Service Impacted section.4. Select the Add Alert Update link in the left hand menu

5. Change/update the Update Title from <Service Name> Interruption to <Service Name> Restored.6. Change the Active field to No.


https://services.huit.harvard.edu/status-dashboard

https://services.huit.harvard.edu/hugate

7. Change Status to Green


8. Use on-screen example text to update the What field.9. Update the correct Ending date/time.

10. Click on the Save button at the bottom of the screen.

Resolved Upon NotificationFollow the instructions for posting an Initial Notification, found on page 18. Select the correct helper text to reflect that the incident has been resolved.


Appendices


Appendix A: Business Hours and After Hours Major Incident Management

Business Hours vs. After Hours

Major Incident Management duties are handled differently during Business Hours and After Hours.

During Business Hours (between 9:00 a.m. and 5:00 Monday through Friday), the Incident Management Process Manager responds to Major Incidents.

During After Hours (between 5:00 p.m. and 9:00 a.m. Monday through Friday and all weekends and University holidays), a rotation of on-call HUIT personnel responds to Major Incidents.

After Hours ResponsibilitiesThe After Hours on-call week begins and ends on Friday at 5:00 p.m. After Hours Incident Management is scheduled on a weekly basis. The rotation of personnel is maintained in the Google Apps shared calendar called “HUIT Incident Commander Calendar.”

The active After Hours on-call individual is responsible for changing the Incident Management forwarding number to the next scheduled individual via Grasshopper when their on-call rotation ends on Friday. See Appendix B for Grasshopper usage instructions.

If you want to switch on-call periods, you must contact the ISRM Process Owner (or if unavailable, a member of the ITSM team) before you rotation schedule so that the on-call calendar can be updated.

On-call EquipmentHave the following equipment available and ready for your After Hours rotation:

Laptop Verizon Wireless Internet Hotspot (If you do not have one, see the Incident Management Process Manager

for a loaner) On-call reference card Incident Commander Reference Manual (this document)

All on-call personnel should save the on-call numbers for all HUIT groups in their phones.

CompensationBased on the HUIT departmental on-call policy, non-exempt employees are compensated $125 per week for After Hours on-call duties. Exempt employees are equally reimbursed with a salary stipend. Reimbursement forms and submission instructions are available from the HUIT Human Resources staff.


Appendix B: Grasshopper Instructions

Grasshopper Call ForwardingGrasshopper is a web-based call-forwarding application that allows all participants in the Incident Management rotation to receive calls directly to their cell phone. The active on-call individual is responsible for modifying the Incident Management forwarding number to the next scheduled individual via Grasshopper when their on-call weekends on Friday at 5p.m.

Configure On-call Forwarding1. Log into http://grasshopper.com (see Incident Management Process Manager for credentials)2. Click on Settings, located on the menu bar.

3. Click on the Call Forwarding button in the right-hand menu.

4. Click on the Red dot to the right of your number to turn yourself on; click on the green dot to turn off the previous Incident commander.

5. Log out.


http://grasshopper.com/

Appendix C: On-Call Contact Information

All HUIT technical teams have dedicated contact numbers for 24x7 Major Incident response. During Business Hours, your call should be returned within 15 minutes. During After Hours, the technical group on-call staff should respond to your page within 30 minutes. If the on-call staff does not return your call within the designated time, the Incident Commander will escalate the Major Incident to the appropriate on-call Technical Resource Manager. All on-call personnel should save the on-call numbers for all HUIT groups in their phones.

The list of on-call contact numbers is available and kept up-to-date in the following locations:

At https://harvard.service-now.com/cms/Contact+Info.do. This URL can be bookmarked on most smartphones

In ServiceNow, when you open any ticket, under the Quick Links on the left hand side, look for is a link called Escalation and On-Call List

Appendix D:


https://harvard.service-now.com/cms/Contact+Info.do

Appendix E: Critical Services

1. Central Networking Servicesa. DNS, DHCP, Infobloxb. Core / Data Center Routersc. Core / Data Center Firewallsd. Load Balancer

2. Data Centera. Facilities / Powerb. Shared Storagec. Virtualization

3. Email4. PIN / LDAP5. University website6. College website7. Phone System / Voicemail / i38. PeopleSoft9. Oracle Financials10. HarvIE11. CAADS12. iSites13. C-Cure / Physical Door Access Systems


Appendix F: Internal Notification Templates

Manual NotificationsIn the event that ServiceNow is not available, send manual internal e-mail notifications from the [email protected] account, using Unix Webmail at https://webmail.fas.harvard.edu.

Send internal notifications to [email protected]. Use the following template(s) for internal notifications. These templates are also located in the Templates folder of the [email protected] account.

Type: NEW INCIDENTUse this template for INTIAL notification of a Major Incident, unless the Incident is already resolved. For Incidents resolved prior to Initial Notification, use the “Resolved upon Notification” template.

Template: NEW INCIDENTFROM: Harvard University Information Technology ([email protected])

TO: [email protected]

SUBJECT: <Name of Service> Interruption OR Degradation

Current Status: New

-- Incident Summary --

Customer Impact:

Time Reported:

Incident Start Date/Time:

Services Affected:

Technical Group(s) Involved:

Service Owner Group:

Major Incident Conference Bridge: 866-890-3820 38793366#


https://webmail.fas.harvard.edu/


-- Technical Detail --

Diagnosis:

Current Actions:

[Your Name]

HUIT Incident Management

617-496-2831


Type: UPDATEUse this template for internal UPDATE notification when an initial notification has already been sent. If a Major Incident has been resolved before initial notification has been sent, use the Resolved upon Notification template instead.

Template: UPDATEFROM: Harvard University Information Technology ([email protected])

TO: [email protected]

SUBJECT: UPDATE: <Name of Service> Interruption OR Degradation

Current Status: In Progress

Update:

-- Incident Summary --

Customer Impact:

Time Reported:

Incident Start Date/Time:

Services Affected:

Technical Group(s) Involved:

Service Owner Group:

Major Incident Conference Bridge: 866-890-3820 38793366#

-- Technical Detail --


Diagnosis:

Current Actions:

[Your Name]

HUIT Incident Management

617-496-2831


Type: RESOLUTION OR RESOLVED UPON NOTIFICATIONUse this template for internal RESOLUTION notification.

Template: RESOLUTIONFROM: Harvard University Information Technology ([email protected])TO: [email protected]: <Name of Service> Interruption OR Degradation

Current Status: Resolved

-- Incident Summary --Customer Impact:

Time Reported:Incident Start Date/Time:Incident End Date/Time:Duration: # Hours # Minutes

Services Affected:

Technical Group(s) Involved:Service Owner Group:

-- Technical Detail --Diagnosis:Resolution Actions:Root Cause:Next Steps (if applicable):

[Your Name]HUIT Incident Management617-496-2831


External Notification Templates

Manual NotificationsIn the event that the System Status web tool is not available, send external email notifications manually from the [email protected] account, via Unix Webmail at https://webmail.fas.harvard.edu.

External notifications are sent to the appropriate mailing list, as documented on page 20. The following templates for external notification are also located in the Templates folder of the [email protected] account.

Type: NEW INCIDENTUse this template for INITIAL notification of a Major Incident, unless the incident is already resolved. For incidents resolved before an initial notification, use the Resolved upon Notification template.

Template: NEW INCIDENTFROM: Harvard University Information Technology ([email protected])

TO: [external distribution list]

SUBJECT: [Service Name] Interruption OR Degradation

Beginning: [Date/Time]

Services or Regions Affected: [Service(s) Name]

Summary

Access to [service name, the services above] is currently [degraded, interrupted]. Those who are trying to connect to [the network in [Building Name or Regional Area], specific applications, email clients, websites] will be unable to do so at this time. HUIT technical staff are actively [investigating the problem, working to restore service].

Contact Information

Please contact HUIT Support at 617-495-9000 or by e-mail at [email protected] with any questions or concerns (Monday through-Friday: 9 a.m.-8 p.m., Saturday: 10 a.m.-6 p.m., Sunday: 12 p.m.-5 p.m.).




Sincerely,

Harvard University Information Technology


Type: RESOLUTIONUse this template for external RESOLUTION notification when an initial notification has already been sent. If a Major Incident has been resolved before initial notification has been sent, use the Resolved upon Notification template instead.

Template: RESOLUTIONFROM: Harvard University Information Technology ([email protected])


SUBJECT: [Service Name] Restored


Ending: [Date/Time]

Services or Regions Affected: [Service Name]

Summary

[HUIT, UNSG] has restored [service name, the services above] to normal operation as of [End Time]. Those who wish to connect to [the network in [Building Name or Regional Area], specific applications, email clients, websites] are once again able to do so. We regret any inconvenience this interruption may have caused.

Contact Information


Sincerely,



Type: RESOLVED UPON NOTIFICATIONUse this template for external RESOLUTION notification when a Major Incident has been resolved before an initial notification is sent.

Template: RESOLVED UPON NOTIFICATIONFROM: Harvard University Information Technology ([email protected])


SUBJECT: [Service Name] Restored


Ending: [Date/Time]


Summary

Access to [service name, the services above] was interrupted from [Start Time] to [End Time]. Those who were trying to connect to [the network in [Building Name or Regional Area], specific applications, email clients, websites] were unable to do so during this time.

[HUIT, UNSG] technical staff have now restored [service name, the services above] to normal operation. We regret any inconvenience this interruption may have caused.

Contact Information


Sincerely,



Appendix G: After Hours Emergency Changes

Emergency ChangesThe After Hours Incident Commander is also responsible for notification of After Hours emergency (Immediate) changes. In the event of an after-hours immediate change, the technical group representative (Change Implementer) will call the on-call Incident Commander.

The change implementer will provide the following information:

What work is being performed? Has the director of the technical group given approval? What impact will the change(s) have on services? Will any services be interrupted? When will the work begin and end?

NOTE: The on-call Incident Commander is responsible for communications only, not approval. The HUIT Change Management process allows for director approval (tacit or implicit) of immediate emergency changes after hours.

Emergency Change Notification–EXTERNAL IMPACTIf the change will have an external impact:

Post web notification and send e-mail using the Major Incident Initial Communications instructions on page 17.

The web System Status posting tool includes Subject and Summary templates for Emergency Change/Maintenance.

Use the Email Options section to send email notification. This is a single email to all appropriate distribution lists. Put the external distribution list in the To: field and the internal distribution list [email protected] in the blind copy (bcc) field.

Emergency Change Notification–NO EXTERNAL IMPACTIf the change will have NO external impact:

Send an internal HUIT e-mail notification via the System Status tool per Major Incident Initial Communications instructions on page 18, without publishing the event to the web.

The web System Status posting tool includes Subject and Summary templates for Emergency Change/Maintenance.

Use the Email Options section to send email notification to [email protected].

NOTE: In the Publishing Options section, uncheck the Published option. This will allow the internal email to be sent without creating a web posting.


After-Hours Emergency Change Template

Manual NotificationsIn the event that the System Status web tool is not available, send emergency change email notifications manually from the [email protected] account or the Unix Webmail at https://webmail.fas.harvard.edu.

Template: AFTER-HOURS EMERGENCY CHANGEFROM: FAS Information Technology ([email protected])


BCC: [email protected]

SUBJECT: [Service Name] Urgent Maintenance


Ending: [Date/Time]


Summary

To [protect, restore] critical services, HUIT will be performing urgent maintenance to [service name]. Access to [the network in ___, specific applications or services] will be [unavailable, briefly interrupted] during this event.

Contact Information


Sincerely,





Document Change Control - Harvard University Information...

Documents

Transcript of Document Change Control - Harvard University Information...