7/30/2019 IG_00270055
1/154
Troubleshooting HP Enterprise NetworksFacilitator Guide
Version 11.11
7/30/2019 IG_00270055
2/154
Copyright 2010 Hewlett-Packard Development Company, L.P.The information contained herein is subject to change without notice. The only warranties for HP products and
services are set forth in the express warranty statements accompanying such products and services. Nothing
herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial
errors or omissions contained herein.
This is an HP copyrighted work that may not be reproduced without the written permission of HP. You may not use
these materials to deliver training to any person outside of your organization without the written permission of HP.
Troubleshooting HP Enterprise Networks
Facilitator Guide
Rev. 11.11
7/30/2019 IG_00270055
3/154
Rev. 11.11 i
Contents
Facilitator IntroductionInstructional methods used in this course ....................................................... Intro-1
What is an activity? .......................................................................... Intro-1Why use activities instead of lecture? .................................................... Intro-2What is expected of me in this type of training? .................................... Intro-2What resources are provided to help me become a facilitator? ............. Intro-3Introducing the HP networking Facilitator Guide ..................................... Intro-3Facilitator Guide icons for activities ...................................................... Intro-3
References ............................................................................................... Intro-5Books ............................................................................................... Intro-5Other references ................................................................................ Intro-5
Teaching this course ................................................................................. Intro-6Format offered ................................................................................... Intro-6Audience .......................................................................................... Intro-6Prerequisites ...................................................................................... Intro-6Equipment setup ................................................................................. Intro-7Class setup ........................................................................................ Intro-7Configuration instructions .................................................................... Intro-7
Working with a USB drive ............................................................ Intro-7Working with a tftp server ............................................................ Intro-7
When the learners arrive in the class .................................................... Intro-8Tips for teaching this course: ......................................................... Intro-8
Course topics and objectives ...................................................................... Intro-9Course topics ..................................................................................... Intro-9Course objectives ............................................................................... Intro-9
Course agenda ....................................................................................... Intro-10
Module 1: Troubleshooting Methodologies and Practices
Objectives .................................................................................................... 1-2Troubleshooting methodology .......................................................................... 1-4
Methodology .......................................................................................... 1-5Skill Sets ................................................................................................ 1-7
Problem solving methodology .......................................................................... 1-9Identification and analysis ............................................................................. 1-12Hypothesis and validation ............................................................................. 1-17Implementation and verification ..................................................................... 1-21Summary .................................................................................................... 1-22
7/30/2019 IG_00270055
4/154
Troubleshooting HP Enterprise Networks
ii Rev. 11.11
Module 2: Intelligent Resilient Framework
Objectives ....................................................................................................2-3Lab 2: IRF Troubleshooting and Problem Resolution ............................................ 2-5Lab Activity 2.1: Trouble Ticket 1 ...................................................................... 2-7
Problem resolution ................................................................................... 2-8IRF Misconfigured ................................................................................... 2-8
Broken ............................................................................................. 2-8Corrected ........................................................................................ 2-9
Master Priority ........................................................................................ 2-9Broken ............................................................................................. 2-9Corrected ........................................................................................ 2-9
Lab Activity 2.2: Trouble Ticket 2 ................................................................... 2-13MAD IP Addresses not in same subnet ..................................................... 2-15
Broken ........................................................................................... 2-15Corrected ...................................................................................... 2-15
MAD VLAN misconfigured ..................................................................... 2-15Broken ........................................................................................... 2-15Corrected ...................................................................................... 2-15
MAD Exclude ....................................................................................... 2-16Broken ........................................................................................... 2-16Corrected ...................................................................................... 2-16
Module 3: Rapid Ring Protection ProtocolBasic Concepts
Overview .................................................................................................... 3-3RRPP topologies ............................................................................................ 3-5RRPP single ring ........................................................................................... 3-6RRPP single ring control and data VLANs .........................................................3-7RRPP single ring primary and secondary ports ................................................. 3-8RRPP mechanisms: Ring health ....................................................................... 3-11
RRPP PDUs ........................................................................................... 3-11HELLO ........................................................................................... 3-11LINK-DOWN .................................................................................. 3-11COMMON-FLUSH-FDB ................................................................... 3-12COMPLETE-FLUSH-FDB .................................................................... 3-12EDGE-HELLO .................................................................................. 3-12MAJOR-FAULT ................................................................................ 3-12
RRPP mechanisms: Ring recovery ................................................................... 3-13RRPP single ring configuration example.......................................................... 3-15MSTP configuration ..................................................................................... 3-17RRPP port configuration ................................................................................ 3-19RRPP master node configuration .................................................................... 3-21RRPP transit nodes configuration .................................................................... 3-23RRPP intersecting rings.................................................................................. 3-25RRPP intersecting rings subring ...................................................................... 3-26RRPP intersecting rings subring control VLAN .................................................. 3-29
7/30/2019 IG_00270055
5/154
Contents
Rev. 11.11 iii
RRPP intersecting rings L2 frame flow ............................................................. 3-31RRPP intersecting ring configuration example.................................................. 3-32RRPP port configuration ............................................................................... 3-34RRPP master node configuration ................................................................... 3-35RRPP transit node configuration .................................................................... 3-36RRPP configuration edge node ...................................................................... 3-37RRPP configuration assistant edge node ......................................................... 3-39RRPP configuration subring master node ......................................................... 3-41RRPP display commands .............................................................................. 3-43Lab 3: RRPP Troubleshooting and Problem Resolution ...................................... 3-45Lab Activity 3: Trouble Ticket 3: .....................................................................3-47
Module 4: BGP Troubleshooting and Problem Resolution
Objectives ................................................................................................... 4-3Lab 4: BGP Troubleshooting and Problem Resolution ......................................... 4-5Lab Activity 4.1: Trouble Ticket 4 ..................................................................... 4-7Lab Activity 4.2: Trouble Ticket 5 ................................................................... 4-11
Module 5: MPLS L3VPN
Objectives ................................................................................................... 5-3MPLS L3VPN Troubleshooting and Problem Resolution ....................................... 5-4Lab 5: MPLS L3VPN Troubleshooting and Problem Resolution ............................. 5-5Lab Activity 5.1: Trouble Ticket 6 ..................................................................... 5-6Lab Activity 5.2: Trouble Ticket 7 ................................................................... 5-10
Module 6: MPLS L2VPN
Objectives ................................................................................................... 6-3MPLS L2VPN Troubleshooting and Problem Resolution ....................................... 6-4Lab 6: MPLS L2VPN Troubleshooting and Problem Resolution ............................. 6-5Lab Activity 6.1: Trouble Ticket 8 ..................................................................... 6-6Configuration for Lab Activities 6.2 and 6.3 .................................................... 6-10Lab Activity 6.2: Trouble Ticket 9 ................................................................... 6-13Lab Activity 6.3: Trouble Ticket 10 ................................................................. 6-17
7/30/2019 IG_00270055
6/154
Troubleshooting HP Enterprise Networks
iv Rev. 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
7/154
Rev. 11.11 Intro 1
Facilitator Introduction
Instructional methods used in this courseAll HP certification courses rely on instructional techniques that emphasize learningactivities and simulations over instructor lecture. The focus on participant engagementencourages learning through interactions among the learners and guided discussionsrather than relying solely or heavily upon interactions between the instructor and theclass as a whole. This reliance on learner interactions and engagement is supportedby contemporary research into adult learning which suggests adults learn best whenthey are challenged through interaction and learn the least when they are subjectedto long lectures based on presentation slides. (For more background on theinstructional techniques and strategies employed in HP networking courseware, see
the References section).
What is an activity?The simulations and activities used in HP networking courseware are structuredlearning segments that enable learners to explore important concepts individuallyand in small groups. These simulations and activities enable learners to review andclarify important material which in turn helps increase knowledge retention andlearning transfer. The activities are incorporated into each module at strategic pointsthat offer opportunities to enhance learning through interaction. Typical locations inmodules include:
Introductory reviews of prerequisite materials Discussions and explorations of the business and technical rationale and
applications for key technologies and products
Hands-on labs where learners put into practice important concepts Lab debriefs where learners clarify and explore the key points of lab activities Learning checks and validation pointsThe specific types of activities used in HP networking training include discussions androle-playing for exploration of business applications and games for review and
learning checks. Some activities use a technique known as teachback, wherelearners are encouraged to briefly present on important topics. These activities arecarefully structured to ensure learners have adequate time and resources to generateany required content. The teachback activities typically require learners to presentdebriefs of group discussions, enabling them to develop and present as a team.Team presentations help to alleviate learner anxiety about presenting, encourageteamwork, and establish a relaxed learning environment for both the facilitator andthe learners.
7/30/2019 IG_00270055
8/154
Troubleshooting HP Enterprise Networks
Intro 2 Rev. 11.11
Why use activities instead of lecture?What I hear, I forget; what I see, I remember; what I do, I understand. This 2,500year old aphorism by the Chinese philosopher Confucius summarizes the keyrationale for participant-centered learning. All learners, especially adults, retain more
of what they learn through doing or through their own explorations of concepts andtechnologies. In a University of Maryland study of 24-hour retention rates,researchers found that learners retained 50 to 90 percent of the material theylearned from group discussions, lab activities, and teachbacks. By contrast, learnersretained only five to 10 percent of material learned through lecture and reading.
Of course, technical content will sometimes require that instructors present lecturematerial. Accordingly, the courseware includes periods of lecture intentionallyseparated by learning activities. The goal in this design is to ensure that HPNcourseware deliveries adhere to the 90/20/8 rule. Developed by widely knowntraining expert Robert W. Pike, the 90/20/8 rule says:
1. Adults can listen with attention for about 90 minutes.2. Adults can listen with retention for about 20 minutes.3. Adults need opportunities to digest or apply what theyve learned every eight
minutes.
With this rule in mind, HP networking courses attempt to ensure that no module lastslonger than 90 minutes (excluding labs) and that lecture periods are regularlyinterspersed with activities to ensure changes of pace every 20 minutes and everyeight minutes. When lecture must run longer than eight minutes without activities, theFacilitators Guide includes cues for brief class discussions and impromptu learning
checks to ensure learners are not disengaged by the lecture.What is expected of me in this type of training?
Up until now, this guide has referred to you as the instructor. From this pointonward, you will be referred to as the facilitator. This small change in wordingsummarizes your primary task in the participant-centered classroom: to facilitate thelearning experiences of the learners. To further explore the meaning of this task,consider these definitions drawn from Webster's Third New International Dictionary:
Lecturer:Gives a discourse before an audience, especially for instruction
Facilitator:Makes a task easier or less difficult, free from difficulty or impediment
As these definitions suggest, the facilitator is the learners guide during the learningexperience, sometimes called the guide on the side. The lecturer on the other handstands in front of, often slightly above the learners, and is sometimes called the sageon the stage.
7/30/2019 IG_00270055
9/154
Facilitator Introduction
Rev. 11.11 Intro 3
As a certified facilitator authorized to conduct HP networking courseware, you areexpected to follow the course agenda and utilize the course Facilitator Guide withoutexception. Any deviation from the prescribed curriculum or activities must beapproved in advance by your Region Training Manager.
What resources are provided to help me become a facilitator?HP networking training materials and programs are designed to provide you with thetools and background you require to successfully facilitate our certification courses.All that is required of you, at least to start, is a willingness to learn these techniquesand to apply them to your extensive knowledge of technology and of lecture-driventraining.
The Train-the-Trainer programs for HP networking courses include instruction in PCRtechniques and theory as well as in the technologies and products presented in eachcourse. The course materials, especially the Facilitator Guide and props for activities,provide you with step-by-step instructions for presenting each element in each course.
Introducing the HP networking Facilitator GuideThe key document in an HP networking certification course is the Facilitator Guide,which provides you with all the instructions necessary to successfully present an HPnetworking certification course. The Facilitator Guide consists of each page in theLearner Guide accompanied by facilitation instructions. All pages intended for thefacilitator are indicated by a black line in the outside margin.
For many items, especially lecture periods, the Learner Guide page appears on theright hand (odd-numbered) page and the facilitation instructions on the left hand(even-numbered page). In some cases, such as this Introduction and in extensive
participant activities and simulations, the facilitation pages will extend for multiplepages in sequence.
Facilitator Guide icons for activitiesThe script for each activity is accompanied by an icon that identifies the type ofactivity. This icon is presented at the top of the first page of each activity along withinformation about the time that should be allocated to the activity and the materialsthat will be required to present it. For example, an activity calling for a facilitatedgroup discussion would provide the following information.
Timing: 1-2 minutes
Materials required:
Module 1 PowerPoint presentation
7/30/2019 IG_00270055
10/154
Troubleshooting HP Enterprise Networks
Intro 4 Rev. 11.11
Table Introduction-1 presents all activity icons used in HP networking FacilitatorGuides. Typically, a single training course includes only a subset of these activitytypes.
Table Introduction-1: Activity icons
7/30/2019 IG_00270055
11/154
Facilitator Introduction
Rev. 11.11 Intro 5
References
BooksPike, Robert W. Creative Training Techniques Handbook. Amherst, MA. HRD Press,
Inc., 2003.Silberman, Mel. Active Training. San Francisco, CA. Pfeiffer, 2006.
Stolovitch, Harold, D. and Erica J. Keeps. Telling Aint Training. Alexandria, VA.ASTD Press, 2002.
Stolovitch, Harold, D. and Erica J. Keeps. Training Aint Performance. Alexandria,VA. ASTD Press, 2004.
Other referencesBob Pike Group, www.bobpikegroup.com
Participant Centered Results, Cindy Pautzke, www.participantcenteredresults.com
Rimer, Sarah. At M.I.T., Large Lectures Are Going the Way of the Blackboard. NewYork Times, January 12, 2009.http://www.nytimes.com/2009/01/13/us/13physics.html?_r=1
7/30/2019 IG_00270055
12/154
Troubleshooting HP Enterprise Networks
Intro 6 Rev. 11.11
Teaching this courseWelcome learners to the Troubleshooting HP Enterprise network course and explainthat this course provides the skills and knowledge necessary to troubleshoot andresolve customer problems on A-Series switching products. This course focuses on
troubleshooting methodologies and practices in medium to enterprise networks.Through hands-on lab, learner activities, discussions, and facilitator lecture, theTroubleshooting HP Enterprise Networks course covers troubleshootingmethodologies and practices for the following technologies:
HP Intelligent Resilient Framework (IRF) Rapid Ring Protection Protocol (RRPP) Border Gateway Protocol (BGP) Multiprotocol Label Switching (MPLS) Layer 3 Virtual Private Network (3VPN)
MPLS L2VPNFormat offered
This course is a four-day instructor-led training with 25 percent lecture and 75 percentlab and classroom activities. The course may be delivered using a remote labenvironment.
AudienceThe Troubleshooting HP Enterprise Networks course is designed for professionalswho support network solutions based on HP A-Series routers and switches, including
systems engineers, systems designers, customer IT staff, HP services field and callcenter support engineers.
PrerequisitesThe required prerequisites for Troubleshooting HP Enterprise Networks course are theASE Network Infrastructure certification and the HP Enterprise Networking training.
7/30/2019 IG_00270055
13/154
Facilitator Introduction
Rev. 11.11 Intro 7
Equipment setupThe equipment will be accessed remotely, and configurations will be loaded bylearners per trouble ticket.
This lab consists of the following HP A-Series network equipment:
Two HP A7506 chassis switches serving as P1 and P2 routers interconnected viaGigabit Ethernet.
Two student pods, each consisting of One HP A7503 chassis switch serving as a PE router interconnected to one
P router and two CE Layer 3 switches via Gigabit Ethernet
Two HP A5800 series switches serving as CE routers for two differentcustomers connected to their local PE router via Gigabit Ethernet
Class setupRemote access to https://shrub.usa.hp.com
Configuration instructionsFrom the Command Line Interface (CLI) of the A-Series, in the level usethese commands as necessary.
Working with a USB drive
Delete the startup configuration on your router
< A- Ser i es >r eset saved- conf i gur at i on
Copy a file to the USB device< A- Ser i es > copy usba: /
Copy a file from USB
< A- Ser i es > copy usba: /
Working with a tftp server
Save file to a TFTP server on a laptop
< A- Ser i es > t f t p put
Restore file from a TFTP server to a router
< A- Ser i es > t f t p get
Create directories for all the labs / Trouble tickets:
< A- Ser i es > mkdi r
Copy files from TFTP server to A-Series devices
< A- Ser i es > t f t p get /
7/30/2019 IG_00270055
14/154
Troubleshooting HP Enterprise Networks
Intro 8 Rev. 11.11
Copy files from USB drive to A-Series devices
< A- Ser i es > copy usba: / /
Make the file the current configuration
Once files are copied to different directories in flash of the A-Series devices, to makethat the current configuration file, use:
< A- Ser i es > syst em- vi ew
[ A- Ser i es] conf i gur at i on r epl ace f i l e /
When the learners arrive in the classUse your usual course introduction.
Ask each student to test lab access so that you can get any problems fixed as quickly aspossible.
Tips for teaching this course: Encourage open questions. Optionally, mix up colleagues into different teams. Optionally, pair the students such that a team will have one skilled person with
one person with fewer skills. Encourage mentoring.
Optionally, pair students such that a team will have at least one native English(or local language) speaking person.
Suggest that during the activities the teams members participate equally
7/30/2019 IG_00270055
15/154
Facilitator Introduction
Rev. 11.11 Intro 9
Course topics and objectivesTiming: 2-3 minutes
Materials required:
Introduction PowerPointpresentation
Display slides 2 to 3 in the Introduction.pptx.
Briefly explain the primary topics to be covered in this course and cover the courseobjectives. Each module will have module objectives.
Emphasize that this course covers how to troubleshoot and maintain HP A-Seriesswitches.
Explain that the course is mostly labs. Explain that learners will be given troubletickets and expected to be able to resolve the problems described in each one.
To facilitate problem resolution the courses uses the HP troubleshooting methodology.Learners will both learn about common troubleshooting problem, and learn how totake a structured approach to troubleshooting.
Course topics Troubleshooting methodologies and practices HP IRF
RRPP BGP MPLS L3VPN MPLS L2VPN
Course objectivesAfter completing the Troubleshooting HP Enterprise Networking course, learners willbe able to:
Formulate and discuss proper methodology to Troubleshoot and resolveproblems
Create a plan for resolving issues Justify the documentation used and discuss it with others on how the problem
resolution should occur
Proficiently construct a troubleshooting action plan given a set of problem(s)
7/30/2019 IG_00270055
16/154
Troubleshooting HP Enterprise Networks
Intro 10 Rev. 11.11
Course agendaTiming: 2-3 minutes
Materials required:
Introduction PowerPointpresentation
Display slides 4 to 6 in the Introduction.pptx.
The MASE Troubleshooting HP Enterprise Networks course includes a number offocused troubleshooting exercises to learn and reinforce how to troubleshoot variousenterprise level protocols
The labs do not build on one another. Each lab is unique and separate althoughproblems introduced in one lab maybe seen again in another lab. There are multiple
problems introduced in each lab, and all must be resolved to ensure properinstruction.
7/30/2019 IG_00270055
17/154
Rev 11.11 1 1
Troubleshooting Methodologies and PracticesModule 1
This module will introduce the troubleshooting process that will be practiced in the
labs for the rest of the class.
Approximate time to complete this module: 20 minutes
7/30/2019 IG_00270055
18/154
Troubleshooting HP Enterprise Networks
1 2 Rev 11.11
ObjectivesTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 2 in 03-M01-Methodologies_and_Practices.pptx.
Every person in this class has likely had to troubleshoot a network problem in thepast. The idea behind this first module is to apply a process to troubleshooting anyproblem. This process can help to reduce the time to resolve a problem.
7/30/2019 IG_00270055
19/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 3
Troubleshooting Methodologies and PracticesModule 1
ObjectivesNo network or networking technology operates smoothly all of the time. Everynetwork technician will be required at some time to troubleshoot issues in networkconfiguration and performance. This module introduces basic techniques for networktroubleshooting.
After completing this module, you will be able to:
Describe a framework for basic network troubleshooting
7/30/2019 IG_00270055
20/154
Troubleshooting HP Enterprise Networks
1 4 Rev 11.11
Troubleshooting methodologyTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 3.
Emphasize that this first module is designed to formalize a process that mostnetwork technicians have probably already developed on their own, possiblywithout even knowing it.
Troubleshooting is a skill just like any other. However, rarely are networkadministrators given the time to practice it.
7/30/2019 IG_00270055
21/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 5
Troubleshooting MethodologyNetwork troubleshooting benefits from having:
Methodology
A discipline for evaluating, analyzing and investigating problem conditions Includes determining the scope of the problem, developing a hypothesis,
testing it out, and if successful, implementing a resolution
Skill sets Familiarity with network devices, how they operate and how they are
managed
Technical tools that may be useful for investigating and verifying problems;from CLI commands and protocol analyzers
Good Q&A skills
Experience Over time, applying a methodology and the technical tools helps develop
your own library of problem recognition capabilities and yields a moreefficient problem resolution process
The basics of troubleshooting any kind of networking trouble might be succinctlystated as keep eliminating obvious causes until the real cause presents itself. Butunderstanding what this means requires a systematic approach and real disciplinewhen attempting to identify causes from symptoms and apply the right fixes orworkarounds.
Troubleshooting is a skill that all networking professionals learn by trial and error. Butskipping some of the more painful or obvious errors can make your learningsomewhat less trying than it might be otherwise. The most important characteristic tocultivate when solving problems is calmness. If you can keep a clear head whenthings fail or start degrading seriously, youll be better able to assess your situationand better equipped to solve whatever problems you discover.
MethodologyDevelopment of problem solving techniques is often an on-the-job acquisitionprocess. Few of us can expect much along the lines of formal networktroubleshooting training in our job positions for a number of reasons. These reasonsmay include:
The relatively fast pace of the day-to-day job tasks and challenges yields littletime to pursue formal training on troubleshooting aspects such as technical toolslike a protocol analyzer.
7/30/2019 IG_00270055
22/154
Troubleshooting HP Enterprise Networks
1 6 Rev 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
23/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 7
Few business environments provide the luxury of a test lab and the time tohone your skills where a progression of test problems can be examined, workedthrough, and resolutions tried out.
In the absence of a more ideal situation, a problem solving methodology canincrease the effectiveness of support staff by standardizing the approach used to
some extent. With a fairly modest amount of discipline, network technicians canimprove their problem resolution efficiency in terms of the effort needed and thenumber of other people that must be directly involved.
Skill SetsThere are a variety of skill sets that can enhance a network technicians success inproblem solving. Some of these skills are purely technical in nature. For instance, it isimportant to understand the fundamentals of how network devices operate and howthey are managed. Having proficiency in reading logs or interpreting a protocolanalyzer display are examples of having familiarity with the potential tools you may
need to call upon from your toolbox.Other skills are much less technical, but still very important. As part of the probleminvestigation process, a network technician may need to talk with various levels ofstaff. The staff may include non-technical end-users and business unit managers tosoftware and hardware vendor support people. Having sufficient interpersonal skillscoupled with good investigative reporter-like skills can expedite the isolation of aproblem and eliminate the noise that often conceals the real problem.
Proactive IT support groups tend to spend time on developing procedures and toolsto facilitate problem resolutions. Some examples of technical tools used by thenetwork technicians are:
Device logsArchived instances of the logs as well as the current one mayprovide hints of where the problem may be. At the very least, familiarity with alog files typical contents helps you differentiate normal from abnormalsituations.
Device statistics and status informationBeing able to determine the health of asystem or the network is important for gathering the vital signs. This type ofinformation can include anything from port statistics and CPU utilization tonetwork reachability results.
Protocol analyzerAlthough this may not be a frequently used tool, it can beinvaluable for examining what conversations are or are not occurring betweencommunicating devices.
A problem solving methodology that is refined over time can be very beneficial tonetwork technicians. Being methodical and learning from the macro and micro levelsof mistakes can help network technicians improve problem recognition capabilitiesand yield a more efficient application of a problem resolution process.
7/30/2019 IG_00270055
24/154
Troubleshooting HP Enterprise Networks
1 8 Rev 11.11
Problem solving methodologyTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 4.
These are the steps that will be used to troubleshoot problems in the labs in thisclass. Depending on the problem some of these steps may be skipped and othertimes it may be necessary to repeat steps or, in the worst case, repeat the wholeprocess.
7/30/2019 IG_00270055
25/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 9
Problem solving methodology
Figure 1-1
A problem solving methodology is a process for managing problem resolution.Although there is no one specific model that may be useful for all problem situations,a general framework can provide guidelines and help ensure efficiency in the effortsmade to solve a problem. Applying a methodology can improve the probability of asuccessful resolution. This graphic illustrates the framework for a general problemsolving methodology that has many applications, including in todays contemporarynetwork environments.
There are six steps to the problem solving methodology outlined here. The steps mustbe executed in order starting with identification. The rules of the methodology statethat if a step fails, you must return to the preceding step above or possibly return tothe top level step.
The six steps are:
IdentificationUnderstand and document the problem from both a user andtechnical perspective. Sometimes it is possible to lose sight of what the potentialproblem is before searching for a cause when we dont consider multipleperspectives.
AnalysisEvaluate the situation by investigating using problem resolution tools,product documentation and user input.
HypothesisDevelop possible resolutions based on the analysis and documenta possible resolution. This documentation may be fairly informal, but it isimportant to be able to explain it in writing. Doing so can reveal a hypothesisthat is unclear and for which a possible resolution may not be plausible.
1. Identification
2. Analysis
3. Hypothesis
4. Validation
6. Verification
1. Identification
2. Analysis
3. Hypothesis
4. Validation
5. Implementation
6. Verification
7/30/2019 IG_00270055
26/154
Troubleshooting HP Enterprise Networks
1 10 Rev 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
27/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 11
ValidationRun a validation process to prove or disprove the hypothesis. Thismay not be particularly feasible, for example, if you have no test lab equipmentto try out your hypothesis. At the very least, performing a walk-through of thehypothesis in an articulate manner with other team members may help.
ImplementationDevelop an implementation plan along with a back-out plan,just in case, and then implement the resolution. For example, have a backupconfiguration and software image readily available.
VerificationVerify the success or failure of the implementation. If it fails,implement the back-out plan.
7/30/2019 IG_00270055
28/154
Troubleshooting HP Enterprise Networks
1 12 Rev 11.11
Identification and analysisTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 5.
Lets look at the identification and analysis steps more closely. This further defineseach step.
This entire process can benefit from a well-documented network. Having a currentand detailed network map including VLANs, addressing and port assignments canmake the troubleshooting process much easier and lead to faster resolution.
Each of these steps is further discussed in your learner guide.Before taking the test for this class, make sure you have read and understand thecontent in the learner guide.
7/30/2019 IG_00270055
29/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 13
Identification and analysis
Figure 1-2
The first step of the six-step methodology is identification, which is an observationprocess. Try to observe everything, not just the apparent problem, and avoidassuming something. Because network troubleshooting primarily involves evaluatingand resolving connectivity issues, the general procedure begins with an analysis ofsymptoms to determine the scope of the issue.
For example, it is important to determine whether the problem is affecting a singlehost, a group of hosts, or the entire network. If many hosts are affected, determine
what they have in common. For instance, if a host can communicate with local hosts,but not remote hosts, verify connectivity with its default gateway. If all hosts in thesame VLAN can communicate with local hosts, but not remote hosts, the issue maybe a logical problem with the default gateway or a physical problem concerningconnectivity with the default gateway. Although the default gateway performs Layer 3forwarding on behalf of local hosts, their communication with the default gateway isdone using Layer 2 addressing.
Isolate failingcomponent
Successful?
Review previousproblem resolutions and
appropriatedocumentation
Document probablecauses
Understand theenvironment
Understand thecustomer/user need
Develop problemdefinition
Prioritize the problem
Identification
No
Yes
Start Analys is
Go to
Analys is step
Go to
Hypothesis step
7/30/2019 IG_00270055
30/154
Troubleshooting HP Enterprise Networks
1 14 Rev 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
31/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 15
The identification process consists of doing tasks that can include:
Documenting the physical settings. The specifics will of course vary dependingon the problem scenario, but some examples are the following:
What client, server and network device hardware and software are in use? What is the network topology between the client and server? Where are the applications and services located? Determine the effects the
problem has on the user/customer and the business.
Developing a problem definitionDocument probable failures. Prioritizing the problemPrioritize based on defined user/customer policies. Is
this a problem that must be investigated immediately or can it wait until you canassemble a strategy using the problem solving methodology?
Step 2 is analysis. Analysis is the process of isolating the problem with the objectiveto narrow down the different possibilities.
The analysis process considers such factors as the following: Does the system work without the problem Previous changes to the system Something new, such as networking equipment, that may have been introduced Any changes to peripheral equipment that may have been made Whether the hardware or software is being used correctlyWith the scope of the problem having been narrowed down, that can help suggestthe type of network troubleshooting tools you may want to use to test probablecauses. For example, the problem investigation may involve using simple networkreachability tools, such as traceroute or ping, or examination of the logs of multipleswitches, or even use of a protocol analyzer.
7/30/2019 IG_00270055
32/154
Troubleshooting HP Enterprise Networks
1 16 Rev 11.11
Hypothesis and validationTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 6.
Lets look at the hypothesis and validation steps more closely. This further defineseach step.
The important part here is to look at every possible cause to the problem and turneach cause into a hypothesis.
These steps may be skipped if the possibility of causing more problems is not amajor concern. In this case, you can go straight to the implementation stage and testeach hypothesis in the live network.
7/30/2019 IG_00270055
33/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 17
Hypothesis and validation
Figure 1-3
Step 3 is hypothesis. The hypothesis step involves the evaluation of the informationacquired from the analysis step to determine a number of probable causes.
Some things to keep in mind are:
What is the technical reason for the business problem? You need a validation procedure for the hypothesis to be usable. Although your
intuition may prove to be correct at times, in the business world, relying on that
primarily makes it difficult for management to feel confident about the process.
Eventual resolution of the problem could create side effects, some that are notimmediately obvious.
Validation, step 4, typically involves experimentally determining whether thehypothesis is reasonable. It increases the confidence level that the problem will in factbe resolved after implementation of a potential solution.
Test hypothesis
Formulate hypothesis
Develop validationprocess
Evaluate possible sideeffects
More probablecauses?
Yes
No
Discard hypothesis
More probablecauses tovalidate?
Perform technical
evaluation
Yes
No
Yes
No
Hypothesis Validation
Validate?
Go toValidation step
Return to Identification
step or go toImplementation step
7/30/2019 IG_00270055
34/154
Troubleshooting HP Enterprise Networks
1 18 Rev 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
35/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 19
The validation step involves:
Testing each hypothesis until you validate a probable cause with a high degreeof certainty. The objective is not necessarily to be 100% sure, but to balance thetime criticality of resolving a problem with the information you have available.
If validation fails for all probable causes you developed, then you may need toreturn to the problem definition phase and start over. Despite what may appearto be time wasted, you will likely have improved your awareness of the problemsituation and will have some additional facts to use when you attempt toredefine the problem.
7/30/2019 IG_00270055
36/154
Troubleshooting HP Enterprise Networks
1 20 Rev 11.11
Implementation and verificationTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 7.
The last two steps are implementation and verification. Arguably, the most importantpart of the implementation step is to develop a back-out plan. In other words, whatdo you do if your solution doesnt work, or worse yet, causes more problems.
After you have verified that the solution solves the problem and doesnt cause anyside effects, then it is important to document the changes. This goes back to havinga well-documented network.
7/30/2019 IG_00270055
37/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 21
Implementation and verification
Figure 1-4
Implementation, step 5, requires planning for installation of some form of system ornetwork fix or modification along with preparation for failure. If an implementationfails, you must be able to restore the system to a previous stable state.
The planning involves:
Development of a specific implementation plan. Development of a verification process to prove the implementation was
successful.
Development of a back-out plan to ensure the implementation can be removed, ifit fails. It should also address how to handle side effects.
Verification, step 6, is the process of proving the implementation was successful anddetermining that any side effects are acceptable. If verification fails or side effectsare unacceptable, the back-out plan developed in the implementation phase isexecuted.
Upon successful completion, the user or customer must be informed and the problemresolution should be documented in a trouble log. Lack of documentation can lead tolengthy resolution for recurring problems.
Yes
Successful?
Run implementation plan
Develop verificationprocess
Develop back-out plan
Schedule implementation
Implementation
Run back-out plan andreturn to identification
Develop implementation
plan
No
Yes
No
Run verification procedure
Any sideeffects?
Document and distribute toeffected personnel
Verification
Go to
Verification step
7/30/2019 IG_00270055
38/154
Troubleshooting HP Enterprise Networks
1 22 Rev 11.11
SummaryTiming: 1-2 minutes
Materials required:
Learner Guide Module 1 PowerPoint presentation
Display Slide 8.
Troubleshooting, just like any other skill, will become easier with experience. Andeven the most complex problems can be resolved in a timely manner while reducingdisruption following these processes.
7/30/2019 IG_00270055
39/154
Troubleshooting Methodologies and Practices
Rev 11.11 1 23
Summary Network troubleshooting benefits from having a methodology, skill sets and
experience
General problem solving methodology consists of six steps: Identification: Develop a problem statement Analysis: Narrow the scope Hypothesis: Define procedures to validate Validation: Test probable causes Implementation: Make changes with back-out plan ready Verification: Ensure that changes resolve problem without side effects
7/30/2019 IG_00270055
40/154
Troubleshooting HP Enterprise Networks
1 24 Rev 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
41/154
Rev. 11.11 2 1
Intelligent Resilient FrameworkModule 2
Facilitation overviewIn this module, you will review some of the common IRF problems.
Approximate time to complete this module: 180 minutes
7/30/2019 IG_00270055
42/154
Troubleshooting HP Enterprise Networks
2 2 Rev. 11.11
ObjectivesTiming: 1-2 minutes
Materials required:
Learner Guide
Display Slide 2 in 04-M02-IRF.pptx.
This module will very quickly review IRF issues. Instruct learners to turn to page 2-1of their learner guide and read the objectives.
7/30/2019 IG_00270055
43/154
Intelligent Resilient Framework
Rev. 11.11 2 3
Intelligent Resilient FrameworkModule 2
ObjectivesIn this module, you will:
Observe IRF operational failures Use the troubleshooting methodology to resolve these failuresThe technologies used in the lab for this module include:
IRF MAD BFD LACP
7/30/2019 IG_00270055
44/154
Troubleshooting HP Enterprise Networks
2 4 Rev. 11.11
Lab 2: IRF Troubleshooting and ProblemResolution
Timing: 60 minutes
Materials required:
Lab GuideDisplay Slide 3 in 04-M02-IRF.pptx.
Introduce this lab, in which learners will use a structured troubleshootingmethodology to solve an IRF trouble ticket.
The following table establishes a timeline for accomplishing lab tasks, and will helpyou guide learners through the lab.
After the lab has been completed, you will lead a demonstration to resolve the
trouble ticket. The facilitator notes for the demonstration begin on page 2-8.
Table 2-1: Lab Activity 2.1 task overview
Minute Task
0 Identification: Load configurations.
15
Analysis: Provide aid on how to replicate theproblem.
Access both devices and watch consoles:di spl ay i r f
30Hypothesis: Determine what learners think theproblems are. Why? What is the resolution?
45
Validation:Discuss which steps are beingused:di spl ay i rf conf i gurati ondi spl ay i rf t opol ogy
50Implementation
55Verification
60End of the lab- begin demonstration
7/30/2019 IG_00270055
45/154
Intelligent Resilient Framework
Rev. 11.11 2 5
Lab 2: IRF Troubleshooting and Problem ResolutionLab 2 is designed to ensure you can use a structured troubleshooting methodology toresolve IRF routing issues. The lab consists of two trouble tickets that you will work toresolve. Refer to your lab guide for instructions on how to do the lab.
Figure 2-1: Lab 2: IRF Troubleshooting and Problem Resolution
w
PX-58sw1
PX-58sw2
A5800
A5800
7/30/2019 IG_00270055
46/154
Troubleshooting HP Enterprise Networks
2 6 Rev. 11.11
Lab Activity 2.1: Trouble Ticket 1Timing: 5 minutes
Materials required:
Learner Guide
Display Slide 4 in 04-M02-IRF.pptx.
Discuss Trouble Ticket 1 with learners. Based on the feedback you get, determinewhether a review of IRF is required. If so, refer learners to the Implementing HP A-Series Networks course.
7/30/2019 IG_00270055
47/154
Intelligent Resilient Framework
Rev. 11.11 2 7
Lab Activity 2.1: Trouble Ticket 1In Lab Activity 2.1, you will use the troubleshooting methodology to resolve the issuein Trouble Ticket 1.
Figure 2-2: Lab Activity 2.1: Trouble Ticket 1
7/30/2019 IG_00270055
48/154
Troubleshooting HP Enterprise Networks
2 8 Rev. 11.11
Lab Activity 2.1 demonstration: Problem resolutionTiming: 30 minutes
Materials required:
Learner Guide
Access a learner pod and load the Trouble Ticket 2 configurations to demonstratethe resolution.
Problem resolutionThere are two major issues in this ticket. First is the fact that the IRF ports aremisconfigured. When virtual IRF ports are configured on an individual device, theycan be numbered as one (1) or two (2). In order for IRF to work correctly, the IRF
port1 of one device must be connected to the IRF port2 of the adjacent IRF member(e.g. IRF member 1, port 2 - irf-port 1/2 - would be connected to IRF member 2, port1 - irf-port 2/1). These devices are both only configured with a port 1 (i.e. irf-port1/1 and irf-port 2/1). This causes IRF to fail.
This is resolved by deleting the IRF port on one of the two member switches, andadding the two 10Gbps ports used to connect the A5800 switches to an IRF port 2instead (i.e. create irf-port 1/2 or irf-port 2/2).
The other problem is the fact that the IRF Master device has not been manuallyconfigured via priority. This results in the IRF Master being elected based on uptime,rather than having a specifically selected Master device.
This is resolved by configuring the 1st switch as member #1 with a priority of 32 (themaximum), while configuring the 2nd switch as member #2 with a lower priorityvalue (e.g. 16) via the irf member X priority Y command.
IRF Misconfigured
BrokenTT1-PODX-58sw1
i r f - por t 1/ 1por t gr oup i nt er f ace Ten- Gi gabi t Et hernet1/ 0/ 27 mode enhancedpor t gr oup i nt er f ace Ten- Gi gabi t Et hernet1/ 0/ 28 mode enhanced
TT1-PODX-58sw2
i r f - por t 2/ 1por t gr oup i nt er f ace Ten- Gi gabi t Et hernet2/ 0/ 27 mode enhancedpor t gr oup i nt er f ace Ten- Gi gabi t Et hernet2/ 0/ 28 mode enhanced
7/30/2019 IG_00270055
49/154
Intelligent Resilient Framework
Rev. 11.11 2 9
CorrectedTT1-PODX-58sw1
i r f - por t 1/ 1por t gr oup i nt er f ace Ten- Gi gabi t Et hernet1/ 0/ 27 mode enhancedpor t gr oup i nt er f ace Ten- Gi gabi t Et hernet1/ 0/ 28 mode enhanced
irf-port-configuration active
TT1-PODX-58sw2
i r f - por t 2/ 2por t gr oup i nt er f ace Ten- Gi gabi t Et hernet2/ 0/ 27 mode enhancedpor t gr oup i nt er f ace Ten- Gi gabi t Et hernet2/ 0/ 28 mode enhanced
irf-port-configuration active
------OR------TT1-PODX-58sw1
i r f - por t 1/ 2por t gr oup i nt er f ace Ten- Gi gabi t Et hernet1/ 0/ 27 mode enhanced
por t gr oup i nt er f ace Ten- Gi gabi t Et hernet1/ 0/ 28 mode enhancedirf-port-configuration active
TT1-PODX-58sw2
i r f - por t 2/ 1por t gr oup i nt er f ace Ten- Gi gabi t Et hernet2/ 0/ 27 mode enhancedpor t gr oup i nt er f ace Ten- Gi gabi t Et hernet2/ 0/ 28 mode enhanced
irf-port-configuration active
Master Priority
BrokenTT1-PODX-58sw1
CorrectedTT1-PODX-58sw1
i r f member 1 pr i or i t y 32TT1-PODX-58sw2
i r f member 1 pr i or i t y 16
7/30/2019 IG_00270055
50/154
Troubleshooting HP Enterprise Networks
2 10 Rev. 11.11
di spl ay i r f
di spl ay i r f
Swi t ch Sl ot Rol e Pri or i t y CPU- Mac
*+1 0 Master 32 00e0- f c0a- 15e0
2 1 Sl ave 1 00e0- f c0f - 8c02- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* i ndi cat es t he devi ce i s t he mast er .
+ i ndi cat es t he devi ce t hr ough whi ch t he user l ogs i n.
The Br i dge MAC of t he I RF i s: 000f-e26a-58ed
Auto upgrade : no
Mac pers i st ent : al ways
Li nk- del ay t i mer : 0 ms
Domai n I D : 30
Aut o mer ge : yes
di spl ay i r f conf i g
di spl ay i r f conf i gur at i on
Member I D NewI D IRF-Port1 IRF-Port2
1 1 Ten-GigabitEthernet1/8/0/1 di sabl e
Ten-GigabitEthernet1/8/0/2
2 2 di sabl e Ten-GigabitEthernet2/12/0/1
Ten-GigabitEthernet2/12/0/2
di spl ay i r f t opol ogy
di spl ay i r f t opol ogy
Topol ogy I nf o
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
IRF-Port1 IRF-Port2
Swi t ch Li nk nei ghbor Li nk nei ghbor Bel ong To
1 DI S - - UP 2 00e0- f c0f - 8c0f
2 UP 1 DI S - - 00e0- f c0f - 8c0f
7/30/2019 IG_00270055
51/154
Intelligent Resilient Framework
Rev. 11.11 2 11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
52/154
Troubleshooting HP Enterprise Networks
2 12 Rev. 11.11
Lab Activity 2.2: Trouble Ticket 2Timing: 60 minutes
Materials required:
Lab GuideDisplay Slide 5 in 04-M02-IRF.pptx.
In this lab, learners will again use the structured troubleshooting methodology tosolve another trouble ticket.
Another task table has been provided here for your reference. Use it to guidelearners through the lab. You will lead another demonstration after the lab has beencompleted. (The notes for the demonstration begin on page 2-14.)
Table 2-2: Lab Activity 2.2 task overview
Minute Task0 Identification: Load configurations.
15
Analysis: Provide aid on how toreplicate the problem.
Access both devices and watchconsoles:di spl ay i r f di spl ay mad
30
Hypothesis: Determine what learnersthink the problems are. Why? What
is the resolution?
45
Validation:Discuss which steps arebeing used:di spl ay mad verbosedi spl ay bf d sessi on
50Implementation
55Verification
60Lab over- begin demonstration
7/30/2019 IG_00270055
53/154
Intelligent Resilient Framework
Rev. 11.11 2 13
Lab Activity 2.2: Trouble Ticket 2In Lab Activity 2.2, you will use the troubleshooting methodology to solve the issue inTrouble Ticket 2:
Figure 2-3: Lab Activity 2.2: Trouble Ticket 2
7/30/2019 IG_00270055
54/154
Troubleshooting HP Enterprise Networks
2 14 Rev. 11.11
Lab Activity 2.2 demonstration: Problem resolutionTiming: 30 minutes
Materials required:
Learner Guide
Access a student pod and load the Trouble Ticket 2 configurations to demonstrate theresolution.
Problem resolution:There are multiple problems in this ticket with the BFD MAD configuration. First isthe fact that the physical ports planned for use in BFD MAD are not assigned to theBFD VLAN. This can be demonstrated via display vlan (in this case VLAN
4000 is the planned VLAN).This can be resolved by placing the appropriate ports into VLAN 4000 as accessports.
Next is the fact that the member IP addresses configured under vlan-interface 4000are not in the same subnet. In order for the BFD session to become active, the twoActive Master devices (resulting from the failure of IRF) must be capable of sendingBFD packets to one another. If they are not in the same IP network, that won't bepossible. This can be demonstrated via the display mad verbose command on theIRF device (either of the member A5800 switches).
This can be resolved by configuring MAD IP addresses for each member device that
are all in the same IP broadcast domain, under the interface vlan-interface 4000view, via the mad ip address member command.
Lastly, when MAD does work properly, one member will remain an active Master,while the other switch(es) will go into a recovery state, and all of its ports will ceaseto forward traffic. By default, this includes the physical ports and the VLANinterface used for BFD MAD as well. Once one device disables its ports, the BFDsession will fail and MAD detection does as well.
This can be resolved by excluding the ports and VLAN interface used for BFD MADfrom being put into a recovery state, via the mad exclude interface
command.
7/30/2019 IG_00270055
55/154
Intelligent Resilient Framework
Rev. 11.11 2 15
MAD IP Addresses not in same subnet
Broken
i nt er f ace Vl an- i nt er f ace4000mad bf d enabl e
mad i p address 10. 254. 254. 1 255. 255. 255. 0 member 1mad i p addr ess 10. 252. 254. 2 255. 255. 255. 0 member 2
Corrected
i nt er f ace Vl an- i nt er f ace4000mad bf d enabl emad i p address 10. 254. 254. 1 255. 255. 255. 0 member 1mad i p addr ess 10. 254. 254. 2 255. 255. 255. 0 member 2
MAD VLAN misconfigured
Broken
i nt er f ace g1/ 0/ 3por t access vl an 4000
i nt er f ace g2/ 0/ 3por t access vl an 3000
Corrected
i nt er f ace gi gabi t - et her net 1/ 0/ 3por t access vl an 4000
i nt er f ace gi gabi t - et her net 2/ 0/ 3por t access vl an 4000
di spl ay mad
di spl ay mad
MAD LACP di sabl ed.
MAD BFD enabled.
7/30/2019 IG_00270055
56/154
Troubleshooting HP Enterprise Networks
2 16 Rev. 11.11
MAD Exclude
Broken
Correctedmad exclude interface gigabit-ethernet 1/0/3mad exclude interface gigabit-ethernet 2/0/3
di spl ay mad
di spl ay mad
MAD LACP di sabl ed.
MAD BFD enabled.
di spl ay mad verbose
di spl ay mad ver bose
Curr ent MAD st at us: Detect/Recovery
Excl uded por t s( conf i gur abl e) :
Gi gabi t Et her net 2/ 1/ 0/ 2
Gi gabi t Et her net 2/ 1/ 0/ 3
Excl uded port s( can not be conf i gur ed) :
Ten- Gi gabi t Et hernet1/ 2/ 0/ 25
Ten- Gi gabi t Et hernet2/ 2/ 0/ 26
MAD enabl ed aggregat i on port :
Br i dge- Aggr egat i on2
MAD BFD enabl ed i nter f ace:
Vl an- i nt er f ace4000
mad i p addr ess 10. 0. 0. 2 255. 255. 255. 0 sl ot 1
mad i p addr ess 10. 0. 0. 3 255. 255. 255. 0 sl ot 2
di spl ay bf d sessi on
di spl ay bf d sessi on
Tot al Sessi on Num: 1 I ni t Mode: Down/Active
Sessi on Wor ki ng Under Echo Mode:
LD Sour ceAddr Dest Addr St ate Hol dt i me I nt er f ace
7 10. 0. 0. 2 10. 0. 0. 3 Down/Up 1700ms Vl an4000
The BFD session only comes up when the IRF link is lost to verify that the other side is still up.
7/30/2019 IG_00270055
57/154
Rev. 11.11 3 1
Rapid Ring Protection ProtocolBasic ConceptsModule 3
Facilitation overviewThis module focuses on Rapid Ring Protection Protocol (RRPP).
Approximate time to complete this module: 160 minutes.
7/30/2019 IG_00270055
58/154
Troubleshooting HP Enterprise Networks
3 2 Rev. 11.11
OverviewTiming: 1-2 minutes
Materials required:
Learner Guide
Display Slide 2 of 05-M03-RRPP.pptx.
This module focuses on Rapid Ring Protection Protocol (RRPP). Explain to learners thatthis module differs from other modules in this course because you will go over RRPPconcepts and configuration. Instruct learners to turn to page 3-1 of their LearnerGuide and read the objectives for this module.
7/30/2019 IG_00270055
59/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 3
Rapid Ring Protection ProtocolBasic ConceptsModule 3
OverviewThis module focuses on Rapid Ring Protection Protocol (RRPP).
Many networks built today use the ring structure to improve reliability. With ringnetworking technologies, network devices are connected to form rings. To avoidbroadcast storms, which are common in a ring network, a loop protectionmechanism must be used.
The IEEE spanning tree protocols have been widely used for loop protection.However, as STP topology convergence gets slower, and networks increase in size,
transmission performance can be degraded.
To remove the negative impact of network size on topology convergence and shortentopology convergence time, HP developed RRPP.
At present, HP A-Series RRPP is not an open standard, but greatly reduces outagetime from loop failure, by converging in under 50 milliseconds. Spanning Tree cantake over 50 seconds to converge. RRPP is able to converge quickly because there isno algorithm or state/ peering mechanism. Everything is defined in the configuration.
RRPP prevents loops. It filters data traffic coming across the ring; it doesnt allow datatraffic to circulate indefinitely and create a Layer 2 loop.
After completing this module, you will be able to:
Describe how RRPP prevents network loops
Explain how to configure RRPP to meet the needs of a given network
Observe RRPP operational failures and use troubleshooting methodology toresolve the failures
7/30/2019 IG_00270055
60/154
Troubleshooting HP Enterprise Networks
3 4 Rev. 11.11
RRPP topologiesTiming: 5 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slides 35 of 05-M03-RRPP.pptx.
After finishing the slide deck, you may want to discuss how you can use two domainson a single ring. By doing this you create a Tangent ring that allows for loadbalancing by having one domain go clockwise and the other counter-clockwise.
The example given for Tangent rings is actually the same physical topology openedlike a clam shell to highlight how to implement load-balancing for advanced classes.
7/30/2019 IG_00270055
61/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 5
RRPP topologies
Figure 3-1: RRPP topologies
RRPP rings have three defined topologies.
A single ring has only one ring and contains only Master and Transit nodes. Since itis only one ring that ring is seen as Primary and defined as level 0.
Intersecting rings have multiple rings, with a central ring that all other subringsconnect to. The central ring is Primary and thus a level 0 ring and the subrings aresecondary with level 1. RRPP protocol data units (PDUs) propagate across Level 0and Level 1 rings differently.
Tangent rings are two rings that do not intersect and are thus defined in separatedomains utilizing the same switch or switches. This design is often used to create aload balanced ring, with traffic flowing in both a clockwise and counter-clockwisedirection.
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Masternode
Transitnode
Domain 1
Ring 1Level 0
Domain 2
Ring1Level0
Port 2
Port 1
Port 1
Port 1 Por
t 1
Port 1
B
C
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Masternode
Transitnode
Ring1
ERing 2
Masternode
Edgenode
Domain 1
Assistant edgenode
Single ring
Intersecting rings
A
B
C
Port 2
Port 1
Port 2Port 2
Port 2
Port 2
Masternode
Transitnode
Domain 1
Ring 1Level 0
Tangent rings
7/30/2019 IG_00270055
62/154
Troubleshooting HP Enterprise Networks
3 6 Rev. 11.11
RRPP single ring
Figure 3-2: RRPP single ring
An RRPP domain defines a topology range calculated and controlled by the RRPPprotocol. A master node forwards RRPP PDUs to maintain the ring and TransitNodes forward the Masters RRPP PDUs.
RRPP Single Ring is defined by domain that defines the topology and how RRPP PDUsare flow within the ring.
In the Figure 3-2, the single ring is in RRPP Domain 1 with a Master Node defined onswitch A, the remaining switches are B, C, and D are defined as Transit Nodes andforward the RRPP PDUs from the Master. Only one master is defined per ring.
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
One master nodeper ring
Transit nodes:All nodes exceptmaster node
Domain 1
Ring 1Level 0
7/30/2019 IG_00270055
63/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 7
RRPP single ring control and data VLANs
Figure 3-3: RRPP single ring control and data VLANs
A control VLAN is defined to send the RRPP PDUs around the ring, all switches mustsupport the control VLAN and it should not carry any data traffic. All trunk portsshould carry the control VLAN.
All data traffic must be allowed over the switches by ensuring the VLAN databasesare consistent and all data VLANs traversing the ring are allowed on the trunk links.
In a single ring, the ring has no attachments and is defined as the primary ring. Theprimary ring is defined as level 0 in RRPP.
AB
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
Ring 1Level 0
Domain 1Control VLAN:Dedicated to
transferring RRPPDUs
Secondary controlVLAN: Acts as controlVLAN on all subrings
Data VLAN: Dedicatedto transferring datapackets
7/30/2019 IG_00270055
64/154
Troubleshooting HP Enterprise Networks
3 8 Rev. 11.11
RRPP single ring primary and secondary portsTiming: 5 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 6 of 05-M03-RRPP.pptx.
Traffic flows clockwise.
7/30/2019 IG_00270055
65/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 9
RRPP single ring primary and secondary ports
Figure 3-4: RRPP single ring primary and secondary ports
By defining the Primary and Secondary ports in RRPP, you define these ports todictate the how the data and control VLAN traffic will flow around the ring.
The primary port on the Master sends RRPP PDUs and the secondary port on theMaster blocks all data traffic.
The link on one switch must lead to its opposite, thus Primary to Secondary... Primaryto Secondary
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
Ring 1Level 0
Domain 1
Data packetsLoop-detect packets
Primary port of master nodeSENDS loop-detect packets
Secondary port ofmaster node RECEIVESloop-detect packets
Primary and secondaryports of transit nodes cantransfer data and controlpackets
7/30/2019 IG_00270055
66/154
Troubleshooting HP Enterprise Networks
3 10 Rev. 11.11
RRPP mechanisms: Ring healthTiming: 5 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slides 7 and 8 of 05-M03-RRPP.pptx.
It is not necessary to discuss RRPP PDUs, but they are included here for the sake ofbeing complete.
7/30/2019 IG_00270055
67/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 11
RRPP mechanisms: Ring health
Figure 3-5: RRPP mechanisms: Ring health
RRPP PDUs are used by the switches on the ring to define status and define whetherthe Masters secondary port will continue to block because loop is seen. In this state,the Master switch will show the ring as COMPLETE.
The Master sends out RRPP PDU HELLOs/ Health Packets to determine ring state outits Primary port.
The transit nodes receive and forward the HELLO to the next switch until it is receivedby the Masters Secondary Port, which will block Data Frames, but not ControlFrames - RRPP PDUs.
If the Masters Secondary port receives its HELLOs in a timely manner the ring isseen as COMPLETE.
RRPP PDUs
HELLO
Sent regularly by a master node to check ring completeness. If the sent HELLOmessages can finally reach the secondary port of the master node within thepredefined period, the ring is considered complete; if not, the ring is consideredopen, in which case a link may have failed on the ring.
LINK-DOWN
Sent by a transit, edge, or edge-assistant node to report link failure to the masternode.
MasterNode
TransitNode
TransitNode
TransitNode
HealthPacket
Primary
Secondary
Secondary
Secondary
Secondary
Primary
Primary
Primary
Polling mechanismHealth packets
Link down alarmmechanismSend by transit nodes to themaster node
Ring recoveryFlush MAC mappings
7/30/2019 IG_00270055
68/154
Troubleshooting HP Enterprise Networks
3 12 Rev. 11.11
COMMON-FLUSH-FDB
Sent by a master node to notify the transit nodes to update their MAC address tablesand ARP/ND tables when it transitions to the failed state.
The nodes on the primary ring must update their MAC address table and ARP/ND
tables after receiving COMMON-FLUSH-FDB messages, even if they are from themaster node on a subring.
COMPLETE-FLUSH-FDB
Sent by a master node to notify the transit nodes to update their MAC address tablesand ARP/ND tables when it transitions to the complete state. The transit nodes thustransition to the link-up state, unblocking the temporarily blocked ports.
For the nodes on the primary ring, if the sending master node is on a subring, theywill update the MAC address tables and ARP/ND tables, but will not unblock theblocked ports.
EDGE-HELLO
Sent by the edge node of a subring and received by the assistant-edge node of thesame subring to check whether the SRPTs of the subring are in good condition.
The edge node periodically sends EDGE-HELLO messages out the two common portsto the assistant-edge node across the primary ring. If the assistant-edge node receivesthe messages, the SRPTs are considered as in good condition; if the assistant-edgenode does not receive the messages within a specified period of time, the SRPTs areconsidered as faulty.
MAJOR-FAULTSent by an assistant-edge node to report SRPT failure to the edge node. Uponreceiving a MAJOR-FAULT message, the edge node blocks its edge port.
7/30/2019 IG_00270055
69/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 13
RRPP mechanisms: Ring recovery
Figure 3-6: RRPP mechanisms: Ring recovery
There are two methods to inform the Master of ring failure, Polling and Link Loss. TheMaster will show the ring as FAILED. Transit nodes will continue to show no changein their status for its blank, -. All validation of the Ring state must be done on theMaster.
By Polling, if the HELLOs fail to be received in a timely fashion.
By default, the Hello timer value is 1 second and the Fail timer value is 3seconds. If the fail timer is reached the ring transition to a failed state and aCOMPLETE-FLUSH-FDB message out both primary and secondary port to instruct allthe transit nodes to update their MAC address table entries and ARP/ND entries.
By Link loss, A Transit Link can send a LINK-DOWN RRPP PDU notifying the Masterof link loss and that the Master should transition the ring to a failed state, stopblocking its Secondary Port and sends a COMPLETE-FLUSH-FDB message out itssecondary port to instruct all the transit nodes to update their MAC address tableentries and ARP/ND entries.
MasterNode
TransitNode
TransitNode
TransitNode
HealthPacket
Primary
Secondary
Secondary
Secondary
Secondary
Primary
Primary
Primary
Polling mechanismHealth packets
Link down alarmmechanismSend by transit nodes to themaster node
Ring recoveryFlush MAC mappings
7/30/2019 IG_00270055
70/154
Troubleshooting HP Enterprise Networks
3 14 Rev. 11.11
RRPP single ring configuration exampleTiming: 3 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 9 of 05-M03-RRPP.pptx.
The steps are very much a copy and paste between switches with only the wordsMaster and Transit being replaced upon the appropriate configuration as long thesame switch ports are used.
7/30/2019 IG_00270055
71/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 15
RRPP single ring configuration example
Figure 3-7: RRPP single ring configuration example
The tasks to configure RRPP single ring:
1. Define VLANs (not shown)
2. Defining MSTP regions
3. Configuring RRPP Ports
4. Creating an RRPP Domain on the Master
a. Configuring Control VLANs
b. Configuring Protected VLANs
1) Configuring RRPP Rings
2) RRPP Nodes
5. Activating an RRPP Domain
6. Configuring the RRPP Domain on the Transit Nodes
a. Configuring Control VLANs
b. Configuring Protected VLANs
1) Configuring RRPP Rings
2) RRPP Nodes
7. Activating an RRPP Domain
A B
CD
GE3/0/1 GE3/0/1
Port 2
GE3/0/2 GE3/0/2Master node
Transit node
Ring 1
GE3/0/2 GE3/0/1
GE3/0/1GE3/0/2
Transit node
Transit node
Domain 1
7/30/2019 IG_00270055
72/154
Troubleshooting HP Enterprise Networks
3 16 Rev. 11.11
MSTP configurationTiming: 3 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 10 of 05-M03-RRPP.pptx.
MSTP should be familiar to the class. The VLAN-mapping modulo 2 command is asimpler yet effective method to allow for load-balancing for VLANs.
7/30/2019 IG_00270055
73/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 17
MSTP configuration
Figure 3-8: MSTP configuration
Defining MSTP regions:
MSTP instances are assigned to a Protected VLAN.
The Protected VLANs are used by RRPP to define how ports are forwarded.
Different RRPP domains on the same RRPP ring are configured with differentprotected VLANs, and each RRPP domain controls the forwarding status of portsin it independently. By defining different MSTP instances in separate RRPPdomains protected VLANs, load balancing of the ring can occur.
All VLANs default to being in MSTI Instance 0
Create two MSTI instances:
revision-level 1
instance 1 vlan 1 to 500
instance 2 vlan 501 to 999
active region-configuration
Assign odd VLANs to Instance 1 and even VLANS toInstance 2:revision-level 1
vlan-mapping modulo 2
active region-configuration
7/30/2019 IG_00270055
74/154
Troubleshooting HP Enterprise Networks
3 18 Rev. 11.11
RRPP port configurationTiming: 3 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 11 of 05-M03-RRPP.pptx.
Emphasize that RRPP ports are to Trunk.
7/30/2019 IG_00270055
75/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 19
RRPP port configuration
Figure 3-9: RRPP port configuration
Configuring RRPP Ports requires that spanning tree on the port be disabled and theport support VLANs required, data and control.
The ports can be configured as:
Trunk
Bridge-Aggregation
Must be Ethernet
The ports cannot be configured as:
802.1X authentication
MAC address authentication
Voice VLAN
Remote Loopback
Spanning Tree
Access
Configure the suppression time of physical-link-state changeson Gigabit ports as zero, disable STP, configure the ports as
trunk ports, and assign them to all VLANs.[DeviceA] interface gigabitethernet 3/0/1
[DeviceA-GigabitEthernet3/0/1] link-delay 0
[DeviceA-GigabitEthernet3/0/1] undo stp enable
[DeviceA-GigabitEthernet3/0/1] port link-type trunk
[DeviceA-GigabitEthernet3/0/1] port trunk permit vlan all
[DeviceA-GigabitEthernet3/0/1] quit
7/30/2019 IG_00270055
76/154
Troubleshooting HP Enterprise Networks
3 20 Rev. 11.11
RRPP master node configurationTiming: 10 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 12 and 13 of 05-M03-RRPP.pptx.
Note that the configuration of the Master and Transit only differs by the bolded word.
7/30/2019 IG_00270055
77/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 21
RRPP master node configuration
Figure 3-10: RRPP master node configuration
Creating an RRPP Domain on the Master Node:
Configuring Control VLANs
Configuring Protected VLANs
Configuring RRPP Rings
RRPP Nodes
Activating an RRPP Domain:All the work conducted in prior tasks is brought together to configure the RRPPdomain and define its operation. Activation of the ring and RRPP generate the RRPPPDUs.
Create RRPP domain 1, configure VLAN 4092 as the primary control VLAN ofRRPP domain 1, and configure the VLANs mapped to MSTIs 0 through 47 as theprotected VLANs of RRPP domain 1.
[DeviceA] rrpp domain 1
[DeviceA-rrpp-domain1] control-vlan 4092
[DeviceA-rrpp-domain1] protected-vlan reference-instance 0 to 47
Configure Device A as the master node of primary ring 1, with Gigabit 3/0/1as the primary port & Gigabit 3/0/2 as the secondary port, and enable ring 1.
[DeviceA-rrpp-domain1] ring 1 node-mode master primary-portgigabit 3/0/1 secondary-port gigabit 3/0/2 level 0
[DeviceA-rrpp-domain1] ring 1 enable
[DeviceA-rrpp-domain1] quit
Enable RRPP.
[DeviceA] rrpp enable
7/30/2019 IG_00270055
78/154
Troubleshooting HP Enterprise Networks
3 22 Rev. 11.11
PAGE INTENTIONALLY LEFT BLANK
7/30/2019 IG_00270055
79/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 23
RRPP transit nodes configuration
Figure 3-11: RRPP transit nodes configuration
Configuring an RRPP Domain on the Transit Node:
Configuring Control VLANs
Configuring Protected VLANs
Configuring RRPP Rings
RRPP Nodes
Activating the RRPP Domain:
The key difference between configuring a Master node and a Transit node is thedefinition of the node-mode:
Master
Transit
Create RRPP domain 1, configure VLAN 4092 as the primary control VLAN ofRRPP domain 1, and configure the VLANs mapped to MSTIs 0 through 47 as theprotected VLANs of RRPP domain 1.
[DeviceB] rrpp domain 1
[DeviceB-rrpp-domain1] control-vlan 4092
[DeviceB-rrpp-domain1] protected-vlan reference-instance 0 to 47
Configure Device B as the transit node of primary ring 1, with GigabitEthernet3/0/1 as the primary port and GigabitEthernet 3/0/2 as the secondary port,and enable ring 1.
[DeviceB-rrpp-domain1] ring 1 node-mode transit primary-portgigabitethernet 3/0/1 secondary-port gigabitethernet 3/0/2 level0
[DeviceB-rrpp-domain1] ring 1 enable
[DeviceB-rrpp-domain1] quit
Enable RRPP.
[DeviceB] rrpp enable
7/30/2019 IG_00270055
80/154
Troubleshooting HP Enterprise Networks
3 24 Rev. 11.11
RRPP intersecting ringsTiming: 3 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 14 of 05-M03-RRPP.pptx.
A primary ring can support several subrings and there is no theoretical limit.
7/30/2019 IG_00270055
81/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 25
RRPP intersecting rings
Figure 3-12: RRPP intersecting rings
In the RRPP intersecting rings many of the same requirements as the single ring. As inthe Single ring, the domain is identified by an Index number that identifies itsconfiguration and RRPP PDUs. The master node generates the HELLO RRPP PDUs tomaintain ring state and forwards them out its Primary Port. The transit nodes receiveand forward the HELLO to the next switch until it is received by the MastersSecondary Port, which will block Data Frames, but not Control Frames - RRPP PDUs.
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
Ring 1E
Ring 2
Master node
Edge node
Assistant edge node
Domain 1RRPP domain
RRPP ring 1=Master ring=Ring level 0
RRPP ring 2=Sub ring=Ring level 1
7/30/2019 IG_00270055
82/154
Troubleshooting HP Enterprise Networks
3 26 Rev. 11.11
RRPP intersecting rings subringTiming: 5 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 15 of 05-M03-RRPP.pptx.
The nodes use RRPP PDUs to notify status of the secondary ring. Edge nodesgenerate RRPP PDU EDGE-HELLO that is forwarded onto the Primary Ring to theAssistant edge node. Assistant-edge nodes await the EDGE-HELLO to note the healthof the secondary ring. Sends a MAJOR-FAULT if it does not receive the EDGE-HELLO,causing the edge port to become blocked.
7/30/2019 IG_00270055
83/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 27
RRPP intersecting rings subring
Figure 3-13: RRPP intersecting rings subring
In addition, the subring (secondary ring) has specific requirements for its RRPP PDUsto flow around the subring and is attached to the primary ring via edge node andassistant edge node off the primary rings transit nodes.
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
Ring 1E
Ring 2
Master node
Edge node
Domain 1
Assistant edge node
Edge node:Belongs to primaryand subrings
Assistant edge node:Belongs to primary
and subrings
7/30/2019 IG_00270055
84/154
Troubleshooting HP Enterprise Networks
3 28 Rev. 11.11
RRPP intersecting rings subring control VLANTiming: 1 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slide 16 of 05-M03-RRPP.pptx.
If VLAN 4092 is the control VLAN than the subrings control VLAN is 4093.
7/30/2019 IG_00270055
85/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 29
RRPP intersecting rings subring control VLAN
Figure 3-14: RRPP intersecting rings subring control VLAN
Subring Level 1 RRPP PDUs flow in their own control VLAN which is defined as theprimary rings VLAN + 1.
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
Ring 1E
Ring 2
Master node
Edge node
Assistant edge node
Domain 1Control VLAN:Dedicated totransferring RRPPDUs
Data VLAN: Dedicatedto transferring datapackets
Secondary controlVLAN: Acts as controlVLAN on all subrings
7/30/2019 IG_00270055
86/154
Troubleshooting HP Enterprise Networks
3 30 Rev. 11.11
RRPP intersecting rings L2 frame flowTiming: 10 minutes
Materials required:
Module 3 PowerPoint presentation
Learner Guide
Display Slides 1722 of 05-M03-RRPP.pptx.
PRIMARY Ring Hellos are seen on all nodes. Subring HELLOs only on that subring.
7/30/2019 IG_00270055
87/154
Rapid Ring Protection ProtocolBasic Concepts
Rev. 11.11 3 31
RRPP intersecting rings L2 frame flow
Figure 3-15: RRPP intersecting rings L2 frame flow
The Primarys Master Node control Frames flow around both rings, where as thesubrings master node control frames only reside in its control VLAN which is kept tothe devices in the subring.
A B
CD
Port 1 Port 1
Port 1
Port 1Port 2
Port 2
Port 2 Port 2Master node
Transit node
Ring 1
Domain 1
Data packets
ERing 2
Master node
Port 3
Port 3Edge node
Assistant edge node
Edge ports:Connect edge no