NetworkTroubleshootingandTools

Domain5.0

5.0NetworkTroubleshootingandTools• 5.1Explainthenetworktroubleshootingmethodology.• 5.2Givenascenario,usetheappropriatetool.• 5.3Givenascenario,troubleshootcommonwiredconnectivityandperformanceissues.• 5.4Givenascenario,troubleshootcommonwirelessconnectivityandperformanceissues.• 5.5Givenascenario,troubleshootcommonnetworkserviceissues.

5.1ExplainTheNetwork

TroubleshootingMethodology

• IdentifytheProblem• EstablishaTheoryofProbableCause• TesttheTheorytoDeterminetheCause• EstablishaPlanofActiontoResolvetheProblemandIdentifyPotentialEffects• ImplementtheSolutionorEscalateasNecessary• VerifyFullSystemFunctionalityand,ifApplicable,ImplementPreventiveMeasures• DocumentFindings,Actions,andOutcomes

IdentifytheProblem

GatherInformation

• Lookatsymptomsoftheproblem• Reviewproblemswithusers• Reviewchangesinsoftware,hardware,appliedpolicies(bothnetworkandsecurity

• Lookatthedevice/slogs• Reviewallerrormessages• Reviewallsecuritymessages

• Beawareoflatestsecuritynews• Whatadvisoriesandknownattacksexistthatpertaintoyourenvironment?• Arethereanynewexploitsthatyouneedtobewatchfulfor?

DuplicatetheProblem,includingwithusers

• Workwithuser/sandobserveproblem• Carefullyaskuser/squestionsandlistentotheirresponses• Observeeachstepthatistakentocausetheproblem• Doestheproblemhappentoasingleuser,groupofusers,entirebuildingororganization

• Replicateproblemasanadministratordetective• Usesamestepsasobserved• Tryacompletelydifferentmethodtocompleteatasktoseeifproblemcontinuestoexist

IdentifySymptoms

• Whatsymptomsareobserved• Couldthisbeahardwareissue,connectivityissue,policiesappliedtodevicesorsoftware

• Considersymptomsandpinpointwhatarea/sbeingaffected• Isthisaproblemthatpointstoasingledeviceoruser

• Usererror,deviceerror,acombination• Isthisaproblemthatpointstoaparticularserver• Isthisadirectoryservicesproblem• Isthisasecurityproblem• Isthisafirewallproblem• Isthisacableorwirelessproblem

DetermineIfAnythingHasChanged

• Whenconsideringsymptomsoftheproblem,hasanythingchanged• Isthesymptomsoftheproblemoccurringonasinglemachinethathasrecentlybeenchangedorreplaced• Wasthereachangeinanyconfigurationofswitches,routers,firewalls• Wasthereachangeindirectoryservices• WasthereachangeinDHCP• WasthereachangeinDNS• Wasthereachangeinpoliciesappliestousersorcomputer• Canyouundoorwishtoundoanyofthem

ApproachMultipleProblemsIndividually

• Ifduringobservations,multipleproblemsseemtobeoccurring• Approachonlyoneproblematatime• Attimes,fixingthemostcommonproblemsandfixtheotherobservedproblems

• Attemptingtofixmultipleproblemscanaddconfusionandnotfixanyandaddadditionalproblems

EstablishaTheoryofProbableCause

QuestiontheObvious

• Oftenbestfirststepistoeliminatetheobvious• Theeasiestfixisoftenthebestone

• Sometimesthefirststepisnotthecorrectanswer,butstillhelpswiththesolution• Eachstepusuallytakesyouclosertothesolutionoftheproblem• Networkscanhaveswitchmisconfigurations• Portspeed,duplex/simplex,wrongVLAN,wrongIPinformation,etc.

ConsiderMultipleApproaches

• Therearetwostandardapproaches• Top-to-bottom/bottom-to-topOSIModel• Divideandconquer

Top-to-bottom/Bottom-to-topOSIModel

• Top-to-bottomstartswiththeuserapplicationandworkdownthroughtheOSImodel• Findthelayerwhereaproblemexists• Correcttheproblematthatlayer

• Bottom-to-topstartsatthephysicalanddatalinklayersandworkuptheOSImodel• Downsideismoreworkcheckingalldevices

DivideandConquer

• SelectanOSIlayer• Doahealthcheck• WorkupordowntheOSImodel• ConsidertheTCP/IPDODmodelvs.OSImodel

OSIModelvs.TCP/IP(DOD)Model

OSIModel7Application6Presentation5 Session4 Transport3 Network2DataLink1Physical

TCP/IP(DOD)ModelApplication

Transport

InternetNetwork Interface

TesttheTheorytoDeterminetheCause

OncetheTheoryIsConfirmed,DeterminetheNextStepstoResolvetheProblem• Oncedeterminethetheory,checktoseeifyoucanfixtheproblem• Formulateanddocumentstepsusedwiththetheorytoresolvetheproblem• Ifyoursolutiondoesnotfixtheproblem,BESUREtorestoretheoriginalconfiguration• Youdonotwanttointroducenewproblems/variables

IftheTheoryIsNotConfirmed,EstablishaNewTheoryorEscalate• Ifthetheoryandstepsformulatedtoresolvetheproblemdoesnotfixtheproblemanewtheorymustbeformortheproblemescalated• Devicemisconfigurationshouldbeconsideredanddependingonorganizationpolicies,escalationtoahigherlevelofexpertisewillbedone• Whenandhowescalationwillbedecidedbyanorganization’spoliciesandprocedures

• Examplesmightbeswitchingloops,routermisconfigurations,ARPproblems,powerproblems

EstablishaPlanofActiontoResolvetheProblemandIdentifythePotentialEffects

EstablishaPlanofActiontoResolvetheProblemandIdentifythePotentialEffects• Whentheproblemisidentified,theplanofresolutionsiscarriedout,thesolutionneedstoappliedandtestedforeffectsthroughoutthenetwork

ImplementtheSolutionorEscalateasNecessary

ImplementtheSolutionorEscalateasNecessary• Whenthesolution/fixisappliedandfullfunctionalityofthenetworkisevaluated• Solutionsteps,causeoffailure,completedocumentationneedstobeimplemented• Futurepreventionshouldalsobedocumented• Ifthesolutionisfoundtoaffectothernetworkoperations,anothersolutionshouldbeconsideredaswellasescalation

VerifyFullSystemFunctionalityand,IfApplicable,ImplementPreventiveMeasures

Verifyfullsystemfunctionalityand,ifapplicable,implementpreventivemeasures• Runregressionteststouncoveranychangestothesystemornetwork• Regressiontestsareare-runofanyoriginalfunctionality/securitytests

DocumentFindings,Actions,andOutcomes

DocumentFindings,Actions,andOutcomes

• Thisstepissometimesavoidedandisoneofthemostimportantinthetroubleshootingprocess• Thiscanbeusedinthefuturebyothernetworkadministrators• Importantdocumentationincludes• Whentheproblemoccurredandwhenthesolutionwasimplemented• Whytheparticularsolutionwasused• Whatchangesorfixesweremade• Otherfixesthatmighthavebeenconsideredandwhytheywerenotused• Whodocumentedandappliedthesolution

• EstablishasearchableknowledgebaseofproblemsandsolutionsforallITstafftoreferto

5.2GivenaScenario,Use

theAppropriateTool

• HardwareTools– BasicHandHeld• HardwareTools- Analyzers• SoftwareTools– TestersandAnalyzers• SoftwareTools– CommandLine

HardwareTools– BasicHandHeld

Crimper

• Acrimperisbasictoolusedtoproperlyattachconnectorstotheendofcables• RJ-45onunshieldedtwisted-pair(UTP)• BNCorFoncoaxialcable• Similartoapairofpliersbutspecializedforthecableandends• Eventalentedusersshouldhaveextraends

CableTester

• Acabletesterisusedtotesttheviabilityofthecableandconnector• Open/brokenwires/connections• Shorts• Incorrectpin-out

• High-endtestersalsoreportsignallossoncableandatconnectors• Therearetwocommonwaystotestaconnection:

• A continuitytest• A resistance test

• Therearetwocommonwaystotestforashort:• A lowvoltage test• A highvoltage test

Besuretomovethecablearoundwhiletestingtocheckforloose/intermittentconnections!

Laser/LightSource• YoucanshootalaserorLEDlightsourcedownafiberopticcable• Checktheotherendtoseeifthelightiscomingthrough• Becarefulwithlasers– donotlookdirectlyintothesource

Punch-downTool

• Usedtoterminatecableincableclosets• Pushesindividualwiresintwistedpairintotheircorrespondingconnectorona66- or110-blockpatchpanelorwalljack• Mosthaveabladebuiltintothetiptocutoffexcesswire

LoopbackAdapter

Aloopbackadaptercanberefertoseveralthings:• Ahardwareplugthattakesoutputandredirectsitbacktotheport’sinput• Checkstoseeifsignalcanbesentandreceivedonthatport

• Avirtualinterfaceonahost/device• AssignedanIPaddress• Doesn’tdirectlyconnecttothenetwork• Isreachedthroughaphysicalportonthedevice(thedeviceroutesincomingsignalinternallytotheloopback)• Usedasan“alwaysup”interfaceforremoteaccesstothedevice,diagnosticslikeping,orassigninganIPaddress-baseddeviceID

Multimeter

• Amultimeterisoneofthesimplestcable-testingtools• Checkscontinuity(nobreaks)inacable• CanalsobeusedtocheckDCresistanceonacable

• Canalsobeusedforvoltagetestsonapowersource• ACorDC• Variouspowerranges

BasicElectricityCharacteristics

• Electricalcircuitshavethreebasiccharacteristics:• Voltage=E

• Measuredinvolts• CanbeAC(alternatingcurrent)orDC(directcurrent)

• Resistance=R• Measuredinohms

• Current=I• Measuredinamperes(ormilliamperes)• MostmetersonlymeasureDCamps

OhmsLaw:E=IR(volts=ampsxohms)PowerFormula:P=IE(powerinwatts=ampsxvolts)

-

+

VoltageTests• Putmultimeterprobesinparallelwithtarget• AC– forwalloutlets/powerstrips,ACmotors• DC– forbatteries,powersupplies– putredprobeon+positiveside,blackprobeon– negative/commonside• VoltageRanges– choosetherangetargetisin,ifunknownstartwithhighestandthendialdown!• Devicesusuallyneedtohavethevoltagewithinaparticularrange• Somedevicesneedthefrequencytobeeither50Hzor60Hz

AC DC

Resistance/ContinuityTests

• Placesavoltageonthecircuittocalculatetheresistance• MAKESUREthecircuityouaremeasuringisNOTenergized!• Youcandamagethemeter!

• Putmultimeterprobesoneithersideofthecable/target• SelectResistanceRangeappropriatefortarget• Ifnotsureofrange,startwithhighestanddialdown

Power

Current(Amperage)Tests• Current• Puttheprobesinserieswiththeload,betweenthepowersourcecontacts• Ifuncertain,startwithhighestsettinganddialdown• DoNOTEVERperformacurrenttestonacircuitwithnoload!

• HighCurrent• Specialtestifthecurrentdrawisknowntobeupto10amps• Usesaspecialpositivejackfortheredprobe

• MostmultimetersonlymeasureDCcurrent• Currenttestsareveryunusualforanetworktechnician

-

+

DC

HardwareTools–Analyzers

TDRandOTDR

• Timedomainreflectometer(TDR)sendsasignalthroughacabletocheckcontinuity• Signalbouncesbackatthebreak/end• Thereflectedsignalisanalyzed

• Timeittook• Levelofsignal/light

• Veryusefulforfindingwherethebreak/openpointisininstalledcable• Opticaltimedomainreflectometer(OTDR)ususedforfiber-opticcables

OTDRTest Launchcableconnectstocablebeingtested

OTDRTrace

TypicalFeaturesofanOTDRTrace

Lightmeter

• Lightmeterisasimplertoolusedtocertifyandtroubleshootfiber-opticcable• Canmeasure/detectloss/breakagebysendinglightthroughafiberopticcable

ToneGenerator• Usedtolocateacable

• Onapatchpanel/jack• Inagroupofinstalledcables

• Veryusefulwhenyoudon’tknowwhichisthecableinquestionorwherethecableleadsto• Usethetonertoinjectawarblingsignal• Usethewandtolocatewhichcable/jackhasthesignal• Becareful:crosstalkbetweencablescanbemisleading

• Prefertousethisonnon-livecircuits• Alsoknownas:

• Foxandhound• Telephonetracer• Cabletracer• Toner

SpectrumAnalyzer• Measuresthelevelofsignal(includingnoise)acrossarangeoffrequencies• UsedtofindinterferencelevelsondifferentWi-Fichannels• Usuallyrequires:• A specializedhardwaredonglethatcanprocessANYsignaltype,notjustWi-Fi• Softwarethatcaninterpretthereading

• Somedevicesareself-contained• SomedevicesrequireaPC

Activity5.2.1– TroubleshootingwithHardwareTools• Let’susesomeanalyzertoolstohelpuslocateandfixaproblem

SoftwareTools– TestersandAnalyzers

PacketSnifferTools• Usedtocaptureandanalyzetrafficonanetwork• Requireanetworkadapterinpromiscuousmode• Mostaresoftware-based• Mosthaveprotocolanalysiscapabilities• PartofIDS/IPSfunctionality• Commonexamplesinclude:

• Wireshark• SolarWindowsBandwidthAnalyzer• PTRG• Airmon-ng• Kismet• tcpdump• Snort• MicrosoftNetworkMonitor

PortScanner• Asoftwareapplicationthatscansnetworkhostsforopenports• Anactivereconnaissancetacticbypentestersandhackers

• Usedtosearchfortargets• Openportsimplyservicesonahostthatareacceptingconnections

• Anorganization’snetworkadministratorcanscanthenetworkforopenportstohelpmakesureonlyportslegitimatelyneededarebeingused• Portstates:

• Open/listening– TCPSYNelicitsaSYN/ACKresponse– portisopenforbusiness• Closedordenied– TCPSYNelicitsaRST(reset)response– noserviceislisteningonthatport

• filteredorblocked– noresponseofanykindduetofirewallorthehostdoesnotexistatthatIPaddress

ProtocolAnalyzer

• Ahardware/softwaretoolthatcapturesandanalyzesnetworktraffic• Canidentify:

• Protocolsusedonthenetwork• Percentageofprotocoluse• Bandwidthutilizationbyprotocolorhost• Unauthorized,unknown,orpotentiallymalicioustraffic(byprotocol)• Peaktimesofutilization• Hostswithnetworkinterfacesinpromiscuousmode

• Mostlyusedbysniffers• Examplesinclude:

• SolarWindsDeepPacketInspectionandAnalysisTool• NetFlow• sFlow

ProtocolAnalyzerExample

Wi-FiAnalyzer

• AWi-Fianalyzerissimilartothenetworkanalyzerexceptitisusedforwirelessnetworks• Collectspacketsfromthewirelessnetworksanddetects:

• Acceptablenetworks,hiddennetworks,interferencebyothernetworks,devices,andothermachinery

• Canuseforwirelesssurveysforplacementofwirelessaccesspoints(WAPs)

BandwidthSpeedTester

• Softwarethatallowsyoutocheckthebandwidth(speed)ofanInternetconnection• HelpsidentifyperformanceissueswithyourISP

• Onlymeasuresspeedtoaparticularsite,nottoallwebsitesontheInternet

• Vendorsofferthisserviceasapartoftheirwebsite• Measuresdownloadanduploadspeed• Somesoftwarevendorsalsoofferlinequalitychecks• Looking-glasssitesrunasoftwarethatallowsviewingofroutingdataaswell

Activity5.2.2– TroubleshootingWithSoftwareTools• Let’susesomesoftwaretoolstotroubleshootaproblem

SoftwareTools–CommandLine

CommandLine(CLI)• Atext-baseduserinterfacetoacomputer'soperatingsystemoran application• A usertypesincommandsandreceivestext-basedoutput

• Nomouse• Nographics• Mightincludedcoloredtextormenus

• AlsoknownasaTUI(text-baseduserinterface)• AsopposedtoaGUI(graphicaluserinterface)

• Generallyusedbyadministrators/ITsupport,hackers,Linuxusers,andadvancedusers• Examples:

• CiscoCLI• Windowscmd.exeorMS-DOSprompt• Linuxbashshell

Ping

• An application that uses ICMP echo request and echo response• Used by virtually all operating systems and platforms

• The most basic network connectivity test• Verifies connectivity at Layer 3• Might be blocked by firewalls• Ping6 and Ping -6 tests connectivity on IPv6 networks

Tracert,Traceroute

• CommandlinenetworkdiagnostictoolsthattrackthepathofapacketasittraversesanIPv4network• Windowsusestracert• Unix,Linux,andMacOSusetraceroute

• Tracert-6,traceroute6,andtraceroute-6testconnectivitybetweendevicesonaIPv6network• UsesincreasingTTLvaluesintheIPheadertoinducerouters(hops)downthepathtoexpirethepacketsandsendbackinformationtothesender

HowTracerouteWorks1. Sendersendsaseriesofpackets(eitherICMPorUDP)toadestination2. Startingpacket(s)havetheTime-to-LiveintheIPheadersetto“1”3. ThefirsthoptoreceivethepacketdecrementstheTTLto“0”4. Thathopdiscardsthepacket,sendinganICMPexpiredintransitmessage

tothesender(thehopalsoidentifiesitselfinthatmessage)5. ThesendersendsafewmorepacketstothedestinationwithaTTLof“2”6. ThefirsthopdecrementstheTTLto17. ThesecondhopdecrementstheTTLto0,discardsthepacket,andsends

amessagetothesender8. Theprocessrepeatsuntilthepacketreachesthefinaldestination9. Gaps(***)intheoutputindicatethathopdidnotrespond

• It’seitherafirewallortoobusy

NslookupandDig

• Command-linenetworkutilitiesusedtoqueryaDNSserver• CanquerytheDNSserverforvarioustypesofrecords,includingafullzonetransfer(completedumpofalloftherecordsforadomain)• NslookupisusedwithWindows• Dig(domaininformationgrouper)isusedwithLinuxandUnix

Ipconfig

• AcommandlinenetworkutilityusedbyWindowsthatdisplaysthedevice’scurrentIPconfiguration• Hasvariousswitchestoreturndifferenttypesofinformation• Ipconfig/allreturnsallinformation

• Informationincludes:• IPaddress,subnetmask,defaultgateway,DNS,WINS,DHCPleaseandexpiretimes,hardware(MAC)address,DNSdomainnameonthatinterface• Informationisreturnedforeveryinterface,whetherphysical,virtual,ortunnel

• UsedtoreleaseandrenewDHCPlease• Ipconfig/release;ipconfig/renew

Ifconfig

• TheLinux/Unix/Macequivalentofipconfig• Doesnotshowexactsameinformationasipconfig• Forexample,doesnotshowtheaddressoftheDNSserver

Iptables

• Usedtoconfigure,maintain,andinspectthetablesofIPv4packetfilterrulesintheLinuxkernelfirewall• Multipletablesmaybeconfigured• Eachtablecontainsanumberofbuilt-inchainsandmayalsocontainuser-definedchains• Achainisalistofrulesthatcanmatchasetofpackets• Eachrulespecifieswhattodowithapacketthatmatcheswhichisreferredtoasa`target',whichmaybeajumptoauser-definedchaininthesametable

Netstat• Acommand-linenetworkutilitytoolthatshowsthestatus/statisticsofportsonacomputer• UsedbynearlyallPCtypeoperatingsystems• Dependingontheversion,canshow:

• Listeningports• Portswithestablishedsessions• Thestatusofanestablishedsession(LISTEN,ESTABLISHED,TIME_WAIT,CLOSE_WAIT,etc.)• Knownroutes• Amountofpacketsinandout• Numberofpacketerrors• ThePID(processID)oftheapplicationthatisusingtheport

• Examples:• netstat-nao• netstat--help• netstat/?

CommonMicrosoftNetstatSwitchesSwitch Function-a Displaysallconnectionsandlisteningports.-r Displaysthecontentsoftheroutingtable.

-n SpeedsexecutionbytellingNetstatnottoconvertaddressesandportnumberstonames.

-s Showsper-protocolstatisticsforIP,ICMP,TCP,andUDP.

-p<protocol>

Showsconnectioninformationforthespecifiedprotocol.TheprotocolcanbeTCP,UDP,orIP.Whenusedwiththe-soption,showsstatisticsforthespecifiedprotocol.Inthiscase,theprotocolcanbeTCP,UDP,IP,orICMP.

-e ShowsEthernetstatistics,andcanbecombinedwith-s.

Interval Showsanewsetofstatisticseachinterval(inseconds).YoucanstoptheredisplayingofNetstatstatisticsbytypingCTRL-C.

Tcpdump

• A commandlinepacketanalyzer• DisplaysthecontentsTCP/IPandothernetworkpacketstransmittedfromorreceivedbyahost• Availableon*NIXsystems

PathPing

• Aroutetracingtoolthatcombinestracertwithsomequalityofservicefeatures• PathPing outputincludes:• Eachhop/routerIPaddress• Lengthoftimetoreachdestination• Packetssuccessfully/unsuccessfullysent(loss)

NetworkMapper(Nmap)

• Acommandlinenetworkscannerandsecurityutility• Usedto:• Pingsweepandportscan• Identifyservicesandoperatingsystemsbasedontheirresponsetonetworkpackets• Inventoryhostsandservicesonthenetwork• Performsomevulnerabilitytesting

• Builtinto*NIXoperatingsystems• CanbedownloadedandrunonWindows

Route

• AcommandthatallowsanadministratortoviewandconfigureroutingtablesonWindowsand*NIXhosts• Examples:

routeprint=displaycurrentroutingtable

routeADD157.0.0.0MASK255.0.0.0157.55.80.1METRIC3IF2destination^^mask ^gatewaymetric^^interface#

Arp

• AprotocolformappingMACaddresstoIPaddresses• Acommandtodisplayoreditthehost’sARPcache

Example:

arp-a=displaythecurrentarpcachearp-s157.55.85.21200-aa-00-62-c6-09=addastaticmappingarp-d=clearthearpcacheofaspecificmappingoralldynamicallylearnedmappings

Dig

• A*NIX command-line toolforquerying DNSserversandtroubleshootingDNSfunctionality• Digcanfunctionincommandlinemodeorinbatchmode• Usesaspecifiednameserverordevice’sdefaultresolverconfiguredinthe /etc/resolv.conf file• Digispartofthe BIND domainnameserversoftwaresuite• Digisacomplimentarytooltonslookup

Activity5.2.3– UsingCommandLineTools

• Let’susesomecommandlinetoolstotroubleshootaproblem

5.3GivenaScenario,

TroubleshootCommonWired

Connectivityand

PerformanceIssues

• SignalIssues:• Attenuation• Latency• Jitter• Crosstalk• EMI

• PhysicalIssues:• Open/short• Incorrectpin-out• IncorrectCableType• BadPort• Damagedcable• Bentpins

• TransceiverIssues:• Transceivermismatch• TX/RXreverse• Duplex/speedmismatch

• TrafficFlowIssues:• Bottleneck• VLANmismatch• NetworkconnectionLEDstatusindicators

SignalIssues

Attenuation

• A termthatreferstoanyreductioninthestrengthofa signal• Attenuationoccursnaturallyasasignal, digital or analog,travelsfartherfromitssource• Alsoknownassignalloss• Incopperandfiberopticcables,attenuationismeasureindecibelsperfoot,kilometer,ormile• Lesstheattenuationperunitdistancemeansmoreefficientcable• Repeaterscanbeinsertedtoovercomeattenuation

Latency

• Latencyreferstoatimeintervalordelaywhenadeviceiswaitingforanotherdevicetodosomething• One-waylatencyismeasuredbycountingthetimeittakesapackettotravelfromitssourcetoitsdestination• Round-triplatencyismeasuredbyaddingone-waylatencytimeandthetimeittakesforthepackettoreturntothesource• Usedtodiagnosenetworkperformanceissues

• Sometypesoftraffic(especiallyrealtimevoiceandvideo)cannottoleratemuchlatency

Jitter

• Variablelatencyfrompackettopacket• Especiallyproblematicforreal-timestreamingtransmissions(voiceandvideo)• Makescallqualitychoppy

• Worstformoflatency• Devicesareconstantlychangingtheirreceivebuffersizestotrytoadapttovariabledelaytimes

Crosstalk

• Crosstalkisadisruptioncausedbytheelectricormagneticfieldsofonetelecommunication signal affectingasignalinanadjacent circuit• Crosstalkiscausedbycalledelectromagneticinterference(EMI)• Occurinmicrocircuitswithincomputers,audioequipment,andwithinnetworks• Occurswhenusingcoaxialcable,unshieldedtwistedpair(UTP),andevenattimeswithopticalfiber• Nearendcrosstalk(NEXT)– interferenceclosetotheoriginofthedata• Farendcrosstalk(FEXT)– interferenceatthereceivingendofthedata

• Shieldingandincreasedtwistsintwistedpairhelpreducecrosstalk

ElectromagneticInterference(EMI)

• Interferencecausedbyanelectromagneticfield• Occurswhencablesareinstallednearelectricaldevices,evennormalofficefixtures• Unshieldedtwistedpair(UTP)canbeaffected• Fiberopticcableisresistant

• Commoncausesinclude:• Motors• Elevators• Fans• Fluorescentlights• Anythingthatgeneratesanelectricalfieldarounditself

• CarefulcableplacementisessentialtoavoidEMI

Activity5.3.1– TroubleshootingSignalProblems• Let’stroubleshootsomesignalproblems

PhysicalIssues

Open/Short

• AnOpenfaultisatermthatdescribesaconditionwhereafullcircuitisnotmade• Usuallycausedbycutincableoralooseconnection

• Ashortisatermthatdescribesaconditionwherethereisaunintendedconnectionbetweenthesourceanddestinationallowingthedatatoflowtounintendeddestinations• Usuallycausedbybadwire,cutinwiresthatallowbarewirestotouch

IncorrectPin-out

• Pin-outisatermthatdescribeshowwiresincablesareinstalledinanend• Notaproblemifpurchasingfromareputablevendor• Ifnetworktechnicianmakecablesneedtousecorrectpinouts• Problemscaninclude:

• Noconnectivity,improper/problematicconnectivity,veryshortdistanceconnectivity

• Canbedetectedbyvisualinspectionorbyusingacablechecker• Aconnectorthathasbeencrimpedwiththewrongpin-outwillhavetobecutoff,andanewconnectorcrimpedonproperly

IncorrectCableTypeTherearemanypossibilitiesforchoosinganincorrectcabletype• Usingastraightthroughcablewhenyouneedacrossovercable• Usingacrossovercablewhenyouneedastraightthroughcable• Usingastraightthroughorcrossovercablewhenyouneedarollovercable• Usingacabletypethatisnotstandardscompliant• YoushoulduseEIA/TIA568Aor568B(mostuseB)

• Usingalowercablecategorywhenahigheroneisneeded• Speed,interferenceresistance,orPoEdistanceisinsufficient

• Canalsooccurfrombaddesignchoices• Insufficientcabletypechosen

BadPort

• Abadportcanmeanthataportonarouterorswitchisnotworkinginaphysicalsense• Theport’smetalpinscouldbebentorcorroded• Theelectricalcircuitryforthatportisdamaged• Thelaserdiode/LEDforafiberopticportisnotproducinglightproperly• Whentroubleshootingbadports,ensurethatthedevicedidnotdeliberatelyshuttheportoffasasafetymeasure• Happensalotwhenswitchtrunksdetectswitchingloops

DamagedCables

• Whentroubleshootingnetworkconnectivity,startingwiththesimplestsolutionsfirstisagoodidea• Checkingfordamagedcablesandwiringisagoodplacetostart• Bystartingandchangingoutacable,agreatdealoftimecanbesaved• Cablesandwiringcanbecheckusingamultifunctioncabletester

BentPins

• Pinsinendscanbebentifadeviceischangedorbentduringinstallation• Preventsconnectivity• Useadifferentport

TransceiverIssues

TransceiverMismatch

• Thetransceiverisincompatiblewiththecabling,oranothertransceiverattheotherendofthecabling• Configurationbetweendevicescouldbeincompatible:• Speedmismatch• Duplexmismatch• Singleormultimodefiberopticmismatch• Frequencyorsignaltypemismatch

• Mismatchesusuallydisplayerrorsintheportstatistics

TX/RXReverse

• TXisatermusedfortransmitandRXisatermusedforreceive• TheTXhastoconnecttoRXforeverypairofwireinnetworkcables• Usinganordinarypatchcabletoconnectsimilardevicesusuallycausesaconnectionoftransmittotransmitandreceivetoreceive• Newerdeviceshavethecapacitytoautosensethetypeofcableandcorrecttheproblem;olderdevicesmaynot

Duplex/SpeedMismatch

• Configurationscanbeincorrectifthenetworkadministratordoesnotconsiderportspeedandduplexsettings• Youmayhavemultiplechoices• AutoNegotiation• Static,suchasspeedandhalfduplexorfullduplex

• Withoutthecorrectsettings,communicationcouldbeproblematic(lotsoferrors)orimpossible

Activity5.3.2– TroubleshootingTransceiverProblems• Let’stroubleshoottransceiverproblems

TrafficFlowIssues

Bottlenecks• Thetermbottleneckinanetworkisusedtodescribeaconditionthatinwhichonedevice,interface,ornetworksegmenthastoomuchtraffic• Itholdsuppacketflowfortherestofthenetwork

• Canhavemanycauses:• Growthofnetworkandorganization• Baddeviceornetworkcard• Malware• Securitybreach

• Identifybottlenecksusing:• Network/packetanalyzer• Statusreportsfromthedeviceinterfaces(especiallyswitchandrouterports)• Statusreportsfromserversthatprovideservicesonthesegment

• Examininglogsisagoodwaytolookformalwareandsecuritybreaches

VLANMismatch

• AgeneralconditionwhentwodevicesareerroneouslyconnectedtothesameVLAN• A“NativeVLANMismatch”occurswhenthenativeVLANofaswitchportisdifferentfromthenativeVLANoftheportofanother(connected)switch• IfaswitchdetectsthatanotherswitchisconnectedbutconfiguredwithadifferentnativeVLAN,youwillseeconsoleerrormessages

NetworkConnectionLEDStatusIndicators

• Lightsondevicesthatprovidestatusinformationaboutthedevice• Caninclude:

• Power• Portinanormal(forwarding)state(green)• Portblocked(amber)• Normalactivitydetected(blinking)• Speedorduplexmismatch(rapidblinking)• Currentbandwidth/throughput/duplex• Differentlightsthatflashduringbootuptoindicatedifferentselfdiagnostictests

5.4GivenaScenario,

TroubleshootCommonWireless

ConnectivityandPerformance

Issues

• SignalLoss• Attenuation• Reflection• Refraction• Absorption

• Latency&Jitter• AntennaIssues

• Incorrectantennatype• Incorrectantennaplacement

• IncorrectWAPtype• WAPIssues

• Interference• Channeloverlap

• Overcapacity• Distancelimitations• Frequencymismatch• Powerlevels• Signal-to-noiseratio

• WAPMisconfiguration• WrongSSID• Wrongpassphrase• Securitytypemismatch

SignalLoss

Attenuation

• Signalstrengthweakensnaturallyoverlongerdistance• Absorptive,reflective,andrefractivematerialswillalsodistortorattenuateasignal

Reflection

• Reflectionisatermusedtodescribeasignalbouncingoffanobject• Inawirednetwork,thesignalreflectsoffofabreakinthewire,ortheunterminatedendofthewire• Inawirelessnetwork,thesignalreflectsoffofahardobjectsuchasawall,furniture,concrete,metal,etc.

• Areflectedsignalbouncesbackonitself,causingphasecancellation,attenuation,ordistortion• Occursalotinofficesthathavecomplexandintricatelydesignedstructuresandfurniture/equipmentplacement• Ifalargeamountofreflectionoccurs,signalscanbeweakenedandalsocauseinterferenceatthereceiver

Refraction• Thebendingofasignalwaveformwhenitentersamediumwherethespeedisdifferent• Changesthedirectionofthewave• Forexample,glassorwatercanrefractwaves• ThiscanaffectWAPplacement

• Watchoutforglasswallsorfishtanks!• Ifasignalchangesdirectionintravelingfromsendertoreceiver, thiscancause:• lowerdatarates• highretries• overalllesseningofcapacity

Absorption

• Oneofthemostcommonreactionsawirelesssignalhaswhenitencountersdifferenttypesofmaterial• Thematerialconvertsthesignal’senergyintoheat• Thesignaldoesnotreflectoffoforpassthroughanabsorptivematerial• Thiseffectivelyblocksthereceiverfromreceivingthesignal

RFAbsorptionRatesbyCommonMaterials

Material AbsorptionRate Amountofsignalabsorbed

Amount ofsignalthatpassesthrough

Plasterboard/drywall

3– 5db 50– 70% 30– 50%

Glasswallandmetalframe

6db 75% 25%

Metaldoor 6– 10db 80– 90% 10– 20%Window 3db 50% 50%Concretewall 6– 15db 75– 97% 3– 25%Blockwall 4– 6 db 40– 75% 25– 60%

Latency&Jitter

Latency

• Justasthereislatencyonawirednetwork,thereisalsolatencyonwireless• Usessamecarriersensemultipleaccessaswired,butmustputupwithmuchmorenoiseandobstacles

• Causedby:• Distance• Interferenceandretransmissions• Arrangementandplacementofwirelessaccesspoints(WAP’s)• Typeandpositionofantennae• Numberofusersonthewirelessnetwork

Jitter

• TherootcausesofjitterandlatencyonaWIFInetwork:• availablebandwidth• numberofpeopleusingtheconnection• interference

• Jitteriscausedbyvarianceintheamountofbandwidthbeingusedinthewirelessnetwork

AntennaIssues

IncorrectAntennaType• Antennaselection willhaveamajorimpactonwirelessperformance• TherearetwobasictypesofantennasforWLANs• Directional/Uni-directional

• Themorefocusedthesignalisinaparticulardirection,thestrongerthesignalisinthatdirection

• Canbe90or180degrees,oruni-directionalYAGI(straightline)• Higher gainantennascanbeusedoutdoorstoextend point-to-pointlinksoveralongerdistanceand/orcreatea point-to-multipointnetwork

• Usetohelpcontainsignalinacertainarea• Omni-directional

• Designedtoradiatesignalsequallyinalldirection,butwithaweakersignalforall• Useifyouneedtotransmitfroma centralnodetousersscatteredallaround anarea

• TherearealsoCPE(customerpremisesequipment)antennas

IncorrectAntennaPlacement

• Positionantennawhereitcanprovidethemaximumbenefitwiththeminimuminterference• Createaheatmap/spectrumanalysistolookfordeadspots• Ensureenoughantennas/WAPsexisttoprovidedesiredcoverage• Makesuredirectionalantennasarepointedintherightdirection,andnotevenslightlyoff

• Makesure90degreeantennasareinthecornerofanareapointedinward• Makesure180degreeantennasareontheborderwallpointedinward

• Inpoint-to-pointlinks,maintainline-of-sightbetweenthetransmitterandreceiverantennas asmuchaspossible• Placethereceiverantennasothatit’satacorrectdistancefromthe transmitter• Usetherighttypeofreceiverantenna• Locateantennasawayfromanysuspectedsourcesofinterference• Carefullyaligntheantennasformaximumsignalgain

IncorrectWAPType

IncorrectWirelessDeviceType

• ChooseaWAPtypethatisappropriateforyourenvironment• Don’tuseawirelessPtPbridgeasanAP– frequencymaybewrong• Don’tuseaCPE(customerpremisesequipment)asanAP– beamisnarrowlyfocused,meanttotravel15kmormore;notsuitableforuserswhomaybebroadlyscatteredaroundthesite

• EnsureyouuseWAPsandcontrollersthatcanhandletheaggregatetraffic/numberofconnections• Forexample,aSOHOWAPcanusuallyonlyhandle10connectionsatanyonetime• ACiscoLWAPPWAPcanhandle50ormoreconnectionsatatime• DedicatedwirelesscontrollerscantelltheWAPstoloadbalanceclientconnections(whenpractical)

Non-APWirelessStationExamples

Wirelessconnectivity+PoEforvideosurveillance

CPEwithline-of-sightconnectivitytoISP(couldalsobeaPtPbridgeonyourcampus)

CPEPoint-to-PointWirelessLinkExample

CPErangecanreachprovidertowerat15kmormore

Mightbe2.4GHz,butantennashapeisfordirectional,not

omni-directionallink.NotmeanttobeanAPforusersatthe

customersite.

CPEPoint-to-MultipointWirelessLinkExample

EachcustomerhasaCPEtoreachtheprovider’stower.Again,thisisnotmeanttobeanAPforendusersatthe

customerlocation.

WAPIssues

Interference• Wirelessinterferenceisatermthatreferstoanythingthatwouldimpedethewirelesssignal• Somecauses/solutionsinclude• Physicalobjects– moveantennas• Busychannels– changechannels• RFI/EMI– moveWAPsawayfromsourcesofinterference• Toomanyusersonthewireless– addaccesspoints,configuregoodplacement,loadbalanceusers• Nonwirelessdevices– wirelessphones,microwaves,wirelessvideocameras– changechannel

• BadelectricalconnectionscancausebroadRFspectrumemissions– fixconnections• RFjamming– DDoSattacks– shieldthenetworkifpossible

ChannelOverlap

• Inthe2.4GHzband,1,6,and11aretheonlynon-overlappingchannels• Overlappingchannelsarenotaproblemifnooneisusingtheadjacentchannels• Thereare25non-overlappingchannelsinthe5GHzspectrum• Putachannelplaninplacetoavoidaself-inducedperformanceproblem• Useaspectrumanalyzertoidentifypotentialchanneloverlap

Overcapacity

• WhenplanningaWirelessnetworkitisimportanttodoalegitimatewirelesssurvey• Takeintoconsiderationtheareaofcoverage• Numberofexpectedusersintheareacoverage,includingnumberofdevicesperuserandguests• Typeofantennaneeded• Placementofantennas• Objectsthatmayinterferewiththewirelesssignals

• EnsureyoudopropercapacityplanningincludingaggregatedatathroughputonalluplinksandswitchestheWAPSconnectto• Useyourwirelesscontrollertoenforceconnectionlimitsandpolicies

DistanceLimitations

• Planningwillhelpwithdistancelimitations,butasanorganizationgrowssignalsmayreachlimits• UseanRFamplifiertoincreasesignalstrength• Considerusingarepeater/rangeextenderbeforeinstallingadditionalAPs• ConsiderrelocatingAPs• Considerreplacingantennas• Consideraddingbridges(orinstallingawiredconnection)toreachadditionalareas

FrequencyMismatch

• Makesurethatclientscanusethesamefrequency/channelastheWAP• Don’tuseaJapanesemodel(thatgoesuptoChannel14)intheUS

• Makesureyouprovideforboth2.4GHzand5GHzclients• Considerifsomeclientsalsouse900MHz• CertainproductssuchastheUbiquitiNanoStationNSM3orNSM365usedifferentfrequencies(3GHz,3.65GHz)tocreateawirelesspoint-to-pointbridge• TheymightlooklikearegularAP,buttheyarenotdesignedtohandleclientconnections

PowerLevels

• Somedevicesallowyoutoconfigurehigherpowerlevels• Ifyoucannotincreasethepowerofadevice,upgradethedeviceoraddmoreAPsforcoverage

Signal-to-NoiseRatio

• Therelativepoweroftheradiosignaltothenoisefloor• AkaS/N• Youwanttheradiosignalleveltobeasfarabovethenoiseflooraspossible• Ifitisbelowthenoisefloor,itbecomesharderto“digitout”ofthesurroundingnoise• TheSignaltoNoise(S/N)ratiocanbeincreasedbyprovidingthesourcewithahigherlevelofsignaloutputpower

WAPMisconfiguration

WrongSSID

• YoumightchoosetonotbroadcasttheSSIDforsecurityreasons• Theuserattemptstomanuallyconfiguretheconnection• UseswrongSSID• NotrealizingthattheSSIDiscasesensitive

• Or,theuserisattachedtoaneviltwin

WrongPassphrase

• TheencryptionorpassphraseisnotconfiguredproperlyontheWirelessAccessPoint(WAP)• Theuserdoesnotknowthecorrectpassphraseorthatthepassphraseiscasesensitive

SecurityTypeMismatch

• Clientmightbetryingtousethewrongencryptiontype• MostclientdevicescanautodetectifthesecurityisWEP,WPA,orWPA-2• Olderclientsmighthavetobemanuallyconfigured

Activity5.4- TroubleshootingWAPIssues

• Let’stroubleshootsomecommonWAPissues

5.5GivenaScenario,

TroubleshootCommonNetwork

ServiceIssues

• CommonNetworkIssues• PhysicalConnectivity• IncorrectIPAddress• IncorrectGateway• IncorrectNetmask• NamesNotResolving• UntrustedSSLCertificate

• DHCPIssues• DuplicateIPAddresses• ExpiredIPAddress• RogueDHCP• ExhaustedDHCPScope

• Firewall/ACLIssues• BlockedTCP/UDPPorts• IncorrectHost-basedFirewallSettings

• IncorrectACLSettings

• AdvancedNetworkIssues• DNSServerIssues• DuplicateMACAddresses

• IncorrectTime

• UnresponsiveService• HardwareFailure

CommonNetworkIssues

PhysicalConnectivity

• Verifylinklights• Sendandreceiveonbothsides

• Verifycable• Cablemightbeinsufficientforneed

• Wrongcategory• Toolongforrequirement

• Mightstillseelinklights• Onebrokenwirewillbreakthesignalbalance

• PermittingEMI/RFI

IncorrectIPAddress

• Useipconfigtoverifycurrentconfiguration• Ensureinterfaceisusingappropriateaddressingmethod• DHCP,static

• BounceinterfaceorchangeIPconfigurationmethodtoclearconfig

IncorrectSubnetMask

• Causesahosttomakeawrongforwardingdecision• IfthedestinationIPaddressisinthesamesubnet,thehostARPstofindtheMACaddressofthedestinationandthensendsthepacketdirectlytothedestination• Ifthedestinationisinadifferentnetwork,thehostARPstofindtheMACaddressofthedefaultgateway,andthensendsthepackettothegateway• Subnetmaskmust:• Bethesameforallhostsonasubnet• NotallowIPaddressoverlapbetweensubnets

IncorrectGateway

• Willpreventtrafficfromleavingthelocalnetwork• Hosthasamissingorincorrectdefaultgateway• Routerhasthewrongaddressorsubnetmask

• Verifyconfigurationonlocaldevice• ipconfig/all

• VerifyconfigurationonDHCPserver• Verifyconfigurationonrouter• showipinterfacebrief(Cisco)

NamesNotResolving

• Checkfornetworkconnectivity• Pingbyname• VerifyyourDNSserverIPaddressesarecorrect• PingtheIPaddressofthehostyouaretryingtogetto(ifitisknown)• VerifywhichDNSserverisbeingusedwithnslookupordig• VerifyyourDNSsuffix• ReleaseandrenewyourDHCPServerIPaddress(andDNSinformation)• Rebootyourcomputerand/orrouter

UntrustedSSLCertificate

• AnUntrustedSSLCertificateisacertificatethat• Hasexpired• Isnotsignedbyalegitimatevendor• Shouldnotbetrusted• Canbearesultofusinganolderandnonsupportedwebbrowser

• Usersneedtobeinstructedtonotvisitthesite• Browsersthatusersarerunningshouldbecurrentforsecurityreasons

DHCPIssues

DuplicateIPAddresses

• Thecomputerordeviceshouldshowanerror• ThecomputerordevicecouldhavebeengivenaddressinformationstaticallywhileontheDHCPserverconfigurationwasnotreservedoroutsidetheDHCPrange• AttempttogetadifferentleasefromtheDHCPserver• Rebootthehosttoclearconfig

ExpiredIPAddress

• ClientsthatreceivedIPaddressinformationfromaDHCPserverattempttorenewtheirlease• Usuallywhen½oftheleasetimehasexpired,andthenatregularintervals• TheDHCPservercouldbedownorunavailable• DHCPclientmightnotbeawarethattheDHCPserverwaschanged• AWindowsorMacclientwillself-assignanAPIPAaddress– 169.254.0.0range

• Reboottheclienttoclearoutanyexistingleaseandattempttoobtainanewlease

RogueDHCPServer• ARogueDHCPServerisaserveraddedtothenetworkbyunauthorizedpartyandisnotabletobeconfiguredbytheorganization’slegitimatenetworkadministrator• Usuallyhappenswhensomeoneaddsawirelessroutertothenetwork,leavingthedefaultDHCPservicerunningonthedevice• CancauseaclienttoreceivefalseIPaddressinformationtocreateman-in-the-middleanddenial-of-serviceattacks• Identifyandtakedowntheroguedevice

• LooktoseeiftheSSIDgiveshintsastothelocationorpersonwhoinstalledit• YoumayhavetouseseveralWi-Fimobiledevicesinacoordinatedefforttotriangulatethelocationoftherogue

ExhaustedDHCPScope

• TheDHCPserverranoutofaddresses• TheexhaustionofaDHCPscopeindicatesthattheorganization’snetworkadministratorhasnotplannedforthegrowthofthenetwork• TheleasetimemightalsobetoolonginthecaseofamobileworkforcethatcomesinandoutoftheLAN• NeedtoincreasethenumberofIPaddressinformationinthepool,and/ordecreasetheleasetime

Firewall/ACLIssues

IncorrectHost-basedFirewallSettings

• Asstatedinthepreviousslide,someblockedsettingscouldhavebeenconfiguredonthefirewall• Examinetheuser’spersonalfirewallsettingstoseeiftheypermitthecorrecttrafficin/outofthedevice

BlockedTCP/UDPPorts

• Thiscouldbetheresultoffirewallorrouterrulesandarecorrectlyblocked• Nochangemaybeneeded

• Couldbeamisconfigurationofrulesonthefirewallorrouter• Especiallylikelyonauser’slaptoporsoftwarefirewall

• Anexceptioncouldbeconfiguredforaspecificclientoruser

IncorrectServer/ServiceACLSettings

• ACLsettingsarenotonlyforfirewallsandpacketfilteringrouters• AnAccessControlList(ACL)isusedtodefinewho/whatcanaccessthesystem• Ifcorrectlyset,willblockunwanteddata/packets• BlockshackershavingaspecificIPaddress• Agreatsecuritysolutiontounauthorizedaccessoftheorganization’snetwork

• AserverorprintermighthaveamisconfiguredACL• Causestheusertomistakenlythinkthereisanetworkproblemwhentheycannotconnect

AdvancedNetworkIssues

DNSServerIssues

• Verifynonetworkissuesorobstructivefirewallsettings• Testwithdigornslookup• CheckDNSconfiguration• Verifyrecordsinzone

• RestartDNSserverservice• ClearDNSresolvercache

DuplicateMACAddresses

• MACaddressesareassignedtodevicesandareunique• TwodevicesusingthesameMACaddresswillcauseaswitchtoforwardthetraffictobothdevices• Ifduplicatesarefound,aroguedevicemaybepartofamalwareattack• Locateanddisableunauthorizeddevice/switchport

IncorrectTime

• IfaclientorserverisnotbeingupdatedbyaNetworkTimeServerusingNetworkTimeProtocol(NTP)itisinsecureandneedstobeupdatedtouseNTP• Mayupdates,patches,andsecurityupdatesdependonthetimeontheclientusingthecorrecttime• MicrosoftActiveDirectoryloginsdependontimesynchronizedserversandclients• Thisisacriticalconfigurationthatneedstobekeptuptodate

UnresponsiveService

UnresponsiveService

• Aservicethatdoesnotrespondcouldbedueto• Anoverloadonaserverorservers• Aserverbeingdown• Incorrectconfiguration• Malwareattack

• Testing• Seeifsomeoneelsecanconnect• Telnetorportscantoseeifserviceisresponding• Checkserverconsole/logs

• Solutions• Restartservice• Addcapacity• Replaceorrepairtheserver• PatchtheOSorapplication• Reconfiguretheserverandservice

HardwareFailure

HardwareFailure

• Useastep-by-steplogicalapproachtotracedownahardwarefailureonthenetwork• Divide-and-conquertoeliminatewholenetworksegments

Activity5.5– CaseStudy:TroubleshootinganUnusualNetworkIssue• Let’sexaminearealworldtroubleshootingcasestudy