LTE Test Case App Note

7/31/2019 LTE Test Case App Note

1/32

Application Note

WEBSITE:www.jdsu.com/test

LTE and EPC Test

An Overview of Test Concepts and Tools for Trials

Application Note

Specic LTE Test Areas 3

Overview3

Gainingdeeperinsights4

Performancetesting5

Single-userthroughputtesting5

Cellperformanceverificationwithmulti-userthroughputtesting 6

Realisticmulti-userthroughputtesting8

Idle-to-activetransitiontimes9

Latencytesting10

KPIverificationandcalculation10

ValidatingLTEvoice12

Signalingvalidation13

VoiceQoSandQoE(MOS) 13

TestingQoSandQoEofLTEstreamingvideo14

EvaluatingLTEMIMOandfrequency-selectivescheduling 15

Testingnetworkcoverage16

TestingLTEhandover16

ValidatingLTEbackhaul18

VerifyingLTEhandsetIOT18

Validatingdeviceconfiguration18

Table o Contents


2/32

Application Note: LTE and EPC TestAn Overview of Test Concepts and Tools for Trials 2

Appendix: Outline for a Basic Phase 1 LTE Test Plan 20

Overview20

Achievabledataratesandlatency:single-userthroughputforUL/DLandTCP/UDP20

Achievabledataratesandlatency:cellthroughputandMUthroughputforUL/DLandTCP/UDP21

Achievabledataratesandlatency:latency21

Intra-LTEmobility:mobilityandhandoverperformance 21

Achievabledataratesandlatency:applicationperformance 22

Coverageandcapacityradiofeaturesefficiencyandgainassessment:linkbudget 22

Coverageandcapacityradiofeaturesefficiencyandgainassessment:scheduler22

Evaluationofantennaconfigurationoptions23

Self-configurationandself-organizingnetworkfeatures 23

Evaluationoffrequencyreuse:onedeploymentscenario23

BasicQoS:userdifferentiationbetweennon-GBRuserswithdifferentQCI24

BasicQoS:user-differentiationbetweenGBRandnon-GBRusers24

Basicapplicationperformance:webbrowsing,streaming,voicecalls,e-mail,VPN,on-linegaming25

References 26

EPSspecificationreferences 26

3GPPreferences26

NGMNreference28

ETSIreference28

Glossary 29


3/32


Specifc LTE Test Areas

Overview

Creatinganoverallframeworkforthetesting,evaluation,andoptimizationofLTEandSAEisalargeandcomplextopicThisdocumentprovidesastartingpointthatcoversthetopicatarelativelyhighlevel

LookingacrossthelifecycleofatechnologysuchasLTE/SAE,thetools,processes,andmeasuresmustbetailoredtosuitorganizationalprioritiesatspecifictimeswithinthelifecycle(Figure1)JDSUprovidesacost-effectivesetofsolutionsthatenablereusingassetsacrossthelifecycleThisensuresnotonlycompletecoveragebutalsothereuseofresultsandassets,leadingtoasolidreturnoninvestment(ROI)

TheexampleslistedbelowaddressdifferentaspectsoftestingatvariousstagesintherolloutofLTE/SAETheseareofferedasanoverviewofthemajorelementsoftheLTEdeploymentlifecycle

Technologyeldtrials

EvaluatethetechnologyagainstNGMNorotherindustryrequirements

Testandvalidatetechnologyimplementations

Adoptaprocessthatismoreopenthanfullyclosedbilateraleldtrials Labtesting

EvaluateeNodeBschedulingperformance

Evaluatesecurityandbillingpolicies

Evaluateserviceperformanceincontrolledenvironments

EvaluateUEperformance

EvaluateMIMOperformancegains

Fieldtrialsandvendorevaluation

Evaluateend-to-end(E2E)performance

Evaluatenetworkcoverage

Evaluatecellandnodeperformanceunderloadedconditions

Evaluateself-optimizingnetworkcapabilities EvaluateIRATcapabilitiesandperformance

PerformKPImonitoringandbenchmarking

Figure 1. A simple model of the LTE deployment lifecycle

Technology

eld trialsLab trials

Field trialsand

vendor

evaluation

Friendly

customer

trials

Commercial

launch

Optimzationand

wider

deployment


4/32


Friendlycustomertrials

Understandend-userQoE/QoS

Conrmhistorictroubleshootingcapabilities

TesthandsetIOT,conformanceandpre-conformance

Monitorhandsetperformance

Includetrending,statisticsandVIPreporting

Commerciallaunch

Performservice-andaccesstechnology-awaremonitoring

VerifyE2Evisibilityandtroubleshootingcapabilities

Validateandunderstandsubscriberbehavior

UnderstandoverallnetworkandIRAThandover(HO)performance Includeintegrationwithnode-loggingcapabilities

Optimizationandwiderdeploymentphase

ValidateandensureSONcapabilitiesareworking

Locateareasforexpansion

Monitorbackhaulperformanceimpactonend-userQoE

Benchmarkserviceperformancebetweenmacroandfemtoroll-out

TheremainderofthisdocumentfocusesonthefieldtrialsportionoftheprocessEachsectiondescribesthetestcasesmostoperatorswillwanttoundertakeandalsooutlinesthetoolsthatwillensureeffectiveperformanceofsuchtestsItisassumedthatnotalloperatorswillperformallstepssowillneedtomatchthetestcases(andpossiblesolutions)touniqueneeds

Gaining deeper insights

Foryears,personnelfromacrossAgilentTechnologieshavebeeninvolvedwithtechnologystandardizationandtestdevelopmentforLTEandSAESince2008,AgilenthasactivelyprovidedtoolstohelpdeveloperstakeLTE/SAEforwardtothemarketplaceInApril2009,thecompanypublishedacomprehensivebookcalledLTEandtheEvolutionto4GWireless:DesignandMeasurementChallenges(ISBN:978-0-470-68261-6)ThisresourcehasreceivedsignificantandwidespreadpositivefeedbackfromthewirelessindustryInMay2010,JDSUacquiredtheNetworkSolutionsDivisionfromAgilentWiththisacquisition,severalcontributingauthorstothisLTEbookincludingtheauthorofthisapplicationnotetransitionedfromAgilenttoJDSUThebookisjustoneexampleofhowAgilentandnowJDSUcontributetothe

overalllandscapeoftheLTEandSAEindustryAsacompaniontothebook,thisapplicationnotecontainsrelevantreferencestotheLTEbook,whichcanbestudiedseparatelyinpursuitofadeeperunderstandingofthevarioustopicsandconceptsinvolvedinLTEtesting


5/32


Perormance testing

WithLTE,muchofthefocushasbeenonincreasingsystemperformanceinareassuchasend-userthroughput,latency,andidle-to-activetransitionsAlthoughtheseimprovementsalongwithsignificantlyimpressivesingle-userdataratesareimportantformarketingreasons,inthemselvestheyplayalimitedroleinhowrealuserswillutilizeadeployedLTEnetworkThus,measurementsareneededforbothsingle-userpeakratesandmulti-userthroughputInanalyzingmulti-userthroughputandoverallcellcapacity,itisimportanttounderstandtheimpactofacellsrealdistributionofusersaswellasthemobilityandusagepatternsofthoseusers

Single-user throughput testing

Intheory,single-userthroughputtestingisquitesimpleHowever,inrealityitcanprovetobequitetrickyAsanexample,thefirstaspectstounderstandare what to measure and how to benchmark the result

LookingatthepublicresultsoftheLTESAETrialInitiative(LSTI)ProofofConcept(PoC)group,itcanbeseenthatpeakrateswillvaryfromafewhundredkilobitspersecondatthecelledgetoover150Mbpsinverygoodradioconditions(forexample,ina20MHz2x2MIMOsystem)Inpractice,measuringthiswiderangeofratescouldbeperformedwithadedicatedhardwaretrafficgeneratorthatwillhaveguaranteedperformanceandwillproducetrafficpreciselyaccordingtoananticipatedtrafficprofileSuchaprofilecouldincludevaryingdatarates,differenttypesoftraffic,anddelaycharacteristicssuchasjitterOtheralternativesincludesoftware-basedtrafficgenerators(forexample,iperf)orsimplygeneratingthetrafficfromexistingstandardapplicationserversusedforFTPorvideostreaming

Ifasoftware-basedtrafficgeneratorisused,itisimportanttounderstandtheconditionsunderwhichitwilldeliverreliableresultsIngeneral,currentlyavailablesoftware-basedtrafficgeneratorsproduceacorrectaveragethroughput;however,theinstantaneousvariationscouldbesignificantTogenerate

highrates,dedicatedCPUresourcesmustbeconsistentlyavailabletothegeneratorShouldotherprocessesstartonthetrafficPC,andthisisoftenthecasewithcertainoperatingsystems,theremaybecomplicatedsideeffectsintheEPCortheeUTRANbecausebufferingmightoccurunexpectedlyConsequently,theremaybeagapinthegeneratedtrafficandthusnothingtotransferovertheairinterfaceforagivenTTI

Ifnotmanagedproperly,theuseofanexistingapplicationserver(forexample,anFTPserver)togeneratethetrafficcouldalsoproduceunexpectedsideeffectsForexample,anFTPservernormallyaccessesafilefromatraditionalharddiskIfmultipleuserstrytodownloadthesamefileatthesametime,therecouldbeabottleneckcausedbytheFTPserverratherthantheLTEairinterface(ingoodradioconditions)ItispossibletosetupanFTPservertomanagethisifpropercareistakenandtheappropriatedisksystemisapplied;however,oversightsassimpleasanimproperlyconfigureddiskcanskewtheoverallresults

Whenitcomestomeasuringtheperformance,oneessentialruleofthumbistounderstandthebasic

aspectsthatwillcontroltheresultsForexample,anunderstandingoftheunderlyingradiolayertransportsettingsiscrucialHARQprovidesanexample:ifHARQisnotenabled,thelinkwillactuallydeliverahigherthroughputingoodradioconditionscomparedtotimeswhenHARQisnotswitchedonOntheotherhand,reliabletransportdependsonHARQbeingswitchedonIfnot,problemssuchasasignificantamountofTCPretransmissionwillcropup,leadingtoaverypooreffectivethroughputEvenifHARQisswitchedon,theamountofHARQretransmissioncanbeconfiguredSettingthistoaverylowvaluewillincreasetheinstantaneousthroughputbutwillleadtoapooreffectivethroughputundernon-idealradioconditions


6/32


Foradetailedunderstandingofbearerthroughput,measurementsshouldbemadeatdifferentlayersintheprotocolstackMAClayer,IPlayerandUDPorTCPlayersafterapplicableretransmissionsDuringthesemeasurements,itisimportanttounderstandandrecordtheactualsettingsthatwereconfiguredfortheMAClayerandtheTCPlayer,andifanyspecificserviceorapplicationlayersettingshavebeenapplied(forexample,iftheapplicationusingTCPhadoneormultipleTCPflows)

InsomeveryspecificcasesitmaybeusefultomeasurethehighestpossiblebitratethattheLTEairinterfacecandeliverForthismeasurement,therecommendedapproachistomeasurethelossless(forexample,zeropacket-lossthroughput)ofasustainedUDPstreamInsuchcases,thespecificMAClayersettingsshouldberecordedaswellastheUDPpacketsizeItcouldaddsubstantialvaluetoperformanRFC2544testovertheLTEconnection;thiswillstepthroughseveraldifferentpacketsizesandthroughputratestolocatethelosslessthroughputforeachrelevantpacketsize

Tounderstandwhythroughputischangingindifferentenvironmentsatdifferenttimes,thebestapproachistorecordasetofLTE-relevantparameterswhileperformingthethroughputtestTheseparametersshouldincludetheinputstotheeNodeBschedulingdecisions(forexample,CQIforalloftheranks(widebandandsub-band),thePMI,andtherankindicator)andinformationabouttheresultingeNodeBschedulingdecisions(forexample,selectedmodulationandcoding,MIMOmode,etc)Whentestinguplinkperformance,therelevantpower-controlinformationshouldalsobelogged

JDSUprovidesallofthetoolsandprocessesneededtoplan,perform,evaluateresults,andprovidereportsforsingle-andmulti-userthroughputtestingThisincludesbothtoolstogenerateE2EtrafficaswellastoolstologdatafromhandsetsandrelevantnetworkinterfacesFortestinginthefield,signalsourcesavailablefromothervendorsemulateuplinkinterferenceandloadingcausedbyusersinothercellsBecauseloadinghasasignificantimpactonthroughputratesitshouldbeincludedaspartofanyrealisticevaluationThegeneratedinterferenceshouldideallyberepresentativeofanLTEairinterfaceinbothULandDLAlthoughwhitenoisewasacceptableforUMTS,whichusesanoise-likeCDMAairinterface,itdoesnotrepresentthepowervariationsacrossfrequencyandtimethatcharacterizeLTEsOFDMandSC-FDMAmodulationschemesRapidvariationsmaytrickschedulersintomakingeitheroptimisticorpessimisticpredictionsofthechannelconditionsandsubsequentmodulationandcodingschemethatcanbereliablytransmittedJDSUtoolsprovidethemeanstoevaluatethisimpactbeforenetworksbecomeheavilyloaded

Cell performance verification with multi-user throughput testing

MeasuringtheperformanceofanLTEsysteminasingle-usercaseprovidesabasicunderstandinguponwhichamorerealisticanalysisofthemulti-usercasecanbeperformedItshouldbeunderstoodthat,inmanycases,single-userpeakthroughputperformancewillbemuchhigherthaneventheaggregatecellthroughputwithseveralusersactivelydownloadingFromindustrystudiesperformedbyLSTIandothers,DLcellthroughputisexpectedtobearound35-40Mbps(assuming20MHzwith2x2MIMO)with10usersspreadoverthecell,allwithfullbufferdownloadsandaproportionalfairscheduler 1,2

1eseexpectationsarerelativelyconsistentwithprevious3GPPsimulations2etermfullbuerreferstotheideathattheeNodeBisconstantlyschedulingdatafromtheS1interfaceebuermustalwayscontain

enoughdatatofullysaturatethelinkeveninthecaseofaperfectRFenvironmentandthehighestthroughoutrate


7/32


Whenbasicmulti-usertestingisperformed,theresultsmayvarygreatly,spanningfromvaluesbelowtheanticipatednumberstoratesnearthesingle-userpeakperformanceThereasonforthiswidevariationisduetotwokeyfactors:theoverallbehavioroftheeNodeBsschedulingandtheenvironmentforeachindividualuserItisthereforeimportanttosetupthetestinarepeatablewaytoensurethatresultsareusefulontheirownandcanbecomparedagainstsimilartestsineitheradjacentcellsinthesamenetworkorwithcellsthatusenetworkequipmentfromanothervendor

TheprocessofsettingupatesttoverifycellperformancewithmultipleusershastoomanystepstopresenthereindetailInoutlineform,herearethekeysteps:

1 EvaluatethedistributionofsignalqualityinthenetworkthroughexhaustivedriveorwalktestingtobuildaCDF

a IftheCDFsfordifferentcellsaresignificantlydifferent,theendresultwillbesignificantlydifferent

b ThedistributionwilldependonthetypeofterraincoveredandwherethewalkordrivetestisperformedForexample,includingindoorlocationswillproducelowersignalqualitiesthanoutdoor-onlydrivetesting

c Itisexpectedthatroughly80percentoftheLTEdatatrafficwillbegeneratedindoors

2DistributetheUEsaccordingtothesignalqualityCDFtoobtainapropertracprole

a ThechoiceofdistributionshouldmatchnotonlythemeasuredCDFbutalsotheanticipateddistributionofrealusersintherealnetworkForexample,ifitisanticipatedthatseveralusersarelocatedinagroupinapubliclocation(forexample,anairportorcaf)thenitisadvisabletoplacesomeoftheusersinsimilarpatterns

b DifferentMIMOconditionsmightbestressedduetothistypeofdistribution(forexample,multi-usergainsfromMIMO)

3Intherealnetwork,locatetheusersaccordingtotheplanneddistribution

a Notethatitwill benearlyimpossibletogetanexactmatchsoitisimportanttoinsteadlocatetheusersinanareasimilartotheoneidentifiedMakesurethattheactualradioconditionsofthechosenlocationsareloggedandstoredforthecompletedurationofthetest

b Forcasesinwhichsomeoftheuserswillbeinamovingenvironment,makesurethatthiscanbemanagedinacontrolledway

4EnsureOCNGisenabledintheDLfortheadjacentcellstocreateDLloadinginthecell

5EnsurerealisticULloadingisgeneratedfromUEsinadjacentcells

6GeneratetractothetestUEsandensurethateachUEissetuptoreceiveortransmittracwithfullbuers

7Logalloftherelevantparametersasidentiedearlierincludingthelocation(latitudeandlongitude)ofeachUE

a Logallrelevanttrafficfromthenetworkinterfacestoensurethatfullbufferswerepresentforall

UEsinallconditions

b VerifythatallUEswereactiveandthattheindividualUEthroughputratesarerealisticcomparedtoexpectations

8CorrelatetheUEsentireindividualthroughputforMAC,IP,andUDPorTCPlayers

a IfoneorseveraloftheUEsweremoving,theaggregatedthroughputmaychangeovertimedependingontheschedulingandspecificradioperformanceofeachdevicesenvironment


8/32


ItshouldbenotedthatthisisnotarealisticwaytotestacellsactualperformanceinarealnetworkscenariowhenbuffersarenotalwaysfullInstead,itisatestofhowtheeNodeBwouldbehaveinaveryspecificscenarioThiscouldnormallybeconsideredaworst-casescenarioforaspecificdistributionofusers

Realistic multi-user throughput testing

Despitethecomplexity,therearestilllimitationsinmulti-usertestingthatusesfullbufferdownloadsanduploadsInarealistictrafficscenario,usersareexpectedtohaveburstytrafficprofiles,possiblycomprisingHTTP,voice,FTP,IM,e-mail,etcInthefull-buffercase,userswithgoodconnectionsdownloadmoredatathanuserswithlowdatarates,whichbiasesresultsAlthoughthisispartlytrue,userstendtodomorewithafasterconnectionthefull-buffercaseisperhapstooextremeThequestion,then,iswhatwouldbeamorerealistictrafficmodel?Woulditbevaluabletopursuethis

testingand,ifso,whatistheaddedvalue?TheanswerdependsonwhichphaseofthetechnologythespecificoperatorisinatthetimeIftheoperatorisabouttoperformvendorselection,thiscouldprovetobeanessentialtesttoensurethattheequipmentbehavesappropriatelyintermsofscheduling,abilitytodeliverthedesiredQoS,andoverallfairnesstothedifferentusers

ItisalsoimportanttounderstandthatdeliveringtheexpectedQoSorbeingabletoprovidefairnessinthesystemisnotnecessarilyadifficulttaskforaneNodeBInpractice,thedifficultyistheabilitytodelivertheexpectedQoSwithaminimumofoverheadfactorsthatimpacttheoverallcellcapacity

Thefollowingisaproposedtesttohighlightcertainpossibledeficienciesinasystem

1QoSoverheadprovisioninganalysis

a Setupastaticmulti-usercelldownloadwith6to10usersperformingfull-bufferdownloads

b Measuretheoverallcellcapacity

c Reassignoneuserasamobileuserwithafixed-rateUDPstreamwiththesamemegabit-per-secondratethatwasachievedinthefull-bufferdownload

i SettheQoSparametersforthisusertomatchthefixedrateoftheUDPstream

ii Verifythatthethroughputrateisintherangeof2to5Mbps

d Measuretheoverallcellcapacity

e Movetheuserfrommediumtopoorradioconditioninfivestepsandrepeatthecell-capacitymeasurementsforeachstep

f Movetheusertothecelledgeforthespecificdataratethatwasprovisioned

g Measuretheoverallcellcapacity

ComparethecellcapacityrstwithandthenwithouttheQoSprovisionedforthebandwidththatcanbedeliveredwithoutanyactualimpactontheoverallcellcapacityencompare,stepbystep,theoverallimpactofusermobilityoncellcapacityandguaranteedQoS

2Testtheimpactofrealistictracbehaviorontheoverallscheduling

a Setupagroupofusers(6to10)witharealistictrafficprofileforthetypeofusersenvisionedfortheLTEnetworkThefollowingareexamplesofusescenarios

i Downloadfivewebpageswitha30-seconddelaybetweeneachdownloadEachpagecontains20objectsforatotalof14MBperpage

ii Receiveane-mailmessagewithalarge(forexample,10MB)attachment


9/32


iii Send10e-mailmessages,fourwith2-MBattachmentsandsixsmalloneswithonly50KBofdataineach

iv ListentoInternetradio(256Kbps)

v PerformabackgroundFTPdownloadofsevenlargefiles(totalof100MB,similartoamonthlyoperatingsystemupdate)

vi MakeoneVoIPcallusing122KbpsonaguaranteedQoSbearer

NotethatthetestconditionsmustbesetupsuchthattheradiointerfacewillbecongestedincertainconditionsduringtestexecutionUsethepreviouslymeasuredanticipatedcellcapacitynumberstoselectthecorrecttest-caseparameters

b Thesequenceofeventsoccursinparallelforeachuser;however,theusersareseparatedintimeby10secondseach

c EachuserwouldbelocatedinaspecificradioenvironmentdefinedbytheCDFforthespecificcell

d LetthetestruntocompletionwhileloggingalldatafromboththetraceUEsandthenetworkbasedprobesIfpossible,alsologthedatafromtheLTEUuinterface

ThedatashouldthenbeanalyzedtorevealactualbehaviorduringcongestedconditionsOneshouldspecificallystudytheimpactofthescheduleronboththeguaranteedQoStraffic(VoIP)andthestaticRTPstreamfortheInternetradiobecausethesetwoservicesarethemostlikelytodegrade

Asimpleandbasicmeasureistobenchmarkthetotaltimeofcompletionforthetestsequence(excludingtheInternetradiostreamandtheVoIPsession,whichcouldrunindefinitely)Theminimumcompletiontimeshouldberelativelyeasytocalculatebasedonthepreviouslymeasuredperformanceforthemulti-userandfull-bufferdownloadscenariosThedifferencefromthisminimumtimeshouldbeanalyzedtounderstandtheoverallefficiencyofthesystem

AddingmobilitytothistestcasewouldallowforfurtheranalysisHowever,itwouldprobablyaddsomuchcomplexitythatitcouldnotbejustifiedasabasiccase

AllaspectsoftheschedulingandperformanceoftheradioenvironmentshouldbeanalyzedtounderstandtheefficiencyineachlayerandduringeachprocessThiswillalsohelprevealwhichconditionsleadtosuccessfulschedulingresultsandwhichcauseschedulingissues

JDSUcanprovidethetoolsandtheprocessestoexecuteandanalyzetheresultsfromthesetypesoftestsJDSUtoolscanhelpcharacterizethebehaviorandallowforbothcompetitivebenchmarkingandregressiontestingwhenalaundrylistmustbemaintainedforaspecificsupplier

Idle-to-active transition times

OneofthemainexpectationsofLTEistoprovideanalwaysconnectedexperienceforendusersThisisachievedinpartbyensuringaswifttransitionbetweentheidleandactivemodesTheoverallRRC

statemachinehasbeenoptimizedandthenumberofpossiblestateshasbeenminimizedtoensurereducedcomplexity,lowerpowerconsumption,andfastertransitiontimes

Tomeasuretheidle-to-activetransitiontimesonemustbeabletoeitherfullycontrolUEbehaviororbeabletologalloftheassociatedsignalingtoensurethedataisavailabletobemeasuredfromtheoveralltraffic

JDSUtoolscanmeasureidle-to-activetimeaswellasotherrelevanttransactionandproceduraltimesThesemeasurementscanbeperformedusingeitherdatafromatracemobilealoneordatacombinedfromtheLTEandEPCnetworklinkstoenabletruecorrelatedE2Emeasurements


10/32


Latency testing

TheoverallexperienceismorethanthespecificbandwidththatanendusercanreceiveandhowquicklythenetworkwillenableswitchingbetweentheidleandactivemodesTheexperiencealsodependsontheE2ElatencyofapackettransitioningthroughthecompleteEPCandeUTRAN

KPI veriication and calculation

TheJDSUbookLTE and the Evolution to 4G Wireless: Design and Measurement ChallengesincludesanextensivesectionaboutKPIs,thecalculationmethods,andtheoverallmethodology(pleaserefertoChapter6)Asaresult,theKPImaterialcoveredhereiskepttoaminimum

ItisunfortunatethatthephraseskeyperformanceindicatorandKPIhavebecomecommonlymisunderstoodandmisusedAtthemostbasiclevel,aKPIisnothingmorethanastatisticora

measurementHowever,itisthetestobjectiveormarketrequirementforagivenservicethatallowsaparticularstatisticormeasurementtobeconsideredasakeyindicatorofperformance

EventhetermperformancecanmeanverydifferentthingsdependingonthetestingcontextForexample,serviceperformanceforVoIPmaybemeasuredintermsofjitter,latency,anddroppedpacketsNetworkperformancemaybemeasuredbythenumberconcurrentVoIPusersthatcanbeservedwithanacceptablelevelofjitter,latency,andpacketlossThus,whenmeasuringqualityorperformance,oneofthekeychallengesisagreeingondefinitionsthatenableconsistentinterpretationsofresults

Accordingto3GPP,KPIsgenerallyfitintooneoffivecategories:accessibility,retainability,integrity,availability,andmobilityThelistissometimesexpandedtoincludethecategoriesofutilizationandusability

3GPPKPIstandardizationeffortsarelargelyfocusedonmeasurementsrelatedtotheendusers

perceivedQoSThesemetricstendtobemoreoperator-centricastheyrelatespecificallytomeasuringtheabilityofcustomerstoobtainandmaintainaconnectiontothenetworkandtherebymakeuseofoneormoreavailableservices

KPIsarebestunderstoodinthecontextoftheactualobjectiveofthemeasurementEachpartofthenetworkhasdifferentresponsibilitiesassociatedwithdeliveringasingleserviceTherefore,LTE-specificKPIsfocusontheeUTRANitselfand,inmanycases,relyontheeNodeBtoactuallymeasureitsownperformanceOnechallengeforanNEMisdefiningawaytoverifythatKPIscalculatedbytheeNodeBarecorrect,especiallywhenrunningathighloadorfullcapacityAnotherchallengeforbothNEMsandWSPsismakingtheshiftfromsimplylookingatKPIstotroubleshootingandidentifyingtherootcausesofproblems

ForeachKPIcategory,eachservicemayhaveadifferentQoSprofileorQCIlabelToidentifytheperformanceofeachtypeofservicebeingaccessed,themeasurementsshouldbemadeonaper-QCIbasisAdditionalKPIsineachofthesecategoriesshouldbeconsideredinordertoevaluatetheend-to-endusabilityandmanageabilityofaservice

Keyaspectsofawell-designedend-to-endtestsystemarethedatasourcesandthepossiblemonitoringpointsthatexistinanLTEandSAEnetworkSomeofthetopicsdiscussedherearealreadypartoftheindustrystandardsOtherareasmayormaynotbepartoffuturestandardizationeffortsby3GPP,ETSI,orotherindustrybodies


11/32


Fundamentaltothetopicofdatasourcesandmonitoringpointsisanunderstandingofmeasurementtheory,basicphysicallaws,andhowtheLTE/SAEtechnologyworksandisdeployedConsiderthefollowingexamplecommontobothLTE/SAEandUMTSinwhichtwoengineersneedtomeasuretheRRCconnectionsuccessratiointhenetwork

erstengineer,whoisexperiencedindrivetesting,commissionsatargeteddrive-testcampaign,measuringtheRRCconnectionsetupsuccessratioforawideareaClearly,thenumberofmeasurementpointsisdirectlyrelatedtothedurationofthetestingandthenumberofactualattemptsperunitoftimeForthisexample,theengineerconcludedthattheRRCconnectionsuccessratiowas985percent

esecondengineerisaccustomedtonetworkcountersandlink-monitoringtoolsAsaresult,heextractslogsfromthesystemisprovideshimwithareportofalloftheRRCconnectionattemptsforthefullnetwork;hismeasuredRRCconnectionsuccessratiois995percent

Whyisthereadifferenceofnearlyonefullpercentagepoint?Theanswerisfundamentaltotherestofthisdiscussion

Thedifferenceintheresultsisnotduetoflawsinthedatasource,beitthedrivetest,thenetworkcounters,orthelink-monitoringtoolsThedisparityiscausedbythetwoengineersmeasuringdifferentnetworkproceduresfromdifferentangles

edrive-testmethodanalyzesnetworkperformanceasseenfromasinglehandsetatspecicphysicalpointsinthenetworkatdistinctpointsintime

enetworkcountersandlink-monitoringtoolsrecordallofthetracandalloftheoccurrencesofsignalingandusertractheyaredesignedtomonitorisisadierentframeofreference:itanalyzesnetworkperformanceasseenonthenetworkandatthenetworkmonitoringpoints

Differencesbegintoaccumulateifanydrive-testlocationsarewelloutsidetheactual,andpotentially,intendednetworkcoverageareaAsaresult,RRCconnectionrequestsmadeoutsideofthenetworkcoverageareawillberecordedbythedrive-testsystembutnotbythenetworkcountersorlink-monitoringsolutionsThishighlightsakeypoint:anextensivedrive-testcampaignprovidesadditionalinformationbeyondwhatnetworkcountersorlink-monitoringtoolscanprovide

Intheoptimizationcommunity,itisgenerallyagreedthatKPIsshouldbecomparedtoeachotheronlywhentheyarederivedfromthesamedatasourceorwhentheyarenormalizedtoremoveanybiasduetomethodorsourceThisisespeciallytrueifcomparisonsshowunexpectedresultsToday,thelackofpropercomparisonsisoneofthelargestcontributorstounsoundoptimizationdecisionsinthemobileindustryValidcomparisonsandmeaningfuloptimizationcanbeensuredonlyifastringentandcoherentagreementondatasourcesandmonitoringpointshasbeensettledinadvance 3

WhetherengagedintheR&Dprocessortheoptimizationphase,onemustoftenchoosebetweenseveraldifferentstrategieswhendevelopinganLTEtestplanSelectingthemostcost-effectiveandresults-effectivestrategyisoneofthemostimportantdecisionstobemadeearlyineachphaseofthe

workOnceastrategyisselected,theboundaryconditionsofitsapplicabilitymustbeestablishedItisimportanttonotethatastrategythatisappropriateforonephaseoftheworkprobablyhassignificantshortcomingsinanyotherphaseInotherwords,itisseldomagoodideatousethesamefundamentalKPIsbecausethedatasourceswillprovidedifferentresultsindifferentphasesofanetworksdeploymentandmaintenance

3PleasenotethatthisissimplyamatteroffundamentalmeasurementtheoryItisnotintendedtobeadiscussionaboutthepotentialriskofresultsbeingincorrectduetoacertainmeasurementtoolnotworkingasdened


12/32


Toillustratethisidea,consideracaseinwhichQoSiscontrastedwithQoEMonitoringtheend-userIPtrafficonamobilenetworkprovidesafullanddetailedunderstandingofthetrafficflows(TCPorUDP),theapplications(voice,video,HTTP,e-mail,etc),andpotentiallyperformanceSomebelievethatthemonitoringofonlytheUDPorTCPflowswillprovideenoughinformationabouttheend-userQoStobeabletodeduceagoodapproximationoftheend-userQoEAsthefollowingscenarioshows,thisisnotthecaseforapplicationssuchasstreamingvideo

Auseriswatchingstreamingvideoonhishandset,buttheradioqualityisnotsucienttodeliverthefullbandwidthovertheairinterfaceWhentheUDPstreamismeasuredinthecorenetwork,nodegradationoftheRTP/UDPstreamisobservedreeotherlocationsprovideabetterplacetoobservethedegradation:ontheairinterface,ontheusershandset,orbetweenthetwoendpointsoftheRLCentity(intheUTRANbetweentheUEandtheRNCorintheLTEeUTRANbetweentheUEandtheeNodeB)enetworkmonitoringtoolinthecorewouldreportahighandstable

bitrate(goodQoS);however,theenduserwouldreportapoorQoE Shiingthescenarios,assumethattheendusersapplicationisquality-awareand,duetotheradio

conditions,signalsthatthevideostreamingservershouldchangethebitrateofitscodecWhenthisnewRTP/UDPbitstreamappearsonthenetwork,thenetworkmonitoringtoolwillassociatethechangewithalowerQoSstreambecauseithasalowerbitrateOntheotherhand,theradioconditionsaregoodenoughtodeliverthisadaptedbitstream,andtheenduserQoEhasnowincreased

ThisscenarioshowsthatthecrucialelementintheQoEisnotthebitratemeasuredinthecorenetwork,butrathertheend-to-endabilitytodeliveraspecificservicetotheenduserTheapplicationdomainwill,inthiscase,ensurethatthebestpossibleQoEisachievedTherefore,themonitoringtoolmustbeapplication-awaretodelivercorrectQoSmeasurementsthatleadtothecorrectestimationofend-userQoE

Validating LTE voice

Oneoftheto-be-defineditemsfortheEPSishowcircuit-switchedservicessuchasvoice,CSUDIvideo,

SMS,LCS,andUSSDwillbemanagedFouralternativesarecommonlyconsidered

Circuit-switchedfallback(CSFB)

VoiceoverLTEgenericaccess(VoLGA)

VoiceoverIMS(VoIMS)

Proprietaryoptions

CSFBandVoLGAarebothstandardizedandcouldbereadilyimplementedVoIMSislikelytofollowandisanticipatedtobeawidespread,long-termsolutionAmongthese,CSfallbackinEPSisdescribedindetailbelow;theotheroptionsarecoveredinbrief

Anotheraspectthatisnotyetspecifiedin3GPPR8istheuseofavoicecodecSeveraldifferentoptionsareathand;however,duetoalackofagreement,thispartofthestandardizationmightbedelayeduntilR10Onekeyreasonforthedelayisthelackofclarityontheobjective:shouldthequalitybeimproved

orshouldthecapacitybeimprovedbythechoiceofcodec?ItislikelythatachoiceofcodecforinitialEPSdeploymentswillbebasedonamutualbilateralagreementbetweentheUEvendors,theoperators,andtheEPSproviderItwouldnotbeasurpriseifAMRandAMR-WBwereusedinitially

ItiscrucialtounderstandthattheITUhasalreadymovedaheadwiththedefinitionoftheG718codecG718isbuiltontheAMR-WBfoundationand,forthemostpart,providesthequalityofAMR-WBat1265KbpsonthesamecapacityasAMR795KbpsOverall,thismeansthatG718providesa57percentincreaseincapacitywithaverylimitedimpactonspeechdelayAdetaileddescriptionisoutsidethescopeofthispapersothereaderisencouragedtostudyITU-TRecG718infulltofurtherunderstandthistopic


13/32


Signaling validation

Figure2isatypicalexampleofanIMScallflowforLTEinteractingwiththeSS7networkAswillbeshown,thisisfarfromonlySIPandsomeofthecriticalaspectscannoteasilybeseeninthissimplified

viewOneexampleisthelackofvisibilityintothechoiceofthebearersthathavebeensetupandhowtheymaptoacertainQCI,etc

Anyshort-termanalysisofavoice-over-LTEimplementationwillmostprobablybeimpactedbyoneorseveralshortcutseveniftheintentionistobestandardscompliantThisshouldbeconsideredwhenperforminganalysisanddrawingconclusionsKeyaspectstoconsideraretheusageofROHCfortheairinterface;theusageofacodecforE2EspeechandtheintegrationofapagingprocedurewhentheUEisinidlemode;and,howthemobilitybetweenaccesstechnologieswouldbemanaged

Voice QoS and QoE (MOS)

VoiceQoSandtheresultingQoEisatopicofmanygoodpublicationsandthusthissectionwillnotgointotoomanydetails

Overall,onemustbecarefulaboutwhatistestedunderwhichconditions,andonemustbeawareofthegoverningfactorsthatcontroltheoutcomeIfacertainvoiceserviceistestedwithabearerdeliveringaspecificQoS,thentheresultingvoicequalitywillbelimitedbythequalitydeliveredbythebearer

CALLSETUP

IDLETOA

CTIVE

Originating UE eNB MME Terminating UEPSTNCore IMS

Delay for RACH

scheduling period

Rach preamble

Security ModeCommand + RRC

ConnectionReconguration

RRC ConnectionReconguration

complete

ConnectionRequest

Connection Setup

SIP INVITE

IAMIAM

COTCOT

ACM/CPGACM/CPG

SIP 183 Session Progress

SIP PRACK

SIP 200 OK

SIP 200 OK

SIP 180 Ringing

SIP UPDATE

RRC ConnectionRequest

RRC ConnectionSetup

RRC ConnectionSetup complete +

NAS servicerequest

TA + Scheduling

Figure 2. Typical IMS call ow for LTE interacting with SS7


14/32


Whilethissoundssimpleintheory,itcausessignificantpracticalconcernsinalmostallnewtechnologiesbeforetheE2EstructureiswellunderstoodInUMTS,voicewasnormallydeliveredusingaradiolinkwithaone-percentBLERtargetCertainimplementationsusedonepercentasaminimumqualitytargetbutover-deliveredonqualityifandwhenresourceswereavailableThismeantthatcomparingasituationinwhichtheBLERwasalwaysheldattheconfiguredBLERtarget(regardlessofavailableresources)andBLERwasadjustedtodeliverthebestpossibleservicewithoutcausingdegradationforothers,theresultsinanunloadednetworkwouldalwaysfavortheflexibleimplementationThisisnotalwaystheintendedtestobjectandthusthevalueofvoice-qualitytestingcanbedegradedorworse,beconsidereduselessTherefore,itiscriticaltounderstandtheunderlyingconditionsthatwillimpactvoiceserviceandtoeithercontrolthesefullyorrecordonlythoseparametersusedThiswillensurefaircomparisonswhenbenchmarkingaresulteitherovertimeorbetweenimplementations

JDSUsuppliestoolsthatcanperformE2Evoice-qualitytestingandalsobenchmarkvoicequalitypassivelyinsidethenetwork(Figure3)

Testing QoS and QoE o LTE streaming video

QoSandQoEmonitoringofavideoserviceisdirectlyanalogoustotheearlierdiscussionofKPIsInshort,allofthesamemethodsandissuesidentifiedintheKPIsectionapplytothesemeasurementsRatherthanrepeatingthatmaterial,pleasereferbacktotheearlierdiscussion

Thereisonekeypointtoadd:understandingvideoQoSandtheresultingQoEshouldbedoneacrossallrelevantaspectsoftheLTEandEPCThismeansthatonecannotonlyidentifytheactualQoSatapointbutcanalsoensuretheserviceupstreamfromthispointifthetrafficflowisgoodatthemonitoredlocationOntheotherhand,ifthetrafficflowispooratthemonitoringlocation,thenafurtherinvestigationupstreamshouldbeperformedNotethatinmanycasestheclientcanusejitterbufferstocompensateforacertainamountandtypeofQoSimpairmentsThus,adegradedQoSdoesnotalwaysleadtothedegradationofQoE

eNB eNB

EPC

JDSU VoIP Office End

UE UE

Serving/PDN

GW

Element Management System /

Network Management SystemJDSU J6804A DNA HD

JDSU J7830A Signaling

Analyzer and J6900A

Triple Play Analyzer

JDSU E6474A NiXt JDSU E6474A NiXt

Figure 3. Network architecture for E2E voice-quality testing


15/32


JDSUprovidesalloftherelevanttoolstomonitorvideoQoSandQoEfromeitherapassiveperspectiveorfromanactiveperspective(Figure4)SupportisprovidedforbothstandardIPTVandforMSTV

Evaluating LTE MIMO and requency-selective scheduling

ComparedtoHSPAsystems,muchofthevalueinLTEcomesfromtheeffectiveusageofbothMIMOandfrequency-selectivescheduling(FSS)Inmanyaspectsthesearetechnologiesthathaveyettobe

provedintermsofaddingrealtangiblevaluetothecustomerinafieldenvironmentTheirrealvaluedependsonthreethings:theactualdeploymentscenario;theoveralltrafficmodeling;and,thespecificimplementationandconfigurationofthesystem

AfeaturesuchasFSSmakesittheoreticallypossibletoaddsignificantperformancetothesystemInpractice,however,performanceislimitedbythealgorithmusedtocontrolschedulingThisisduetooperationaltradeoffssuchasreducingsignificantuplinktrafficforsub-bandCQIreporting,ortechnicalfactorssuchasthepracticalavailablecomputationalperformanceintheeNodeB

AsfarasMIMOisconcerned,itshouldbeunderstoodthatMIMOusageorthemeasurementsleadingtoaspecificMIMOconfigurationforspecifictransmissionstoauserduringaspecificTTIistypicallynotastaticbehaviorWhatsmore,anyexpectationthatMIMOcanbecontrolled,modeled,orunderstoodfromoneorafewmeasurementsistypicallyunrealisticAsaresult,itisimportanttoidentifytwokeyelements:thoseaspectsofthetechnologythatshouldbeevaluatedfromastatisticalbehaviorpointofview;and,thoseparametersthatcanbeanalyzedfromasinglediscretemeasurement

withoutlookingatalargersample

ItisvitaltounderstandandcharacterizetheactualbehaviorofbothMIMOandFSStoensuretheproperdimensioningofthecells,thebackhaul,andtheQoSprovisioningThestatisticalnatureofMIMOandFSScanbewellunderstoodonadetailedTTIlevelonlyiftherighttoolsandprocessareappliedThesecapabilitiesareallavailableusingJDSUtoolsandmethodsThetoolsallowadetailedanalysisandcharacterization,enablingaclearunderstandingofactualbehavior

Figure 4. Tools for monitoring end-to-end LTE QoS and QoE ( VoIP, IPTV, data)

Evolved Node B

Evolved UTRAN (E-UTRAN)

UEs

Uu

Evolved Packet Core (EPC)

MME

S-GWP-GW

Core, Services, IMS

PCRF

MRF

Internet

CSCFHSS


16/32


Testing network coverage

Duringnetworkplanning,modelingisperformedandassumptionsaremadeIftheseareatallinaccurate,theundesirableconsequencesincludeeitherunplannedcoverageholesorunwantedsignalleakageintoadjacentcellsEarlyintheplanningandtuningprocess,aclearunderstandingofactualnetworkcoverageversustheplannednetworklayoutenablestheplanningandengineeringteamtodevelopastrongmethodologythatallowsforverycost-effectivenetworkdeploymentandtuningEarlyverificationofthecoveragewillallowcreationofaCDFofthenetworkperformanceandallowforpropertestingofcellperformance

JDSUprovidesdetailedRFcoveragemeasurementsincludingRSSI,RSRP,RSRQ,andRS_CINRThesecapabilitiescaneffectivelycoveralargenumberofbandsinverycomplexnetworktopologiesTheresultsfromthemeasurementscanbepost-processedwitheithercustomsoftwareoranyoftheindustry-standardpost-processingapplications

Testing LTE handover

CoveragetestingiscloselylinkedtothecharacterizationofHOvalidationandperformanceTofullygraspHOperformance,onehastofirststudyandunderstandtheoverallproceduresandprocessesthatunderliethehandoverinLTEAsiswidelyunderstood,nosofthandoverispresentinLTEAsaresult,thehandoverisalwaysahardbreakbeforemaketransitionTheresultingimpactonend-userQoEmustbeunderstoodandoptimizedItisalsoimportanttounderstandthehandoversoverheadimpactontheeNodeBbecauseLTEhandoverscanbeusedtoproactivelymanageserviceloadinginaspecificarea

Severaldifferenttypesofhandoversarepresent:thosebetweencellsinthesameeNodeB;thosebetweencellsindifferenteNodeBswithoutanyX2dataforwarding;and,thoseinwhichX2dataforwardingisensuringaminimuminterruptionofdatatrafficItiscommonlyunderstoodthattheX2dataforwardingfeaturemightbeimplementedinlaterreleasesoftheeUTRANsoftware;however,some

vendorsmightmakethiscapabilityavailableearlyinthelifecycle

AUEcannormallydiscriminatebetweenthetypesofhandoversbymonitoringtheSFNaroundtheHOtimeandcheckiftheSFNisjumpingoriscontinuous

Figure5isagoodexampleofahandoffbetweentwocellsinasingleeNodeBTheplotshowsthattheBLERisincreasingthemomentsbeforetheHOandthatthisresultsindegradedthroughputontheMAClayerAftertheHOisperformed(forexample,anRRCconnectionreconfiguration),MAClayerthroughputisincreasedandnoBLERisreported


17/32


Followingthisfromasignalingperspective,thecalltrace(Figure6)revealstheresultingHOinterruptiontime,whichisontheorderof12to35msNotethatthisanalysisrequiresaccesstoboththeprotocollogsfromtheUE(orfromapassiveUuprobeorfromaeNodeBfeed)withthosefromthenetworklinks(forexample,S1,X2,etc)

Figure 5. MAC throughput (green line) sags before an RRC event but recovers immediately after.

Figure 6. A call trace can reveal the resulting HO interruption time


18/32


Asinallmobiletechnologies,itisrelativelystraightforwardtoachievesuccessfulhandoversHowever,achievinganoptimizednetworkwithoptimizedhandoversrequiresgreatinsightandtremendousskillintheartofoptimization

Validating LTE backhaul

LTEoffersgreatlyincreasedbandwidthtotheenduserInturn,itisveryimportantthatthisbandwidthcapabilityisavailablethrougheverypartofthenetworkOneofthebigareasoffocusforthisisthemobilebackhaulnetwork,typicallyEthernet,betweenthecellsite(eNodeB)andthecorenetwork

InLTEfieldtrialswehavewitnessedtodate,thewirelessbackhaulnetworkisbroughton-linepriortoimplementationoftheLTE-specifictestcasesCommissioningtheEthernetbackhaulquicklyandeasilyisvitaltokeepinganLTEfieldtrialonscheduleTheJDSUNetCompleteServiceAssuranceSolutionforWirelessBackhaulverifiesEthernetserviceoperationtotrialcellsitesinadvanceoftheLTEtrialexecutionAdherencetoRFC-2544standardsismaintainedwithcost-savingderivedthroughautomationandefficientdeploymentoftechniciansevenacrossmultiplecellsitesHowever,sincesuchtestingisnotuniquetoanLTEfieldtrial,itisbeyondthescopeofthisapplicationnoteGeneralinformationisavailablesuchastheIETFsRFC2544(1999),theIEEE,andtheITUwhichdefinetestandperformancemonitoringmethodologiesforEthernetnetworkinterfacedevices

Veriying LTE handset IOT

ForeachoperatorintheearlyphasesofbringingLTEtomarket,itisessentialtoensureproperinteroperabilityofLTEhandsetsordatacardsOfcourse,theseareearlydevicesthatwillcontinuetoevolverapidlyovertimeConsequently,itisimportanttounderstandthecapabilitiesandensureinteroperabilitybetweenthehandsetandtherelevantnetworkelementsItwillnotbepossibletosimplyrelyonpre-conformanceorconformancetestresultsbecausethesewillseldomreflectactualuserbehaviororissuesfoundintheearlyphasesofnetworkdeployment

IndustryforumssuchasLSTIhavedefinedaminimumfeaturesetandacorrespondingIoDTandIOTtestplanTheseareavailableonlytoLSTImembersandmaybeusedonlyforLSTIpurposesAsaresult,non-membersmustrelyonothermeanstosecurehandsetinteroperabilityAsanLSTImember,AgilenthasmadesignificantcontributionstotheIoDTandIOTphasesThroughthisexperiencepersonnelinbothAgilentandJDSUarebetterabletocontributeeffectivelytothewiderindustry

JDSUprovidesmonitoringtoolssuchastheSignalingAnalyzerthatprovidethecapabilitytoanalyzeandcorrelateinformationfromeachinterfaceandhighlightanydiscrepanciesversustheanticipatedbehavioronasignalinglevelTheJDSUNiXTapplicationcanactivelystimulateahandsettoexecutedifferentapplicationsandtasksThiscorrelatestocontroloftherelevantRFparameters(forexample,fading,MIMOprofiles,etc)andaneffectiveLTEhandsetinteroperabilityenvironmentcanbeprovidedIfappropriatelyandeffectivelyautomated,thiswillensurethatcost-effectiveandreliabletestingisperformed,therebymaximizingassetutilization

Validating device coniguration

OnepartofLTEisyettobefullystandardizedandwidelyagreedupon:theeffectiveperformanceofdeviceconfigurationEvenifsignificantinteroperabilitytestinghasbeenperformed,anyoperatorlaunchinganLTEserviceearlywillberequiredtoremotelyupdatehandsetsandparameterssuchasnetworksettings,software,orotherrelevantparametersduringoperationofthenetwork 4

4ManyofthetraditionalOTAsystemsarebasedonshortmessageservices(SMS)thatarenoteasilysuppliedinanLTEenvironmentwithouttheuseofeitherCSFBIMSorVoLGA,bothofwhicharerelativelycomplexanderror-pronetechnologies


19/32


Itisessentialtoprovidefullvisibilityintodevicebehaviorduringpre-configuration,configuration,andpost-configurationphasesandtocomprehendthefullE2EsignalingneededtodelivertheconfigurationenvironmentWithJDSUtools,thepre-configurationstatecanbeanalyzedbothqualitatively(forexample,theperformanceofalloperationalservices)andinsimplego/no-gotesting(whichservicesworkordontwork)Specificnetworkbehaviorsthatidentifyincorrectlyconfigureddevicescanbeanalyzedanddocumentedtobeusedinsubsequentnetwork-widemonitoringsystemsDuringtheconfigurationprocedure,differenttypesofpositiveandnegativetestingcanbeperformedtostimulateerrorconditionsandtoprovethattheconfigurationissuccessfulifcertainconditionsaremetThenegativeconditionsareespeciallycriticalbecauseitisoftendifficulttoevaluatethestateofanend-userdeviceinarealnetwork


20/32


Appendix: Outline or a Basic Phase 1 LTE Test Plan

Overview

Thissectioncontainsabasicsetoftestcasesthatcanbeusedintwoways:toevaluateLTEasatechnology;and,toassessthebasicperformanceofLTEinfrastructurevendors

Thebasictestplandescribedhereiswell-alignedwiththethinkingofmostinfrastructurevendorsItisalsoconsistentwiththeirdetailedtestplansAsaresult,thissectioncanbeusefultooperatorstooTheycanusethisplanwhenaskingvendorstocreateadetailedoutlinethatspellsouthowtestingwillbeconductedTheoperatorcouldthenvalidatethatthedetailedtestplanmeetstheintentionandbasicexpectationsoftheplanTheoveralltestexecution,datacollection,analysis,andreportingcanbemanagedwithintegrity,ensuringavalidevaluation

Subsequentphasesofatestplancanbeprovidedthatwillallowtheoperatortomovetowardvendorselectionandnetworkroll-out

Thesuggestedtestplanhaseightmajorsections

Achievabledataratesandlatency:single-userthroughputforUL/DLandTCP/UDP

Intra-LTEmobility

Coverageandcapacity

Evaluationofantennacongurationoptions

Self-congurationandself-organizingnetworkfeatures

Evaluationoffrequencyreuse

BasicQoS

Basicapplicationperformance

Below,AchievabledataratesandlatencyhasfoursubsectionsandCoverageandcapacityhastwoEachofthe13totalentriespresentsabasictestoverviewthatcanbeeasilyleveragedintoatestplan

Achievable data rates and latency: single-user throughput or UL/DL and TCP/UDP

Basic test overview

Evaluatesingle-userthroughputacrossarangeofradioconditionsGenerateULandDLloadingof70percentfromadjacentcells

PerformseparateULandDLthroughputtestsindividuallywithTCPandUDPMeasureaveragethroughputfor30secondsduringstationaryconditionseTCPandUDPtestsshouldmeasurelosslessthroughputratherthanrawthroughput

LogallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanesfromthenetworkandUEsides

Providerawandnormalizedresultsaccordingtothecommontestdescription


21/32


Achievable data rates and latency: cell throughput and MU throughput or UL/DL and TCP/UDP

Basic test overview

GenerateaCDFofthecellaccordingtothecommontestdescriptionPlace10UEsaccordingtotheCDFdistribution

Verifyplacementofthe10UEsinappropriateandrepresentativelocations

GenerateULandDLloadingof70percentfromadjacentcells

PerformseparateULandDLthroughputtestsindividuallywithTCPandUDPMeasureaveragethroughputfor30secondsduringstationaryconditionseTCPandUDPtestsshouldmeasurelosslessthroughputratherthanrawthroughput


Providerawresultsandnormalizedresultsaccordingtothecommontestdescription

Achievable data rates and latency: latency

Basic test overview

Evaluatetheend-to-endlatencyinarangeofradioconditionsandloadingscenarios

No-loadscenario

Performend-to-pinginarangeofradioconditionsforaminimumof100samplesperlocationandpacketsize(32,1000,1500bytes)

With-loadscenario

GenerateULandDLloadingof70percentfromadjacentcells

Generatesame-cellDLloadfromoneUEingoodradioconditionswithfullUDPdownload

Generatesame-cellULloadfrom1UEinpoorradioconditionswithfullUDPupload Performend-to-endping(32,1000,1500bytes)inarangeofradioconditionsforaminimumof100samplesperlocationandpacketsize

LogallappropriateparametersandconditionsincludingfulltracesofcontrolanduserplanefromthenetworkandUEsides

ProvideRANandEPClatencyand,ifrelevant,thebackhaultransmissionlatency

Providerawresultsplusminimum,maximumandaveragevaluesforeachradioconditionandpacketsize

Intra-LTE mobility: mobility and handover perormance

Basic test overview

TestandcompareresultsforhandoverbasedonbothS1aloneandonX2forwarding;testsamecell

loadedandunloaded Samemethodologyforboth(S1andX2)testcases

Repeattestforatleast20HOofeachtype

PerformthetestwithDLandULTCPandUDPtracsubsequently

MeasureandreportHOsuccessrate

MeasureandreportcontrolplaneHOtime

MeasureandreportuserplaneHOinterruptiontimeandpacketloss


22/32



ForloadedHOperformance,testaddtwoUEsingoodradioconditionsinbothcellsperformingfull-buerDLandUL

Othercellloadshouldbe70percentinDLInthemobilitycase,notethatULloadingisnotrequiredduetopracticalissues

Achievable data rates and latency: application perormance

Basic test overview

Fromidlemode,connectanddownloadtheCopernicuswebsite(seecommontestdescription)andmeasurethedownloadtimeinarangeofradioconditionsAminimumofverepetitionsateach

locationisrequiredAminimumofve(10isrecommended)dierentradioconditionsshouldbecoveredfromgoodradioconditionstocell-edgeconditions

eothercellloadshouldbe70percentinULandDL


Coverage and capacity radio eatures eiciency and gain assessment: link budget

Basic test overview

Testrstinstaticlocations:

eUEislocatedclosetothecellcenterAUDPDLtransferwithfullbuerisinitiated

eusermovesawayfromthebasestation,sopathlossincreasesRepeatthemeasurementsatthenewlocation

isisrepeatedforatleast10dierentlocationsacrossthecellAmajorityofthelocationsshouldbeinpoorradioconditions;continueuntilcoverageislost

Repeatthesametestwhileinmobileconditions:drivefromcellcentertocelledgeandrepeatseveraltimes

LogallappropriateparametersandconditionsincludingfulltracesofcontrolanduserplanesfromthenetworkandUEsides

Coverage and capacity radio eatures eiciency and gain assessment: scheduler

Basic test overview

LocatefourUEsingoodradioconditionsascloseaspossibletoequalradioconditionsEnsurethatallUEshavethesamepriorityandQoSsettings

SetupDLloadingofUDPwiththefollowingtraccharacteristics:UE1=X;UE2=2*X;UE3=3*X;

UE4=4*XEnsureXischosensothattheaggregatedthroughputoftheusersexceedsthecellcapacity Measuretheresultingbehavior

LocatetheUEsatfourdierentlocationsandgeneratethesameUDPstreamthroughputtoeachUEEnsureaggregatedthroughputoftheusersexceedsthecellcapacity

Measuretheresultingbehavior



23/32


Evaluation o antenna coniguration options

Basic test overview

Performthesingle-userthroughputtestcaseandthemultiusercellcapacitytestcasewiththefollowingantennacongurations:

SIMO

2x2MIMO

4x4MIMO(Optional:Performifpossibleforthevendor)

Sel-coniguration and sel-organizing network eatures

Basic test overview

EnsureatleasttwoeNodeBsareoperationalandnoneighborsexistinthetargetcell VerifyintheO&MsystemthatnoneighborsorX2interfacesarecongured

ActivateANRineNodeB

TurnontheUEinacellwherenoneighborsaredenedAerUEreportingofsignalstrengthsofsurroundingsectors,theANRfunctionshalladdneighborstotheneighborlist

Performahandover

VerifyintheO&MsystemthatallcorrectneighborsandX2interfaceshavebeenadded

PerformanHOtoandfromthenewneighbor


Evaluation o requency reuse: one deployment scenario

Basic test overview

LocateaUEatthecelledge(in100percentothercellload)andpreparefull-buerdownloads

Withoutothercellload,measureaveragethroughputfor30secondsforbothTCPandUDP

With50percentothercellload,measuretheaveragethroughputfor30secondsforbothTCPandUDP




Comparethecell-edgeperformanceunderthevariousloadingconditionsandcomparethemeasuredSINRforeachloadingconditionatthesamephysicallocation


24/32


Basic QoS: user dierentiation between non-GBR users with dierent QCI

Basic test overview

OneUE(A)shouldhaveafull-buerdownloadofUDPtracingoodradioconditionsandwiththeQCIoflowestpriority

OneUE(B)shouldhaveadownloadofaUDPstreamof10MbpsandbelocatedinmediumradioconditionswithaQCIofsecond-lowestpriority

OneUE(C)shouldhaveadownloadofaUDPstreamof10MbpsandbelocatedinmediumradioconditionswithaQCIofhighestpriority

Measuretheresultingthroughputandpacketloss(onUEBandC)andreporttheactualreceivedthroughputandtheS1andSGithroughput

Othercellloadshouldbe70percentinULandDL


Note:Toensurethatcongestionoccurs,itmaybenecessarytonormalizethethroughputofUEBandC

Basic QoS: user-dierentiation between GBR and non-GBR users

Basic test overview

Assume:N1>N2>N3>N4(User1hasthehighestpriorityandUser4hastheleastpriority)

LocatefourUEingoodradioconditionsCongureDLbearerswithGBRvaluesforUE1,UE2,UE3,andUE4EnsurethatX1,X2,X3,andX4DLthroughputsareaggregatedabovethecellcapacityatallUElocationsbutthatX1,X2,andX3aggregatedarebelowthecellcapacity

GenerateUDPDLtractoclientsUE1,UE2,UE3,andUE4

StopDLtransferaeratleast30secondsofdatatransfertotheUEclients

MeasureDLthroughput

VerifythattheDLratesforUE1,UE2,andUE3arenolessthantheirpredenedGBRvalues(X1,X2,andX3,respectively)

VerifythatUE4(lowest-prioritytrac)wasthemostnegativelyaected(theDLbitratefailedto

achieveX4,whichisthedesiredDL-GBRforUE4)

ChangetheARPsettingofUE4tobethehighestpriority(N1)

RepeatwithsimultaneousDLtransfertoallfourUEclients

VerifythattheDLdataratetoUE3(lowestpriorityamongallUEs)ismostnegativelyaected(itsdataratewillbelowerthanitsdesiredDL-GBR)

User UE1 UE2 UE3 UE4

GBR DL (Mbps) X1 X2 X3 X4

GBR UL (Mbps) Y1 Y2 Y3 Y4

MBR DL (Mbps) A1 A2 A3 A4

MBR UL (Mbps) B1 B2 B3 B4

ARP (Allocation and Retention Priority is NOT

equal for all ows)

N1 N2 N3 N4


25/32


PerformthesametestbutlimittheS1downlinkperformancetocausecongestionatalevelbelowtheRFcellcapacitybutabovetheaggregatedX1,X2,andX3VerifythroughputisbelowtheS1peakthroughputlimitation

elimitationcanbeachievedviaVLANtuning

Basic application perormance: web browsing, streaming, voice calls, e-mail, VPN, on-line

gaming

Basic test overview

Identifytherelevantservicesforeachlocalenvironment

Selectaheavilytrackedlocalwebsite:

1-5MBforthemainpage(includingimages)

SelectaVoIPapplicationsupportedbytheoperatorandthelocalenvironment(forexample,Skype,GoogleChat,MSN,etc)

SelectalocalcorporatecustomerusingaVPNinamobileenvironment;theVPNconnectionshouldhaveadatarateontheorderof2-10Mbpsend-to-end

Identifythreelocal,well-recognizedservices:

Ahigh-requirement(low-latency)onlinegame

Ahighperformancexed-lineInternetservice(forexample,xDSL,packetcable,etc)

Astreamingvideo(RTP/UDP)service

Executeandcomparetheapplicationperformanceonthefollowingaccesstechnologies:

HSPA(bestlocalcommercialavailableservice)

Fixed-lineInternetservice

LTEingood,medium,andpoorradioconditions Logallappropriateparametersandconditionsincludingfulltracesofthecontrolanduserplanes

fromthenetworkandUEsidesforLTEand,ifpossible,forHSPAandthexed-lineInternetservice

Forthevoiceservice,measureandcompareMOS

Forthestreamingvideoservice,measureandcomparevideoMOSorasubjectivequalitycomparisonwithfriendlyusers

Fortheonlinegame,getasubjectivestatementfromanexperiencedgamerandanE2Elatencymeasurement

Reporttheabsoluteandrelativeperformancebetweenthedierentaccesstechnologies


26/32


Reerences

EPS speciication reerences

3GPP reerences

3GPP TS 23272:CircuitSwitchedFallbackinEvolvedPacketSystem;Stage2

3GPP TS 24301:Non-Access-Stratum(NAS)protocolforEvolvedPacketSystem(EPS);Stage3

3GPP TS 29118:MobilityManagementEntity(MME)VisitorLocationRegister(VLR)SGsinterfacespecification

TS 23401:GeneralPacketRadioService(GPRS)enhancementsforEvolvedUniversalTerrestrialRadioAccessNetwork(E-UTRAN)access

3GPP TS 21905:Vocabularyfor3GPPspecifications

3GPP TS 22278:ServicerequirementsfortheEvolvedPacketSystem(EPS)

3GPP TS 43318:GenericAccessNetwork(GAN);Stage2


TS 24301:Non-Access-Stratum(NAS)protocolforEvolvedPacketSystem(EPS);Stage3

TS 29272:EvolvedPacketSystem;MMEandSGSNrelatedinterfacesbasedonDiameterprotocol

TS 33102:3GSecurity;Securityarchitecture

TS 33203:AccesssecurityforIP-basedservices

TS 33210:3GSecurity;NetworkDomainSecurity;IPnetworklayersecurity

3GPP TS Title Protocol Interace(s)

24.301 Non-Access-Stratum (NAS) Protocol for Evolved Packet System (EPS); Stage 3 EPS NAS S1-C

36.413 E-UTRAN: S1 Application Protocol (S1AP) S1-AP S1-C

36.423 E-UTRAN: X2 Application Protocol (X2AP) X2-AP X2-C

29.118 Mobil ity Man agem ent En tity (MME) - Visitor Location Register (VLR) SGs

Interface Specs.

SGsAP SGs

29.168 Cell Broadcast Center Interfaces with the Evolved Packet Core; Stage 3 SBc-AP SBc

29.272 MME Related Interfaces Based on Diameter Protocol Diameter+ S6a, S6d, S13

29.274 Evolved General Packet Radio Ser vi ce (GPRS)

Tunneling Protocol for Control plane (GTPv2-C);

GTPv2-C S3-C, S4-C, S5/8-C, S10, S11-C

29.275 Proxy Mobile IPv6 (PMIPv6) based Mobil ity and Tunn el in g Protocols; Stage 3 PMIPv6 S5, S8 (PMIP)

29.276 Optim ized Handover Procedures an d Protocols betw een EUTRAN Access and

cdma2000 HRPD

S101-AP S101

29.277 Optim ized Handover Procedures an d Protocols betw een EUTRAN Access and

1xRTT Access

S102-AP S102

29.280 3GPP EPS Sv Interface (MME to MSC) for SRVCC Sv Sv

29.281 GPRS Tunneling Protocol User Plane (GTPv1-U) GTPv1-U S1-U, X2-U, S4-U, S5/8-U, S12-U


27/32


TS 33401:3GPPSystemArchitectureEvolution(SAE):SecurityArchitecture;

TS 36300:EvolvedUniversalTerrestrialRadioAccess(E-UTRA)andEvolvedUniversalTerrestrialRadioAccessNetwork(E-UTRAN);Overalldescription;Stage2



TS 29272:EvolvedPacketSystem;MMEandSGSNrelatedinterfacesbasedonDiameterprotocol

TS 33102:3GSecurity;Securityarchitecture

TS 33203:AccesssecurityforIP-basedservices

TS 33210:3GSecurity;NetworkDomainSecurity;IPnetworklayersecurity

TS 33401:3GPPSystemArchitectureEvolution(SAE):SecurityArchitecture;


TS 36420:X2layer1generalaspectsandprinciples

TS 36421:X2layer1

TS 36422:X2signallingtransport

TS 36423:X2ApplicationProtocol(X2AP)

TS 36424:S2datatransport

TS 29281:GPRSTunnelingprotocolforuserplane(GTPv1-U)



TS 29305:InterworkingFunction(IWF)betweenMAP-basedandDiameter-basedinterfaces

TS 29274:TunnelingprotocolforControlplane(GTPv2-C);Stage3

TS 23060:GeneralPacketRadioService(GPRS);servicedescription;Stage2

TS 36410:S1layer1generalaspectsandprinciples

TS 36412:S1signallingtransport

TS 36413:S1Applicationprotocol(S1AP)

TS 36414:S1datatransport

TS 29281:GPRSTunnelingprotocolforuserplane(GTPv1-U)


28/32



3GPP TR 22968:StudyforrequirementsforaPublicWarningSystem(PWS)

3GPP TS 22168:EarthquakeandTsunamiWarningSystem(ETWS)requirements;Stage1

3GPP TR 23828:EarthquakeandTsunamiWarningSystem(ETWS),requirementsandsolutions

TS 23060:GeneralPacketRadioService(GPRS);servicedescription;Stage2

TS 23107:QualityofService(QoS)conceptandarchitecture

TS 23203:PolicyandChargingControlArchitecture



TS 23002:Networkarchitecture

TS 23003:Numbering,addressingandidentification

NGMN reerence

wwwngmnorg/uploads/media/White_Paper_NGMN_Beyond_HSPA_and_EVDOpdf

ETSI reerence

ETSI TS 102 250-1:Speech Processing, Transmission and Quality Aspects (STQ); QoS aspects for popularservices in GSM and 3G networks: Part 1: Identification of Quality of Service aspects


29/32


Glossary

AAA authentication,authorization,andaccounting

AMR adaptivemulti-rate

AMR-WB adaptivemulti-ratewideband

APN accesspointname

ASP applicationserviceprovider

BICC bearer-independentcallcontrol

BLER blockerrorrate

BSC basestationcontroller

BTS basetransceiverstation

CAPEX capitalexpendituresCDF cumulativedistributionfunction

CDMA codedivisionmultipleaccess

CLID callinglineidentication

CLIP callinglineidenticationpresentation

CSUDI circuit-switchedunrestricteddigitalinformation

CSFB circuit-switchedfallback

DL downlink

DMT dataminingtoolkit

E2E end-to-end

eNodeB evolvedNodeB;NodeBisaUMTSbasetransceiverstation(BTS)

EPC evolvedpacketcore

EPS evolvedpacketsystem

ESN electronicserialnumber

ETSI EuropeanTelecommunicationsStandardsInstitute

eUTRAN evolvedUMTSterrestrialradioaccessnetwork;alsoabbreviatedasE-UTRANorEUTRAN

FSS frequency-selectivescheduling

GGSN gatewayGPRSsupportnode

GPRS generalpacketradioservice

GUI graphicaluserinterface

HA homeagent

HARQ hybridautomaticrepeatrequest

HLR homelocationregister

HO handoverHTTP hypertexttransferprotocol

IM instantmessaging

IMS instantmessagingservice

IMSI internationalmobilesubscriberidentity


30/32


IP-TV Internetprotocoltelevision

IOT inter-ocetrunk

IP Internetprotocol

IRAT inter-radioaccesstechnology

KPI keyperformanceindicator

LCS locationservices

LSTI LTESAEtrialinitiative

LTE Longtermevolution

MGW mediagateway

MIMO multiple-input/multiple-output

MSC mobileswitchingcenter

MSC-S mobileswitchingcenterserver

MSS mobilesoswitch

MSTV MaximumServiceTelevision

NAI networkaddressindicator

NE networkelement

NEM networkequipmentmanufacturer

NGMN next-generationmobilenetworks

OCNG OFDMAchannel-noisegeneration

OFDM orthogonalfrequency-divisionmultiplexing

OPEX operatingexpenses

OLAP onlineanalyticalprocessing

PCF packetcontrolfunctionPDSN-FA packetdataservingnode,foreignagent

PDSN-HA packetdataservingnode,homeagent

PoC proofofconcept

PSTN publicswitchedtelephonenetwork

QCI QoSclassidentier

QoE qualityofexperience

QoS qualityofservice

QoSM QualityofServiceManager

RNC radionetworkcontroller

ROHC robustheadercompression

RRC radioresourcecontrol

SAE SystemArchitectureEvolution

SC-FDMA single-carrierfrequency-divisionmultipleaccess

SCTP StandardControlTransmissionProtocol

SFN single-frequencynetwork

SGSN servingGPRSsupportnode


31/32


SIP sessioninitiationprotocol

SMS shortmessagingservice

SON self-optimizingnetwork

STP signalingtransferpoint

TDR transactiondetailrecord

TTI transmissiontimeinterval

UE userequipment

UMTS UniversalMobileTelecommunicationsSystem

UL uplink

URI uniformresourceindicator

USSD unstructuredsupplementaryser vicedata

VIP veryimportantperson

VoIMS voiceoverinstantmessagingser vice

VoLGA voiceoverLTEgenericaccess

WAP WirelessApplicationProtocol

WSP wirelessserviceprovider


32/32

Test & Measurement Regional Sales


LTE Test Case App Note

Documents

Transcript of LTE Test Case App Note