CSRIC WG#3 Indoor Location Accuracy Test Bed Report

March6, 2013Stephen J. Wisely & Richard Craig

Co-Chairs

2

Work Group 3 Areas of Focus

The WG addressed the FCC’s questions as best we could for now We developed a testbed framework and have solicited interest from suppliers of current

technology on their in interest in having their product go through the TB We are currently working on the finer points of the testing process and logistics

associated with TB program oversight We anticipate conducting testing in two stages:

Stage I – Characterize performance of current technologies Stage II – review of future technologies

We will provide a read out on Stage I efforts to the CSRIC committee at the September 2012 and March 2013 meetings

We believe the Testbed concept could survive beyond the current CSRIC charter as an independently supported entity for evaluating future technologies

3

Participants

4

First Name Last Name Organization

Brett Schneider Bexar Metro 9-1-1 Network DistrictDeWayne Sennett ATIS Norman Shaw Polaris Wireless, Inc.Susan Sherwood Verizon WirelessGirish Sivaram TeleCommunications Systems, Inc. (TSC)John Snapp Intrado, Inc.Dorothy Spears-Dean Virginia Information Technologies Agency Bill Tortoriello US CellularGreg Turetzky CSR Technology Inc.Bruce Wilson Qualcomm Inc.Stephen Wisely APCO InternationalRichard Deh-Min

Wu Nokia Siemens Networks

WG 3 Indoor Location Test Bed Subgroup consisted of the following members:

First Name Last Name Organization

Wayne Ballantyne Motorola Mobility, Inc.Andrew Beck CommScopeRichard Craig Verizon WirelessKhaled Dessouky TechnoCom Corporation Thomas Dombrowsky Wiley Rein LLPJeanna Green SprintRoger Hixson NENARyan Jensen T-MobileMarte Kinder Time Warner CableSandra Lott CenturyLinkMike Loushine Applied Communication Sciences

Barry Martin BoeingKathy McMahon APCO InternationalMartin Moody Metro Emergency Services BoardGary Parsons NextNav LLCGanesh Pattabiranan NextNav LLCGustavo Pavon True Position, Inc.Raghavendhra Rao AT&TChuck Ronshagen Cassidian Communications

Indoor Test Bed –Stage 1• Goal: Independent comparative testing that provides the FCC with meaningful,

objective results on indoor wireless location performance• Constraints: Within time frame of CSRIC III; financially feasible framework• WG3 chose San Francisco Bay Area due to the wide range of wireless

environments (morphologies) present• Invitation extended to a host of location technology vendors to participate• 7 vendors started the process and 3 eventually participated • Two existing and one new location technologies tested:

– NextNav (wireless beacon technology)– Polaris Wireless (RF pattern matching/fingerprinting)– Qualcomm (AGPS/AFLT hybrid)

• Four major wireless carriers participated:– AT&T, Sprint, T-Mobile, Verizon

• Competent, experienced independent test house selected (TechnoCom)

5

Indoor Test bed Methodology• Based on sound statistical test methodology

– Articulated in ATIS-0500013; received wide consensus

• Multiple buildings (19) of different sizes & types across range of environments – Dense urban, urban, suburban, and rural morphologies

• Multiple test points in each building– 2 to 6 test points depending on size and complexity of selected building– Representing typical use scenarios

• Statistically significant, adequately large number of independent test calls at each stationary test point– 100 test calls per technology minimum target per test point– >13,400 test calls per technology placed at 74 test points, well over 100 avg.– Aggregated over different devices and networks used by the location technology under test

at each test point

• Indoor ground truth accuracy < 1 m (professional survey accuracy 2 cm)• Performance attributes tested: accuracy, yield, TTFF, uncertainty, scatter

6

Test Environments and Buildings

7

Dense urbanDowntown Financial DistrictSan Francisco(6 bldgs)

UrbanDowntownSan Franciscoand San Jose(5 bldgs)

SuburbanSanta Claraand Sunnyvale(6 bldgs)

RuralSan Benito County(2 bldgs)

Urban environment had the most challenged horizontal accuracy

followed closely by dense urban

Summary Results-- Accuracy

8

NextNav technology provided vertical results

NextNav 67% 90%Dense Urban 2.9 4Urban 1.9 2.8Suburban 4.6 5.5

Vertical Distance Error, m

Summary Results: Yield, TTFF, and Uncertainty

• Yield varied with the severity of the environment but was generally high– >90% for all technologies in the configurations tested with one minor exception

(86% AGPS/AFLT yield in dense urban)– More yield variation for the technology actually integrated into the wireless

network and has to exchange messages per the standard E911 implementation through severe RF fading

• TTFF was stable and not an issue– Small variation about design values for non-network integrated implantations

• 27 sec. for NextNav and 24 sec. for Polaris

– More variation for AGPS/AFLT, which is actually integrated into the network • 28 sec. avg. with 33 sec. 90% in urban/dense urban vs. 24 sec avg. & 26 sec. 90% suburban/rural

• Uncertainty is still a useful parameter and its reliability generally reflects how well a system is performing in a given environment

9

Location Scatter Relative to Building Sizes & Density

• Traditional benchmarks from outdoor performance may not best suite the indoors, especially in denser cities

• Even traditionally good performance may not meet the unmitigated expectations of some--education on indoor issues is important

10

Test points in the 5-story building partially shadowed by tall bldg Test point 3 (T3) at center of 50/150 m circles

Lessons and Recommendations• The cooperatively funded indoor test bed is a viable framework, which yields very valuable results

– It’s a model that should be repeated with some adjustments, based on the lessons learned– Key to its success are the experience & strong commitment of its participants Create and maintain the CSRIC-type framework that fosters the active and balanced

participation by the various stakeholders• There are a number of difficult challenges that need to be met repeatedly

– Project setup and contractual arrangements are difficult and time consuming– Building access is the most vexing of challenges facing indoor testing Allow for the long lead time of several months required to properly tackle these issues Create a test management resource with contractual authority and some funding capability Develop a more structured approach to dealing with building access and, as needed, use

creative incentive mechanisms for buildings to participate• Deployed location technologies help with indoor performance and new technologies tested have

promising performance, but there is a long way to go to meet expectations for reliable indoor use of wireless location for E911 Continually test new technologies, including next generations of existing systems,

technologies used for LBS (e.g., WiFi) & new technologies targeted to the latest air interfaces• The Bay Area has proven to be a good place for the indoor test bed

Leverage it fully in future test beds

11