ExScal Report

Anish AroraAnish Arora

The ExScal TeamThe ExScal Team

June 28, 2004June 28, 2004

Anish AroraAnish Arora

The ExScal TeamThe ExScal Team

June 28, 2004June 28, 2004

2

Outline

• Requirements and scenarios

• Architecture• Topology: Tiers 1 & 2

• Hardware components• XSM & its acoustic, PIR, magnetometer sensing hardware

• Stargate & its GPS support; power

• Software components• Reprogramming

• Messaging at Tiers 1 and 2

• Integration status

3

Outline of ExScal breakout session

• Site and logistics• Deployment• Timeline

• Xsm schedule

• Workplan

• Metrics• Process• Monitoring & Management• Simulation & Testbed• Beyond extreme scaling:

• Other proposed ExScal experiments for December ?

• What to measure ?

4

Requirements

• Quick, high confidence detection & classification of multiple intruder types over long “linear” region

• Fine-grain localization of intruders initially

• Bounded-uncertainty tracking of intruders subsequently

• Concurrent tracking of small number of separated intruders

(or intruder groups)

• Efficient control/maintenance, especially reprogramming,

localization, health monitoring

• Reliable, extended operation (over a fortnight)

5

Sample scenarios

Intruding persons arrive by boat, enter and cross lines• (pir) detection, classification, and fine-grain localization

Persons run through the line (a 10K event!)• coarse-grain (bounded uncertainty) tracking• monitor line health

ATV weaves across line• (acoustic) detection, classification, and fine-grain localization

Car/bus traverse road• coarse-grain tracking; (magnetometer) classification based on

initial line crossing

Visualize network health/performance statistics

6

Chosen topology wrt to scenario requirements: Grid

• Thick line allows detection & classification w.h.p. as intruders enter the protected region; also allows fine grain intruder localization

• Grid of thin lines allows bounded uncertainty tracking

Thick Entry Line

A S S E T

10 km

500 m

7

Tier 1 (XSM) topology

• Detection: Moving intruder detected by at least 10 PIRs

• Classification: Car detected by ≥5 magnetometers

• Tracking: Car traveling at 40 km/hr will be detected for >4 seconds

• Fault-tolerant coverage:

detection is tolerant to crash failure of <7 motes in a contiguous region

classification is tolerant to crash failure of <4 motes in a contiguous region

higher #failures tolerated if crash failures are uniform

… … … … …

180 m

9 m

9 m

4.5 m

36 m

4.5 m

90 m

• # XSMs per Tier2 node ≤50

• each XSM has >12 comm. neighbors (assuming inner band radius of 20m)

• # of XSM hops to Tier2 node ≤4

• separation 91m

8

Tier 2 (Stargate) topology

A S S E T

9,720m

90 m

Base Station4,860m 4,860m

16 m

180 m

180 m

180 m

360 m180

m

Path length

• # XSM hops to supernode on thick (thin) line is ≤4 (5)

• # Stargate hops to basestation is ≤20 assuming comm. range of 300m

Fault-tolerant coverage:

• thick (thin) lines tolerate contiguous failure of ≥9 (2) Stargates

• more if uniform

275 stargates

9

XSM design considerations

• Improve detection range

• Improve lifetime (previously 3-4 days)

low latency wakeup to support continuous passive vigilance

• Lower cost

• Reliability: deal with 10K nodes with incorrect program

• Fix radio anisotropic radiation and impedance mismatch

10

Main changes over Mica2:• Improved sensing, actuation:

added PIR

integrated magnetometer set/reset; fewer ferromagnetic components & their distance to mag. is more

adjustable frequency sounder

• Low-power operation: wakeup circuits for PIR & acoustic

programmable LPF&HPF (acoustic)

• Lower-latency on mag. circuit   

• Grenade timer added for reliability

• One-touch operation USER, RESET switches

• Centered antenna, impedance matched to deal with irregular RF

XSM

11

Low-power wakeup concept

• Measure signal statistics

• Compute filter cutoffs and signal thresholds

• Program filters and comparator thresholds

• Turn off processor and radio

• Leave signal conditioning circuit running

• Real events and false alarms wakeup MCU

• Perform more complex signal processing in SW

• Reject false alarms

12

XSM enclosure

3” x 3” x 3”

13

XSM power management

• Energy: 1700 mAhr (85% of 2000 mAhr at 2.2V)

• Network reprogramming: 128 kB → 1 mAhr

• Option for 14 day lifetime:

Partially active ≤ 2 mAhr spent in sleep mode ≤ 20 mAhr spent in reprogramming 84 hours of active lifetime remaining → 6

hours of active life per day for 14 days

Low-power radio mode (35.5% duty cycle, 19 pkt/s)

17mAhr in active mode → 7 hours of active life per day for 14 days

Put CPU to sleep during sensing

Other approaches for reducing power consumption in active mode

• Power management built-in TinyOS Use HPLPowerManagement and Timer modules

Module Active Sleep

CPU 8mA 10 A

Radio 8mA, 16mA (Tx)

1 A

Radio (35.5% duty cycle)

3mA, 12mA (Tx)

2 A

Flash 15mA (write)

2 A

Sensors 4 mA 8 A

Total 20 mA (on average)

40 A

14

Tier 2 (Stargate) nodes

CPU: Stargate with Mica2

Comm: SMC2532 HighPower (23dBm) 802.11 card

GPS: TripNAV TN200, WAAS Accuracy~3m

Antenna: 9 dBi collinear omnidirectional antenna

Packaging:Waterproof Stargate Enclosure with

integrated Antenna Mast and Base

Battery: Sealed Lead Acid Battery (105Ah)

Life: Idle Stargate + 802.11 Card: 350 mA

14 Days/12 Hour Network Access: 60 Ah

45 Ah Reserve for Compute + TX

OS: v7.1

15

Wireless link performance

Log-distance path loss model

Experiment overview:

• Open grass (6”) field at OSU Airport

• Antenna height: 4’ (120 cm)

• Distance: 100 - 400 m

• Measurements: RSSI reported by card

for 200 packets

at Max Power level &

fixed rate at 1Mbit/sec

Pr(d) Pr (d0) 10 log(d /d0)

16

Wireless link performance

Log-distance path loss model

Pr(d) Pr (d0) 10 log(d /d0)

Receiver sensitivity8% PER at -89dBm

17

Wireless link performance

Time variation of received signal strength at 400m

18

Power management

• Radio consumes 60% of the power of Stargate (>200mA out of

350mA)

• Wakeup on wireless traffic will not be supported for 802.11

• 10 second deadline over 20 hops to Tier 3

fast powercycling of radios requires firmware modification

wireless connections at Tier 2 up continuously

19

Status: • first release integrated onto XSM; used

in April IPT demo• characterization & calibration of old XSM

acoustic hardware

Concept:

• energy detector with sliding median

filters for tracking mean & variance of

background noise

Typically range 10-15m:• SUV@20mph on grass, higher on asphalt• good impulsive noise rejection• reduced sensitivity for high background

noise• wind causes anisotropic, time-varying range

Acoustic detector (MITRE)

20

Acoustic detector: Improvements

1G:• add AGC to detector• use extra 2-bits of accuracy• band-pass median filter• 8x increase in sensitivity

range increases by

1.5 G:• in quiet environments, signals

drop below 1 ADC unit• will use quantification aware

point estimator• allows for stochastic

resonance or dithering

8

10-3

10-2

10-1

100

101

10-3

10-2

10-1

100

Quantization Estemator (N = 128)

Sigma in A2D Units

Sta

ndard

Devation o

f E

ste

mation in A

2D

Units

Meansqrt(1/12/N)Mediansqrt(1/4/N)Quant MLEQuant Mediansqrt(1/2) / N

100

101

102

103

10-4

10-3

10-2

10-1

100

Law of Large Numbers (sigma = 0.010000)

N

Sta

ndard

Devation o

f E

ste

mato

r

Meansqrt(1/12/N)Mediansqrt(1/4/N)Quant MLEQuant Mediansqrt(1/2) / N

21

Acoustic detector: 2G

Targets-not-of-interest:

• as range grows, number of targets-not-of-interest grows

birds, airplanes, wind tree creek, people talking, etc.

especially acute in urban settings

• to do much better than 1G must do partial classification

probably the real limit on range, not sensitivity

i.e., address targets not of interest problem

Proposed Method:

• do low resolution FFT

• fit simple templates to spectrum a car is flat from 100 Hz to 2K

Hz (approx.) a bus is bi-modal an ATV is low-frequency a bird is very narrow band speech is tonal & mid-band

• use template match as detection statistic

22

Performance• good SNR observed

so far…

• should support long-range,

low-false alarm detection

PIR detector

human 3 crossings @ 35 ft

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

2 S/divPDF of NoiseKurtosis = 2.97 Concepts

• Sensor & signal conditioning hardware• ADC• PIR Driver (ADC, Wakeup)• Sampling• Long-term moving and 2 computation

• Decide H = H0 if x ≤ , H1 if x > where

= + and is a function of PFA

• Apply digital hysteresis to H

xS N

23

Localization

Routing

Monitoring

messaging

Time sync

Network prog Power man

Application

node locations

location timestampsof events

failure info

Reliable Comm

Other Tier 1 “foundation” components

24

Messaging at Tier 1

• Substantial performance improvement since last year

• Grid Routing enhancements: reduced # of potential parents (only motes that are closer to

supernode) no inversion

load balancing of parents: switch parent when beacon from potential parent received fast recovery in case of failure

broadcast protocol now piggybacks network parameters in beacons

each mote has primary & secondary supernodes to tolerate supernode failure

• Reliable Comm enhancements

25

Network testbed

• 7 * 7 grid of MICA2 motes in 3-4 inches tall grass field

Base station

26

Application traffic (trace driven)

• Car moving across the network from left to right at a speed of 15 MPH

• A mote generates a “start” message at the beginning of an event; the mote generates an “stop” message at the end of the event

• All messages are sent to the base station, which performs higher-level detection & classification

27

Application trace (contd.)

• Distribution of packet generation

• Highest burst rate = 14.07 packets/sec exceeds limit of BMAC in multi-hop networks ~42.93/4 =

10.73 packets/sec (note: even an ideal MAC only gives ¼ throughput in the case of multi-hop)

• Optimal event goodput: 6.66 packets/sec

0 5 10 150

20

40

60

80

100

Time (seconds)

The

num

ber

of p

acke

ts g

ener

ated

28

Exscal GR & RC (up to 2 retransmissions)

Average delivery ratio = 98.8%

up from 46.5% last year

0.95

1

1

1

1

1

1

1

1

1

1

0.95

0.95

1

1

1

1

1

1

1

0.95

1

1

1

1

1

1

0.95

1

1

1

1

1

1

1

1

1

0.9

1

1

1

1

1

1

0.9

1

0.95

0.95

29

Exscal GR & RC: distribution of packet reception

0 2 4 6 8 10 12 14 160

10

20

30

40

50

60

70

80

90

100

Time (seconds)

The

num

ber

of u

niqu

e pa

cket

s re

ceiv

ed

RT = 0RT = 1RT = 2

Max. event goodput: 6.37 packets/sec

(close to optimal goodput: 6.66 packets/sec)

Average delay: 1.31 seconds

30

Distribution of packet generation and reception

0 5 10 15 20 25 30 350

10

20

30

40

50

60

70

80

90

100

Time (seconds)

The

num

ber

of u

niqu

e pa

cket

s re

ceiv

ed

Traffic traceExscal version GR & RCLiTes version GR & RC

32

Network Programming: Deluge (UCB)

• Reliable Pipelined Epidemic Distribution of series of pages Robust to lossy or asymmetric links Very low maintenance bandwidth

• Page Advertise, Request/Fix, Xfer Density-aware suppression and snoop on each

• Packet CRC + Page CRC

• 159 Byte memory footprint

• Packed image (no 64k xnp limit)

• Multiple Program images

• Extensive simulation of dissemination Alg. and many variants

• Tested extensively on 77 nodes

• Simulated on >1,000 nodes in EXSCAL configuration with multiple sources

• 4 mins for 1-hop 33k image to 33 nodes

• Command line host tools

flash

…

Maintain

Request

Transmit

33

Localization

Routing

messaging

Time sync

Monitoring-Man.

Data aggreg

node locations

Tier2 architecture

Network prog

Tier1-Loc.

34

Messaging at Tier 2

• First implementations tested in both testbed and simulations

• Three contexts: Unicast

Preliminary experimental results

Stream Broadcast ns2 simulation results

Single Packet Broadcast Preliminary experimental results

35

Indoor experimental testbed

2 4 6 8 10 12 140

10

20

30

40

50

60

70

80

90

100

The distance (inches)

Link

relia

bilit

y (%

)

• Indoor table-top with 15 Stargates

• Radio: SMC IEEE802.11b card without antenna

• Topology 3 X 5 grid 4 inches separation

Link reliability based on ping

36

Unicast routing

• Low power consumption

• Alternate path routing on link failure

• No packet losses

• No periodic beaconing

• Multiple neighbors can be chosen as next hop node

• Exploit:

MAC failure events: iwevent feedback

Buffer failure event: Emstar feedback

Goals Design Principles

37

Unicast protocol

• Learn information about neighbors during “network initialization” location of neighbor identity of neighbor reliability on link to neighbor

• Use greedy packet forwarding ignore neighbors with low reliability pick up next-hop that minimizes distance to destination

38

Packet delivery percentage

• Current unicast protocol mechanisms for detecting node & link failure identified but

not yet implemented each packet is simply transmitted twice as otherwise end-to-

end reliability is around 60%• For transmission from source to diagonally opposite sink

• Each node sends 1 packet every 10 seconds Four farthest nodes sending packets to the sink

Average packet delivery percentage = 96.88% All nodes sending packet

Average packet delivery percentage = 97.9%

Inter-packet interval (seconds)

10 5 2

Packet delivery 93.91% 100% 98.90%

39

Unicast: plans

• Implement techniques for handling node and link failure MAC and buffer losses

• Experiment with various metrics for selecting next-hop node

• Test on larger topologies

40

Broadcast for reliable reprogramming

• Minimize number of data transmissions

• Minimize contention between new data transmissions and old data re-transmissions

• Use a minimum connected dominating subset (MCDS) of nodes for data transmissions

• Phase I for streaming new data and phase II for recovery

Goals Design Principles

41

Reliable broadcast protocol

• Pre-determined parent child relationship Between MCDS nodes Between non-MCDS nodes and MCDS nodes

• Phase I (Only MCDS nodes participate) Tier-3 node streams packets at a fixed streaming rate Each node broadcasts new data packets, with following piggybacked

information a bit-vector with 1’s for the packets it has received total number of packets

Each node estimates the time at which phase I ends

• Phase II (All nodes participate) Periodically unicast bit-vector to parent if packets are missing Unicast data to children requesting recovery

42

Ns2 simulation results for 219 nodes: Latency

• 128 51-byte packets in the final 219 node stargate topology

• Finding the right streaming period to minimize latency

• Disk model, contention, capture effects, not link loss

Latency

02468

1012141618

0.012 0.013 0.014 0.015 0.016 0.017

Streaming Period

Tra

nm

issi

on

Tim

e

Recovery

Streaming

43

Ns2 simulation results for 219 nodes: reliability

Reliability

121122123124125126127128129

0.012 0.013 0.014 0.015 0.016 0.017

Streaming Period

Mes

sag

es R

ecei

ved

Recovery

Streaming

44

Overhead

0

5000

10000

15000

20000

25000

0.012 0.013 0.014 0.015 0.016 0.017

Streaming Period

Mes

sage

s Tr

ansm

itted

Recovery

Streaming

Ns2 simulation results for 219 nodes: overhead

45

Reliable broadcast plans

• Simulate lossy channel So far losses are only due to collisions or capture

effects at MAC

• Extend the protocol dynamic streaming-rate computation mechanisms to deal with node failures k-connected structure instead of 1-connected

• Experiment with stargates

46

Single packet broadcast: Motivation

• Reliably distributing commands Start the network Put nodes in doze mode Put nodes in fully awake mode Commands for T1 nodes

• Reliably distributing parameters New transmit power New data rate Parameters for T1 networks

• Reliably querying the T2 nodes How many T2 nodes are up? How many T2 nodes got the last broadcast successfully? Status report of T1 nodes

Is T1 localization over? How many T1 nodes are up?

47

Single packet broadcast

• Highly Robust: reliable delivery and ack

• Minimize number of overhead messages

• Minimize interference between request and response messages

• Periodic request and response messages

• Use aggregation of responses

• Use separate phases for request and response messages

Goals Design Principles

48

Single packet broadcast(Two Phase Protocol)

• Phase I: inform T2 nodes; create a structure for phase II T3 node sends k periodic beacons Each T2 node on hearing the first beacon starts sending k periodic

beacons Each T2 node estimates the end of first phase Each T2 node registers with a parent to which it will send

aggregated response

• Phase II: aggregate responses from T2 nodes If node is a leaf node

Send periodic responses to parent

else Aggregate responses from children and send periodically to parent

Parent acknowledges children for each response Child stops sending response on hearing ACK

49

Single packet broadcast: Status and plans

• Status Experiments on 15-node indoor grid network

A preliminary version of the protocol has been testedo Response is transmitted only once (not periodically)o Phase II is merged with Phase I

Beacon period: 10 sec Response message period sent only at 5 sec intervals

With aggregation (average of 5 runs) 13 out of 15 nodes are heard back in 14.5 sec

Without aggregation (average of 5 runs) 9 out of 15 nodes are heard back in 11.5 sec Slightly faster but low reliability

• Plans Experiments with the more robust protocol Experiments with larger topologies

50

12 hand-placed XSMs known

locations

w/ active acoustic sensing

2 stargates for “left” & “right” 6

XSMs

System can:

• detect intruders (including golf-cart, car, SUV, tractor, plane)

• visualize intruder on Tier 3 node

• classify some intruder types, using multi-modal distributed influence fields

Standalone reprogramming demo : 50 nodes

Standalone localization demo (46nodes, improved June 15th)

April 30th demo layout & features

50 m

~10 m

~10 m

51

Integration status

• Sensing signal chains Mag.(2-3m for metal-bearing humans, 5-7m for vehicles) Acoustic (15-25m for vehicles) PIR (10m for humans)

• Tier 1 network services GridRouting ReliableComm TimeSync (modified version) Deluge (Multihop reprogramming) Suspend application during reprogramming

• Tier 2 Mica2 “transceiver”: talks to Emstar “hostmoted” component on Stargate

Emstar “routing” component for application to communicate w/ Tier 3 node