Outdoor Distributed Computing Liviu Iftode Department of Computer Science Rutgers University ...

Outdoor Distributed ComputingOutdoor Distributed Computing

Liviu IftodeLiviu IftodeDepartment of Computer ScienceDepartment of Computer Science

Rutgers UniversityRutgers University

http://discolab.rutgers.eduhttp://discolab.rutgers.edu

Supported by NSF through ITR grant ANI-0121416Supported by NSF through ITR grant ANI-0121416

http://discolab.rutgers.edu/

Traditional Programming ModelTraditional Programming Model

Programs access data through variablesPrograms access data through variables

Variables mapped to physical memory locationsVariables mapped to physical memory locations

Runtime + OS guarantees Runtime + OS guarantees reference consistencyreference consistency

Access time has an (acceptable) upper boundAccess time has an (acceptable) upper bound

Program

Virtual Address Space

Page Table + OS

Physical Memory

Variable access

““Indoor” Distributed ComputingIndoor” Distributed Computing

Computing distributed for performance or fault tolerance Computing distributed for performance or fault tolerance Nodes are computationally equivalentNodes are computationally equivalent Configuration is stable (failures are exceptions)Configuration is stable (failures are exceptions) Networking is robust and has acceptable delaysNetworking is robust and has acceptable delays Relatively easy to programRelatively easy to program

Message passing or shared memoryMessage passing or shared memory Naming easy to translate to network addressesNaming easy to translate to network addresses Correct programs complete and return deterministic resultsCorrect programs complete and return deterministic results

Distributed object Distributed object tracking over a largetracking over a largegeographical areageographical area

Cars collaborating Cars collaborating for a safer and for a safer and more fluent trafficmore fluent traffic

Computers “Move” OutdoorsComputers “Move” Outdoors

Linux CarSensors Linux CameraLinux Watch

Outdoor Distributed SystemsOutdoor Distributed Systems

Nodes are volatile and/or mobile: dynamic configurations Nodes are volatile and/or mobile: dynamic configurations Networking is wireless and ad-hoc: unpredictable delaysNetworking is wireless and ad-hoc: unpredictable delays Location dictates the computation: naming must include locationLocation dictates the computation: naming must include location How to program and execute distributed applications outdoors?How to program and execute distributed applications outdoors?

Traditional distributed computing does not workTraditional distributed computing does not work

How to Program Outdoor Distributed How to Program Outdoor Distributed Embedded Systems?Embedded Systems?

Recent research in networked embedded Recent research in networked embedded

systemssystems HardwareHardware Operating SystemsOperating Systems Network ProtocolsNetwork Protocols Data collection/dissemination in sensor networksData collection/dissemination in sensor networks

Our research focuses on programmabilityOur research focuses on programmability How to code distributed applications to execute How to code distributed applications to execute

over networks of embedded systems?over networks of embedded systems?

Traditional Distributed Computing Traditional Distributed Computing Does Not Work Well over Ad-HocDoes Not Work Well over Ad-Hoc

Fixed address naming and routing (e.g., IP) Fixed address naming and routing (e.g., IP)

are too rigidare too rigid

End-to-end data transfer may hardly End-to-end data transfer may hardly

completecomplete

Outdoor distributed computing requires Outdoor distributed computing requires

novel system architectures and novel system architectures and

programming modelsprogramming models

ExampleExample

““Water the hottest spot on the Left Hill”Water the hottest spot on the Left Hill” Number and location of mobile sprinklers is Number and location of mobile sprinklers is

unknownunknown Configuration is not stable in timeConfiguration is not stable in time

sprinklers movesprinklers move temperature changestemperature changes

What is a good result?What is a good result?

Mobile sprinkler with temperature sensor

Left Hill Right Hill

Hot spot

Quality of ResultQuality of Result

QoR

Outdoors Adversity0 100%

100% ideal

real

Talk at a GlanceTalk at a Glance

Outdoor Programming ModelOutdoor Programming Model

Outdoor Execution ModelOutdoor Execution Model

ApplicationsApplications

Outdoor ProgrammingOutdoor Programming

– How to program an unknown number of mobile systems to execute an intrusion detection application in a certain physical space?- Programming model must be simple to use - Networking aspects should be hidden from programmer- Access to mobile systems as simple as access to variablesAccess to mobile systems as simple as access to variables- A virtual address space for the physical space?


Sequential Programming ExampleSequential Programming Example

struct temp a[N];for(i=0;i<N;i++) {

if (a[i].temp> Max_temp) {Max_temp = a[i].temp;Max_id = i; }

}a[Max_id].water = ON;

Distributed Programming using Distributed Programming using Message PassingMessage Passing

Struct temp a[N/P], b[P];for(i=0;i<N/P;i++) {


}send(0,Max_temp,sizeof(int));If (my_node_id == 0) {

for(i=0;j<P;i++)recv(i,&b[i],sizeof(int));

<compute max of b: b_max is in b_max_id>

…

Applications access distributed data through shared Applications access distributed data through shared variablesvariables

Runtime system translates variable accesses into Runtime system translates variable accesses into message passing (when necessary)message passing (when necessary)

Software Distributed Shared MemorySoftware Distributed Shared Memory

Distributed Application

Shared virtual address space

Page Table + Message Passing

Physical memories

Variable accesses

Shared Memory Distributed Shared Memory Distributed ComputingComputing

struct a[N], b[P];Thread worker(k) {

int Max_temp, Max_id,i;for(i=k*N/P; i< (k+1)*N/P; i++) {


}b[k]=Max_temp;barrier()if (k==0) {

<calculate b_max and b_max_id>

From Indoor to Outdoor From Indoor to Outdoor ComputingComputing

Virtual Address Space Space Region

Variables Spatial References

Variables mapped to physical memory

Spatial references mapped to systems embedded in the physical space

Reference consistency

Bounded access time

?

?

Spatial Programming (SP) at a Glance Spatial Programming (SP) at a Glance

Outdoor distributed applications are Outdoor distributed applications are

programmed using spatial referencesprogrammed using spatial references

A virtual address space hides networking A virtual address space hides networking

Embedded systems/nodes named by their Embedded systems/nodes named by their

expected locations and propertiesexpected locations and properties

Space RegionSpace Region

radius

Hill = new Space({lat, long}, radius);

{lat,long}

Virtual representation of a physical spaceVirtual representation of a physical space Similar to a virtual address space in a Similar to a virtual address space in a

conventional conventional

computer systemcomputer system

Spatial ReferencesSpatial References

– Defined as {space:property} pairs–Similar to variables in conventional programmingSimilar to variables in conventional programming

–Used to name systems embedded in physical space

–Indexes used to distinguish among similar systems in the same space region

-Same index maps to the same system (reference consistency)-Distinct indexes map to distinct systems

{Hill:robot[0]}

{Hill:robot[1]}

{Hill:truck}

Hill

Reference ConsistencyReference Consistency

At the time of the first access, a spatial reference At the time of the first access, a spatial reference

is mapped to an unmapped embedded system is mapped to an unmapped embedded system

located in the specified space located in the specified space

Mappings maintained in per-application Mapping Mappings maintained in per-application Mapping

Tables (MT)Tables (MT)

A spatial reference refers the same system as A spatial reference refers the same system as long as it is located in the same space, fails long as it is located in the same space, fails otherwiseotherwise

{space, property, index}{space, property, index} {network_address, location}{network_address, location}

Reference ConsistencyReference Consistency

For (i=0;i<1000;i++)For (i=0;i<1000;i++)

{Left_Hill:robot[i]}.stop =ON{Left_Hill:robot[i]}.stop =ON;;


{Hill:robot[0]}

{Hill:robot[1]}

Reference Consistency (con’t)Reference Consistency (con’t)

Left HillLeft Hill Right HillRight Hill

{Left_Hill:robot[0]}.move = ON;{Left_Hill:robot[0]}.move = ON;

Left HillLeft Hill Right HillRight Hill

{Left_Hill:robot[0]}.move = OFF;{Left_Hill:robot[0]}.move = OFF;Time

Space CastingSpace Casting



{Right_Hill:robot[0]}

{Left_Hill:{Right_Hill:robot[0]}}

time

Relative SpacesRelative Spaces


{rangeOf({Left_Hill:robot[0]}, radius):robot[0]}

Space CompositionSpace Composition


{(Left_Hill + Right_Hill):robot[0]}

Bounding the Access TimeBounding the Access Time A spatial reference can take an infinite timeA spatial reference can take an infinite time

Discover an unmapped system for a new spatial Discover an unmapped system for a new spatial

referencereference

Mapped systems may move, go out of space, or Mapped systems may move, go out of space, or

disappear disappear

How to bound the time to access a spatial How to bound the time to access a spatial

reference?reference? Associate an explicit timeout with each spatial reference Associate an explicit timeout with each spatial reference

accessaccess

try{ {Hill:robot[0], timeout}.camera = ON;}catch(TimeoutException e){ // the programmer decides the next action}

Hills

Hills

Configuration Changes During Configuration Changes During Execution Execution

ExecutionTime

SystemState

{Hills:robot[0]}

{Hills:robot[1]}

{Hills:robot[2]}

Hills

for(i=0;i<1000;i++) try{

if ({Left_Hill:Hot[i], timeout}.temp > Max_temp)Max_temp = {Left_Hill:Hot[i], timeout}.temp;Max_id = i;

}catch(TimeoutException e) break; {Left_Hill:Hot[Max_id]}.water = ON;

Spatial Programming ExampleSpatial Programming Example

Application: Water the hottest spot on the Left HillApplication: Water the hottest spot on the Left Hill

Mobile sprinkler with temperature sensor


Hot spot

Smart Message ArchitectureSmart Message Architecture

Smart Message (SM)Smart Message (SM) User-defined distributed applicationUser-defined distributed application Executes on nodes of interest named by Executes on nodes of interest named by

propertiesproperties Migrates between nodes of interestMigrates between nodes of interest

Self-RoutingSelf-Routing Application-controlled routing to perform Application-controlled routing to perform

migrationmigration Executed on every node on the path between Executed on every node on the path between

nodes of interestnodes of interest Cooperative NodesCooperative Nodes

Execution environment (Virtual Machine)Execution environment (Virtual Machine) Memory addressable by names (Tag SpaceMemory addressable by names (Tag Space

Smart vs. “Dumb” MessagesSmart vs. “Dumb” Messages

Data migration

Mary’s lunch:

AppetizerEntreeDessert

•`Execution migration

Application ExampleApplication Example

n=0n=0while (n<NumTaxis)while (n<NumTaxis) migrate(Taxi);migrate(Taxi); if (readTag(Available))if (readTag(Available)) writeTag(Available, false);writeTag(Available, false); writeTag(Location, myLocation);writeTag(Location, myLocation); n++;n++;

n=0n=0

TaxiTaxi TaxiTaxi

n=0n=0 n=0n=0 n=1n=1 n=1n=1 n=2n=2

needneed2 taxis2 taxis

data brickdata brick

application application code brickcode brick

routing routing code brickcode brick

Cooperative Node ArchitectureCooperative Node Architecture

NetworkNetwork

SMSMPlatformPlatform

AdmissionAdmissionManagerManagerAdmissionAdmissionManagerManager

NetworkNetworkSMSM SMSM

Operating System & I/OOperating System & I/O

CodeCodeCacheCacheCodeCodeCacheCache

VirtualVirtualMachineMachine

VirtualVirtualMachineMachine

Tag Tag SpaceSpace

Tag Tag SpaceSpace

SM ReadySM ReadyQueueQueue

Smart Message AdmissionSmart Message Admission

Prevents SMs from migrating to nodes where Prevents SMs from migrating to nodes where they cannot executethey cannot execute

SMs specify resources requirementsSMs specify resources requirements Lower bounds for virtual machine cycles, tag space Lower bounds for virtual machine cycles, tag space

memory, runtime memory, network bandwidth, etc.memory, runtime memory, network bandwidth, etc. Specified by programmer or derived using compiler Specified by programmer or derived using compiler

supportsupport SMs accepted if the node can satisfy SM SMs accepted if the node can satisfy SM

requirementsrequirements SMs transfer only the missing code bricksSMs transfer only the missing code bricks More resources can be granted according to More resources can be granted according to

admission policyadmission policy If not granted, SM is allowed to migrateIf not granted, SM is allowed to migrate

Smart Message ExecutionSmart Message Execution

Takes place over a virtual machineTakes place over a virtual machine

Non-preemptive but time boundedNon-preemptive but time bounded

Ends with migration or terminatesEnds with migration or terminates

During execution, SMs canDuring execution, SMs can Create new SMs Create new SMs Spawn new SMsSpawn new SMs Access the tag spaceAccess the tag space Block on a tags to be updatedBlock on a tags to be updated

Tag SpaceTag Space Local memory that SMs can useLocal memory that SMs can use

SM-created tags: limited lifetime, persistent across SM SM-created tags: limited lifetime, persistent across SM

executionsexecutions

I/O tags: maintained by the systemI/O tags: maintained by the system

Multiple RolesMultiple Roles

Node addressing Node addressing migrate (tag,timeout)migrate (tag,timeout)

I/O port access I/O port access read (temperature)read (temperature)

Store dataStore data write (tag,value)write (tag,value)

Inter -SM communication Inter -SM communication

Synchronization on writeSynchronization on write block(tag,timeout)block(tag,timeout)

Routing Routing

Protection Domains for Tag SpaceProtection Domains for Tag Space

Owner: SM that creates the tagOwner: SM that creates the tag Family: all SMs having a common ancestor with the SM Family: all SMs having a common ancestor with the SM

that creates the tagthat creates the tag Origin: all SMs created on the same node as the family Origin: all SMs created on the same node as the family

originator of the tag owner originator of the tag owner Code: all SMs that carry the same code brick(s)Code: all SMs that carry the same code brick(s) Others: all SMsOthers: all SMs

FamilyFamily

CodeCodeOriginOrigin

Others

OwnerOwner

Access Control Example Access Control Example (Code-based protection domain) (Code-based protection domain)

Node5

Node3

Node4

SM2SM2

SM2

C2 Cr

Node1

Node2Code

Bricks

Tag

Owner = SM1[Hash(Cr),

RW]

SM1C1 Cr

SM1 SM1SM3

C3C4

Access permission granted for SM2

Access permission denied for SM3

Cr Same routing used by SM1 and SM2

Application ExampleApplication Example

n=0n=0while (n<NumTaxis)while (n<NumTaxis) migrate(Taxi);migrate(Taxi); if (readTag(Available))if (readTag(Available)) writeTag(Available, false);writeTag(Available, false); writeTag(Location, myLocation);writeTag(Location, myLocation); n++;n++;

n=0n=0

TaxiTaxi TaxiTaxi

n=0n=0 n=0n=0 n=1n=1 n=1n=1 n=2n=2

needneed2 taxis2 taxis

data brickdata brick

application application code brickcode brick

routing routing code brickcode brick

migrate()migrate() – implements routing algorithm– migrates application to next node of interest– node of interest named by tags

– sys_migrate()sys_migrate()– one hop migration– used in migrate to implement routing

MigrationMigrationmigrate(Taxi)migrate(Taxi)

sys_migrate(2)sys_migrate(2) sys_migrate(3)sys_migrate(3) sys_migrate(4)sys_migrate(4)

TaxiTaxiTaxiTaxi

11 22 33 44

NetworkNetwork

Routing ExampleRouting Example 1 2 i

RouteToTaxi = 2

TaxiRouteToTaxi = ?migrate(Taxi){

while(!readTag(Taxi)) if (readTag(RouteToTaxi)) sys_migrate(readTag(RouteToTaxi)); else create_SM(DiscoverySM, Taxi); createTag(RouteToTaxi, lifetime, null); block_SM(RouteToTaxi, timeout);}

RouteToTaxi = j

Self-RoutingSelf-Routing

SMs carry the routing brick and execute it at SMs carry the routing brick and execute it at each nodeeach node

SMs control their routingSMs control their routing Select routing algorithm (migrate primitive)Select routing algorithm (migrate primitive)

Multiple library implementationsMultiple library implementations

Implement a new oneImplement a new one

Change routing algorithm during execution in Change routing algorithm during execution in

response toresponse to

Adverse network conditionsAdverse network conditions

Application’s requirementsApplication’s requirements

Dynamic Change of RoutingDynamic Change of Routing

SM starts with SM starts with proactive routingproactive routing

migrate(Taxi, timeout1)migrate(Taxi, timeout1)

Dense networkDense networkLow mobilityLow mobility

Proactive routingProactive routing

Sparse networkSparse networkHigh mobilityHigh mobility

On-demand routingOn-demand routing

migrate timeoutsSM changes routing to on-demand

migrate(Taxi, timeout2)

Prototype ImplementationPrototype Implementation Modified version of Sun’s Java K Virtual MachineModified version of Sun’s Java K Virtual Machine

Small memory footprint (160KB)Small memory footprint (160KB)

SM and tag space primitives implemented inside SM and tag space primitives implemented inside

virtual machine as native methods (efficiency)virtual machine as native methods (efficiency)

Implemented I/O tags: GPS location, neighbor Implemented I/O tags: GPS location, neighbor

discovery, image capture, light sensor, system statusdiscovery, image capture, light sensor, system status

Prototype Node with GPS receiver and video camera

Lightweight Migration Lightweight Migration Traditional process migration difficultTraditional process migration difficult

Strong coupling between execution entity and hostStrong coupling between execution entity and host OS state (e.g., open sockets, file descriptors) hard to transferOS state (e.g., open sockets, file descriptors) hard to transfer

Tag space decouples the SM execution state from the Tag space decouples the SM execution state from the

OS stateOS state

SM migration transfersSM migration transfers Data bricksData bricks explicitly specified by programmer as mobile explicitly specified by programmer as mobile

datadata Minimal execution control stateMinimal execution control state required to resume the SM at required to resume the SM at

the next node (e.g., instruction pointer, operand stack the next node (e.g., instruction pointer, operand stack

pointer) pointer)

Experimental Results for Simple Experimental Results for Simple Routing AlgorithmsRouting Algorithms

Completion Time

Routing algorithmCode not cached (ms)Code cached (ms)

Geographic

On-demand

415.6 126.6

506.6 314.7

user nodenode of interestintermediate node

Self-Routing SimulationSelf-Routing Simulation

starting nodestarting node node of interest other nodeother node

3 nodes of interest located in the corners have to be visited in clockwise order vary the radius from 100m to 700m

0

0.5

1

1.5

2

2.5

3

0 100 200 300 400 500 600 700 800

Region Radius (meters)

Com

plet

ion

Tim

e (s

ec)

On-Demand Routing Geographic+On-Demand Routing

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 100 200 300 400 500 600 700 800

Region Radius (meters)

Byt

es S

ent

in t

he

Net

wo

rk (

KB

ytes

)

On-Demand Routing Geographic+On-Demand Routing

On-Demand Routing versus Geographic and On-Demand Routing

SP Implementation using SMSP Implementation using SM

SP application -> SMSP application -> SM Embedded system properties -> TagsEmbedded system properties -> Tags SM self-routing SM self-routing

Content-based routing using tagsContent-based routing using tags Geographical routingGeographical routing

Reference consistencyReference consistency unique tag created on the node which a spatial reference is unique tag created on the node which a spatial reference is

mapped tomapped to MappingTable (MT) maps spatial reference to the unique MappingTable (MT) maps spatial reference to the unique

tag and expected locationtag and expected location MT is carried by the SM that implements the SP applicationMT is carried by the SM that implements the SP application

Spatial reference access -> an SM migration to the Spatial reference access -> an SM migration to the node where the spatial reference is mapped tonode where the spatial reference is mapped to

Max_temp = {Left_Hill:Hot[1], timeout}.temp;

SP using SMs: ExampleSP using SMs: Example

Mobile sprinkler with temperature sensors


Hot spot

Spatial Reference Access

Smart Message

ret = migrate_geo(location, timeout);if ret == LocationUnreachable ret = migrate_tag(yU78GH5, timeout);if (ret == OK) && (location == Left_Hill) return readTag(temp);else throw TimeoutException

{Left_Hill,Hot,1} {yU78GH5,location}MT

CodeBrick

SP Application: Intrusion SP Application: Intrusion detectiondetection

Code Size breakdown for SM Code Size breakdown for SM (Application + SP Library)(Application + SP Library)

Code Size breakdown for Code Size breakdown for SP librarySP library

Testbed: 10 HP iPAQs with 802.11 cards and GPS devicesTestbed: 10 HP iPAQs with 802.11 cards and GPS devices

user nodeuser nodelight sensorlight sensorcamera nodecamera node

regular noderegular node

monitoredmonitoredspacespace

Execution Time BreakdownExecution Time Breakdown

Intelligent Distributed Intelligent Distributed Transportation SystemsTransportation Systems

Two projects:Two projects: EZCab: An Automatic EZCab: An Automatic

System for Booking CabsSystem for Booking Cabs Traffic Viewer: A Scalable Traffic Viewer: A Scalable

Traffic Monitoring SystemTraffic Monitoring System

Infrastructure-free approach based on car-to-car communication

short-range wireless communication location information from GPS receiver vehicle’s data from sensors through on-board diagnostic system (OBD-II)

EZCab IdeaEZCab Idea allow people to book cabs in dense cab allow people to book cabs in dense cab

area area using ad-hoc networking using ad-hoc networking

Finding Free CabsFinding Free Cabs

Booking a cabBooking a cab

Phase 2 & 3: Report and Phase 2 & 3: Report and ValidationValidation

TrafficViewTrafficView

• Enable drivers to monitor traffic in front of their cars, farther Enable drivers to monitor traffic in front of their cars, farther than they can seethan they can see

• Based exclusively on vehicle-to-vehicle ad hoc communicationBased exclusively on vehicle-to-vehicle ad hoc communication

How TrafficView WorksHow TrafficView Works

Receive data fromremote vehicle

Non-validateddataset

Validate

Validateddataset

Local data

Display

Broadcast data

Data AggregationData Aggregation

How to aggregate data to collect monitor How to aggregate data to collect monitor

vehicles as far as possible with “acceptable” vehicles as far as possible with “acceptable”

accuracy lossaccuracy loss

Natural SolutionNatural Solution Aggregate data for vehicles that are close to Aggregate data for vehicles that are close to

each othereach other Perform more aggregation as distance increasesPerform more aggregation as distance increases

Smart Phones as Universal Smart Phones as Universal Personal ServersPersonal Servers

Mobile phones and PDAs converge into Smart Mobile phones and PDAs converge into Smart PhonesPhones ProgrammableProgrammable Dual-connectivity (BT & GPRS)Dual-connectivity (BT & GPRS)

How to build pervasive applications on them?How to build pervasive applications on them?

SummarySummary

Spatial Programming makes outdoor distributed Spatial Programming makes outdoor distributed

computing simplecomputing simple

Volatility, mobility, configuration dynamics, ad-Volatility, mobility, configuration dynamics, ad-

hoc networking are all hidden from programmerhoc networking are all hidden from programmer

Spatial Programming can be implemented using Spatial Programming can be implemented using

Smart MessagesSmart Messages

Outdoor applications: intelligent transportation, Outdoor applications: intelligent transportation,

pervasive computing using smart phones pervasive computing using smart phones

AcknowledgementsAcknowledgements FacultyFaculty

Uli Kremer (check our PLDI’05 paper!)Uli Kremer (check our PLDI’05 paper!) Graduate StudentsGraduate Students

Cristian Borcea (currently on the NJIT faculty)Cristian Borcea (currently on the NJIT faculty) Porlin KangPorlin Kang Nishkam RaviNishkam Ravi Pravin ShankarPravin Shankar Peng ZhouPeng Zhou Deepa IyerDeepa Iyer Akhilesh SaxenaAkhilesh Saxena

VisitorsVisitors Chalermek Intanagonwiwat (post-doc, currently on the Chalermek Intanagonwiwat (post-doc, currently on the

faculty of Chulalongkorn University, Thailand)faculty of Chulalongkorn University, Thailand) Oriana Riva (grad student, University of Helsinki)Oriana Riva (grad student, University of Helsinki) Tamer Nadeem (grad student, UMD)Tamer Nadeem (grad student, UMD) Marios Dikaiakos (faculty, Univ of Cyprus)Marios Dikaiakos (faculty, Univ of Cyprus)

Undergraduate students:Undergraduate students: Niket Desai, Peter Stern, Yan NuriyevNiket Desai, Peter Stern, Yan Nuriyev

Thank you!

http://discolab.rutgers.edu

Outdoor Distributed Computing Liviu Iftode Department of Computer Science Rutgers University ...

Documents

Transcript of Outdoor Distributed Computing Liviu Iftode Department of Computer Science Rutgers University ...