Classification and Analysis of Distributed Event Filtering Algorithms Sven Bittner Dr. Annika Hinze...

Classification and Analysis Classification and Analysis of Distributed of Distributed

Event Filtering Algorithms Event Filtering Algorithms

Sven BittnerDr. Annika HinzeUniversity of WaikatoNew Zealand

Presentation at CoopISPresentation at CoopISCyprusCyprus27. 10. 200427. 10. 2004

2/25

MotivationMotivation

Event Event Notification Notification

SystemSystem

e2: t=30°C

e3: r=0,2 liter

ee44:: r=2 r=2

literliter

ee11 :: t=15°C

t=15°C

EventsEvents

FilteringFiltering

Efficient scalable Efficient scalable filteringfiltering

MessagesMessages

((ee 22))

(e1)

((ee33 ), ), ((ee

44 ))

Profiles Profiles SubscribersSubscribers

p1=(t>22°C)

p2=(t<18°C)

p3=(r>0,1 lit.)

ProvidersProviders(Sensors)(Sensors)

Facilty Mangement (one building)Facilty Mangement (one building)• >10>1044 ProfilesProfiles• >10>1033 Events/ Events/SecondSecond

3/25Classification Theoretical analysis Experiments Open Research

Distributed FilteringDistributed Filtering

1

3 4

5

ENSENS

DistributedDistributed

62 Central Filter-

components

62

Acyclic overlay network for

distribution of profiles and

events

- manage set of exclusive local clients - forward matching event messages to clients

Clients communicate with Brokers

S

P

4/25

State of the Art: Distributed State of the Art: Distributed algorithmsalgorithms

Rendezvous nodes: Rendezvous nodes: certain brokers specialise in filtering of certain event types, certain brokers specialise in filtering of certain event types, meeting meeting points for profiles and event messages meeting meeting points for profiles and event messages [11,12][11,12]

Event/profile forwarding:Event/profile forwarding:– brokers know all profiles, propagates events down the tree [1]brokers know all profiles, propagates events down the tree [1]– Brokers only know profiles registered at it‘s children, event Brokers only know profiles registered at it‘s children, event

forwarded up to root and then down to leaves [3,14]forwarded up to root and then down to leaves [3,14]– Point2point: broker knows about profiles of neighbours, Point2point: broker knows about profiles of neighbours,

forwarding of events or profiles [3]forwarding of events or profiles [3]

Optimizations:Optimizations:– Covering and merging of overlapping profiles [9,10]Covering and merging of overlapping profiles [9,10]– Immediate covering computation [3]Immediate covering computation [3]– Covering computation of request [7]Covering computation of request [7]

5/25

Problem definition: Efficient Problem definition: Efficient algorithmsalgorithms

Evaluations so farEvaluations so far– Mostly based on simulated network topologiesMostly based on simulated network topologies– Restricted analysis of influencial factorsRestricted analysis of influencial factors– Independently for single algorithms Independently for single algorithms

(ie under different conditons) (ie under different conditons)

Open issues: Open issues:

1. classification scheme for distributed filter algorithms1. classification scheme for distributed filter algorithms

2. uniform performance analysis of filter algorithms that 2. uniform performance analysis of filter algorithms that allows for a comparison of the algorithms’ efficiencyallows for a comparison of the algorithms’ efficiency

6/25

RoadmapRoadmap


Classification Classification – Schema for distributed filter algorithms Schema for distributed filter algorithms – Classification of filter algorithmsClassification of filter algorithms

Theoretical performance analysis Theoretical performance analysis

Experimental performance analysisExperimental performance analysis

Conclusion and OutlookConclusion and Outlook

Classification Theoretical analysis Experiments Open Research

7/25

Classification: DimensionsClassification: Dimensions

Location of filteringLocation of filtering– Subscribers, Publishers, Arbitrary Subscribers, Publishers, Arbitrary

Spreading of Filter complexitySpreading of Filter complexity– Exclusive Exclusive – Distributed Distributed

• Storage strategies: preventive vs optimitsticStorage strategies: preventive vs optimitstic

Communication with subscribersCommunication with subscribers– Direct, forwarding, transparentDirect, forwarding, transparent


8/25


Filter LocationFilter Location

– Close to subscribers: Close to subscribers: Profile submitted only to Profile submitted only to closest broker, flooding of closest broker, flooding of event messages over the network (event flooding)event messages over the network (event flooding)

– Close to publishers:Close to publishers:Events published only to closest broker, flooding Events published only to closest broker, flooding of profiles over the network (profile flooding) of profiles over the network (profile flooding)

1

3 4

5

62 ENSENS


P S

– Arbitrary Arbitrary dedicated brokers responsible for filtering of dedicated brokers responsible for filtering of specific message types, forwarding of events specific message types, forwarding of events and profiles to these brokers (rendezvous nodes)and profiles to these brokers (rendezvous nodes)


9/25


Spreading of Filter complexitySpreading of Filter complexity

– Exlusive filtering at certain brokers Exlusive filtering at certain brokers

– Distributed: broker filters so that the neighbours Distributed: broker filters so that the neighbours with matching profiles are found with matching profiles are found

1

3 4

5

62 ENSENS


P S

– Storage Strategies:Storage Strategies:• Preventive – store all available profiles, also duplicates Preventive – store all available profiles, also duplicates

and covered ones and covered ones (more memory, faster unsubscriptions)(more memory, faster unsubscriptions)

• Optimistic – minimize number of stored profiles by Optimistic – minimize number of stored profiles by discarding covered ones discarding covered ones (less memory)(less memory)


10/25


Communication with subscribersCommunication with subscribers

– Direct: only filtering broker Direct: only filtering broker and subscriber are involvedand subscriber are involved(costly estabishing communication)(costly estabishing communication)

1

3 4

5

62 ENSENS


P S

– Forwarding: only neighbor brokers and local Forwarding: only neighbor brokers and local clients (higher memory consumption)clients (higher memory consumption)

– Transparent: Transparent: • communication via broker proxies communication via broker proxies • Brokers act as subscribers to their neighborsBrokers act as subscribers to their neighbors

(limits number of subscribers to each broker) (limits number of subscribers to each broker)


11/25Classification Theoretical analysis Experiments Open Research

Classification: SchemaClassification: Schema

17 possible 17 possible variationsvariations

Existing Existing algorithmsalgorithms& systems:& systems:

Precise classification Precise classification difficult (lack of difficult (lack of detail)detail)

Often several Often several options possibleoptions possible

Not all variations Not all variations implementedimplemented

12/25

RoadmapRoadmap


Classification Classification





13/25

Theoretical Evaluation: DimensionsTheoretical Evaluation: Dimensions

– Network Traffic (worst case: blocking)Network Traffic (worst case: blocking)– Memory Usage (worst case: swapping)Memory Usage (worst case: swapping)– Efficiency / PerformanceEfficiency / Performance– ScalabilityScalability

14/25

Theoretical Evaluation: ExampleTheoretical Evaluation: Example

Event floodingEvent flooding

– Filtering close to subscribersFiltering close to subscribers– Exclusive filteringExclusive filtering– Direct communicationDirect communication

1

3 4

5

62

P1e

S

p

p3

(e)

– High traffic due to flooding with high frequencyHigh traffic due to flooding with high frequency– Low memory usage (profiles stored just once)Low memory usage (profiles stored just once)– Medium performanceMedium performance– Low scalability due to traffic increaseLow scalability due to traffic increase


15/25

RoadmapRoadmap







16/25

Practical evaluationPractical evaluation

Selection of 3 algorithmsSelection of 3 algorithms

1.Event Flooding:1.Event Flooding: Close to subscribers, exclusive Close to subscribers, exclusive filtering, direct communicationfiltering, direct communication

2.Profile flooding2.Profile flooding: Close to publishers, distributed : Close to publishers, distributed filtering, optimistic storage, transparent filtering, optimistic storage, transparent communicationcommunication

3.Rendezvous nodes3.Rendezvous nodes: Arbitrary, distributed : Arbitrary, distributed filtering, optimistic storage, transparent filtering, optimistic storage, transparent communicationcommunication


17/25

Practical evaluation: ExperimentsPractical evaluation: Experiments

ParametersParameters– Number of profiles Number of profiles – Proportion of matching eventsProportion of matching events– Proportion of matching profilesProportion of matching profiles– Number of brokersNumber of brokers– Covering between profilesCovering between profiles– Number of different event types Number of different event types – Locality bewteen events and profilesLocality bewteen events and profiles

Implementation of algorithms in system DAS Implementation of algorithms in system DAS (distributed version of A-mediAS) (distributed version of A-mediAS)

Number of profiles Number of profiles

Proportion of matching eventsProportion of matching events

Proportion of matching profilesProportion of matching profiles

Number of brokersNumber of brokers


18/25

Practical evaluation: Experiments Practical evaluation: Experiments (1)(1)

Influence of number of profiles Influence of number of profiles


PFPFEFEFRNRN

19/25


Influence of proportion of matching eventsInfluence of proportion of matching events


PFPFEFEFRNRN

20/25


Influence of number of brokersInfluence of number of brokers


PFPFEFEFRNRN

21/25

RoadmapRoadmap







22/25

Conclusion & Outlook: Selected Conclusion & Outlook: Selected resultsresults

Proposed classification schemaProposed classification schema First classification of algorithmsFirst classification of algorithms

Theoretical evaluation of 17 variationsTheoretical evaluation of 17 variations– limited information availablelimited information available– various optionsvarious options

Practical evaluation of 3 variationsPractical evaluation of 3 variations– Evaluated 7 parametersEvaluated 7 parameters– Restriction in practicality of number of brokersRestriction in practicality of number of brokers– Limitation of topology variationsLimitation of topology variations


23/25

Conclusion & Outlook: Selected Conclusion & Outlook: Selected resultsresults

Profile forwardingProfile forwarding– Mostly best efficiency and lowest network loadMostly best efficiency and lowest network load– Largest memory consumptionLargest memory consumption

Event forwardingEvent forwarding– Very high network loadVery high network load– Optimal memory usage Optimal memory usage – High proportion of matching profiles High proportion of matching profiles best best

efficiencyefficiency– Large number of profiles Large number of profiles best scalability best scalability

Rendezvous nodesRendezvous nodes– In none of the configurations better than the other In none of the configurations better than the other

algorithms (influence of topology?)algorithms (influence of topology?)Classification Theoretical analysis Experiments Open Research

24/25

Conclusion & OutlookConclusion & Outlook

Optimal algorithm depends on Optimal algorithm depends on – system load, system load, – system usage, and system usage, and – application application

--> System should support various filter algorithms --> System should support various filter algorithms and adapt dynamically to changing situationsand adapt dynamically to changing situations

Outlook:Outlook:– Integrate in adaptive system A-mediASIntegrate in adaptive system A-mediAS– Composite filtering algorithms in grids and mobile Composite filtering algorithms in grids and mobile

environmentsenvironments


Thank you for your attention!Thank you for your attention!

And now…questions.And now…questions.

Contact:Contact:

Annika HinzeAnnika Hinze

[email protected]@cs.waikato.ac.nz

www.annikahinze.dewww.annikahinze.de

Classification and Analysis of Distributed Event Filtering Algorithms Sven Bittner Dr. Annika Hinze...

Documents

Transcript of Classification and Analysis of Distributed Event Filtering Algorithms Sven Bittner Dr. Annika Hinze...