POP Publisher Optimal Placement

Publisher Relocation Algorithms for Minimizing Delivery Delay and Message LoadAlex Cheung and Hans-Arno JacobsenAugust, 14th 2009

MIDDLEWARE SYSTEMSRESEARCH GROUP

AgendaProblem statement and goalRelated approachesHow GRAPE and POP works?Experiment resultsConclusions and Future Work

First I will describe the problem that were solving,Go through related approaches.2ProblemPublishers can join anywhere in the networkClosest brokerImpact:High delivery delayHigh system utilizationMatchingBandwidthSubscription Storage

PSSPure forwardersIn many pub/sub systems, publishers. Whatever closest means: network wise, geographic wise, etc.3GoalAdaptively move publisher to the area of highest-rated subscribers or highest number of publication deliveriesKey properties of solution:DynamicTransparentScalableRobust

SSPOur goal is to dynamically compute the best location for the publisher and transparently move the publisher to the desired location. The solution should complement existing pub/sub system, including being able to scale with the number of brokers and clients, and robust by not introducing any single points of failure.4Existing ApproachesFilter-based Pub/Sub:R.Baldoni et al. Efficient publish/subscribe through a self-organizing broker overlay and its application to SIENA. The Computer Journal, 2007.Migliavacca et al. Adapting Publish-Subscribe Routing to Traffic Demands. DEBS 2007.

Multicast-based Pub/Sub:Such as Riabovs subscription clustering algorithms (ICDCS02 and 03), SUB-2-SUB (one subscription per peer), TERA (topic-based)Assign similar subscriptions to one or more cluster of serversOne-time-match at the dispatcherSuitable for static workloadsMay get false-positive publication deliveryArchitecture is fundamentally different than filter-based approaches

There is a different type of pub/sub architecture that solves our problem. This architecture is referred to as multicast-based. 5TerminologyB1B2B3B4B5PReference brokerupstreamdownstreamPublication flowGRAPE - IntroGreedy Relocation Algorithm for Publishers of EventsGoal: Move publishers to area with highest-rated subscribers or highest publication deliveries based on GRAPEs configuration.

GRAPEs ConfigurationThe configuration tells GRAPE what aspect of system performance to improve:Prioritize on minimizing average end-to-end delivery delay or total system message rate (a.k.a. system load) Weight on prioritization falls on a scale between 0% (weakest) and 100% (full).Example: Prioritize on minimizing load at 100% (load100)Just to recap, the goal of POPIn order to achieve this, POP utilizes a 3 phase algorithmChallenges. Computational: dont want super elaborate probabilistic scheme for tagging8Minimize Delivery Delay or Load?SSSSSS[class,=,`STOCK], [symbol,=,`GOOG], [volume,>,1000000]P[class,`STOCK], [symbol,`GOOG], [volume,9900000][class,=,`STOCK], [symbol,=,`GOOG], [volume,>,0]4 msg/s1 msg/s100% Load 0%0% Delay 100%9GRAPEs 3 PhasesOperation of GRAPE is divided into 3 phases:Phase 1: Discover location of publication deliveries by tracing live publication messages in trace sessionsRetrieve trace and broker performance informationPhase 2: In a centralized manner, pinpoint the broker that minimizes the average delivery delay or system loadPhase 3: Migrate the publisher to the broker decided in phase 2Transparently with minimal routing table update and message overhead

Just to recap, the goal of POPIn order to achieve this, POP utilizes a 3 phase algorithmChallenges. Computational: dont want super elaborate probabilistic scheme for tagging10Phase 1 Logging Publication HistoryEach broker records, per publisher, the publications delivered to local subscribersGthreshold publications are traced per trace sessionEach trace session is identified by the message ID of first traced publication message of that session

B34-M213B34-M215B34-M216B34-M217B34-M220B34-M222B34-M225B34-M226Publications received from start of trace session B34-M212 B34-M21201011111110000000Trace session IDStart of bit vectorGRAPEs data structure representing local delivery pattern. Requires each publication to store the trace session IDPhase 1 Trace Data and Broker Performance RetrievalB1B5B7B6B8PSSS1x9x5xS1xReplyB8ReplyB7ReplyB8, B7, B6ReplyB8, B7, B6, B5 at the end of a trace sessionPhase 1 Contents of Trace Information Broker IDNeighbor ID(s)Bit vector (for estimating total system message rate)Total number of local deliveries (for estimating end-to-end delivery delay)Input queuing delayAverage matching delayOutput queuing delays to neighbor(s) and binding(s)

GRAPE adds 1 reply message per trace session.Phase 2 Broker SelectionEstimate the average end-to-end delivery delayLocal delivery counts, and queuing and matching delaysPublisher ping times to the downstream brokersEstimate the total broker message rateBit vectorsPhase 2 Estimating Average End-to-End Delivery DelayB1B8B6B7PSSS95S21Input Q:Matching:Output Q (RMI):Output Q (B5):

Input Q:Matching:Output Q (RMI):Output Q (B5):Output Q (B7):Output Q (B8):Input Q:Matching:Output Q (RMI):Output Q (B6):Input Q:Matching:Output Q (RMI):Output Q (B6):30 ms20 ms100 ms50ms20 ms5 ms45 ms25 ms40 ms35 ms30 ms10 ms70 ms30 ms35 ms15 ms75 ms35 msSubscriber at B1:10+(30+20+100) 1 = 160 msSubscribers at B2:10+[(30+20+50)+(20+5+45)] 2 = 350 ms Subscribers at B7:10+ [(30+20+50)+(20+5+40)+(30+10+70)] 9 = 2,485 msSubscribers at B8:10+[(30+20+50)+(20+5+35)+(35+15+75)] 5 = 1,435 msAverage end-to-end delivery delay:(150+340+2475+1425) 17 = 268 ms10 msPing time:Phase 2 Estimating Total Broker Message RateB1B8B6B7PSSS95S21 10000 00001 01111 00100Bit vector capturing publication deliveries to local subscribers 11111 01111 01111 00100Message rate through a broker is calculated by using the OR-bit operator to aggregate the bit vectors of all downstream brokersPhase 2 Minimizing Delivery Delay with Weight P%Get ping times from publisherCalculate the average delivery delay if the publisher is positioned at any of the downstream brokersNormalize, sort, and drop candidates with average delivery delays greater than 1-P (0 P 1).Calculate the total broker message rate if the publisher is positioned at any of the remaining candidate brokersSelect the candidate that yields the lowest total system message rate.

Phase 3 Publisher Migration ProtocolRequirements:Transparent to the end-user publisherMinimize network and computational overheadNo additional storage overhead

Phase 3 - ExampleB1B3B2B5B4B7B6B8SSSSSS2x4x3x1x9x5xPS1x(1) Update last hop of P to B6-xUpdate last hop of P to B6Remove all S with B6 as last hopUpdate last hop of P to B6Remove all S with B5 as last hopForward (all) matching S to B5How to tell when all subs are processed by B6 before P can publish again?DONE19POP - IntroPublisher Optimistic PlacementGoal: Move publishers to the area with highest publication delivery or concentration of matching subscribers

POPs Methodology Overview3 phase algorithm:Phase 1: Discover location of publication deliveries by probabilistically tracing live publication messagesOngoing, efficiently with minimal network, computational, and storage overheadPhase 2: In a decentralized fashion, pinpoint the broker closest to the set of matching subscribers using trace data from phase 1Phase 3: Migrate the publisher to the broker decided in phase 2Same as GRAPEs Phase 3

Just to recap, the goal of POPIn order to achieve this, POP utilizes a 3 phase algorithmChallenges. Computational: dont want super elaborate probabilistic scheme for tagging21Phase 1 Aggregated RepliesB43B615B1B3B2B5B4B7B6B8PSSSSSS2x4x3x1x9x5xS1xB1B2B4B516315B32B3B224B89B75B6B7B8159Publisher Profile TableMultiple publication traces are aggregated by :Si = Snew + (1 - ) Si-1Reply9Reply5Reply15Reply15In terms of message overhead, POP introduces 1 reply message per traced publication 9x means 9 matching subscribers are represented by the S symbol.

Any questions?22Phase 2 Decentralized Broker Selection AlgorithmPhase 2 starts when Pthreshold publications are tracedGoal: Pinpoint the broker that is closest to highest concentration of matching subscribersUsing trace information from only a subset of brokersThe Next Best Broker condition:The next best neighboring broker is the one whose number of downstream subscribers is greater than the sum of all other neighbors' downstream subscribers plus the local broker's subscribers.

Trace information stored in Publisher Profile Cache23Phase 2 ExampleB43B615B1B3B2B5B4B7B6B8SSSSSS2x4x3x1x9x5xPS1xB1B2B4B516315B32B3B224B89B75B6B7B8159AdvId: PDestId: nullBroker List:B1, B5, B610B6Notice there are a number of brokers whose PPCs were not used, which means POP does not need detailed global knowledge of the entire system. Just the aggregated result.24Experiment SetupExperiments on both PlanetLab and a cluster testbed

PlanetLab: 63 brokers 1 broker per box 20 publishers with publication rate of 10 40 msg/min 80 subscribers per publisher 1600 subscribers in total Pthreshold of 50 Gthreshold of 50Cluster testbed: 127 brokers Up to 7 brokers per box 30 publishers with publication rate of 30 300 msg/min 200 subscribers per publisher 6000 subscribers in total Pthreshold of 100 Gthreshold of 100

Average Input Utilization Ratio VS Subscriber Distribution Graph4/25/200926

Average Delivery Delay VS Subscriber Distribution Graph4/25/200927Results SummaryUnder random workloadNo significant performance differences between POP and GRAPEPrioritization metric and weight has almost no impact on GRAPEs performanceIncreasing the number of publication samples on POP Increases the response timeIncreases the amount of message overheadIncreases the average broker message rateGRAPE reduces the input util ratio by up to 68%, average message rate by 84%, average delivery delay by 68%, and message overhead relative to POP by 91%.

Conclusions and Future WorkPOP and GRAPE moves publishers to highest-rated or highest number of matching subscribers to:Reduce load in the system, and/orScalabilityReduce average delivery delay on publication messagesPerformance

Subscriber relocation algorithm that works in concert with GRAPE

Questions and Notes