Benchmarking Event Processing Systems: Current State and Future Directions

FINCoS Framework TutorialMarcelo R.N. Mendes

1What is FINCoS?A set of tools for data generation, load submission, and performance measurement of CEP systems;Main Characteristics:Flexible;Independent of particular workloads;Neutral (not bound to any specific CEP product);Correctness check;Scalable.2CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 20122FINCoS: Architecture3CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

Scalability:It is easy to increase the load: simply add more Drivers!

Validation and Performance Measuring: Plug-and-Play Validators

Independence of CEP Products and workloads

Flexibility:Several Test Options

3FINCoS: ComponentsControllerMain Interface with the user. Allows to:Create reusable test setups;Control Drivers and Sinks (e.g., load, start, stop);Alter workload parameters on-the-fly (e.g., event submission rates, switch phases)

4CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

4FINCoS: ComponentsDriverSimulates external sources of events;Submits load to the SUT;Includes data generation mechanism;SinkReceives query results from CEP engine;Stores output events in log files or send them via network for subsequent validation;

5CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

5FINCoS: ComponentsPerformance MonitorCollect Metrics of Performance RunsTwo Modes:Online (network);Offline (log files).

6CISUC, University of Coimbra - mnunes@dei.uc.ptJanuary, 2012

6FINCoS: ComponentsJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt7

AdapterOn Input: converts from the framework neutral representation (CSV) to CEP systems native representationOn Ouput: converts output events from CEP systems native format to neutral representation (CSV)

7Lesson 1: Creating a Test SetupConfigure a DriverDecomposable WorkloadSynthetic vs. Thirdy-Party DatasetData Generation OptionsEvent Arrival ProcessConfigure a SinkJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt88January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt9

9January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt10

10January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt11

unique identifier

number of threads used for load generationsave generated data to disk?real time performance monitoring?to where Driver send the eventsmachine that hosts the Driver11January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt12

Workload can be split in independent phases 12January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt13

13January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt14

Chooses the type of workload for this phaseSets events arrival processEvent MixEvents are generated in a ordered wayWhen data generation happensTo ensure repeatability14January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt15

Configure event types15January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt16

16January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt17

17January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt18

Configure Event Type Schema18January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt19

19January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt20

20January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt21

Column Name

21January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt22

Column Type

22January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt23

Data Generation Options

23January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt24

Add Values24January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt25

25January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt26

26January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt27

27January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt28

28January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt29

29January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt30

30January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt31

31January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt32

32January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt33

33January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt34

34January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt35

35January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt36

36January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt37

37January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt38

unique identifierList of event types this Sink subscribes tofrom where Sink receive eventsmachine that hosts the SinkPort used by Sink to receive eventssave generated data to disk?real time performance monitoring?38January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt39

39January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt40

40January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt41

41January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt42

Done!42Lesson 2: Connecting with a CEP engineConfigure Connection PropertiesLoad FINCoS Adapter applicationJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt4343January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt44

List of input streamsList of output streamsNo Information Available Yet!44January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt45

Retrieve list of streams45January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt46

from the setup file created earlier46January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt47

47January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt48

Load vendor-specific connection properties 48January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt49

Engine=EsperConfiguration_file_path=C:\\FINCoS\\queries\\esper\\Q1\\Stream_Set.xmlQueries_path=C:\\FINCoS\\queries\\esper\\Q1\\Query_Set.xmlEvent_format=Map49January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt50

Connect to CEP engine50January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt51

The list of streams in the CEP engine is retrieved and matched against the configured ones in the test setup51January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt52

Loads the Adapter:Open Server Socket for Drivers (Input)Subscribes to streams at the CEP engine (Output)52January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt53

Done!53Lesson 3: Running TestsLoad Drivers and SinksStart Load SubmissionControl Load Submission

NOTE: At this point the following steps must have been completed:Creation of a test setup;Connection with a CEP engine;Initialization of the FINCoS Daemon Service at the machines where Drivers and Sinks are intended to run;January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt5454January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt55

Load components55January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt56

56January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt57

Start Load Submission57January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt58

Sent/Received counts58January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt59

59January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt60

60January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt61

61Lesson 4: Collecting Performance MetricsOffline Performance MeasurementAfter test completion, use FINCoS Performance Monitor to process log files

Online Performance MeasurementEnable realtime monitoring at Drivers and Sinks;Initialize FINCoS Performance Monitor in the Realtime Mode before test starts.

January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt6262January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt63

Offline Performance Measurement63January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt64

Add log files64January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt65

Log file created after test completion65January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt66

Configure M.I. here:Optionally, save collected metrics to a fileProcess log files66January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt67

View Results67January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt68

68January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt69

Select the stream69January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt70

Select the metric70January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt71

Select exhibition scale71January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt72

Add Metrics to the Chart72January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt73

Change Visualization Options73Lesson 4: Collecting Performance Metrics (...cont)Offline Performance MeasurementAfter test completion, use FINCoS Performance Monitor to process log files

Online Performance MeasurementEnable realtime monitoring at Drivers and Sinks;Initialize FINCoS Performance Monitor in the Realtime Mode before test starts.

January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt7474January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt75

Edit Driver Configuration75January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt76

Enable Runtime Monitoring

76January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt77

To where send eventsSend only a fraction of events

77January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt78

78January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt79

Edit Sink Configuration79January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt80

Enable Runtime Monitoring

80January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt81

81January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt82

Select a test setup file82January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt83

the one we have just modified83January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt84

Performance Monitor is now listening to connections of Drivers and Sinks84January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt85

Add counters85January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt86

86January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt87

87January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt88

Drivers and Sinks are now connected with Performance Monitor88January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt89

89January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt90

90Lesson 5: Setting Test OptionsResponse Time Measurement ModeCommunication ModeLog FlushingJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt9191January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt92

92January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt93

93January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt94

94Response Time MeasurementJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt95DRIVERCEP ENGINESINKADAPTER234sendconvertinputconvertoutputprocessinputforwardgenerate resultforwardsend195Response Time MeasurementJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt96DRIVERCEP ENGINESINKADAPTERsendconvertinputconvertoutputprocessinputforwardgenerate resultforwardsendt = t4 t1End-to-end, in milliseconds:From Driver up to Sink 234196Response Time MeasurementJanuary, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt97DRIVERCEP ENGINESINKADAPTERsendconvertinputconvertoutputprocessinputforwardgenerate resultforwardsendt = t3 t2Inside Adapter, in nanoseconds:From immediately before the input event is forwarded to CEP engine to immediately after the result is received.234197When to use End-to-End RT Measurement ModeWhen communication mode is set to DIRECT (mandatory);When it is acceptable to have a less accurate response time measurement in order to reduce the overhead in the server machine;When you want to consider communication and conversion time as part of response time;When Drivers and Sinks run in the same machine (no problems with clock synchronization).January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt9898When to use Inside Adapter RT Measurement ModeWhen accurate response time measurement is required;When you want exclude communication and conversion times from response time;When Drivers and Sinks run in different machines (requires synchronization between machines: inherently inaccurate 10ms);When the server machine can afford with the additional overhead of response time measurement.January, 2012CISUC, University of Coimbra - mnunes@dei.uc.pt9999