EVENT DRIVEN CUBE MONITORING.
description
Transcript of EVENT DRIVEN CUBE MONITORING.
![Page 1: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/1.jpg)
EVENT DRIVEN CUBE MONITORING.David Prime & David ElliottSQLBits 6
![Page 2: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/2.jpg)
WHO WE ARE.
Who we are– David Prime – Betfair Research– David Elliott – Information Management & Analytics Architect
Betfair– Launched June 2000 around an exchange betting platform– You can bet that an outcome will happen (back) or that it won't happen (lay). – You can choose the odds at which you want to play. – You can bet whilst the game is in play. – You can play on a range of products and games other than sports wagering
What this means in terms of data– Bets: >5 million bets daily– Latency: 99.9% bets processed <1 sec– More trades than all of the European stock exchanges combined
![Page 3: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/3.jpg)
OUR OBJECTIVES.
Background– Early look at StreamInsight in Deep-dive– Architectural direction: EDSOA– Real Time requirements: Anti-Fraud, Legislation, Exposure Monitoring– BI / OI– Analytics API / Continuous ETL
Cube Monitoring– A good use-case and an opportunity to assess using SI with the rest of the BI stack– Provide light-weight aggregated usage information for the business
Real-Time– Alerts: name and shame greedy users, discover broken code– Aggregate session data– Using time windows to run complex monitoring scenarios
![Page 4: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/4.jpg)
COMPLEX EVENT PROCESSING.
Betfair is awash with events
Your online business probably is too
![Page 5: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/5.jpg)
COMPLEX EVENT PROCESSING.
![Page 6: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/6.jpg)
STREAMINSIGHT.
What is StreamInsight?
NewIntegrated
FastImproving
![Page 7: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/7.jpg)
OVERVIEW.
DB
Input adaptor
Outputadaptors
Alerting
Cubes
Trace events
CEP Server
Trace & Real-Time ETL
StreamInsight
![Page 8: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/8.jpg)
WHAT DO WE WANT TO CAPTURE?
![Page 9: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/9.jpg)
TRACE EVENTS.
ERROR
QUERY SUBCUBE
QUERY BEGIN
QUERY END
EXISTING SESSION
SESSION INTIALIZE
AUDITLOGIN
AUDIT LOGOUT
![Page 10: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/10.jpg)
TRACING ANALYSIS SERVICES.
String connString = "Provider=MSOLAP;Data Source=bigbox;Initial Catalog=AdventureWorks Sample;Integrated Security=SSPI;";
// Create AS server objectserver = new Microsoft.AnalysisServices.Server();
// Connect serverserver.Connect(connString);
Trace trace = server.Traces.Add();
TraceEvent sessionInit = trace.Events.Add(TraceEventClass.SessionInitialize);sessionInit.Columns.Add(TraceColumn.TextData);sessionInit.Columns.Add(TraceColumn.ConnectionID);sessionInit.Columns.Add(TraceColumn.NTDomainName);sessionInit.Columns.Add(TraceColumn.NTUserName);sessionInit.Columns.Add(TraceColumn.ApplicationName);sessionInit.Columns.Add(TraceColumn.StartTime);sessionInit.Columns.Add(TraceColumn.CurrentTime);sessionInit.Columns.Add(TraceColumn.DatabaseName);
etc...
C#
![Page 11: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/11.jpg)
TRACING ANALYSIS SERVICES.
trace.Update();
//engage the traces TraceEventHandler onTraceEvent = new TraceEventHandler(OnTraceEvent); TraceStoppedEventHandler onTraceStopped = new TraceStoppedEventHandler(OnTraceStopped);
trace.OnEvent += new TraceEventHandler(OnTraceEvent); trace.Stopped += new TraceStoppedEventHandler(OnTraceStopped); trace.Start();
C#
![Page 12: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/12.jpg)
TRACING ANALYSIS SERVICES.
private void OnTraceEvent(object sender, TraceEventArgs e) {
siAdapter.PutEvent(e); //send the event out to streaminsight dbwriter.putEvent(e); //the dbwriter constructs a load of inserts based on the shape of the event //and dumps to our DB for cube-ness
switch (e.EventClass.ToString()) { case "SessionInitialize": break;
case "ExistingSession": break;
case "QueryEnd": break;
case "QuerySubcube": decodeQuery(e, querySubCubeID); break;
.
.
.
C#
![Page 13: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/13.jpg)
NOW WHERE?
![Page 14: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/14.jpg)
//filters out the events we want CepStream<QuerySumm> querySumm = from e in producer.AlterEventDuration(e => TimeSpan.FromMinutes(1)) where e.eventClass == "QueryEnd" select new QuerySumm { userName = e.userName, allTime = e.duration, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime }; //detects slow queries so we can go and moan at the user CepStream<SlowAlert> slowProducer = from e in querySumm where e.allTime.Milliseconds > 1000 select new SlowAlert { userName = e.userName, allTime = e.allTime, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime };
LINQ
STREAMINSIGHT.
![Page 15: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/15.jpg)
LINQ
STREAMINSIGHT. //filters out the events we want CepStream<QuerySumm> querySumm = from e in producer.AlterEventDuration(e => TimeSpan.FromMinutes(1)) where e.eventClass == "QueryEnd" select new QuerySumm { userName = e.userName, allTime = e.duration, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime }; //detects slow queries so we can go and moan at the user CepStream<SlowAlert> slowProducer = from e in querySumm where e.allTime.Milliseconds > 1000 select new SlowAlert { userName = e.userName, allTime = e.allTime, cpuTime = e.cpuTime, startTime = e.startTime, endTime = e.endTime };
![Page 16: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/16.jpg)
AND THEN?
Output adaptors are nice
• Nagios• Splunk• Homebrew
• MORE CUBES :)
![Page 17: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/17.jpg)
OUTPUT CUBE.
How do we do this?– FACTS– Dimensions
![Page 18: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/18.jpg)
DEMO.
![Page 19: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/19.jpg)
NEXT STEPS: SECURITY.
Security Monitoring / Auditing– Alerting on suspicious querying activity / disallowed querying
Alerting Reporting Analysis
– Provide an audit trail of querying on sensitive attributes– Regulatory Reporting– Dynamic Security
![Page 20: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/20.jpg)
NEXT STEPS: PERFORMANCE.
Performance Recommendations– Provide data to enable assessment of ‘hot’ areas within the cubes
Alerting Reporting Analysis
– Feed into third party monitoring tools– Identify heavy users– Identify poorly performing queries for tuning– Automatic aggregation generation
![Page 21: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/21.jpg)
EXTENDING THE FRAMEWORK.
![Page 22: EVENT DRIVEN CUBE MONITORING.](https://reader033.fdocuments.us/reader033/viewer/2022051317/568163c3550346895dd4eb39/html5/thumbnails/22.jpg)
QUESTIONS.