DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks...
Transcript of DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks...
![Page 1: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/1.jpg)
![Page 2: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/2.jpg)
FUNCTIONALDEVOPSEVENT PROCESSING IN CLOJURE
Andy Marks @andee_marks
![Page 3: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/3.jpg)
Background
![Page 4: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/4.jpg)
we all code!
![Page 5: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/5.jpg)
(prn “Hello World”)
![Page 6: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/6.jpg)
(ns hello.core)
(defn -main [] (prn “Hello World”))
![Page 7: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/7.jpg)
Jetty
(ns hello.core)
(defn -main [] (prn “Hello World”))
✓access requests ✓queues ✓sessions
![Page 8: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/8.jpg)
AWS EC2
Jetty
(ns hello.core)
(defn -main [] (prn “Hello World”))
✓disk ✓network interfaces ✓memory ✓cores/cpus
![Page 9: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/9.jpg)
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
✓records ✓replication ✓resolution
![Page 10: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/10.jpg)
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
DB
✓performance ✓indexing ✓locking ✓connections
![Page 11: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/11.jpg)
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
DB
Load Balancer
AWS EC2
Jetty
AWS EC2
Jetty
✓connections ✓DoS ✓Clients
![Page 12: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/12.jpg)
AWS EC2
Jetty
AWS EC2
Jetty
AWS EC2
Jetty
DNS (Master)
DNS (Slave)
DB
Load Balancer
AWS EC2
Jetty
AWS EC2
Jetty
Message Q
ueue
✓queues ✓timeouts
![Page 13: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/13.jpg)
that is a lot of information…
![Page 14: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/14.jpg)
from a lot of different places
![Page 15: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/15.jpg)
this is a problem
![Page 16: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/16.jpg)
this is the problem Riemann helps
solve
![Page 17: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/17.jpg)
Riemann is written in
Clojure
52 source files7256 lines
377 functions30 macros
![Page 18: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/18.jpg)
Riemann is written and configured
in Clojure
![Page 19: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/19.jpg)
And so… to Riemann
![Page 20: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/20.jpg)
kyl
e ki
ngsb
ury (
@ap
hyr)
![Page 21: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/21.jpg)
OVERVIEW
![Page 22: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/22.jpg)
Correlate events from disparate sources
What is the latency of the web servers when the DB servers are generating timeouts?
Filter events based on key criteria Ignore all network events with a normal status
Aggregate events to perform statistical analysis (e.g., percentiles) What is the average response rate for 95% of requests?
Monitor status of sources by availability of events
Which services haven’t sent any events for the last 5 minutes?
Route event streams to multiple consumers
Send all error events from DB servers to the DBA team as well as the main operational dashboards
SAMPLE RIEMANN USE CASES
![Page 23: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/23.jpg)
i = f(e)e: thundering herd of events i: information ready for downstream processing
![Page 24: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/24.jpg)
Hypothesis
Event processing is function application at large scale… using
Clojure to build/configure Riemann is a logical choice.
![Page 25: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/25.jpg)
(defrecord Event [ service state description metric tags time ttl
KEY CONCEPTS: EVENTS
“reg1.foo.bar” “reg-service http” “ok” “apache-log” 201 [“reg1” “http”] 104534892 60
riem
ann
sour
ce
![Page 26: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/26.jpg)
KEY CONCEPTS: STREAMS
(logging/init {:file "riemann.log" :console true})
(instrumentation {:interval 5}) ;; self monitor every 5 seconds (periodically-expire 1) ;; check for expired events every second
(tcp-server) (repl-server)
(streams ;; insert magic here!!! prn)
riem
ann
confi
g
![Page 27: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/27.jpg)
KEY CONCEPTS: INDEX
flow of events
index
riemann process
![Page 28: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/28.jpg)
General FP goodness
![Page 29: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/29.jpg)
HOMOICONICITY
“[It] is where a program's source code is written as a basic data structure that the
programming language knows how to access.” Wikipedia
(defn validate-config [file]
(try
(read-strings (slurp file)) (catch clojure.lang.LispReader$ReaderException e (throw (logging/nice-syntax-error e file))))) ri
eman
n so
urce
![Page 30: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/30.jpg)
HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable number of child streams and returns a function
that takes an event and responds to the event it is passed when it is invoked.”
http://riemann.io/quickstart.html
“a higher-order function… does at least one of the following:
[1] takes one or more functions as an input,
[2] outputs a function” Wikipedia
![Page 31: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/31.jpg)
HIGHER ORDER FUNCTIONS
“A stream is just a function that takes a variable number of child streams and returns a function
that takes an event and responds to the event it is passed when it is invoked.”
http://riemann.io/quickstart.html
(streams
(rate 5 prn) )
riem
ann
confi
g
![Page 32: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/32.jpg)
IMMUTABILITY
“Streams ‘change’ events by sending altered, shared-structure *copies* of events
downstream.” http://riemann.io/howto.html#split-streams
(streams
(where (host "db04") (with :service "foo" prn) (with :service "bar" prn)) )
riem
ann
confi
g
![Page 33: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/33.jpg)
Specific Clojure goodness
![Page 34: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/34.jpg)
“clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.”
“clojure's remarkably fast for what it is, sexprs make the tree structure of the streams visually apparent, it makes writing the concurrent algos much simpler, and macros allow us to express things like 'by and 'pipe in ways that would be awkward in other languages without building our own AST & interpreter or transpiler, etc.”
@APHYR SAID…
![Page 35: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/35.jpg)
S-EXPRESSIONS
“a notation for nested list (tree-structured) data… popularised by the programming
language Lisp” Wikipedia
![Page 36: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/36.jpg)
S-EXPRESSIONS
(streams
(where (and (service #"^riak") (state “critical”)) (email “[email protected]")) )
1 Filter certain events…
2 Let people know…
1
2
riem
ann
confi
g
“make the tree structure of the streams visually apparent”
![Page 37: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/37.jpg)
3
1
2
S-EXPRESSIONS
1 Split on event fields…
2 Detect state changes…
3 Collate and email(streams
(by [:host :service] (changed :state (rollup 5 3600 (email “[email protected]")))) ) ri
eman
n co
nfig
“make the tree structure of the streams visually apparent”
![Page 38: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/38.jpg)
MACROS
(defmacro pipe [marker & stages] `(let [~@(->> stages reverse (interpose marker) (cons marker))] ~marker))
(streams
(pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5
(graphite {:host “127.0.0.1” :port 2003}))) )
riem
ann
confi
g
riem
ann
sour
ce
![Page 39: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/39.jpg)
MACROS: PIPE
(streams
(let [aggregate (rate 5 (graphite {:host “127.0.0.1” :port 2003}))] (splitp = service "http req rate" aggregate "0mq req rate" (scale 2 aggregate))) )
or
riem
ann
confi
gri
eman
n co
nfig
(streams
(pipe - (splitp = service "http req rate" - "0mq req rate" (scale 2 -)) (rate 5
(graphite {:host “127.0.0.1” :port 2003}))) )
![Page 40: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/40.jpg)
Conclusion
![Page 41: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/41.jpg)
Hypothesis
Event processing is function application at large scale: using Clojure to build/configure Riemann is a logical
choice.
homoiconicityhigher order functions
s-expressionsmacros
immutability
![Page 42: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/42.jpg)
CLOJURE AD ABSURDUM
(defn do-you-trust-me? [event] (prn (load-string (:description event))))
(streams
(where (service "event-gen") do-you-trust-me?) )
(ns event-gen.core (:require [riemann.client :refer :all]) (:gen-class :main true))
(defn -main [& args] (let [c (tcp-client {:host "127.0.0.1"})]
(send-event c {:service "event-gen" :description "(+ 1 1)"}) (close-client c)))
riem
ann
clie
ntri
eman
n co
nfig
![Page 43: DEVOPS - YOW! Conference...EVENT PROCESSING IN CLOJURE Andy Marks @andee_marks amarks@thoughtworks.com. Background. we all code! ... Event processing is function application at large](https://reader033.fdocuments.us/reader033/viewer/2022060900/609de17dfc54501aba477189/html5/thumbnails/43.jpg)
CLOJURE AD ABSURDUM
(defrecord Event [service description ])
“event-gen” “(+ 1 1)”
Riemann (Clojure)
riemann.config (Clojure)
Client (Clojure)
Clojure
=> “2”