Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Agenda*
Intro*
Overview*of*architecture*
JMeter*
OEP*
ELK*
Demo*
Q&A*
1"
2"
3"
4"
5"
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 2*
6"
7"
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Intro*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 3*
Jon"Hall*Oracle*ConsulNng*
liSlegrid*developer*and*maintainer*
Ivan"Cikic*Oracle*ConsulNng*
*
*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
• Load*test*tool*–*JMeter*
• Queues*–*AcNveMQ*
• Event*processing*–*OEP*
• Caching*–*Coherence,*of*course*:U)*
• Data/log*analysis*–*ELK*(ElasNcSearch,*LogStash*and*Kibana) * **
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 4*
MiniUme*architecture*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
MiniUme*architecture*(conNnued)*
! JMeter*–*reads*source*data*from*a*CSV*and*publishes*it*to*a*topic*
! AcNveMQ*–*provides*the*demo*messaging*infrastructure*
! OEP*–*subscribes*to*the*topic,*parses*the*payload*and*performs*event*
processing*
! Coherence*–*acts*as*inUmemory*storage/data*grid*(could*be*queried)*
! ELK*–*visualisaNon*of*requests*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 5*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
JMeter*U*concepts/terminology*
! Test*plan*–*your*test*creaNon*area*
! Workbench*–*temporary*'paste'*area,*also*used*to*configure*recording*
proxy*
! Sampler*(request*type):*Web*U*HTTP(S),*SOAP,*FTP,*JDBC,*JMS,*TCP,*others*
! Logic*controllers*(if,*loop*etc.)*
! Config*Element*(e.g.*HTTP*Request*Details)**
! Timers*
! Pre/pro*processing*
! AsserNons*(response*checks)*
! Listener*(reporNng,*summaries,*graphs,*write*to*file*etc.)*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 6*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
JMeter*U*example*
! Building*an*example:*
• Reading*source*data*from*CSV*
• JMS*publish*and*subscribe*
• Looking*at*output*
*
! Toggle*is*your*friend*:U)*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 7*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
JMeter*–*avoiding*hardUcoding*
! User*defined*variables:*
• Name:*numberOfMessages*
• Value:*${__P(numberOfMessages,*20)}*
! Loop*count:*${numberOfMessages}*
! jmeter*UJnumberOfMessages=25*
! DescripNon*field*as*temporary*paste*area*
*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 8*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
JMeter*summary*
! Distributed*mode*
! Lots*of*other*features*not*covered*here*
! BeanShell*is*useful*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 9*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Oracle*Event*Processing*
• LightUweight*Java*ApplicaNon*Server*(embeddable)*
• Based*on*three*simple*concepts:*
• Event*Adapters*–*Inbound*and*outbound*external*connecNons*• Event*Channels*–*To*connect*things*together*• Event*Processors*–*To*process*informaNon*in*realUNme*
• These*components*connect*to*form*an*Event*Processing*Network*(EPN)*
• The*heart*of*processing*is*ConNnuous*Query*Language*(CQL)*• Good*integraNon*with*Oracle*Coherence**
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 10*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Event*Processing*Network*(EPN)*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 11*
Inbound Adapter Listens to Car
Insurance Quote Requests
Inbound Channel Connects the request to the Processor and
Calculator
Quote Calculator Simulates a premium being calculated for successful requests
Quote Processor Creates a moving average of calculated premiums in
the past hour
Outbound Adapter Sends the Response
to downstream systems
Fraud Processor Looks for patterns in the
inbound stream for suspicious behaviour and creates
messages when conditions are matched
Outbound Adapter Sends the response back into the risk calculation
engine and updates a real-time dashboard
Data
Data
Data
Data
Data
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Event*Adapters*
• Adapters*manage*data*entering*and*leaving*the*EPN*
• A*number*of*different*inbound*/*outbound*adapters*is*provided*OOTB:**
• JMS*
• REST**• EDN*• CSV*• HTTP*PublishUSubscribe*• High*Availability*Adapters*• Write*your*own*Adapter*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 12*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
• A*channel*represents*the*logical*conduit*through*which*events*flow*between*other*types*of*components*
• Channels*provide*buffering,*queuing*and*concurrency*• Event*parNNoning**• Channel*selector*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 13*
Event*Channels*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Event*Processors*
• Processes*incoming*events*from*various*input*channels*and*other*data*
sources*
• Processors*use*Oracle*ConNnuous*Query*Language*(CQL)*to*write*the*business*logic*in*the*form*of*conNnuous*queries*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 14*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ConNnuous*Query*Language*(CQL)*
• CQL*supports:*• Filtering,*AggregaNon,*ProjecNon*• Time*and*Count*based*windows**
• Slides*• Joining*streams***
• PaSern*matching *(with*MATCH_RECOGNIZE)*
• Top*/*BoSom*N*
• Up*/*Down*Trend*• FluctuaNon*• Eliminate*/*Detect*Duplicates*
• Detect*Missing*Event*
• W*/*Inverse*W**
*Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 15*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ConNnuous*Query*Language*(CQL)*
• CQL*supports*joining*with*external*sources*• Coherence*cache*• Database*table**
• Extendable*via*data*cartridges**• JDBC*• Hadoop*• Oracle*NoSQL*• Java**
*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 16*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Oracle*Coherence*and*OEP*
• Access*to*nonUstreaming*data,*event*enrichment*for*example*
• Cache*as*event*sink*
*
• *Cache*as*event*source*(pull*or*push*model)*
• Perform*data*grid*operaNons*within*Event*Beans*!
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 17*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Performance*Tuning*and*Scalability*
• Channel*threading*and*buffering*• Pass*Through*(maxUthreads=0,*maxUsize=0).*Event*ordering*preserved.**
• Synchronous*Handoff*(maxUthreads>0,*maxUsize=0)*
• Concurrent*Queue*(maxUthreads>0,*maxUsize>0)*
• Event*ParNNoning*Channel*• Batching*channel*• Parallel*CQL*execuNon**
• ORDERED,*UNORDERED,*PARTITION_ORDERED*
• Parallel*execuNon*of*adapters*and*event*beans**(work*manager*threads)*
*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 18*
Scale"Up"
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|* Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 19*
Performance*Tuning*and*Scalability*
• Scalability*with*NoNficaNon*Groups*
• ParNNon*incoming*JMS*
stream*to*mulNple*servers*
• Clustering*supported*with**Oracle*Coherence*
* <cluster> <server-name>oep-server-1</server-name> <enabled>coherence</enabled> <groups>LondonGroup_1</groups> </cluster> <cluster> <server-name>oep-server-2</server-name> <enabled>coherence</enabled> <groups>LondonGroup_2</groups> </cluster>
***
Scale"Out"
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Types*of*High*Availability*(in*context*of*CEP*systems)**
• CEP*systems*are*characterised*by*very*dynamic,*constantly*changing*data*
• CEP*systems*are*oven*highly*stateful*
• Typical*soluNons*to*statefulness*problem:*
• AcNve/acNve*–*replicate*the*behaviour*of*the*system*
• AcNve/passive*–*replicate*the*state*of*the*system*
• Upstream*backup*–*saving*the*stream*of*events*that*produced*the*state*so*that*it*can*be*rebuilt**
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 20*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Oracle*Event*Processing*HA*
• Relies*on*Oracle*Coherence**• Supports*acNve/acNve*architecture*
• Primary*instance*responsible*
for*sending*events*
• Secondary*instances*buffer**output*events*
• High*Availability*Adapters**acNng*as*proxies*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 21*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
High*Availability*Adapters*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 22*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
High*Availability*Quality*of*Service*
Quality"of"Service" Missed"Events"
Duplicate"Events"
Performance"Overhead"
Simple*Failover* Yes*(many)* Yes*(few)* Negligible*
Simple*Failover*with*Buffering* Yes*(less)* Yes*(many)* Low*
LightUWeight*Queue*Trimming* No* Yes*(few)* Medium*
Precise*Recovery*with*JMS* No* No* High*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 23*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
LightUWeight*Queue*Trimming*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 24*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
High*Availability*Design*PaSerns*
• Select*the*minimum*high*availability*your*applicaNon*can*tolerate*
• Limit*the*applicaNon*state*
• Ensure*applicaNons*are*idempotent*
• Make*events*universally*idenNfiable**
• Understand*the*importance*of*event*ordering*(queue*trimming)**
• Prefer*determinisNc*behaviour*
• Prefer*monotonic*event*idenNfiers*
• Plan*for*server*recovery*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 25*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ELK*
! ElasNcSearch*–*stores*data*inUmemory*for*querying*
! Logstash*–*agent,*captures*and*uploads*data*to*ElasNcSearch*
! Kibana*–*frontend*GUI*to*ElasNcSearch*
! ELK?**Or*should*it*be*LEK?***
• But*that*doesn't*sound*as*cool*:Up*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 26*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Logstash*
! Tool*for*receiving,*processing*and*outpuyng*logs.*
! WriSen*in*JRuby.*
! Inputs*
� Over*40*types,*e.g.*file,*stdin,*JMX,*Log4J,*etc.*
! Filters*
� Over*50*types,*e.g.*Grok,*etc.*
! Outputs*
� Over*55*types,*e.g.*ElasNcSearch,*file,*stdout,*etc.*
*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 27*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Logstash*configuraNon*
! ConfiguraNon*string*
• logstash*Ue*"input*{*stdin{}*}*output*{*stdout{}*}"*
! ConfiguraNon*file:*logstash*Uf*simple.conf*
input&{&stdin{}&}&&&filter&{&grok&{&match&=>&[&"message",&"%{NUMBER:quote:float}"&]&}&}&&
&output&{&&
&&& &stdout&{&codec&=>&json&}&&
&&& &elasticsearch&{&cluster&=>&testcluster&&index&=>"es&&codedec&=>&json}&&
&}&
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 28*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Logstash*configuraNon*
input&{&
jmx&{&
&&&&path&=>&"myjmx"&
&&&&polling_frequency&=>&30&
&&&&type&=>&"jmx"&
&&&&nb_thread&=>&4&
&&}&
}&
output&{&
&&stdout&{}&
&&elasticsearch{&
&&&&&cluster&=>&"testcluster"&
&&}&
}&
&
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 29*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ElasNcSearch**overview*
! Built*on*Lucene*(high*performance*text*search*engine).*
! Stores*data*in*JSON*
! REST*API*for*puyng,*querying*and*deleNng*data*
! Scale*out*with*HA*
! Document*oriented,*schemaUfree*
*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 30*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ElasNcSearch*terminology*
! Index*is*like*a*database,*examples:*
• quotes*
• logstashU2015.03.13*
• LogstashU2015.03.17*
! You*can*have*as*many*as*you*want*
! Can*be*created*dynamically*or*upfront*
! Type*is*a*bit*like*a*table*
• Such*as*'logs'*
• hSp://localhost:9200/quotes/_all/_mapping?preSy=true*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 31*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ElasNcSearch*types*{&"quotes"&:&{&
&&&&"mappings"&:&{&
&&&&&&"logs"&:&{&
&&&&&&&&"properties"&:&{&
&&&&&&&&&&"@timestamp"&:&{&"type"&:&"date",&"format"&:&"dateOptionalTime"&&},&
&&&&&&&&&&"@version"&:&{&"type"&:&"string"&},&
&&&&&&&&&&"host"&:&{&"type"&:&"string"&},&
&&&&&&&&&&"message"&:&{&"type"&:&"string"&},&
&&&&&&&&&&"quote"&:&{&"type"&:&"double"&},&
&&&&&&&&&&"tags"&:&{&"type"&:&"string"&}&
&&&&&&&&}&
}}}}&
&
&
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 32*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
ElasNcSearch*documents*
! Document*is*like*a*row*in*a*table*
{"_index":&"quotes",&
&&&&"_type":&"logs",&
&&&&"_id":&"ASWcW8RoShaOjWHddcc8DQ",&
&&&&"_version":&1,&&"_score":&1,&"_source":&{&
&&&&&&&&"message":&"123",&
&&&&&&&&"@version":&"1",&
&&&&&&&&"@timestamp":&"2015[03[18T18:29:48.405Z",&
&&&&&&&&"host":&"lenovo2",&
!!!!!!!!"quote":!123!
}}&
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 33*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Kibana*
! BrowserUbased*GUI*for*use*with*ElasNcSearch*
! Used*for*analysis*of*data*(Discover)*
! Used*for*visualizaNon*of*data*(Visualize)*
• Buckets*
! Building*cool*dashboards*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 34*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
Kibana*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 35*
Copyright*©*2014,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|*
QuesNons?*
Thank*you!*
Oracle*ConfidenNal*–*Internal/Restricted/Highly*Restricted* 36*
Top Related