Meetup ElasticSearch : « Booster votre Magento avec Elasticsearch »
Productionalizing Elasticsearch at Scale - DeveloperMarch · 2018-07-27 · like Elasticsearch,...
Transcript of Productionalizing Elasticsearch at Scale - DeveloperMarch · 2018-07-27 · like Elasticsearch,...
About me
Myself Jeeva, enthusiastic engineer, working as technical lead in Paypal.
I have been a backbone in designing and implementing a highly scalable system that powers the elastic search engine to process billions of records every day.
Having intense hands on experience on multiple cutting-edge technologies like Elasticsearch, Kafka, Akka, Scala and Reactive programming, I provide excellent expertise in building resilient distributed systems.
Contact me at subramaniajeeva [gmail, twitter, fb, linkedin]
Agenda
• Elasticsearch - How it works
• Scaling Elasticsearch in production
• Resilient by design
• Do’s and Don’ts
• Monitoring and Alerting
• Summary
Elasticsearch – Data Organization
Shard 0
Shard 1 Replica
Shard 1
Shard 0 Replica
Node 2 Node 1
cluster Index
Disk xx GB Disk xx GB
Shard 0 Primary
Elasticsearch – Routing
Shard 1 Replica
Shard 1 Primary
Shard 0 Replica
Node 2 Node 1
Indexing request Document A
Index document A to shard1
Replicate document A
{"text": "brown fox jumped into the well"}
Dissecting Indexing
The <em>Quick brown</em> fox jumped over the lazy dog
The Quick brown fox jumped over the lazy dog
The two <em>lazy</em> dogs were slower than the less lazy <em>dog</em>, Rover
The two lazy dogs were slower than the less lazy dog, Rover
Source Text
html_strip filter
Standard tokenizer
Stop words filter
Lower case filter
The Quick jumped
over
fox brown
the lazy dog
the quick jumped
over
fox brown
the lazy dog
than
slower were two The lazy dogs
dog the less lazy Rover
than
slower were two the lazy dogs
dog the less lazy rover
quick jumped
over
fox brown
lazy dog
slower two lazy dogs
dog less lazy rover
Term Document
quick 1
brown 1
fox 1
jumped 1
over 1
lazy 1, 2
dog 1, 2
two 2
dogs 2
slower 2
less 2
rover 2
Document 1 Document 2 Inverted Index
Dissecting Search
The lazy Fox
The lazy Fox html_strip
filter
Standard tokenizer
Stop words filter
Lower case filter
Fox lazy The
Term Document
quick 1
brown 1
fox 1
jumped 1
over 1
lazy 1, 2
dog 1, 2
two 2
dogs 2
slower 2
less 2
rover 2
Search for “The lazy Fox”
Inverted Index
fox lazy the
fox lazy
lazy
fox
matches document 1 and 2
matches document 1
Back-pressure
Kafka to ES stream
Kafka Source Transformer Bulkizer ES client Sink
pull pull pull pull
Back-pressure
Kafka to ES stream
Kafka Source Transformer Bulkizer ES client Sink
push push push push
pull pull pull
References
•
•
•
•
•