(R)Evolution - FOSDEM...
Transcript of (R)Evolution - FOSDEM...
![Page 1: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/1.jpg)
(R)EvolutionPhilipp Krenn@xeraa
![Page 2: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/2.jpg)
Revolution
![Page 4: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/4.jpg)
![Page 5: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/5.jpg)
https://www.reddit.com/r/SQL/comments/7i828i/ms_sql_query_is_too_complex/
![Page 6: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/6.jpg)
Who uses Elasticsearch?
![Page 7: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/7.jpg)
Infrastructure | Developer Advocate
![Page 8: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/8.jpg)
http://thedudeabides.com/articles/the_birth_of_compass
![Page 9: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/9.jpg)
![Page 10: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/10.jpg)
$ curl http://localhost:9200
{
"name": "elasticsearch1",
"cluster_name": "docker-cluster",
"cluster_uuid": "sW2BcTZRRCSVVH2sDsWF8A",
"version": {
"number": "5.6.7",
"build_hash": "4669214",
"build_date": "2018-01-25T21:14:50.776Z",
"build_snapshot": false,
"lucene_version": "6.6.1"
},
"tagline": "You Know, for Search"
}
![Page 11: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/11.jpg)
$ curl http://localhost:9200
{
"name": "elasticsearch1",
"cluster_name": "docker-cluster",
"cluster_uuid": "sW2BcTZRRCSVVH2sDsWF8A",
"version": {
"number": "5.6.7",
"build_hash": "4669214",
"build_date": "2018-01-25T21:14:50.776Z",
"build_snapshot": false,
"lucene_version": "6.6.1"
},
"tagline": "You Know, for Search"
}
![Page 12: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/12.jpg)
![Page 13: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/13.jpg)
![Page 14: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/14.jpg)
![Page 15: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/15.jpg)
![Page 16: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/16.jpg)
![Page 17: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/17.jpg)
![Page 18: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/18.jpg)
![Page 19: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/19.jpg)
![Page 20: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/20.jpg)
Evolution
![Page 21: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/21.jpg)
TermsCluster, Node, Index, Shard, Document
![Page 22: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/22.jpg)
![Page 23: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/23.jpg)
Strictness*
5.0
* Demo
![Page 24: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/24.jpg)
![Page 25: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/25.jpg)
Parameters & configs
![Page 26: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/26.jpg)
Bootstrap checks
![Page 27: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/27.jpg)
Rolling Upgrades*
6.0
* Demo
![Page 28: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/28.jpg)
![Page 29: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/29.jpg)
Floodstage Watermark*
6.0
* Demo
![Page 30: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/30.jpg)
![Page 31: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/31.jpg)
Low 85%
High 90%
Floodstage 95%
![Page 32: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/32.jpg)
Sequence Numbers*
6.0
* Demo
![Page 33: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/33.jpg)
![Page 34: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/34.jpg)
![Page 35: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/35.jpg)
63 bits ought to beenough for anyone.
![Page 36: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/36.jpg)
Cross datacenter replication6.x
![Page 37: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/37.jpg)
Types*
5.6 to 8.0
* Demo
![Page 38: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/38.jpg)
![Page 39: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/39.jpg)
Why
Data types
Sparsity
Scoring
![Page 40: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/40.jpg)
How
5.6 opt-in single type
6.x single type
7.x type optional in API
8.x no more types
https://www.elastic.co/guide/en/elasticsearch/reference/6.1/removal-of-types.html#_schedule_for_removal_of_mapping_types
![Page 41: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/41.jpg)
Automatic Queue Resizing6.0
![Page 42: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/42.jpg)
Reject and retry instead of long queues
thread_pool.search.target_response_rate: 2s
Serving 50 requests/s
Queue size:
![Page 43: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/43.jpg)
Adaptive Replica Selection6.1
![Page 44: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/44.jpg)
![Page 45: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/45.jpg)
Pick best shardExponentially Weighted Moving Average
(EWMA)
Piggyback on requests
![Page 46: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/46.jpg)
![Page 47: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/47.jpg)
Shrink & Split*
5.06.1
* Demo
![Page 48: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/48.jpg)
![Page 49: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/49.jpg)
Shrink
Combine shards by a factor
![Page 50: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/50.jpg)
![Page 51: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/51.jpg)
Split
Split into a factor of number_of_routing_shards
![Page 52: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/52.jpg)
Benchmarks∞
![Page 53: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/53.jpg)
![Page 54: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/54.jpg)
![Page 56: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/56.jpg)
Conclusion
![Page 57: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/57.jpg)
Strictness
Rolling Upgrades
Floodstage Watermark
Sequence Numbers
![Page 58: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/58.jpg)
Types
Automatic Queue Resizing
Adaptive Replica Selection
Shrink & Split
![Page 59: (R)Evolution - FOSDEM...](https://reader034.fdocuments.us/reader034/viewer/2022042205/5ea79a8bb672a25b8455d713/html5/thumbnails/59.jpg)
Questions?Philipp Krenn@xeraa