Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?....
Transcript of Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?....
![Page 1: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/1.jpg)
1
Donggeng Yu12/07/2019, Pronto, eBay
ProntoElasticsearch Extension Practice in eBay
![Page 2: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/2.jpg)
22
Agenda
Overview of Elasticsearch in eBay
Use Cases & Challenges
Tools Extension for Clusters Management
Service Extension for Clusters Capability
1
2
3
4
![Page 3: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/3.jpg)
33
Elastic Stack• ELKB
‒ Elasticsearch - Search & Aggregation‒ Logstash – ETL‒ Kibana – Visualization‒ Beats – Data Shipper
• X-Pack‒ security, alerting, monitoring,
reporting, machine learning and etc.
• Use Cases & OOTB Solutions‒ Logs / Metrics‒ APM / Uptime‒ SIEM / Endpoint Security‒ Site Search / App Search / Enterprise‒ Maps
![Page 4: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/4.jpg)
44
Pronto Ecosystem in eBay
62%Supporting text goes here under the number
![Page 5: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/5.jpg)
5
100+ clusters6k+ nodes
VM(openstack) / Container(k8s)
![Page 6: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/6.jpg)
66
Agenda
Overview of Elasticsearch in eBay
Use Cases & Challenges
Tools Extension for Clusters Management
Service Extension for Clusters Capability
3
4
2
1
![Page 7: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/7.jpg)
7
Use Cases in eBay
• Use Cases:‒ Near real time search / aggregation
‒ Virtual Shop / Tire Installation / Terapeak / SEO
‒ On-Site Traffic
‒ Metrics & Logs‒ UFES / Ceilometer / SRE / UMP‒ More than 20T/day for a single cluster
![Page 8: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/8.jpg)
88
Vertical Shop & Tire Installation
![Page 9: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/9.jpg)
99
Terapeak - eCommerce Data Insights
• Terapeak‒ SAAS based tool for providing
ecommerce data insights to online sellers
‒ Acquired by eBay
• Tech Stack‒ From RMDB + SOLR to ELK‒ S3 and Hadoop for data staging‒ Spark for data ETL‒ Kafka for data queue‒ Postgres for Data Warehouse‒ Elasticsearch for indexing and
search‒ ReactJS for front-end application
![Page 10: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/10.jpg)
1010
UFES - Anomaly Detection for SLB
• Goal‒ Unified Front-End Services - Move eBay
Closer to Users so that the world shops first on eBay. UFES team built out 8 new Internet Points of Presence(POP) across the globe
‒ Need to route traffic via UFES PoPs by replacing the Netscaler Hardware SEO Load Balancers with Envoy Proxy based Software Load Balancers.
• Elastic Stack‒ Filebeats + Kafka + Elasticsearch Clusters‒ Dashboard for monitoring and comparison‒ Anomaly Detection for SLB
![Page 11: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/11.jpg)
1111
Ceilometer - IT Operation Analytics
![Page 12: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/12.jpg)
1212
Challenges of Managing Clusters Fleets at Scale
• Integrated with eBay’s Platform & follow the standards
‒ Configuration management & Change management‒ Full lifecycle management
• Easy onboarding and integration‒ Elasticsearch as a Service‒ How to free customer to focus on domain business
• Performance & High Availability‒ Search: Site facing application response time should less than
100 ms‒ Ingesting: 20T per day for a single cluster ‒ Different deployments, like cross region deployment
• Cost Control‒ Hardware cost‒ License fee (support some features like security, alert and ML)‒ Human resource‒ Support (7*24 on-call support & on-site support, etc.)
Performance
HA
Onboarding Integration
Cost
![Page 13: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/13.jpg)
1313
Solutions for Challenges
• From VM to Container‒ VM (Openstack)
‒ Fixed flavor‒ Puppet Foreman infrastructure‒ Puppet module for Elasticsearch
‒ Container (K8s)‒ Flexible flavor (request/limit)‒ Operator Pattern
‒ Deployment + Statefulset + Service
• Best practices & Different deployments‒ Important System Configuration & Best practices‒ Anti-Affinity (High availability)‒ Cross region deployment (High availability)‒ Flavor chosen by traffic (Cost saving)‒ Hot-warm architecture (Cost saving)‒ LB for write / read
Cluster Provision & Management
Performance
HA
Onboarding Integration
Cost
![Page 14: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/14.jpg)
1414
Solutions for ChallengesTooling and Service Extension
Performance
HA
Onboarding Integration
Cost
![Page 15: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/15.jpg)
1515
Agenda
Overview of Elasticsearch in eBay
Use Cases & Challenges
Tools Extension for Clusters Management
Service Extension for Clusters Capability
2
1
4
3
![Page 16: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/16.jpg)
1616
Use Case Onboarding
• Capacity planning ‒ What’s the use case and use scenarios
‒ Data retention / active period‒ Performance
‒ Index rate / search rate‒ Document & bulk size
‒ Deployment & Cost‒ How many nodes?‒ What’s the hardware configuration?‒ What kind of deployment should be used?
‒ Best practices‒ Software configuration‒ Deployment in different Region‒ Keep the margin to ensure that traffic
becomes large without performance issues
Node Storage Memory CPU Network
Master Low Low Low Low
Data Extreme High High Medium
Ingest Low Medium High Medium
Coordinator Low Medium Medium Medium
Machine Learning Low Extreme Extreme Medium
![Page 17: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/17.jpg)
1717
Onboarding Self-Service and Sizing ToolOnboarding Integration
![Page 18: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/18.jpg)
1818
Customer Support
• Support model‒ Different SLA for different use cases
‒ Search response time should less than 100ms‒ Cluster should NOT be in RED
‒ 7*24 support for Site-facing or Tier 2 above‒ SEC call / Pagerduty
• Support case‒ Cluster in RED
‒ Node missing and replica is 0‒ Dangling index
‒ Response time‒ Full GC because of Machine check error (MCE)‒ Too many shards and fields
Onboarding Integration Cost
![Page 19: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/19.jpg)
1919
Data Ingestion Pipeline
• Added Value for customers ‒ Self-service, no coding/testing‒ No onboarding required
• Shared cluster ‒ 30+ use cases / 3T per day
• Shared data assets‒ Partition by application name
• Shared dashboard‒ 30+ Dashboards‒ 300+ Charts/Visualizations
Onboarding Integration
![Page 20: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/20.jpg)
2020
Simple Steps - service onboarding a new use case
pom.xml
web.xml
![Page 21: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/21.jpg)
2121
Data Management & Optimization
• Backup & Restore‒ Snapshot lifecycle management (SWIFT as
the repository )
• Time series data‒ Benefits of using time-based indices
‒ Delete index is faster than delete by query‒ Use hot-warm architecture‒ Close indices or force-merge read-only
indices‒ Time series data
‒ Treapeak v.s UFES (different needs)
• LifeCycle management‒ Central policy management / Web UI /
OOTB Policies
Performance
Onboarding
Integration
Cost
![Page 22: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/22.jpg)
2222
Index Management Tool vs. Curator vs. ILM
Function Curator Pronto Index Mgmt. Tool Elastic ILM
High Availability N/A YES YES
Web UI N/A YES YES
Version Compatibility N/A 2.x/5.x/6.x/7.x 6.8+
Multi-Clusters N/A YES N/A
![Page 23: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/23.jpg)
2323
Diagnostic Tool
• Features‒ Find Improper settings or usage‒ Job scheduler & Diagnostic report for
potential issues
• Rules‒ Too many indices / Too many shards /
Index have too many fields‒ Shard size check (20GB to 40GB)‒ Imbalance shards‒ Replica number should bigger than 0‒ Node missing / Rack Id attribute missed /
Minimum master ‒ Machine check error / Server disk full‒ Alias & index template checking
Performance Cost
![Page 24: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/24.jpg)
2424
Performance & User Scenarios
• Many Factors:‒ Index / Shard ‒ Query / Scripting ‒ Mapping / Setting
Behavior Use Cases
Index heavy Logging / Metrics / Security / APM
Search heavy App Search / Site Search / Analytics
Update heavy Caching / Systems of Record
![Page 25: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/25.jpg)
2525
Performance Issues & Optimization
• Wildcard search‒ Customer use beginning patterns
with * and ?.‒ Avoid to use * or ?.
• Stopwords & Shard Size‒ Reindex with the stop words‒ Use more shards to improve the
throughput
• Too many indices / shards / fields‒ Close or delete the unused indices‒ Improve the document modeling‒ Disable the dynamic mapping
• Performance Optimization‒ Disable swapping & give memory to the
file system cache‒ Unset or increase the refresh interval‒ Disable refresh and replicas for initial
loads‒ Use auto generated Ids‒ Disable the features you do not need‒ Don’t use default dynamic string mapping‒ Watch your shard size / shrink index‒ Force Merge ‒ Pre-Index data‒ Avoid scripts‒ Force-merge read-only indices‒ Warm up global ordinals‒ Replicas might help with through, but not
always
![Page 26: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/26.jpg)
2626
Performance Testing Tool
• Performance testing‒ Testing data‒ Testing scripts‒ Test report for analysis
• Web based tool‒ Developed based on the Gatling‒ Web UI to select the testing scripts
and testing data‒ Test report for analysis
Performance
![Page 27: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/27.jpg)
2727
Agenda
Overview of Elasticsearch in eBay
Use Cases & Challenges
Tools Extension for Clusters Management
Service Extension for Clusters Capability
2
1
3
4
![Page 28: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/28.jpg)
2828
Solution and security plugin for Elasticsearch
• Pronto Security Plugin‒ TLS for encrypted communications‒ Cluster / Index level RBAC control‒ Follow eBay’s standard
‒ API Key for Application ‒ 2FA for user login‒ Audit logs
• Security Consideration‒ Authentication / RBAC‒ Certification retention‒ Firewall / White IP list‒ Vulnerability management
Cost
![Page 29: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/29.jpg)
2929
X-Pack Subscription
• License cost‒ License fee is based on
the node count
• How to Extend ‒ Develop the Kibana
Application‒ Integrate with the
alerting and anomaly detection service
Cost
![Page 30: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/30.jpg)
3030
Alerting Service
• Schedule‒ A schedule for running a query and
checking the condition.• Query
‒ The query to run as input to the condition. Watches support the full Elasticsearch query and aggregation
• Condition‒ A condition that determines whether
or not to execute the actions. You can use simple conditions (always true), or use scripting for more sophisticated scenarios
• Action‒ One or more actions, such as sending
email, pushing data to 3rd party systems through a webhook
‒ Throttling
Biz Data Alert
Cost
![Page 31: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/31.jpg)
3131
Cost
![Page 32: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/32.jpg)
3232
ML for Anomaly DetectionCost
![Page 33: Pronto Elasticsearch Extension Practice in eBay · ‒Customer use beginning patterns with * and ?. ‒Avoid to use * or ?. •Stopwords& Shard Size ‒Reindexwith the stop words](https://reader033.fdocuments.us/reader033/viewer/2022041715/5e4afb706084e4360c7b04e2/html5/thumbnails/33.jpg)
3333
Review
• Easy onboarding and integration
• High Availability & High Performance
• Low cost for hardware / license fee / support efforts
Performance
HA
Onboarding Integration
Cost