NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB,...
Transcript of NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB,...
![Page 1: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/1.jpg)
NoSQL@Mobiliar – How we started
Christian Hählen
![Page 2: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/2.jpg)
Content
1. Background
1.1 Short introduction
1.2 Why did we start with NoSQL?
2. NoSQL@Mobiliar
3.1 Self-developped telemetry framework on classic hardware
3.2 Toolbox / Templates for usage in Docker / Kubernetes
3. Looking back
11.09.2017NoSQL@Mobiliar 2
![Page 3: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/3.jpg)
Background
11.09.2017NoSQL@Mobiliar 3
![Page 4: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/4.jpg)
About the company…
Swiss Mobiliar Insurance
• Own applications as well as external applications
• Operate Windows and Linux,
directly and on Docker/Kubernetes
• Observe and anticipate development
on new technologies
Mobiliar and databases
• Oracle, SQL Server, DB2
• Linux, Windows, z/OS
11.09.2017NoSQL@Mobiliar 4
![Page 5: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/5.jpg)
… and the speaker
Christian Hählen
• 9 years working experience
• on SQL Server as data architect & database administrator
• as a database interface developer (.NET)
• 4 years working experience as System Administrator /
Head of Infrastructure and Operations
• Working at Mobiliar as database specialist / DBA for one year
• Member of PASS and co-organizer Swiss Chapter Events in Bern
• Interested in the future of
• SQL Server
• NoSQL
• Information system technologies
11.09.2017NoSQL@Mobiliar 5
![Page 6: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/6.jpg)
History – Why did we start with NoSQL?
• NoSQL «everywhere»
• NoSQL already in the house
• With upcoming technologies like docker
RDBMS have some struggles in adapting
How to scale?
Multi-replica write
For some applications RDBMS are not the right tool
Images (BLOBs), Objects, Graphs, JSON
Transient workflow data
11.09.2017NoSQL@Mobiliar 6
![Page 7: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/7.jpg)
• Being a DBA…
different opinions about DBA in NoSQL and agile world
developers ask if they may use NoSQL system XYZ
be ready to answer the first developer’s question
either with
"No, we don't want this in our company because …”
or with
"yes, we made a Pilot / PoC, leading to this toolbox
and templates"
History – Why did we start with NoSQL?
11.09.2017NoSQL@Mobiliar 7
![Page 8: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/8.jpg)
NoSQL@Mobiliar
11.09.2017NoSQL@Mobiliar 8
![Page 9: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/9.jpg)
NoSQL@Mobiliar
Performance Monitoring
• Self-developped telemetry framework (ElasticSearch, spoud)
• Operational data from applications (Prometheus)
11.09.2017NoSQL@Mobiliar 9
![Page 10: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/10.jpg)
NoSQL@Mobiliar
Performance Monitoring
• Self-developed telemetry framework
Linked in application
Runs on every (~800) employee laptop to monitor
performance of busines application
v1: Oracle
v2: ElasticSearch - performance
v3: spoud - flexibility
11.09.2017NoSQL@Mobiliar 10
![Page 11: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/11.jpg)
NoSQL@Mobiliar
Performance Monitoring
• Self-developed telemetry framework
Transmits data about every step, network, I/O …
90 to 100 Millions of documents per day
Last 500 days stored
For performance: index per day
Analysation / visualisation done with Kibana
11.09.2017NoSQL@Mobiliar 11
![Page 12: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/12.jpg)
NoSQL@Mobiliar
Performance Monitoring
• Operational data from applications
Applications are developed exposing data for
Prometheus
Docker containers
currently only showing overall data
in future should expose internal data (processes etc.)
11.09.2017NoSQL@Mobiliar 12
![Page 13: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/13.jpg)
NoSQL@Mobiliar
Usage in Docker and Kubernetes
• Change application to use MongoDB!
Why MongoDB? – We heard about it
Does data management already have a tool of choice?
• Let’s do an evaluation of NoSQL systems
Let’s define a set of systems to be supported
Prepare toolbox and guidelines
• But we need it now…
11.09.2017NoSQL@Mobiliar 13
«If you don’t have
the time to do something
right, where are you
going to find the time
to fix it?»
Stephen King
![Page 14: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/14.jpg)
NoSQL@Mobiliar
Toolbox / Templates for usage in Docker
• Evaluation:
RDBMS vs. NoSQL, classic infrastructure
vs. VMWare vs. Container
Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB,
Couchbase, Cockroach, ArangoDB
Focus on scalability, community, container size / startup
time, UI, schema / typing
Interviews with stakeholders
• Result:
Vision of the future
recommendation of RDBMS and NoSQL systems
11.09.2017NoSQL@Mobiliar 14
![Page 15: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/15.jpg)
NoSQL@Mobiliar
Toolbox / Templates for usage in Docker
• Pilot / PoC:
Rocket.Chat
Not off the shelf
scalable NoSQL system
automated backup
performance monitoring
deployment «as one» on our platform
DevOps Project:
Developer
Container Operator
Database specialist
11.09.2017NoSQL@Mobiliar 15
![Page 16: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/16.jpg)
![Page 17: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/17.jpg)
• Pilot targets:
Ready-to-use template for Docker /
Kubernetes Application
AMW & Kubernetes & Docker Deployment
Using Helm chart
Persistent storage claim
Automated backup external storage
Kubernetes Job
Covered by company backup solution
Integration in our Prometheus
health / performance of DB
GUI for DevOps to have «a feeling»
Collectable by Data Management team
NoSQL@Mobiliar
Toolbox / Templates for usage in Docker
11.09.2017NoSQL@Mobiliar 17
![Page 18: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/18.jpg)
• Pilot state:
Ready-to-use template for Docker /
Kubernetes Application
AMW & Kubernetes & Docker Deployment
Using Helm chart
Persistent storage claim
Automated backup external storage
Kubernetes Job
Covered by company backup solution
Integration in our Prometheus
health / performance of DB
GUI for DevOps to have «a feeling»
Collectable by Data Management team
NoSQL@Mobiliar
Toolbox / Templates for usage in Docker
11.09.2017NoSQL@Mobiliar 18
![Page 19: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/19.jpg)
Looking back
11.09.2017NoSQL@Mobiliar 19
![Page 20: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/20.jpg)
Conclusion
11.09.2017NoSQL@Mobiliar 20
• NoSQL
Might look scary to a DBA
Hot topic
Not yet on it’s peak
![Page 21: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/21.jpg)
Conclusion
• NoSQL
Might look scary to a DBA
Hot topic
Not yet on it’s peak
You can’t hide / ignore
If you dive into it, you feel how…
11.09.2017NoSQL@Mobiliar 21
![Page 22: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/22.jpg)
Conclusion
• NoSQL
Might look scary to a DBA
Hot topic
Not yet on it’s peak
You can’t hide / ignore
If you dive into it, you feel how
interesting
powerful
tempting
11.09.2017NoSQL@Mobiliar 22
![Page 23: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/23.jpg)
Conclusion
• NoSQL
Might look scary to a DBA
Hot topic
Not yet on it’s peak
You can’t hide / ignore
If you dive into it, you feel how
interesting
powerful
tempting
• Implementation @ Mobiliar
Not everybody was enthusiastic at first
Toolset and guidelines are important
Pilot ist still ongoing
More to do than we first thought
Very interesting work
11.09.2017NoSQL@Mobiliar 23
![Page 24: NoSQL@Mobiliar How we started · 2017. 9. 11. · Oracle, SQL Server, MariaDB, PostgreSQL, MongoDB, Couchbase, Cockroach, ArangoDB Focus on scalability, community, container size](https://reader034.fdocuments.us/reader034/viewer/2022052104/603f034dc9584f1a4726c489/html5/thumbnails/24.jpg)
Questions
Thanks for listening …
Want to learn more about our future experiences?
Follow our new blog at www.mobiliar.ch/db-blog
11.09.2017NoSQL@Mobiliar 24