cloud-based data lake Evolution of a modern · O’Reilly Software Architecture Conference Berlin,...

Evolution of a moderncloud-based data lake

Viacheslav Inozemtsev

O’Reilly Software Architecture Conference

Berlin, 07.11.2019

[email protected]

mailto:[email protected]

OUTLINE

Introduction of Zalando

Why data lake?

What is a data lake?

Evolution of the data lake

Future

3

Please write the title in all capital letters

Put images in the grey dotted box "unsupported placeholder"Who am I?

Viacheslav Inozemtsev

● 2010 - Specialist in Applied Mathematics and Computer Science at the Tomsk State University, Russia

● 2014 - Master of Computer Science at the University of Bonn, Germany● Total of 8 years of experience as Data and Software Engineer● Last 3 years at Zalando

4

Introduction of Zalando

5


Put images in the grey dotted box "unsupported placeholder"Introduction of Zalando

● 29M active customers

6



● 29M active customers● 14000 employees

7



● 29M active customers● 14000 employees● 2000 engineers and data scientists

8



● 29M active customers● 14000 employees● 2000 engineers and data scientists● 200 engineering teams

9



● 29M active customers● 14000 employees● 2000 engineers and data scientists● 200 engineering teams● Variety of major data systems

○ Messaging Bus○ BI Data Warehouse○ Google Analytics platform○ Custom datasets

10

Why data lake?

11


Put images in the grey dotted box "unsupported placeholder"Why data lake?

12



13



14



15



16



17



18



19



20



● To enable sharing of the data so, that it is:

21



● To enable sharing of the data so, that it is:○ easy to publish and consume

22



● To enable sharing of the data so, that it is:○ easy to publish and consume○ compliant

23



● To enable sharing of the data so, that it is:○ easy to publish and consume○ compliant○ secure

24



● To enable sharing of the data so, that it is:○ easy to publish and consume○ compliant○ secure○ cost-efficient

25

What is a data lake?

26


Put images in the grey dotted box "unsupported placeholder"What is a data lake?

● Central exchange system○ connects producers and consumers of the data○ defends data from malicious misuse or accidental leaking

27



● Central exchange system○ connects producers and consumers of the data○ defends data from malicious misuse or accidental leaking

● Central big data system○ provides as much different data as possible○ has to be fast and easy to use

28



● Central exchange system○ connects producers and consumers of the data○ defends data from malicious misuse or accidental leaking○ open vs hide!

● Central big data system○ provides as much different data as possible○ has to be fast and easy to use

29




● Central big data system○ provides as much different data as possible○ has to be fast and easy to use○ scale vs performance!

30

Evolution of the data lake

31


Put images in the grey dotted box "unsupported placeholder"Evolution of the data lake - Stage 1: Inception

32



33



34



35



36



37



38



39


Put images in the grey dotted box "unsupported placeholder"Evolution of the data lake - Stage 2: Optimization

40



41



42



43



44



45



46



47



48



49


Put images in the grey dotted box "unsupported placeholder"Evolution of the data lake - Stage 3: Revolution

https://martinfowler.com/articles/data-monolith-to-mesh.html

https://martinfowler.com/articles/data-monolith-to-mesh.html

50


Put images in the grey dotted box "unsupported placeholder"Evolution of the data lake - centralized

51


Put images in the grey dotted box "unsupported placeholder"Evolution of the data lake - federated

52


Put images in the grey dotted box "unsupported placeholder"What is data lake?


● Central big data system○ provides as much different data as possible○ has to be fast and easy to use○ scale vs performance!

53



● Federated exchange system○ connects producers and consumers of the data○ defends data from malicious misuse or accidental leaking○ open vs hide!

● Federated big data system○ provides as much different data as possible○ has to be fast and easy to use○ scale vs performance!

54



● Federated exchange system○ connects producers and consumers of the data○ defends data from malicious misuse or accidental leaking○ open vs hide!○ resolved by centralized governance

● Federated big data system○ provides as much different data as possible○ has to be fast and easy to use○ scale vs performance!

55



● Federated exchange system○ connects producers and consumers of the data○ defends data from malicious misuse or accidental leaking○ open vs hide!○ resolved by centralized governance

● Federated big data system○ provides as much different data as possible○ has to be fast and easy to use○ scale vs performance!○ resolved by decentralized ownership

56

Future

57


Put images in the grey dotted box "unsupported placeholder"Future

● Make data lake fully federated

58


Put images in the grey dotted box "unsupported placeholder"Future

● Make data lake fully federated● Abstract physical storage into logical layer of datasets

59

Evolution of a moderncloud-based data lake

Viacheslav InozemtsevO’Reilly Software Architecture Conference

Berlin, 07.11.2019

[email protected]

www.zalando.com

jobs.zalando.com/tech

mailto:[email protected]

https://www.zalando.com/

https://jobs.zalando.com/tech/

cloud-based data lake Evolution of a modern · O’Reilly Software Architecture Conference Berlin,...

Documents

Transcript of cloud-based data lake Evolution of a modern · O’Reilly Software Architecture Conference Berlin,...