Agile Lab_BigData_Meetup
-
Upload
paolo-platter -
Category
Documents
-
view
211 -
download
1
Transcript of Agile Lab_BigData_Meetup
![Page 1: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/1.jpg)
Mattia Bertorello
![Page 2: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/2.jpg)
Who is Agile Lab?
GO BIG (data) or GO HOME
![Page 3: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/3.jpg)
Summary
•Why streaming matters
•Why prediction?
•Streaming architecture
•Spark streaming
•Demo time
![Page 4: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/4.jpg)
Why streaming
matters
Ⓒ2015 Agile Lab S.r.l.
![Page 5: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/5.jpg)
Why streaming matters
Data Big Data
Business Reaction
Batch Analysis
Typical BigData Workflow
![Page 6: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/6.jpg)
Why streaming matters
DataReal Time Processing
FASTER REACTIONS MORE PROFITS
Business Reaction
Streaming BigData Workflow
![Page 7: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/7.jpg)
Why streaming matters
• Fleet Management• Insurance• Recommendation• Etc...
![Page 8: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/8.jpg)
Why prediction?
Ⓒ2015 Agile Lab S.r.l.
![Page 9: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/9.jpg)
Why prediction?
• Rule based categorization and clustering is obsolete
• Pattern discovery
• Adaptation to fast changing data
• Smart thinking: no dummies
• Prediction is more valuable
![Page 10: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/10.jpg)
Streaming
architecture
Ⓒ2015 Agile Lab S.r.l.
![Page 11: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/11.jpg)
Streaming architecture
Ingestion Layer
Processing Layer
Serving Layer
![Page 12: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/12.jpg)
Spark Streaming
![Page 13: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/13.jpg)
Spark Notebook
Use Apache Spark straight from the Browser
![Page 14: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/14.jpg)
Demo time...
Ⓒ2015 Agile Lab S.r.l.
![Page 15: Agile Lab_BigData_Meetup](https://reader033.fdocuments.us/reader033/viewer/2022052709/58ac4e381a28ab99028b6431/html5/thumbnails/15.jpg)
Card transaction analysis
PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP
Classificazione delle transazioni
online/offline
PAN CIFRATO | AMOUNT | DESCRIPTION | TIMESTAMP | ISONLINE
fraud detection algorithm
SQL aggregation
Generazione di allarmi in tempo reale