Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create...

26
Confidential Customized for Lorem Ipsum LLC Version 1.0 Spark na Google Cloud Friends don't let friends build data centers

Transcript of Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create...

Page 1: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Confidential Customized for Lorem Ipsum LLC Version 1.0

Spark na Google CloudFriends don't let friends build data centers

Page 2: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Data Scientist @Avenue Code

Evandro CaldeiraCientista de dados na Avenue Code. Formado em Engenharia da Computação e louco por café

E-mail: [email protected]

mailto:[email protected]
Page 3: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

BrasilBelo HorizonteSão PauloPorto Alegre

EUA

Canadá

Page 4: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Confidential Customized for Lorem Ipsum LLC Version 1.0

TOC

Overview

On premise vs cloud

Como migrar

Demo

#
Page 5: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Por que Spark?

Page 6: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 7: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 8: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

On premise1 Equipamentos

2 Gerenciamento

3 Picos de uso

4 $$$

Page 9: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Migragação para GCP:Descomissionamento de um datacenter em 2018

Spotify

Page 10: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Primeiros passos para GCP

Page 11: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 12: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 13: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

O que fazer1 Mova os dados

2 Experimente

Page 14: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

O que fazer3 Use clusters efêmeros

5 Delete o cluster ao finalizar

4 Workers preemptivos

https://cloud.google.com/dataproc/docs/concepts/compute/preemptible-vms?hl=pt-br
Page 15: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Hands on

Page 16: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 17: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Créditos grátis!

https://cloud.google.com/free/?hl=pt-br
Page 18: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Instalação1 Google SDK

2 Spark standalone

https://spark.apache.org/downloads.html
Page 19: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Criação do cluster

Page 20: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
https://cloud.google.com/dataproc/docs/guides/create-cluster
Page 21: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 22: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Execução de job

Page 23: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
https://cloud.google.com/dataproc/docs/guides/submit-job?hl=pt-br
Page 24: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)
Page 25: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Sourcehttps://github.com/evandroc/tdc-spark

https://github.com/evandroc/tdc-spark
Page 26: Spark na Google Cloud - Amazon Web Services · Labels + Add label Max restarts per hour ... Create a cluster Name cluster-I Region global Cluster mode Standard (1 master, N workers)

Obrigado.


Quicklabel System - Candy Labels and Chocolate Labels

Quicklabel System - Candy Labels and Chocolate Labels

MySQL Cluster 7 - O'Reilly Mediaassets.en.oreilly.com/1/event/36/Discover What_s New in... · 2010-04-15 · MySQL Cluster 7.1 Monitor Reliability Manage Access Platform Faster restarts

MySQL Cluster 7 - O'Reilly Mediaassets.en.oreilly.com/1/event/36/Discover What_s New in... · 2010-04-15 · MySQL Cluster 7.1 Monitor Reliability Manage Access Platform Faster restarts

CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems …loshchilov.com/publications/CEC2013.pdf · 2014. 2. 28. · CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems

CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems …loshchilov.com/publications/CEC2013.pdf · 2014. 2. 28. · CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems

Independent Record Labels Update die Labels

Independent Record Labels Update die Labels

Risk-based restarts of untreated subsea oil and gas ... · Risk-Based Restarts of Untreated Subsea Oil and Gas Flowlines in the GoM 6 required to achieve successful understanding

Risk-based restarts of untreated subsea oil and gas ... · Risk-Based Restarts of Untreated Subsea Oil and Gas Flowlines in the GoM 6 required to achieve successful understanding

Parent Club Start Up or Restarts

Parent Club Start Up or Restarts

Mindware · Manufacturer of Labels, Product labels,clear to clear labels,mark sensing labels , shrink labels, movie tickets, taffeta suppliers, thermal ribbon convertors, labels manaufatcurer

Mindware · Manufacturer of Labels, Product labels,clear to clear labels,mark sensing labels , shrink labels, movie tickets, taffeta suppliers, thermal ribbon convertors, labels manaufatcurer

THT Labels: General Identification...THT Labels: General Identification General Identification Labels 88 Benchtop Thermal Transfer Labels 3” Core General ID Labels B-423 Polyester

THT Labels: General Identification...THT Labels: General Identification General Identification Labels 88 Benchtop Thermal Transfer Labels 3” Core General ID Labels B-423 Polyester

Chapter 1: Understanding Kubernetes Architecture · PDF fileSave . all 11.10 Time 11 Labels 11. ... Restarts 11. 47 Memory usage history 11-40 ... (bytes) run: echo Restarts Pods Name

Chapter 1: Understanding Kubernetes Architecture · PDF fileSave . all 11.10 Time 11 Labels 11. ... Restarts 11. 47 Memory usage history 11-40 ... (bytes) run: echo Restarts Pods Name

ST. NICHOLAS NATIONAL SHRINE AUG RESTARTS CONSTRUCTION

ST. NICHOLAS NATIONAL SHRINE AUG RESTARTS CONSTRUCTION

Exploratory Data Analysis for Microarrayscompdiag.molgen.mpg.de/ngfn/docs/2006/sep/rahne... · learning) and cluster analysis (unsupervised learning) is important: If the class labels

Exploratory Data Analysis for Microarrayscompdiag.molgen.mpg.de/ngfn/docs/2006/sep/rahne... · learning) and cluster analysis (unsupervised learning) is important: If the class labels

Fast Database Restarts at Facebook

Fast Database Restarts at Facebook

Laser Labels • Ink Jet Labels EDP Labels Stock or Custom ...

Laser Labels • Ink Jet Labels EDP Labels Stock or Custom ...

Production restarts threaten to spoil bumper year for Europe’s crackers

Production restarts threaten to spoil bumper year for Europe’s crackers

2013 Section Meeting Coaching Workshop Restarts/Set Pieces For the Intermediate Coach.

2013 Section Meeting Coaching Workshop Restarts/Set Pieces For the Intermediate Coach.

TABLE OF CONTENTS— PLANT LABELS & TAGS · 2018-07-16 · TABLE OF CONTENTS— PLANT LABELS & TAGS PLANT LABELS & TAGS Thin Plastic Labels Heavy Plastic Labels Plastic Computer Labels

TABLE OF CONTENTS— PLANT LABELS & TAGS · 2018-07-16 · TABLE OF CONTENTS— PLANT LABELS & TAGS PLANT LABELS & TAGS Thin Plastic Labels Heavy Plastic Labels Plastic Computer Labels

Online Job Scheduling Mechanisms: How Bad are Restarts?

Online Job Scheduling Mechanisms: How Bad are Restarts?

REStARTS: Das Experimentierbuch · REStARTS: Das Experimentierbuch A u t o r en: VKI - Patricia Corieri – Philippe Léonard – Michel Riethmuller – Mario Carbonaro CIRA – Marika

REStARTS: Das Experimentierbuch · REStARTS: Das Experimentierbuch A u t o r en: VKI - Patricia Corieri – Philippe Léonard – Michel Riethmuller – Mario Carbonaro CIRA – Marika

CMA-ES with Restarts for Solving CEC 2013 Benchmark ...CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems Ilya Loshchilov Laboratory of Intelligent Systems Ecole Polytechnique

CMA-ES with Restarts for Solving CEC 2013 Benchmark ...CMA-ES with Restarts for Solving CEC 2013 Benchmark Problems Ilya Loshchilov Laboratory of Intelligent Systems Ecole Polytechnique

An Update on Fukushima Recovery and Reactor Restarts

An Update on Fukushima Recovery and Reactor Restarts