GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File...

69
GFS/HDFS 15-440 Distributed Systems

TAGS:

Transcript of GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File...

Page 1: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

GFS/HDFS

15-440 Distributed Systems

Page 2: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

2

Building Blocks

• Google WorkQueue (scheduler)• Google File System• Chubby Lock service (paxos-based)• Two other pieces helpful but not required

• Sawzall (languate)• MapReduce

• BigTable: Build a more application-friendly storage service using these parts

Page 3: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

Overview

• Google File System (GFS) and Hadoop Distributed File System (HDFS)

• BigTable

3

Page 4: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

Google Disk Farm

Early days…

…today

4

Page 5: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

Google Platform Characteristics

• Lots of cheap PCs, each with disk and CPU• High aggregate storage capacity• Spread search processing across many CPUs

• How to share data among PCs?

5

Page 6: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

Google Platform Characteristics

• 100s to 1000s of PCs in cluster• Many modes of failure for each PC:

• App bugs, OS bugs• Human error• Disk failure, memory failure, net failure, power supply

failure• Connector failure

• Monitoring, fault tolerance, auto-recovery essential

6

Page 7: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

7

Page 8: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

Data-Center Network

Page 9: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 10: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 11: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 12: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 13: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 14: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 15: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 16: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 17: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 18: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 19: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 20: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 21: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 22: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 23: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 24: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 25: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 26: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 27: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 28: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 29: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 30: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 31: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 32: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 33: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 34: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 35: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 36: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 37: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 38: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 39: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 40: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 41: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 42: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 43: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 44: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 45: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 46: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 47: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 48: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 49: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 50: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 51: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 52: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 53: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 54: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 55: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 56: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 57: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 58: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 59: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 60: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 61: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 62: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 63: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 64: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 65: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 66: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 67: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 68: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.
Page 69: GFS/HDFS 15-440 Distributed Systems. 2 Building Blocks Google WorkQueue (scheduler) Google File System Chubby Lock service (paxos-based) Two other pieces.

Chubby Ifox

Chubby Ifox

Paxos Made Simpleajayk/paxosfinal.pdf• Google Megastore used for 3 billion writes and 20 billion read transactions daily. • It uses Paxos to manage synchronous replication between

Paxos Made Simpleajayk/paxosfinal.pdf• Google Megastore used for 3 billion writes and 20 billion read transactions daily. • It uses Paxos to manage synchronous replication between

Virtual Synchrony, Paxos , and Beyond

Virtual Synchrony, Paxos , and Beyond

Stone house holiday villa paxos

Stone house holiday villa paxos

Chubby woman

Chubby woman

Paxos Made EPR: Decidable Reasoning about …padon/paxos-made-epr-oopsla17.pdf108 Paxos Made EPR: Decidable Reasoning about Distributed Protocols ODED PADON, Tel Aviv University, Israel

Paxos Made EPR: Decidable Reasoning about …padon/paxos-made-epr-oopsla17.pdf108 Paxos Made EPR: Decidable Reasoning about Distributed Protocols ODED PADON, Tel Aviv University, Israel

Chubby cardinal

Chubby cardinal

Chubby Checker v. Macy's complaint.pdf

Chubby Checker v. Macy's complaint.pdf

WPaxos: Ruling the Archipelago with Fast Consensus · including in Chubby [2] based on Paxos [3], Apache ZooKeeper [4] based on Zab [5], and etcd [6] based on Raft [7]. All of these

WPaxos: Ruling the Archipelago with Fast Consensus · including in Chubby [2] based on Paxos [3], Apache ZooKeeper [4] based on Zab [5], and etcd [6] based on Raft [7]. All of these

Paxos Made Simple

Paxos Made Simple

CHUBBY and PAXOS

CHUBBY and PAXOS

MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

MetaSync - courses.cs.washington.edu · Metasync: Simulate Paxos • Use an append-only list to log Paxos messages – Client sends normal Paxos messages – Upon arrival of message,

Paxos introduction

Paxos introduction

Team Chubby Chasers

Team Chubby Chasers

Chubby Football player

Chubby Football player

Presentation a Journey to Paxos Englishboicosfinearts.com › Ajourneytopaxos › Press_Release.pdf · “A Journey to Paxos” is a book of 83 watercolors of the island of Paxos

Presentation a Journey to Paxos Englishboicosfinearts.com › Ajourneytopaxos › Press_Release.pdf · “A Journey to Paxos” is a book of 83 watercolors of the island of Paxos

Glyfada beach villas paxos

Glyfada beach villas paxos

The Chubby Lock Service for Loosely-coupled Distributed systems · 2014. 11. 21. · The Chubby Systems Introduction Background Overview of Chubby System Design 13 Architecture Client

The Chubby Lock Service for Loosely-coupled Distributed systems · 2014. 11. 21. · The Chubby Systems Introduction Background Overview of Chubby System Design 13 Architecture Client

Implementing Replicated Logs with Paxos

Implementing Replicated Logs with Paxos

Noleen Chubby / Chubby LT Owner’s Manualidriders.com/proflex/resources/98chubby_manual.pdf · 2001. 1. 18. · 2. Using the Chubby Wrench (preload adjuster removal tool), loosen

Noleen Chubby / Chubby LT Owner’s Manualidriders.com/proflex/resources/98chubby_manual.pdf · 2001. 1. 18. · 2. Using the Chubby Wrench (preload adjuster removal tool), loosen