Building Distributed Systems

decision guide

Oleksiy Kurnenkov

Core InfrastructureDomain leadOnApp

Building Distributed Systems

decision guide

Introduction

Part I

Distributed system

is a set of interconnected functional units with separate runtime context

MONOLITH : DISTRIBUTED

1 Runtime context

Coroutines

Threads

Process

N Runtime contexts

Nodes

Components

Clusters

Grids

SCALE

Small ->

Middle ->

Global ->

OS, LAN

LAN

WAN

TYPES

CDN

Grids

Clouds

Clusters

Enterprise

CDN

Grid

Globus Toolkit 6

BOINC

x-Torrent

Cloud

Cluster

Enterprise

TASKS

MMOG

Analysis

Sensors

Real Time Enterprise

Media

Storage

Computation

Transactions

Challenges

AvailabilityScalabilitySecurityOpennessHeterogeneityConcurrencyTransparencyFeasibility

It IS there

99.9999999%

Availability

It grows and shrinks

Scalability

U ain’ gonna crack it >:|

Security

It is open for development and integration

Opennes

Everything changes

Heterogeneity

Parallel workloads

Concurrency

Less know - less depend

Transparency

Simplify as possible

Feasibility

Fallacies

The network is reliableLatency is zeroBandwidth is infiniteThe network is secureTopology doesn't changeThere is one administratorTransport cost is zeroThe network is homogeneous

Mechanics

Part II

MechanicsI

MembershipConcurrency / ParallelismSynchronizationCoordinationConsensusClustering

Who’s there?

Membership

Membership

Heartbeats

Up-2-date list of nodes

We are Legion

Concurrency / Parallelism

Concurrency / Parallelism

Actors

Process Calculi

Shared memory

MY precious!

Synchronization

Synchronisation

Mutex | Semaphore

Distributed Lock

Roger that!

Coordination

Coordination

Leader election

Orchestration

the ‘Two Generals Problem’

Deal

Consensus

Consensus

Quorum

Consistency Availability Partition tolerance

Altogether

Clustering

Clustering

Load Balancing

Redundancy

Availability

Replication

Partition tolerance

MechanicsII

ScalingFault DetectionFailoverRecoverFail back

Scaling

Vertical

Horisontal

Automatic

Fault Detection

Monitoring

Supervision

Activity Analysis

Failover

Data | State Accessibility

Resource allocation

Spare Capacity

Business Continuity

Recovery

State Reconstruction

Data backup

State dump

Failback

Data consistency

Conflict resolution

Apache YARN

Apache Zookeeper

Apache Mesos

Apache Drill

Google Chubby

CoreOS

Google Kubernetes

Google Dremel

Architecture

Part III

Client - Server

N-tier

Service oriented Architecture

Event Driven Architecture

P2P

Context

System Allocation

Architecturalentitiesdecomposition

SubsystemServiceComponentProcessObjectMiddleware

Fractal

Subsystem -> Component -> Process -> Object

Subsystem -> Component -> Process -> Object

Subsystem -> Component -> Process -> Object

CommunicationInfrastructure

aka Middleware

IPC-----------------------------Remote InvocationGroup CommunicationMessaging

IPC

Share data

Send data

Remote Invocation

Request - Reply

RPC

RMI

Group Communication

{ Uni | Multi | Broad } cast

Publish | Subscribe

Messaging

Message Queue

Message Broker

Pipe | Filter | Translator

Implementation

Part IV

Microcosm -> Macrocosm

Middleware

[ * ] Process

Host

Network

Within a process

Threads

Shared memory

Mutex | Monitor

require 'thread'

class RaceCondition def initialize(resource, concurrency_level) @resource = resource @concurrency_level = concurrency_level end

def perform @concurrency_level.times do threads << Thread.new { action } end

threads.each(&:join) end

private

def threads @threads ||= [] end

def action @resource += 1 puts @resource endend

RaceCondition.new(0, 10).perform

# 12# 3# 5# 6# 7# 8# 910# 4

require 'thread'

class Sycnhronizer def initialize(resource, concurrency_level) @resource = resource @concurrency_level = concurrency_level end

def perform @concurrency_level.times do threads << Thread.new { action } end

threads.each(&:join) end

private

def threads @threads ||= [] end

def action lock.synchronize do @resource += 1 puts @resource end end

def lock @lock ||= Mutex.new endend

Sycnhronizer.new(0, 10).perform

# 1# 2# 3# 4# 5# 6# 7# 8# 9# 10

Microcosm -> Macrocosm

Process

[ * ] Host

NetworkMiddleware

Within a host

Processes

MQ | Shared memory | Pipes | UNIX Socket

Semaphore

# http://bogomips.org/ruby_posix_mq/

require 'posix_mq'

class Producer attr_reader :mq

def initialize(mq_name) @mq = POSIX_MQ.new("/foo", :rw) end

def send(message, prio = 0) puts "Send: #{message}. Priority #{prio}" mq.send("#{message} #{prio}", prio) endend

p = Producer.new("/test_ipc")

p.send("Hello from #{Process.pid}", 10)p.send("Hello from #{Process.pid}", 2)p.send("Hello from #{Process.pid}", 0)p.send("Hello from #{Process.pid}", 1)p.send("Hello from #{Process.pid}", 20)

# ruby posix_mq/producer.rb## Send: Hello from 12635. Priority 10# Send: Hello from 12635. Priority 2# Send: Hello from 12635. Priority 0# Send: Hello from 12635. Priority 1# Send: Hello from 12635. Priority 20

# http://bogomips.org/ruby_posix_mq/

require 'posix_mq'

class Consumer attr_reader :mq

def initialize(mq_name) @mq = POSIX_MQ.new("/foo", :rw) end

def receive mq.receive.first end

def receive_non_block mq.nonblock = true

begin receive rescue Errno::EAGAIN mq.nonblock = false puts "Nothing" end end

def shift mq.tryshift endend

c = Consumer.new("/test_ipc")

while m = c.shift puts "got: #{m}"end

# ruby posix_mq/consumer.rb

# got: Hello from 12635 10# got: Hello from 12635 20# got: Hello from 12635 2# got: Hello from 12635 1# got: Hello from 12635 0

# https://github.com/pmahoney/process_shared

require 'process_shared'

mutex = ProcessShared::Mutex.newmem = ProcessShared::SharedMemory.new(:int)mem.put_int(0, 0)

pid1 = fork do puts "in process 1 (#{Process.pid})" 10.times do sleep 0.01 mutex.synchronize do value = mem.get_int(0) sleep 0.01 puts "process 1 (#{Process.pid}) incrementing" mem.put_int(0, value + 1) end endend

pid2 = fork do puts "in process 2 (#{Process.pid})" 10.times do sleep 0.01 mutex.synchronize do value = mem.get_int(0) sleep 0.01 puts "process 2 (#{Process.pid}) decrementing" mem.put_int(0, value - 1) end endend

Process.wait(pid1)Process.wait(pid2)

# ruby shm.rb## in process 1 (8038)# in process 2 (8041)# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# process 1 (8038) incrementing# process 2 (8041) decrementing# value should be zero: 0

rd_child, wr_parent = IO.piperd_parent, wr_child = IO.pipe

pid = fork do rd_parent.close wr_parent.close

wr_child.puts "sent from child process" puts rd_child.getsend

rd_child.closewr_child.close

wr_parent.write "sent from parent process"puts rd_parent.gets

# ruby pipes/pipes.rb## sent from child process# sent from parent process

require 'eventmachine'

module UnixServer def receive_data(data) puts data

EM.stop if data.chomp == "exit"

send_data("Server #{Process.pid}: Got #{data.chomp} from you") close_connection_after_writing endend

EM.run do puts "Started UNIX socket server on /tmp/sock"

EM::start_unix_domain_server("/tmp/sock", UnixServer)end

# ruby server.rb## Started UNIX socket server on /tmp/sock## HELLO! My pid is 13847

require 'socket'

UNIXSocket.open("/tmp/sock") do |c| c.write("HELLO! My pid is #{Process.pid}")

puts c.readend

# ruby client.rb## Server 13843: Got HELLO! My pid is 13847 from you

Microcosm -> Macrocosm

Process

Host

[ * ] NetworkMiddleware

Over the Network

{ TCP | UDP } Socket

HTTP Client - Server

Async messaging

require "bunny"

conn = Bunny.newconn.start

ch = conn.create_channelq = ch.queue("hello")

puts " [*] Waiting for messages in #{q.name}. To exit press CTRL+C"q.subscribe(:block => true) do |delivery_info, properties, body| puts " [x] Received #{body}"

# cancel the consumer to exit delivery_info.consumer.cancelend

require "bunny"

conn = Bunny.new(:hostname => "rabbit.local")conn.start

ch = conn.create_channel

q = ch.queue("hello")ch.default_exchange.publish("Hello World!", :routing_key => q.name)puts " [x] Sent 'Hello World!'"

conn.close

require "bunny"

conn = Bunny.newconn.start

ch = conn.create_channelx = ch.fanout("logs")

msg = ARGV.empty? ? "Hello World!" : ARGV.join(" ")

x.publish(msg)puts " [x] Sent #{msg}"

conn.close

require "bunny"

conn = Bunny.newconn.start

ch = conn.create_channelx = ch.fanout("logs")q = ch.queue("", :exclusive => true)

q.bind(x)

puts " [*] Waiting for logs. To exit press CTRL+C"

begin q.subscribe(:block => true) do |delivery_info, properties, body| puts " [x] #{body}" endrescue Interrupt => _ ch.close conn.closeend

Apache Kafka

RabbitMQ

Next time

To be continued:

Algorithms, patterns,

code!

Thank you!

Q&A