Topics in Distributed Systems CS350, CS550, CS580F.

52
Topics in Distributed Systems CS350, CS550, CS580F

description

©2010 D. J. Foreman 3 Introduction

Transcript of Topics in Distributed Systems CS350, CS550, CS580F.

Page 1: Topics in Distributed Systems CS350, CS550, CS580F.

Topics in Distributed Systems

CS350, CS550, CS580F

Page 2: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman2

Topics

1. Types of Distributed systems2. Communications3. Naming4. Synchronization5. Consistency & Replication6. Fault Tolerance7. Security

Page 3: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman3

Introduction

Page 4: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman4

Definition of a Distributed System

A collection of functions implemented across a set of independent devices that appears to its users as a single coherent system.

Horizontal - all devices are peers Vertical - there is a control hierarchy

Note: Devices may be computers or ….

Page 5: Topics in Distributed Systems CS350, CS550, CS580F.

Types of Distributed systems

Client-Server Peer-peer Processor Pool Network Operating Systems

©2010 D. J. Foreman5

Page 6: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman6

Examples of Distributed Systems

An "ATM machine" A remote file access mechanism A database A chat room A computing "grid" (like SETI)

Page 7: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman7

Design

All “usually” have: Multi-threaded/Multi-process operation User interface (need not be graphical) Point(s) of control (any system shuts-down all)

Page 8: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman8

Some Problems (topics)

System Character codes (ASCII/EBCDIC) Number representation (big- vs. little-Endian)

Size of an "int" (16/32/64 - bit) Float

Shared Memory Operational Compatibility

Thread control System services Process handling

Pointers & References Objects

Page 9: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman9

Hardware Systems

Multiprocessors (modularized shared memory) Bus traffic cache incoherence Limited scalability (network cost/speed) NUMA – Non-Uniform Memory Access

(some local, some shared via network)

Homogeneous Multicomputers Private memory + switching circuits Bus-based connection: ethernet routing Switch-based: routing via h/w net (mesh, n-cube)

e.g., Clusters of Workstations

Heterogeneous Multicomputers No commonality, no global view

Page 10: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman10

Software

Tightly coupled (true "Distributed System") Maintains global view

Multiprocessor (one O/S)Multicomputer (homogeneous vs. heterogeneous)

Loosely coupled (NOS) Each heterogeneous computer runs its own O/S Not much transparency as seen by applications BUT: Local services are shared on the network

Page 11: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman11

Some Classical Distributed Systems

Locus – distributed UNIX Lightweight kernel systems w/UNIX emulation

Amoeba – groups of threads (processor pool) Mach – synch msg-based IPC, threads

Argus ('79) – "Guardians" for transactions Clouds – object-based (like methods) V-system (1984)– IPC via kernel (msgs only),

true multi-threading (unavailable in UNIX '84) Cedar - µcoded VM, IDE, concurrent execution

(see list of papers)

Page 12: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman12

Classical Systems, continued

XDFS - ca. 1977 - servers Cambridge File Server – ca.1982 - indexes on

servers used by OS to build its directory Sun NFS – networked UNIX FS

Page 13: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman13

Open vs Closed Systems

Open Extensible Services external to kernel

Closed Non-extensible Services internal to kernel

Services are: File system Process mgmt Network communications Names Time Security/authentication

Page 14: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman14

Atomicity

Semaphores Unstructured – requires app to test/signal

Mutexes Monitors

A programming constructShared variablesProcedures

lock and unlock functions (may be kernel-provided) Threads (Pthreads, POSIX threads)

Butenhof, "Programming with POSIX threads", AW Lewis & Berg, "Multithreaded Programming with

Pthreads", PTR-PH

Page 15: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman15

Number Representations

What is an "int" 16 bits??? 32 bits???

FP number storage Many mechanisms, depending on h/w Less portable than integers

Page 16: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman16

An example using int's

32-bit compiler, treats "int" as 32 bitsint x; x=1; // x is 00000000 00000000 00000000 00000001

Y(x); // all 32 bits ---> 16-bit target system Target of call compiled on 16-bit system

function Y (int val) // "int" is 16 bits{print val;} // prints 0 ---- WHY???? ---

'Y' only gets the 1st 16 bits of x because it doesn't KNOW there are more to get !!! ---

WHY??? -- Same problem for 32-bit 64-bit calls and

whatever comes next. Remember Y2K!

Page 17: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman17

Operational Compatibility

Thread controls Pthreads on Windows (compatible w/POSIX) POSIX threads

seen on Linux & Solaris MS threads Solaris threads others (OS/2, MVS)

System services (commands, signals, file manipulation)

Process handling (creation, deletion) Communication (IPC: msgs vs pipes)

Page 18: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman18

Memory

Page 19: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman19

Memory Problems

a. Little Endian (Intel): lowest # byte = highest valueb. Big Endian (Sun): highest # byte = highest value

Receiver must know data types

Original(Sun)

as received(Windows)

improper correction

5*2245

Page 20: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman20

Shared Memory

Inhibits distribution (coherency lost, cost to build) When NOT shared, problems arise:

Consistency Access (speed, mechanisms) Compatibility (endian) etc.

Atomicity Program level

Compare & Swap Test & OP (add, subtract, set-bit)

Kernel/library supported Semaphores, Mutexes, Condition variables

Page 21: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman21

Distributed Memory

Partial solutions for heterogeneous systems Small local (non-shared) memory Bus and caches to a main store Bus and caches to each processor

Full solutions for multi-computer Interconnection (h/w switching) networks

Limited scalability VERY expensive, but VERY fast

Page 22: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman22

Communications

2

Page 23: Topics in Distributed Systems CS350, CS550, CS580F.

Primitives - 1

Reliable – e.g.; TCP Ack Ordered Guaranteed delivery

Unreliable – e.g.; UDP No ack No guarantee No ordering (bottles in the ocean)

©2010 D. J. Foreman23

Page 24: Topics in Distributed Systems CS350, CS550, CS580F.

Primitives - 2

Blocking/non-blocking Buffered/unbuffered Distinguished messages

Send_get (sender is blocked) Get_request Send_reply (original sender is unblocked)

©2010 D. J. Foreman24

Page 25: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman25

Methods

Shared variables - coherence?? Message passing

RPC – Remote Procedure Call Very good for Client-server Hides message passing

RMI – Remote Method Invocation (also Remote Object Invocation)

Like RPC, but with system-wide objects Java server implements a Security Manager & registry

Very much like DCE MOM – Message Oriented Middleware

Streams (TCP/IP) Needed for continuous information flow (video, audio)

Page 26: Topics in Distributed Systems CS350, CS550, CS580F.

RPC Protocols

Requestor Receiver (server)

©2010 D. J. Foreman26

Request

Ack ReplyAck

Request

Ack Reply

Request

Reply

T0

Tn

Page 27: Topics in Distributed Systems CS350, CS550, CS580F.

Marshaling

Heterogeneous systems No guarantee of data conformity On guarantee of directionality (endian-ness)

Data must be converted Data format must be standardized Communication types

Virtual circuits –reliable, flow controlled Datagrams unreliable, no TCP “connect”

©2010 D. J. Foreman27

Page 28: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman28

RPC Steps

1. Client procedure calls client stub in normal way2. Client stub builds message, calls local OS3. Client's OS sends message to remote OS4. Remote OS gives message to server stub5. Server stub unpacks parameters, calls server6. Server does work, returns result to the stub7. Server stub packs it in message, calls local OS8. Server's OS sends message to client's OS9. Client's OS gives message to client stub10.Stub unpacks result, returns to client

2

Page 29: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman29

Passing Value Parameters

a) Original message on the PCb) The message after receipt on a SPARC WSc) The message after being inverted. Numbers in

dotted-boxes indicate the address of each byte

2

Page 30: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman30

Berkeley Sockets (1)

Delete the connectionClose

Input dataReceive

Output dataSend

Allow a specific connection request to an ethereal port, selected by the O/S, freeing the "listen" port

Accept

Signal availability for connection requestsListen

Assign a "handle" to the socketBind

Create an endpoint from a connectionSocket

Basic TCP/IP 'commands'

2

Page 31: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman31

Connection-oriented communication pattern using sockets. Note the ordering of Reads & Writes vs time.

Berkeley Sockets: Client-Server

S o c ket B ind Lis ten A c c ep t R ead W rite C lo s e

S e rve r

C lien tS o c ket C o nnec t W rite R ead C lo s e

2

t0

Page 32: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman32

Berkeley Sockets: Peer-Peer

S o c ket, B ind ,L is ten , A c c ep t R ead W rite

P e e r 1

P eer 2

C o nnec t W riteR eadS o c ket, B ind ,L is ten , A c c ep t

C o nnec t

Read & Write are independent threads

2

Page 33: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman33

Berkeley Sockets: Producer-Consumer

S o c ket B ind L is ten A c c ep t R ead C lo s e

C onsum e r

P ro d u cerS o c ket, B ind C o nnec t W rite C lo s e

Note the one-way nature of communication

2

Page 34: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman34

Naming

4

Page 35: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman35

What is a name?

Basic objects File Value Program

Data structures across a network How to reference How to modify

Types Human readable Machine readable

Page 36: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman36

Synchronization

5

Page 37: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman37

Mechanisms

Explicit signals Semaphores, conditions, events

Test and Set shared variables Consider chip cycle vs. bus cycle

Page 38: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman38

Methods

Semaphores Conditional critical sections Messages Event counts and sequencers Monitors Modules, common procedures and entries Path expressions Managers I/O commands

Page 39: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman39

Problems

Race conditions - cause loss of data

Action sequences - hierarchy to be maintained

Data protection

Page 40: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman40

Consistency and Replication

6

Page 41: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman41

World view

Users must see same data at all times Updates must be controlled Rollbacks must maintain consistency

Page 42: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman42

Fault Tolerance

7

Page 43: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman43

Partitioning

Problem: when is the system partitioned? Response time? Keep-alive? Other?

Solutions Rollback? Re-start? Cut-off? Other?

Page 44: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman44

Security

8

Page 45: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman45

Whose data is it?

Must protect data and programs Viruses Worms Errors Theft

Page 46: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman46

Virtualization

8

Page 47: Topics in Distributed Systems CS350, CS550, CS580F.

Problems

Hardware differences on ends of real systems Time delays cannot be simulated

©2010 D. J. Foreman47

Page 48: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman48

Papers

8

Page 49: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman49

Introductory Papers

The papers listed in the following section, while dated in the early 1970s and 80's, form some of the foundations of contemporary implementations and research.

Page 50: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman50

Basic Concepts

Synchronizing Resources, G. Andrews, 1981 Specification of Synchronizing Processes, K.

Ramanathan, 1983 Concepts and Notations for Concurrent

Programming, G. Andrews, 1983 Distributed Operating Systems, A.

Tanenbaum, 1985

Page 51: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman51

Programming & Concept Papers

The Temporal Semantics of Concurrent Programs, A. Pneuli, 1981

Time, Clocks, and the Ordering of Events in a D.S., L. Lamport, 1978

Determining the Last Process to Fail, D. Skeen, 1985

Correctness Proofs of D. Termination Algorithms, K. Apt, 1986

Monitors: An OS Structuring Concept, C.A.R. Hoare, 1974

Page 52: Topics in Distributed Systems CS350, CS550, CS580F.

©2010 D. J. Foreman52

Actual System Papers

An Introduction to the V System, E. Berglund, 1986

The Design of the Saguaro D.O.S., G. Andrews,1987

XMS: A Rendezvous-based D. S. Software Architecture, N. Gammage, 1985