Synchronization

CSCI 4900/6900

Importance of Clocks & Synchronization

• Avoiding simultaneous access of resources– Cooperate to grant exclusive access for small durations

• Process may need to agree upon ordering of events– First person to complete assignment will receive bonus– How to infer who completed first?

• Synchronization & ordering is difficult in distributed setting– We have seen this several times !!!

Chapter Outline• Actual time-based synchronization

– Physical clocks– Synchronizing multiple clocks

• Relative ordering-based synchronization– Logical clocks, Lamport’s algorithm, Vector time stamps

• Global state

• Leader election algorithms (time permitting)

• Mutual exclusion in distributed systems

• Distributed transactions

Clocks in Systems• Notion of time in centralized system is unambiguous

– Do a system call to get current time

• Important property of time – Always moves forward– If B executes the system call after A, B will never get a

lower value

• Many system operations need some notion of time for correctness

• Example – “Make” program in Unix– Re-compiles only files that were changed since last

compilation– What happens when there is no global agreement on

time?

Clock Synchronization

When each machine has its own clock, an event that occurred after another event may nevertheless be assigned an earlier time.

System Clocks & Clock Skew• Almost all computers have clocks (timers)

• A precisely cut quartz crystal kept under tension– Oscillates at a well-defined frequency

• Two registers – Counter & holding register– Each oscillation of crystal decrements counter by 1– Interrupt when counter is zero & reload from holding reg.– Each interrupt is a tick

• Clocks at multiple CPUs cannot be guaranteed to oscillate at exact same frequency

• Difference between various clocks – Clock Skew

How is Time Actually Measured?• Solar time – Based on earth’s rotation

– Transit of sun: Sun reaching the highest apparent in sky

– Solar day: Time b/w two consecutive sun transits

– Solar second: 1/86400th of a solar day

– Mean solar second: 1/86400th of a mean solar day

• Atomic time– Second: Time for cesium-133 atom to make 9,192,631,770

transitions

– International atomic time (TAI)

• Leap seconds to resolve difference b/w TAI & solar time (UTC)

• NIST broadcasts UTC on radio station (WWV)

Physical Clocks (1)

Computation of the mean solar day.

Physical Clocks (2)

TAI seconds are of constant length, unlike solar seconds. Leap seconds are introduced when necessary to keep in phase with

the sun.

Clock Synchronization Algorithms

• Two related problems– If one machine has WWV receiver: synchronize all

machines to machine with the WWV receiver– No WWV receiver: Keep all machines relatively

synchronized

• Many algorithms with some key assumptions– Each machine has timer that causes interrupt H times/sec– Increments software clock on each interrupt

– Cp(t) indicates clock value when UTC time is t

– In ideal world dC/dt = 1

Drift & Max. Drift Rate

The relation between clock time and UTC when clocks tick at different rates.

• In real world dC/dt is not one

• Maximum drift rate: ρ such that 1- ρ ≤ dC/dt ≤ 1 + ρ– Specified by manufactures

Clock Synchronization Algorithms• Two clocks drifting from UTC in opposite

directions at rate of ρ need to be synchronized every δ/2ρ secs.

• Chritian’s Algorithm:– Suited when one machine has WWV receiver– Each machine sends a request to time server periodically

(period < δ/2ρ ) seconds– Time server responds with its current time (CUTC)

• Simple scheme– Set receivers time to CUTC

– Two problems• Clock might run backward !!!• Doesn’t consider processing time

Cristian's Algorithm• Introduce change gradually – Reduce time by a small

amount

• Add T0 – T1/2 to CUTC

The Berkeley Algorithm

a) The time daemon asks all the other machines for their clock values

b) The machines answer

c) The time daemon tells everyone how to adjust their clock

Logical Clocks• For many applications it is sufficient if all machines

agree upon some time– Synchronization with UTC not needed

• Logical clocks

• Lamport showed that in many cases clock synchronization is not needed– What actually is needed is agreement with regards to

ordering of events– Example – Compilation occurred before file editing

Concepts in Logical Clocks• Happens-before relation

• a->b if one of the following is true– a and b are events in same process and a occurs before b– If a is an event of sending a message and b is the event of

receiving the same message is another process a->b.• Implies that message cannot be received before it is sent

• Happens-before is transitive– If a->b and b->c then a->c

• If two events in x and y are in two processes that never exchange messages then x & y are concurrent

Lamport’s Timestamps Algorithm• We need a way of measuring time such that for

every event a we can assign a time C(a) on which all processes agree.

• If a->b than C(a) < C(b)

• Alternatively– If a and b are in same process and a precedes b C(a) <

C(b)– If a representing sending a message and b represents

receiving the same message C(a) < C(b)

• Clock C should always move forward

Lamport’s Algorithm• Each process runs with its own clock

– Clocks need not be synchronized

• Processes sends messages that are time-stamped with the local clock time

• When a process receives a message– If its local clock is more than time-stamp of message no

need of any adjustment– If local clock less than message’s time-stamp, local clock

incremented to one more than message’s time-stamp

• Between every two events the clock is incremented• Use decimal point followed by process number for

global uniqueness

Example: Totally-Ordered Multicasting

Updating a replicated database and leaving it in an inconsistent state.

Globally Ordered Multicast• Database is replicated at multiple locations

• Query is forwarded to nearest copy, where as update is sent to all replicas

• Suppose client deposits $100 to account (having $1000) and bank starts a process to add 1% interest

• Two replicas may get these update requests in different order– One replica shows $1110, where as the other $1111

• What we need is globally ordered multicast

Lamport’s Algorithm for GOM• Messages timestamped with sender’s local time• Assumptions

– When message is multicast it is also sent to sender– Messages from same sender arrive sequentially

• When a process receives a message, it is put into local queue

• Receiver multicasts acknowledgement to everyone• Local times adjusted through Lamport’s algorithm• Each process will have the exact same local queue• Deliver the first message if all the ACKs are

received

Causality• Lamport’s algorithm leads to all events being

ordered irrespective of whether they are dependent on each other

• Example – Electronic bulletin board– Users may post new articles or respond to existing ones– No dependence between different threads

• Lamport’s algorithm imposes order even between independent events

• CAUSALITY – Two events are ordered only if one is dependent on the other

Vector Timestamps for Causality• Each event is assigned a vector timestamp

– If VT(a) < VT (b) then a is said to causally precede b

• Each process maintains a vector Pi maintains Vi

– Vi[i] – Number of events occurred at Pi

– Vi[j] = k then Pi knows that k events have occurred at Pj

• When Pi sends message to Pj, it piggybacks Vi• Pj knows about how many events have occurred at Pi• Pj also knows how many events have occurred at other

processes– Pj adjusts its local vector by setting Vj[k] = max{Vj[k], vt[k]}– Increments Vj[i] by 1

• Delivers a message r only if vt(r) [j] = Vk[j] + 1 and vt(r) [i] <= Vk[i] for all i <> j

Global State• Local states of each processes together with

messages in transit

• Global states is needed for detecting termination conditions/deadlocks

• Distributed snapshot algorithm (Chandy & Lamport)– Reflects state where distributed system might have been

• Always represents a consistent global state– If it is recorded that a process received a message we

should have also recorded that the message has been sent

• Distributed snapshots via Cuts

Cuts for Global State

a) A consistent cutb) An inconsistent cut

Distributed Snapshots Algorithm• Assumption – Processes are connected via uni-directional

point-to-point links (channels)• Any process (p) may initiate algorithm• P records its local state and sends a marker to along each

outgoing channel– Marker indicates that receiver should participate in the snapshot

algorithm

• When process Q receives a marker– It saves its local state, if it has already not done so, and sends

marker along each outgoing channel

– If Q had recorded local state, it records the state of the channel

– DONE to indicate that process has done its part

Global Snapshots

a) Organization of a process and channels for a distributed snapshot

Global State (3)

b) Process Q receives a marker for the first time and records its local state

c) Q records all incoming messaged) Q receives a marker for its incoming channel and finishes recording

the state of the incoming channel

Termination Detection• Detecting the termination of a distributed

computation• Initiator starts the process by sending a marker to all

processes• Each process records its state• On completing its part of global snapshots, it checks

whether there are no messages in transit• If no transit messages process sends DONE to

initiator• Else sends CONTINUE

– Initiator has to restart the detection algorithm

Leader Election in Distributed Systems• Many distributed systems need one process to act

as a coordinator/initiator– “First among equals” not “one above the rest”

• Does not matter which process takes up responsibility– But all process should agree about who would be the

leader

• Assumption: Each process has a unique identifier

• Bully Algorithm

• Ring Algorithm

Bully Algorithm• When a process P notices that current coordinator

has failed, it sends an ELECTION message to all processes with higher IDs

• If no one responds, P becomes the leader

• If a higher-up receives P’s message, it will send an OK message to P and execute the algorithm

• Process with highest ID takes over as coordinator by sending COORDINATOR message

• If a process with higher ID comes back, it takes over leadership by sending COORDINATOR message

Bully Algorithm - Example

The bully election algorithm• Process 4 holds an election• Process 5 and 6 respond, telling 4 to stop• Now 5 and 6 each hold an election

Bully Algorithm - Example (2)

d) Process 6 tells 5 to stope) Process 6 wins and tells everyone

Ring Algorithm

• When a process P notices current coordinator’s failure, it builds an ELECTION message– Message contains the P’s ID

• P sends message to it successor– Skip if the successor is down

• Each process adds its ID to the ELECTION message

• When election message reaches the initiator, it converts message to COORDINATOR– The process ID with highest ID is declared as new

coordinator

Ring Algorithm - Example