Consistency and Replication - Boise State...

1/65

Consistency and Replication

2/65

Replicas and Consistency???

Tatiana Maslany in the show Orphan Black: The story of a group ofclones that discover each other and the secret organization Dyad, whichwas responsible for their creation.

3/65

Reasons for Replication

I To increase the reliability of a system.I To increase the performance of a system. This can be of two

types:I Scaling in numbersI Scaling in geographical area

I Having multiple copies leads to the problem of consistency.When and how the copies are made consistent determines theprice of replication.

I Example: Client caching of web pages in web browser gainsperformance for the client at the cost of consistency.

4/65

Replication as a Scaling Technique

I Placing copies of data close to client processes can help withscaling. But keeping copies up to date requires more networkbandwidth. Updating too often may be a waste. Not updatingoften enough is the flip side.

I How to keep the replicas consistent? Use global ordering usingLamport timestamps or use a coordinator. This may require alot of communication for a large system.

I In many cases, the real solution is to loosen consistencyconstraints. E.g. The updates do not have to be atomic. Towhat extent we can loosen depends highly on the access andupdate patterns as well as the application.

I A range of consistency models are available.

5/65

Data Store

I Examples:I distributed shared memoryI distributed shared databaseI distributed file system

I Each process has a local or nearby copy of the whole store orpart of it.

I A data operation is classified as a write when it changes thedata, and is otherwise classifies as a read operation.

6/65

What is a Consistency Model?

I A consistency model is essentially a contract betweenprocesses and the data store. It says that if processes agree toobey certain rules, the store promises to work correctly.

I Models with minor restrictions are easy to use while modelswith major restrictions are more difficult to use. But then easymodels don’t perform as well. So we have to make trade-offs.

7/65

Consistency Models

I Data-centric consistency modelsI Continuous consistencyI Sequential consistencyI Causal consistencyI Entry Consistency with Grouping operations

I Client-centric consistency modelsI Eventual consistencyI Monotonic readsI Monotonic writesI Read your writesI Writes follow reads

8/65

Sequential Consistency (1)

A data store is sequentially consistent when:

The result of any execution is the same as if the (read and write)operations by all processes on the data store were executed in somesequential order and the operations of each individual processappear in this sequence in the order specified by its program.

9/65


I Behavior of two processes operating on the same data item.The horizontal axis is time.

10/65


(a) A sequentially consistent data store.(b) A data store that is not sequentially consistent.

11/65


I Three concurrently-executing processes.

12/65


I Four valid execution sequences for the three processes. Thevertical axis is time.

I Signature is output of P1, P2, P3 concatenated.

13/65

Causal Consistency (1)

For a data store to be considered causally consistent, it is necessarythat the store obeys the following condition:

I Causal Consistency: Writes that are potentially causallyrelated must be seen by all processes in the same order.Concurrent writes by different processes may be seen in adifferent order on different machines. Write by the sameprocess are considered to also be causally related.

I Weaker than sequential consistency.

14/65


I This sequence is allowed with a causally-consistent store, butnot with a sequentially consistent store. Why?

15/65


I A violation of a causally-consistent store. Why?

16/65


I A valid sequence of events in a causally-consistent store.Concurrent writes do not have to be globally ordered.

17/65


I Implementing causal consistency requires keeping track ofwhich processes have seen which writes.

I It effectively means a dependency graph of which operation isdependent on which other operations must be constructed andmaintained.

I This can be done using vector timestamps.

18/65

Entry Consistency with Grouping Operations (1)

Necessary criteria for correct synchronization:

I Acquiring a lock can succeed only when all updates to itsassociated shared data have completed.

I Exclusive access to a lock can succeed only if no other processhas exclusive or nonexclusive access to that lock.

I Nonexclusive access to a lock is allowed only if any previousexclusive access has been completed, including updates on thelock’s associated data.

This, in effect, is linearizing the usage of locks, adhering tosequential consistency.

19/65

Entry Consistency with Grouping Operations (2)

L(x) W(x)a L(y) W(y)b U(x) U(y)

L(x) R(x)a R(y) NIL

L(y) R(y)b

P1:

P2:

P3:

I A valid event sequence for entry consistency.

20/65

Client-Centric Consistency Models

I Being able to handle concurrent operations on shared datawhile maintaining sequential consistency is fundamental todistributed systems. For performance reasons, this may beguaranteed only when processes use synchronizationmechanisms.

I A special, important class of distributed data stores have thefollowing characterization:

I Lack of simultaneous updates (or when such updates happen,they can be easily resolved)

I Most operations involve reading data.

I A very weak consistency model, called eventual consistency, issufficient in such cases. This can be implemented relativelycheaply by client-centric consistency models.

21/65

Eventual Consistency (1)

I Consider the following example scenarios:I A database where most operations are reads. Only one, or very few

processes perform update operations. The question then is how fastupdates should be made available to only-reading processes.

I DNS system has an authority for each domain so write-write conflictsnever happen. The only conflicts we need to handle are read-writeconflicts. A lazy propagation is sufficient in this case.

I Web pages are updated by a single authority. There are normally nowrite-write conflicts to resolve. Clients such as Web browsers andproxies cache pages so they may return out-of-date Web pages. Userstolerate this inconsistency.

I If no updates take place, all replicas will gradually become consistent.This form of consistency is called eventual consistency.

I Eventual consistency essentially requires only that updates areguaranteed to propagate to all replicas. Write-write conflicts are easyto solve when assuming that only a small group of processes canperform updates. Eventual consistency is therefore often cheap toimplement.

22/65

Eventual Consistency (2)

I The above problem can be alleviated by using client-centricconsistency, which provides guarantees for a single clientconsistency in accessing the data store.

23/65

Client-centric Consistency Model Notation

I Version xi of a data item is the result of a series of writeoperations that took place since initialization.

I We denote these operations by the write set W (xi )

I By appending write operations to that series, we obtainanother version xj and say that xj follows from xi . We denotethis as W (xi ;xj).

I If we do not know if xj follows from xi , we use the notationW (xi | xj).

24/65

Monotonic Reads (1)

A data store is said to provide monotonic-read consistency if thefollowing condition holds:

I If a process reads the value of a data item x , then anysuccessive read operation on x by that process will alwaysreturn that same value or a more recent value.

I But no guarantees on concurrent access by different clients.

25/65

Monotonic Reads (2)

W (x )1 1

W (x x )2 1 2;

R (x )1 1

R (x )1 2

L1:

L2:

I The read operations performed by a single process P at twodifferent local copies of the same data store. Amonotonic-read consistent data store.

I L1 and L2 are two local data stores.I W1(x1): Process P1 produces version x1 without knowing

anything about other versions.I W2(x1;x2): Process P2 produces version x2 that follows from

x1.

26/65

Monotonic Reads (3)

W (x )1 1

W (x x )2 1 2|

R (x )1 1

R (x )1 2

L1:

L2:

I The read operations performed by a single process P at twodifferent local copies of the same data store. A data store thatdoes not provide monotonic reads.

I W2(x1 | x2) denotes that process P2 produced version x2concurrently to version x1, a potential write-write conflict.

27/65

Monotonic Writes (1)

In a monotonic-write consistent store, the following condition holds:

I A write operation by a process on a data item x is completedbefore any successive write operation on x by the sameprocess.

28/65


W (x )1 1

W (x x )2 1 2; W (x x )1 2 3;

L1:

L2:

I A monotonic-write consistent data store.

29/65


W (x )1 1

W (x x )2 1 2| W (x x )1 31|

L1:

L2:

I A data store that does not provide monotonic-writeconsistency.

30/65


W (x )1 1

W (x x )2 1 2| W (x x )1 2 3;

L1:

L2:

I A data store that does not provide monotonic-writeconsistency as we have both WS(x1 | x2) and WS(x1 | x3).

31/65


W (x )1 1

W (x x )2 1 2| W (x x )1 31;

L1:

L2:

I Consistent because WS(x1;x3) although x1 has apparentlyoverwritten x2.

32/65

Read Your Writes (1)

A data store is said to provide read-your-writes consistency, if thefollowing condition holds:

I The effect of a write operation by a process on data item xwill always be seen by a successive read operation on x by thesame process.

33/65


W (x )1 1

W (x x )2 1 2; R1 2(x )

L1:

L2:

I A data store that provides read-your-writes consistency.

34/65


W (x )1 1

W (x x )2 1 2| R1 2(x )

L1:

L2:

I A data store that does not follow Read-Your-Writesconsistency model.

35/65

Writes Follow Reads (1)

A data store is said to provide writes-follow-reads consistency, if thefollowing holds:

I A write operation by a process on a data item x following aprevious read operation on x by the same process isguaranteed to take place on the same or a more recent valueof x that was read.

36/65


R2(x )1W (x )1 1

W (x x )3 1 2; W (x x )2 2 3;

L1:

L2:

I A Writes-Follow-Reads consistent data store.

37/65


W (x )1 1 R2(x )1

W (x x )3 1 2| W (x x )2 31|

L1:

L2:

I A data store that does not follow Writes-Follow-Readsconsistency model.

38/65

Replica Management

I A key issue for any distributed system that supports replicationis to decide where, when and by whom replicas should beplaced, and subsequently the mechanisms to use for keepingthe replicas consistent. Two separate problems:

I placing replica servers: handled automatically by data centersI placing content: permanent replicas, server-initiated,

client-initiated

39/65

Content Replication and Placement

I The logical organization of different kinds of copies of a datastore into three concentric rings.

40/65

Server-Initiated Replicas

I Counting access requests from different clients.I Three possible actions: migrate, delete, replicate.

41/65

Client-Initiated Replicas

I Same as client caches.I Used to improve access times.I Sharing a cache between clients may or may not improve the

hit rate. E.g. Web data, file serversI Shared caches can be at departmental or organization level

42/65

Content Distribution (1)

I Design issue: What is actually to be propagated?I Propagate only a notification of an update (better for low

read-to-write ratios)I Transfer data from one copy to another (better for high

read-to-write ratios)I Propagate the update operation to other copies (active

replication)

43/65


I Push Protocol: Updates are propagated to other replicaswithout being asked. This is useful for keeping a relativelyhigher degree of consistency.

I A push-based approach is efficient when the read-to-updateratio is relatively high.

I Pull Protocol: A server or a client requests another server tosend it any updates it has at the moment. A pull-basedapproach is efficient when the read-to-update ratio is relativelylow.

I A comparison between push-based and pull-based protocols inthe case of multiple-client, single-server systems.

44/65


I A lease is a promise by the server that it will push updates tothe client for a specified time.

I When a lease expires, the client is forced to poll the server forupdates and pull in the modified data if necessary. We can useleases to dynamically switch between push and pull.

I Types of leases:I Age-based leases. Grant long lasting leases to data that is

expected to remain unmodified.I Renewal-frequency based leases. Give longer leases to clients

where its data is popular.I State-space overhead based leases. Start lowering expiration

times as space runs low.

I Using multicasting can be much more efficient than unicastingfor the updates.

45/65

Implementations: Consistency Protocols

I Bounding numerical deviationsI Implement using a push model based on views on values of

variables.I Bounding staleness deviations

I Use Vector Clocks, where on server Sk , RVCk [i ] = ti impliesthat Sk has seen all writes that have been submitted to Si upto time ti .

I Whenever a server notices that tk −RVCk [i ] is about to exceeda specified limit, it simply starts pulling in writes from Si witha timestamp later than RVCk [i ].

I Bounding ordering deviationsI Can be bounded by specifying the maximal length of the queue

of tentative writes.I When this is exceeded, the server no longer accepts writes, but

will instead attempt to commit tentative writes by negotiatingwith other servers in which order its writes should be executed.

46/65

Implementations: Primary-based Protocols

I Implementations tend to prefer simpler consistency models.I Primary-based protocols are used for implementing sequential

consistency.I In a primary-based protocol, each data item x in the data store

has an associated primary, which is responsible for writeoperations on x

I Primary can be fixed at a remote server or write operationscan be carried out locally after moving the primary to theprocess where the write operation was initiated.

47/65

Remote Write Protocol

I Also known as a primary-backup protocol. Implements sequentialconsistency.

I Non-blocking version: the primary acknowledges after updating itscopy and informs backup servers afterwards

48/65

Local Write Protocol

I Primary-backup protocol in which the primary migrates to theprocess wanting to perform an update. Updates have to bepropagated back to other replicas.

49/65

Examples of Local Write Primary-Backup Protocols

I Mobile devicesI Before disconnecting, the mobile device becomes the primary server for

each data item it expects to update.I While disconnected, update operations happen locally on the device

while other processes can still perform read operations (but withoutupdates).

I Later, when connecting again, the updates are propagated from theprimary to backups, bringing the data store in a consistent state again.

I Distributed File SystemsI Normally, there may be a fixed central server through which all write

operations take place (as in remote write primary backup).I However, the server temporarily allows one of the replicas to perform a

series of local updates to speed up performance.I When the replica server is done, the updates are propagated to the

central server, from where they can be distributed to other replicaservers.

50/65

Replicated-Write Protocols

I Write operations can be carried out at multiple replicas insteadof just one.

I Active replication: the write operation is forwarded to allreplicas. It requires operations to be carried out in the sameorder everywhere. So it requires totally ordered multicastsusing either Lamport timestamps or a central coordinator.

I Quorum-based protocol: the write operation is done bymajority voting.

51/65

Quorum-based Replicated-Write Protocol (1)

Quorum-based protocol

I To read a file with N replicas, a client needs to assemble aread quorum, an arbitrary collection of NR servers. Similarly,to write to a file, a write quorum of at least NW servers. Thevalues of the quorums are subject to the following rules:

I NR +NW > NI NW > N/2

I Only one writer at a time can achieve write quorum.I Every reader sees at least one copy of the most recent read

(takes one with most recent version number or logicaltimestamp)

52/65

Quorum-based Replicated-Write Protocols (2)

I Quorum-based Protocol. Three examples of the votingalgorithm.(a) A correct choice of read and write set.(b) A choice that may lead to write-write conflicts.(c) A correct choice, known as ROWA (read one, write all).

I In-class Exercise. A file is replicated on 10 servers. List allcombinations of read and write quorums that are permitted bythe voting algorithm.

53/65

Quorum-based Replicated-Write Protocols (3)

I ROWA: R = 1, W = NFast reads, slow writes

I RAWO: R = N, W = 1Fast writes, slow reads

I Majority: R = W = N/2 + 1Both moderately slow, but extremely high availability

I Weighted voting:Give more votes to "better" replicas

54/65

Implementing Client-Centric Consistency (1)

I Each write operation has a globally unique identifier. Such anidentifier can be assigned by the server where the write wassubmitted.

I Each client tracks identifiers for two sets of writes:I read set: consists of writes relevant for the read operations

performed by the client.I write set: consists of the writes performed by the client.

55/65


I Implementing Monotonic Reads: When a client performs aread operation at a server, that server is handed the client’sread set to check whether all the identified writes have takenplace locally. If not, it can contact other servers to be broughtup to date. Or it can forward the request to another serverwhere the write operations have already taken place.

I After the read operation is performed, the write operationsthat are relevant to the read operation are added to theclient’s read set.

I How to determine exactly where the write operations in theclient’s read set have taken place? The write identifier cancontain the server’s identifier. Servers can be required to logtheir writes so they can replayed at another server. Theclient’s can generate a globally unique identifier that isincluded in the write identifier.

56/65


I Implementing Monotonic Writes: Whenever a client initiates anew write operation at a server, the server is handed over theclient’s write set. It then ensures that the identified writeoperations are performed first and in the correct order. Afterperforming the new write operation, that write operation’sidentifier is added to the write set.

I Implementing Read-Your-Writes: Requires that the serverwhere the read operation is performed has seen all the writesin the client’s write set. They can be fetched from otherservers or the client can search for the appropriate server toimprove performance.

I Implementing Writes-Follow-Reads: Bring the selected serverup to date with the write operations in the client’s read set,and then later adding the identifier for the write set to thewrite set, along with the identifiers of the read set (which havenow become relevant for the write operation just performed).

57/65


I Problems with the implementation: The read set and write setassociated with each client can become very big.

I Improving efficiency: keep track of the sets only for the lengthof a session. This isn’t sufficient. Why?

I The main problem is the representation of the read and writesets. It can be represented more efficiently by means of vectortimestamps.

58/65

Representing Read/Write Sets Using Vector Timestamps (1)

I Whenever a server accepts a new write operation W , it assignsit a globally unique identifier along with a timestamp ts(W ).That server is identified as the origin(W ). Subsequent writesto that server get higher timestamps.

I Each server Si maintains a vector timestamp WVCi , whereWVCi [j ] is equal to the timestamp of the most recent writeoperation originating from Sj that has been processed by Si .Assume that the writes from Sj are processed in the order theywere submitted.

I Whenever a client issues a request to read or write operationat a specific server, that server returns its current timestampalong with the results of the operation.

59/65


I Read and write sets are represented by vector timestamps. Foreach session A, we construct a vector timestamp SVCA withSVCA[i ] set equal to the maximum timestamp of all writeoperations in A that originate from server Si .

SVCA[j ] =max{ts(W ) |W ∈ A& origin(W ) = Sj}

I The timestamp of a session always represents the latest writeoperations that have been seen by the applications that areexecuting as part of that session.

I The compactness is obtained by representing all observedwrites originating from the same server through a singletimestamp.

60/65


I Suppose a client, as part of session A, logs in at server Si andpasses SVCA to Si .

I Assume that SVCA[j ]> WVCi [j ]. That is, Si has not yet seen allthe writes originating from Sj that the client has seen.

I Depending upon the required consistency, the server may haveto fetch these writes before being able to consistently reportback to the client.

I Once the operation is performed, the server Si will return itscurrent timestamp WVCi . At that point, SVCA is adjusted to:

SVCA[j ] =max{SVCA[j ],WVCi [j ]}

61/65

Exercises (1)

I Problem 1. Access to shared Java objects can be serialized bydeclaring its methods as being synchronized. Is this enough toguarantee serialization when such an object is replicated?

I Problem 2. Explain in your own words what the main reasonis for actually considering weak consistency models.

I Problem 3. It is often argued that weak consistency modelsimpose an extra burden for programmers. To what extent isthis statement actually true?

I Problem 4. What kind of consistency model would you use toimplement an electronic stock market? Explain your answer.(Hint: Sequential versus Causal)

I Problem 5. Consider a personal mailbox for a mobile user,implemented as part of a wide-area distributed database.What kind of client-centric consistency model would be themost appropriate?

62/65

Exercises (2)

I Problem 6. When using a lease, is it necessary that the clocksof a client and server, respectively, are tightly synchronized?

I Problem 7. Consider a non-blocking primary-backup protocolused to guarantee sequential consistency in a distributed datastore. Does such a store always provide read-your-writesconsistency?

I Problem 8. For active replication to work in general, it isnecessary that all operations be carried out in the same orderat each replica. Is this ordering always necessary? (Hint:Consider read and idempotent write operations)

I Problem 9. A file is replicated on 10 servers. List allcombinations of read and write quorums that are permitted bythe voting algorithm.

63/65

Summary of Consistency Models (1)

Data-Centric Consistency Models

I Sequential Consistency: The result of any execution is thesame as if the (read and write) operations by all processes onthe data store were executed in some sequential order andthe operations of each individual process appear in thissequence in the order specified by its program.

I Causal Consistency: Writes that are potentially causallyrelated must be seen by all processes in the same order.Concurrent writes by different processes may be seen in adifferent order on different machines. Writes by the sameprocess are also considered to be causally related.

64/65

Summary of Consistency Models (2)

Eventual consistency essentially requires only that updates areguaranteed to propagate to all replicas. Client-CentricConsistency Models that are used to implement eventualconsistency.

I Monotonic Reads: If a process reads the value of a data itemx , then any successive read operation on x by that process willalways return that same value or a more recent value.

I Monotonic Writes: A write operation by a process on a dataitem x is completed before any successive write operation on xby the same process.

I Read Your Writes: The effect of a write operation by aprocess on data item x will always be seen by a successive readoperation on x by the same process.

I Writes Follow Reads: A write operation by a process on adata item x following a previous read operation on x by thesame process is guaranteed to take place on the same or amore recent value of x that was read.

65/65

References

I Eventually Consistent by Werner Vogels (CTO, Amazon)I How eventual is eventual consistency? posted on Basho BlogI 10 Lessons from 10 Years of Amazon Web Services by Werner

Vogels (CTO, Amazon)

http://www.allthingsdistributed.com/2008/12/eventually_consistent.html

http://basho.com/how-eventual-is-eventual-consistency/

http://www.allthingsdistributed.com/2016/03/10-lessons-from-10-years-of-aws.html

Consistency and Replication - Boise State...

Documents

Transcript of Consistency and Replication - Boise State...