Post on 24-May-2020
LECTURE 6 Consistency and Replication
Replication and why is it hard
¨ Data needs to be replicated for reliability or to improve performance ¤ If a replica crashes, possible to continue to have access. ¤ Performance improves by accessing nearer data stores ¤ Aids scaling
¨ However, a key challenge is ensuring the consistency of data across replicas. ¤ Hard to realize in large distributed systems. `
Handling Failures
¨ Logical clocks enable all replicas to execute updates in same order ¤ But, cannot handle failure of any replica!
¨ To deal with crash failures we saw: ¤ Chandy-Lamport snapshot for coordinated checkpoint ¤ Log sources of non-determinism between checkpoints
3
Types of failures
¨ Crash failures ¤ Can resume with previously saved state
¨ Fail stop ¤ All state is lost upon failure ¤ Need to replicate state
4
Primary Backup Replication 5
Client Primary Backup
Backup
Backup
Primary Backup Replication
¨ How to handle primary failure? ¤ Promote one of the backups as the primary
¨ How to handle backup failure? ¤ Add another machine as a backup
6
Primary Backup Sync
¨ When should primary sync up with backups?
¨ What should be transferred when syncing?
7
Example: MapReduce Master
RegisterServer() { while (1) {
receive msg and parse addr
pick task to assign
mark task as assigned
respond with task assignment }
}
8
Primary & Backups in sync
When to sync
again?
Takeaways
¨ Cannot tolerate primary failure after update to its state is externally visible
¨ Corollary: Okay for primary to be out of sync with backup until change is externally visible ¤ External consistency
¨ Implications for when primary should sync with backups?
9
Primary-Backup Sync 10
C1
C2
P
B
Reads vs. Updates
¨ For operations that do not update state, primary need not consult backups ¤ When is this true?
¨ If backup is externally consistent with primary ¤ à If backup takes over as primary, it will generate
identical response as primary may have
11
What to transfer in sync?
¨ Snapshot of primary’s state ¤ Slow ¤ When is this necessary? ¤ Necessary when bootstrapping new backup
¨ Every operation ¤ Why is this okay? ¤ Leverage determinism of state machine
12
Service Development
¨ Getting coordination right between primary and backups is tricky ¤ Easy to mess up
¨ Must make replication transparent to developer
13
RSM with Logical Clocks 14
Replicated State Machine
Application
Updates
Ordered Updates
Replicated State Machine
Application
Updates
Server1 Server2
Transparent Primary Backup
¨ Application relies on library to keep primary and backups in sync ¤ Receive message from client ¤ Sync with backups before sending response to client
¨ Will this solution work? ¨ Does not capture non-determinism in execution
15
Virtual Machines
Operating System
Hardware
Applications
CPU Disk RAM
Process File system Virtual memory
Virtual Machine
Virtual Machine Monitor
16
Primary VM
Backup VM
Logging channel
Shared Disk
�!�,)� �� �*!� �� �&%��,)�+!&%�
@8AG4G<BA B9 94H?GGB?8E4AG .%F 9BE G;8 ( (�*!+� C?4G9BE@� 'HE 4CCEB46; <F F<@<?4E 5HG J8 ;4I8 @478 FB@89HA74@8AG4? 6;4A:8F 9BE C8E9BE@4A68 E84FBAF 4A7 <AI8FG<:4G87 4 AH@58E B9 78F<:A 4?G8EA4G<I8F� !A 477<G<BA J8 ;4I8;47 GB 78F<:A 4A7 <@C?8@8AG @4AL 477<G<BA4? 6B@CBA8AGF<A G;8 FLFG8@ 4A7 784? J<G; 4 AH@58E B9 CE46G<64? <FFH8FGB 5H<?7 4 6B@C?8G8 FLFG8@ G;4G <F 8�6<8AG 4A7 HF45?8 5L6HFGB@8EF EHAA<A: 8AG8ECE<F8 4CC?<64G<BAF� +<@<?4E GB @BFGBG;8E CE46G<64? FLFG8@F 7<F6HFF87 J8 BA?L 4GG8@CG GB 784?J<G; 94<?FGBC 94<?HE8F 1��3 J;<6; 4E8 F8EI8E 94<?HE8F G;4G 64A58 78G86G87 589BE8 G;8 94<?<A: F8EI8E 64HF8F 4A <A6BEE86G 8KG8EA4??L I<F<5?8 46G<BA�,;8 E8FG B9 G;8 C4C8E <F BE:4A<M87 4F 9B??BJF� �<EFG J8
78F6E<58 BHE 54F<6 78F<:A 4A7 78G4<? BHE 9HA74@8AG4? CEBGB6B?F G;4G 8AFHE8 G;4G AB 74G4 <F ?BFG <9 4 546>HC .% G4>8FBI8E 49G8E 4 CE<@4EL .% 94<?F� ,;8A J8 78F6E<58 <A 78G4<? @4AL B9 G;8 CE46G<64? <FFH8F G;4G @HFG 58 477E8FF87 GB5H<?7 4 EB5HFG 6B@C?8G8 4A7 4HGB@4G87 FLFG8@� /8 4?FB78F6E<58 F8I8E4? 78F<:A 6;B<68F G;4G 4E<F8 9BE <@C?8@8AG<A:94H?GGB?8E4AG .%F 4A7 7<F6HFF G;8 GE478B�F <A G;8F8 6;B<68F�&8KG J8 :<I8 C8E9BE@4A68 E8FH?GF 9BE BHE <@C?8@8AG4G<BA9BE FB@8 58A6;@4E>F 4A7 FB@8 E84? 8AG8ECE<F8 4CC?<64G<BAF��<A4??L J8 78F6E<58 E8?4G87 JBE> 4A7 6BA6?H78�
2. BASIC FT DESIGN�<:HE8 � F;BJF G;8 54F<6 F8GHC B9 BHE FLFG8@ 9BE 94H?G
GB?8E4AG .%F� �BE 4 :<I8A .% 9BE J;<6; J8 78F<E8 GB CEBI<7894H?G GB?8E4A68 �G;8 -.'*�.4 .%� J8 EHA 4 �!(1- .% BA4 7<�8E8AG C;LF<64? F8EI8E G;4G <F >8CG <A FLA6 4A7 8K86HG8F<78AG<64??L GB G;8 CE<@4EL I<EGH4? @46;<A8 G;BH:; J<G; 4F@4?? G<@8 ?4:� /8 F4L G;4G G;8 GJB .%F 4E8 <A 2'.01�) ),!(�/0#-� ,;8 I<EGH4? 7<F>F 9BE G;8 .%F 4E8 BA F;4E87 FGBE4:8�FH6; 4F 4 �<5E8 �;4AA8? BE <+�+! 7<F> 4EE4L� 4A7 G;8E89BE8 4668FF<5?8 GB G;8 CE<@4EL 4A7 546>HC .% 9BE <ACHG 4A7BHGCHG� �/8 J<?? 7<F6HFF 4 78F<:A <A J;<6; G;8 CE<@4EL 4A7546>HC .% ;4I8 F8C4E4G8 ABAF;4E87 I<EGH4? 7<F>F <A +86G<BA ����� 'A?L G;8 CE<@4EL .% 47I8EG<F8F <GF CE8F8A68 BAG;8 A8GJBE> FB 4?? A8GJBE> <ACHGF 6B@8 GB G;8 CE<@4EL .%�+<@<?4E?L 4?? BG;8E <ACHGF �FH6; 4F >8L5B4E7 4A7 @BHF8� :BBA?L GB G;8 CE<@4EL .%��?? <ACHG G;4G G;8 CE<@4EL .% E868<I8F <F F8AG GB G;8
546>HC .% I<4 4 A8GJBE> 6BAA86G<BA >ABJA 4F G;8 ),%%'+%!&�++#)� �BE F8EI8E JBE>?B47F G;8 7B@<A4AG <ACHG GE4�6<F A8GJBE> 4A7 7<F>� �77<G<BA4? <A9BE@4G<BA 4F 7<F6HFF8758?BJ <A +86G<BA ��� <F GE4AF@<GG87 4F A868FF4EL GB 8AFHE8G;4G G;8 546>HC .% 8K86HG8F ABA78G8E@<A<FG<6 BC8E4G<BAF<A G;8 F4@8 J4L 4F G;8 CE<@4EL .%� ,;8 E8FH?G <F G;4G G;8546>HC .% 4?J4LF 8K86HG8F <78AG<64??L GB G;8 CE<@4EL .%� BJ8I8E G;8 BHGCHGF B9 G;8 546>HC .% 4E8 7EBCC87 5LG;8 ;LC8EI<FBE FB BA?L G;8 CE<@4EL CEB7H68F 46GH4? BHGCHGFG;4G 4E8 E8GHEA87 GB 6?<8AGF� �F 78F6E<587 <A +86G<BA ��� G;8CE<@4EL 4A7 546>HC .% 9B??BJ 4 FC86<�6 CEBGB6B? <A6?H7<A:8KC?<6<G 46>ABJ?87:@8AGF 5L G;8 546>HC .% <A BE78E GB8AFHE8 G;4G AB 74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F�,B 78G86G <9 4 CE<@4EL BE 546>HC .% ;4F 94<?87 BHE FLF
G8@ HF8F 4 6B@5<A4G<BA B9 ;84EG584G<A: 58GJ88A G;8 E8?8I4AGF8EI8EF 4A7 @BA<GBE<A: B9 G;8 GE4�6 BA G;8 ?B::<A: 6;4AA8?�!A 477<G<BA J8 @HFG 8AFHE8 G;4G BA?L BA8 B9 G;8 CE<@4ELBE 546>HC .% G4>8F BI8E 8K86HG<BA 8I8A <9 G;8E8 <F 4 FC?<G5E4<A F<GH4G<BA J;8E8 G;8 CE<@4EL 4A7 546>HC F8EI8EF ;4I8?BFG 6B@@HA<64G<BA J<G; 846; BG;8E�!A G;8 9B??BJ<A: F86G<BAF J8 CEBI<78 @BE8 78G4<?F BA F8I
8E4? <@CBEG4AG 4E84F� !A +86G<BA ��� J8 :<I8 FB@8 78G4<?FBA G;8 78G8E@<A<FG<6 E8C?4L G86;AB?B:L G;4G 8AFHE8F G;4G CE<@4EL 4A7 546>HC .%F 4E8 >8CG <A FLA6 I<4 G;8 <A9BE@4G<BAF8AG BI8E G;8 ?B::<A: 6;4AA8?� !A +86G<BA ��� J8 78F6E<584 9HA74@8AG4? EH?8 B9 BHE �, CEBGB6B? G;4G 8AFHE8F G;4G AB74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F� !A +86G<BA ��� J8 78F6E<58BHE @8G;B7F 9BE 78G86G<A: 4A7 E8FCBA7<A: GB 4 94<?HE8 <A 46BEE86G 94F;<BA�
2.1 Deterministic Replay Implementation�F J8 ;4I8 @8AG<BA87 E8C?<64G<A: F8EI8E �BE .%� 8K8
6HG<BA 64A 58 @B78?87 4F G;8 E8C?<64G<BA B9 4 78G8E@<A<FG<6 FG4G8 @46;<A8� !9 GJB 78G8E@<A<FG<6 FG4G8 @46;<A8F 4E8FG4EG87 <A G;8 F4@8 <A<G<4? FG4G8 4A7 CEBI<787 G;8 8K46G F4@8<ACHGF <A G;8 F4@8 BE78E G;8A G;8L J<?? :B G;EBH:; G;8 F4@8F8DH8A68F B9 FG4G8F 4A7 CEB7H68 G;8 F4@8 BHGCHGF� � I<EGH4? @46;<A8 ;4F 4 5EB47 F8G B9 <ACHGF <A6?H7<A: <A6B@<A:A8GJBE> C46>8GF 7<F> E847F 4A7 <ACHG 9EB@ G;8 >8L5B4E74A7 @BHF8� &BA78G8E@<A<FG<6 8I8AGF �FH6; 4F I<EGH4? <AG8EEHCGF� 4A7 ABA78G8E@<A<FG<6 BC8E4G<BAF �FH6; 4F E847<A:G;8 6?B6> 6L6?8 6BHAG8E B9 G;8 CEB68FFBE� 4?FB 4�86G G;8 .%�FFG4G8� ,;<F CE8F8AGF G;E88 6;4??8A:8F 9BE E8C?<64G<A: 8K86HG<BA B9 4AL .% EHAA<A: 4AL BC8E4G<A: FLFG8@ 4A7 JBE>?B47���� 6BEE86G?L 64CGHE<A: 4?? G;8 <ACHG 4A7 ABA78G8E@<A<F@A868FF4EL GB 8AFHE8 78G8E@<A<FG<6 8K86HG<BA B9 4 546>HC I<EGH4? @46;<A8 ��� 6BEE86G?L 4CC?L<A: G;8 <ACHGF 4A7 ABA78G8E@<A<F@ GB G;8 546>HC I<EGH4? @46;<A8 4A7 ��� 7B<A:FB <A 4 @4AA8E G;4G 7B8FA�G 78:E478 C8E9BE@4A68� !A 477<G<BA @4AL 6B@C?8K BC8E4G<BAF <A K�� @<6EBCEB68FFBEF ;4I8HA78�A87 ;8A68 ABA78G8E@<A<FG<6 F<78 8�86GF� �4CGHE<A:G;8F8 HA78�A87 F<78 8�86GF 4A7 E8C?4L<A: G;8@ GB CEB7H68G;8 F4@8 FG4G8 CE8F8AGF 4A 477<G<BA4? 6;4??8A:8�.%J4E8 78G8E@<A<FG<6 E8C?4L 1��3 CEBI<78F 8K46G?L G;<F
9HA6G<BA4?<GL 9BE K�� I<EGH4? @46;<A8F BA G;8 .%J4E8 I+C;8E8C?4G9BE@� �8G8E@<A<FG<6 E8C?4L E86BE7F G;8 <ACHGF B9 4 .%4A7 4?? CBFF<5?8 ABA78G8E@<A<F@ 4FFB6<4G87 J<G; G;8 .%8K86HG<BA <A 4 FGE84@ B9 ?B: 8AGE<8F JE<GG8A GB 4 ?B: �?8� ,;8.% 8K86HG<BA @4L 58 8K46G?L E8C?4L87 ?4G8E 5L E847<A: G;8?B: 8AGE<8F 9EB@ G;8 �?8� �BE ABA78G8E@<A<FG<6 BC8E4G<BAFFH�6<8AG <A9BE@4G<BA <F ?B::87 GB 4??BJ G;8 BC8E4G<BA GB 58E8CEB7H687 J<G; G;8 F4@8 FG4G8 6;4A:8 4A7 BHGCHG� �BEABA78G8E@<A<FG<6 8I8AGF FH6; 4F G<@8E BE !' 6B@C?8G<BA <A
31
Primary VM
Backup VM
Logging channel
Shared Disk
�!�,)� �� �*!� �� �&%��,)�+!&%�
@8AG4G<BA B9 94H?GGB?8E4AG .%F 9BE G;8 ( (�*!+� C?4G9BE@� 'HE 4CCEB46; <F F<@<?4E 5HG J8 ;4I8 @478 FB@89HA74@8AG4? 6;4A:8F 9BE C8E9BE@4A68 E84FBAF 4A7 <AI8FG<:4G87 4 AH@58E B9 78F<:A 4?G8EA4G<I8F� !A 477<G<BA J8 ;4I8;47 GB 78F<:A 4A7 <@C?8@8AG @4AL 477<G<BA4? 6B@CBA8AGF<A G;8 FLFG8@ 4A7 784? J<G; 4 AH@58E B9 CE46G<64? <FFH8FGB 5H<?7 4 6B@C?8G8 FLFG8@ G;4G <F 8�6<8AG 4A7 HF45?8 5L6HFGB@8EF EHAA<A: 8AG8ECE<F8 4CC?<64G<BAF� +<@<?4E GB @BFGBG;8E CE46G<64? FLFG8@F 7<F6HFF87 J8 BA?L 4GG8@CG GB 784?J<G; 94<?FGBC 94<?HE8F 1��3 J;<6; 4E8 F8EI8E 94<?HE8F G;4G 64A58 78G86G87 589BE8 G;8 94<?<A: F8EI8E 64HF8F 4A <A6BEE86G 8KG8EA4??L I<F<5?8 46G<BA�,;8 E8FG B9 G;8 C4C8E <F BE:4A<M87 4F 9B??BJF� �<EFG J8
78F6E<58 BHE 54F<6 78F<:A 4A7 78G4<? BHE 9HA74@8AG4? CEBGB6B?F G;4G 8AFHE8 G;4G AB 74G4 <F ?BFG <9 4 546>HC .% G4>8FBI8E 49G8E 4 CE<@4EL .% 94<?F� ,;8A J8 78F6E<58 <A 78G4<? @4AL B9 G;8 CE46G<64? <FFH8F G;4G @HFG 58 477E8FF87 GB5H<?7 4 EB5HFG 6B@C?8G8 4A7 4HGB@4G87 FLFG8@� /8 4?FB78F6E<58 F8I8E4? 78F<:A 6;B<68F G;4G 4E<F8 9BE <@C?8@8AG<A:94H?GGB?8E4AG .%F 4A7 7<F6HFF G;8 GE478B�F <A G;8F8 6;B<68F�&8KG J8 :<I8 C8E9BE@4A68 E8FH?GF 9BE BHE <@C?8@8AG4G<BA9BE FB@8 58A6;@4E>F 4A7 FB@8 E84? 8AG8ECE<F8 4CC?<64G<BAF��<A4??L J8 78F6E<58 E8?4G87 JBE> 4A7 6BA6?H78�
2. BASIC FT DESIGN�<:HE8 � F;BJF G;8 54F<6 F8GHC B9 BHE FLFG8@ 9BE 94H?G
GB?8E4AG .%F� �BE 4 :<I8A .% 9BE J;<6; J8 78F<E8 GB CEBI<7894H?G GB?8E4A68 �G;8 -.'*�.4 .%� J8 EHA 4 �!(1- .% BA4 7<�8E8AG C;LF<64? F8EI8E G;4G <F >8CG <A FLA6 4A7 8K86HG8F<78AG<64??L GB G;8 CE<@4EL I<EGH4? @46;<A8 G;BH:; J<G; 4F@4?? G<@8 ?4:� /8 F4L G;4G G;8 GJB .%F 4E8 <A 2'.01�) ),!(�/0#-� ,;8 I<EGH4? 7<F>F 9BE G;8 .%F 4E8 BA F;4E87 FGBE4:8�FH6; 4F 4 �<5E8 �;4AA8? BE <+�+! 7<F> 4EE4L� 4A7 G;8E89BE8 4668FF<5?8 GB G;8 CE<@4EL 4A7 546>HC .% 9BE <ACHG 4A7BHGCHG� �/8 J<?? 7<F6HFF 4 78F<:A <A J;<6; G;8 CE<@4EL 4A7546>HC .% ;4I8 F8C4E4G8 ABAF;4E87 I<EGH4? 7<F>F <A +86G<BA ����� 'A?L G;8 CE<@4EL .% 47I8EG<F8F <GF CE8F8A68 BAG;8 A8GJBE> FB 4?? A8GJBE> <ACHGF 6B@8 GB G;8 CE<@4EL .%�+<@<?4E?L 4?? BG;8E <ACHGF �FH6; 4F >8L5B4E7 4A7 @BHF8� :BBA?L GB G;8 CE<@4EL .%��?? <ACHG G;4G G;8 CE<@4EL .% E868<I8F <F F8AG GB G;8
546>HC .% I<4 4 A8GJBE> 6BAA86G<BA >ABJA 4F G;8 ),%%'+%!&�++#)� �BE F8EI8E JBE>?B47F G;8 7B@<A4AG <ACHG GE4�6<F A8GJBE> 4A7 7<F>� �77<G<BA4? <A9BE@4G<BA 4F 7<F6HFF8758?BJ <A +86G<BA ��� <F GE4AF@<GG87 4F A868FF4EL GB 8AFHE8G;4G G;8 546>HC .% 8K86HG8F ABA78G8E@<A<FG<6 BC8E4G<BAF<A G;8 F4@8 J4L 4F G;8 CE<@4EL .%� ,;8 E8FH?G <F G;4G G;8546>HC .% 4?J4LF 8K86HG8F <78AG<64??L GB G;8 CE<@4EL .%� BJ8I8E G;8 BHGCHGF B9 G;8 546>HC .% 4E8 7EBCC87 5LG;8 ;LC8EI<FBE FB BA?L G;8 CE<@4EL CEB7H68F 46GH4? BHGCHGFG;4G 4E8 E8GHEA87 GB 6?<8AGF� �F 78F6E<587 <A +86G<BA ��� G;8CE<@4EL 4A7 546>HC .% 9B??BJ 4 FC86<�6 CEBGB6B? <A6?H7<A:8KC?<6<G 46>ABJ?87:@8AGF 5L G;8 546>HC .% <A BE78E GB8AFHE8 G;4G AB 74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F�,B 78G86G <9 4 CE<@4EL BE 546>HC .% ;4F 94<?87 BHE FLF
G8@ HF8F 4 6B@5<A4G<BA B9 ;84EG584G<A: 58GJ88A G;8 E8?8I4AGF8EI8EF 4A7 @BA<GBE<A: B9 G;8 GE4�6 BA G;8 ?B::<A: 6;4AA8?�!A 477<G<BA J8 @HFG 8AFHE8 G;4G BA?L BA8 B9 G;8 CE<@4ELBE 546>HC .% G4>8F BI8E 8K86HG<BA 8I8A <9 G;8E8 <F 4 FC?<G5E4<A F<GH4G<BA J;8E8 G;8 CE<@4EL 4A7 546>HC F8EI8EF ;4I8?BFG 6B@@HA<64G<BA J<G; 846; BG;8E�!A G;8 9B??BJ<A: F86G<BAF J8 CEBI<78 @BE8 78G4<?F BA F8I
8E4? <@CBEG4AG 4E84F� !A +86G<BA ��� J8 :<I8 FB@8 78G4<?FBA G;8 78G8E@<A<FG<6 E8C?4L G86;AB?B:L G;4G 8AFHE8F G;4G CE<@4EL 4A7 546>HC .%F 4E8 >8CG <A FLA6 I<4 G;8 <A9BE@4G<BAF8AG BI8E G;8 ?B::<A: 6;4AA8?� !A +86G<BA ��� J8 78F6E<584 9HA74@8AG4? EH?8 B9 BHE �, CEBGB6B? G;4G 8AFHE8F G;4G AB74G4 <F ?BFG <9 G;8 CE<@4EL 94<?F� !A +86G<BA ��� J8 78F6E<58BHE @8G;B7F 9BE 78G86G<A: 4A7 E8FCBA7<A: GB 4 94<?HE8 <A 46BEE86G 94F;<BA�
2.1 Deterministic Replay Implementation�F J8 ;4I8 @8AG<BA87 E8C?<64G<A: F8EI8E �BE .%� 8K8
6HG<BA 64A 58 @B78?87 4F G;8 E8C?<64G<BA B9 4 78G8E@<A<FG<6 FG4G8 @46;<A8� !9 GJB 78G8E@<A<FG<6 FG4G8 @46;<A8F 4E8FG4EG87 <A G;8 F4@8 <A<G<4? FG4G8 4A7 CEBI<787 G;8 8K46G F4@8<ACHGF <A G;8 F4@8 BE78E G;8A G;8L J<?? :B G;EBH:; G;8 F4@8F8DH8A68F B9 FG4G8F 4A7 CEB7H68 G;8 F4@8 BHGCHGF� � I<EGH4? @46;<A8 ;4F 4 5EB47 F8G B9 <ACHGF <A6?H7<A: <A6B@<A:A8GJBE> C46>8GF 7<F> E847F 4A7 <ACHG 9EB@ G;8 >8L5B4E74A7 @BHF8� &BA78G8E@<A<FG<6 8I8AGF �FH6; 4F I<EGH4? <AG8EEHCGF� 4A7 ABA78G8E@<A<FG<6 BC8E4G<BAF �FH6; 4F E847<A:G;8 6?B6> 6L6?8 6BHAG8E B9 G;8 CEB68FFBE� 4?FB 4�86G G;8 .%�FFG4G8� ,;<F CE8F8AGF G;E88 6;4??8A:8F 9BE E8C?<64G<A: 8K86HG<BA B9 4AL .% EHAA<A: 4AL BC8E4G<A: FLFG8@ 4A7 JBE>?B47���� 6BEE86G?L 64CGHE<A: 4?? G;8 <ACHG 4A7 ABA78G8E@<A<F@A868FF4EL GB 8AFHE8 78G8E@<A<FG<6 8K86HG<BA B9 4 546>HC I<EGH4? @46;<A8 ��� 6BEE86G?L 4CC?L<A: G;8 <ACHGF 4A7 ABA78G8E@<A<F@ GB G;8 546>HC I<EGH4? @46;<A8 4A7 ��� 7B<A:FB <A 4 @4AA8E G;4G 7B8FA�G 78:E478 C8E9BE@4A68� !A 477<G<BA @4AL 6B@C?8K BC8E4G<BAF <A K�� @<6EBCEB68FFBEF ;4I8HA78�A87 ;8A68 ABA78G8E@<A<FG<6 F<78 8�86GF� �4CGHE<A:G;8F8 HA78�A87 F<78 8�86GF 4A7 E8C?4L<A: G;8@ GB CEB7H68G;8 F4@8 FG4G8 CE8F8AGF 4A 477<G<BA4? 6;4??8A:8�.%J4E8 78G8E@<A<FG<6 E8C?4L 1��3 CEBI<78F 8K46G?L G;<F
9HA6G<BA4?<GL 9BE K�� I<EGH4? @46;<A8F BA G;8 .%J4E8 I+C;8E8C?4G9BE@� �8G8E@<A<FG<6 E8C?4L E86BE7F G;8 <ACHGF B9 4 .%4A7 4?? CBFF<5?8 ABA78G8E@<A<F@ 4FFB6<4G87 J<G; G;8 .%8K86HG<BA <A 4 FGE84@ B9 ?B: 8AGE<8F JE<GG8A GB 4 ?B: �?8� ,;8.% 8K86HG<BA @4L 58 8K46G?L E8C?4L87 ?4G8E 5L E847<A: G;8?B: 8AGE<8F 9EB@ G;8 �?8� �BE ABA78G8E@<A<FG<6 BC8E4G<BAFFH�6<8AG <A9BE@4G<BA <F ?B::87 GB 4??BJ G;8 BC8E4G<BA GB 58E8CEB7H687 J<G; G;8 F4@8 FG4G8 6;4A:8 4A7 BHGCHG� �BEABA78G8E@<A<FG<6 8I8AGF FH6; 4F G<@8E BE !' 6B@C?8G<BA <A
31
VMM-based Primary Backup
¨ Primary and backup execute on two virtual machines
¨ Primary logs inputs and outputs ¨ Backup applies inputs from log ¨ Primary waits for backup output
¨ Primary-backup monitor each other ¤ If primary fails, backup takes over
¨ VMM at primary logs external inputs and causes of non-determinism
¨ Example: Log results of non-deterministic instructions ¤ e.g., timestamp counter read (RDTSC)
¨ Random number generation?
Log-based VM replication 18
¨ VMM at primary sends log entries to VMM at backup over the logging channel
¨ Backup hypervisor replays log entries ¤ Stops backup VM at next input event or non-deterministic instruction
¤ Delivers same input as primary ¤ Delivers same result to non-deterministic instruction as primary
Log-based VM replication 19
¨ VM inputs ¤ Incoming network packets ¤ Disk reads ¤ Keyboard and mouse events ¤ Clock timer interrupt events ¤ Results of non-deterministic instructions
¨ VM outputs ¤ Outgoing network packets ¤ Disk writes
¨ Why log outputs?
Virtual Machine I/O 20
Example Scenario
Primary
¨ Write input to log ¨ Apply input ¨ Produce output ¨ Fail!
Backup
¨ Take over as primary ¨ Read from log and
apply input ¨ Timer interrupt fires ¨ Produce output
Backup does not know timing of output relative to timer interrupt
Execution of interrupt handler may affect output
Optimizing Performance
¨ Primary must buffer output until ACK from backup ¤ Why? ¤ Slow from client perspective!
n If replicas are not in close proximity.
¨ How to optimize performance? ¨ Pipeline sync with backup and execution at primary
22
VMM-based Replication 23
C
P
B
Primary Backup Replication
¨ Promote one of the backups if primary fails ¨ Replace any failed backup
¨ When should primary sync with backups? ¤ Before making state change externally visible ¤ Primary and backups must be externally consistent
¨ What to sync? ¤ Entire state when bootstrapping new backup ¤ Thereafter, all sources of non-determinism
24
RSM with Primary Backup 25
Operating System
Application
Operating System
Application
Server1 Server2
Virtual Machine Monitor
Virtual Machine Monitor
Hardware Hardware
Client perspective
¨ What does a worker need to know in order to register itself with replicated master?
¨ Needs to know which machine is primary
26
Primary Backup Replication
Client Primary Backup
Backup
Backup
27
Client perspective
¨ What does a worker need to know in order to register itself with replicated master?
¨ Needs to know which machine is primary
¨ Can primary be hard coded into client code? ¨ No, primary gets replaced when it fails
¨ How does client discover current primary?
28
Primary Backup Replication
Client Primary Backup
Backup
Backup
View service
29
View service
¨ Maintains current membership of primary-backup service (called view) ¤ View number, primary, backup
¨ When does view service change view? ¨ When primary or any backup fails ¨ Periodically exchange heartbeat messages to
detect failures
¨ What if view service is down or not reachable?
30
Transitioning between views
¨ How to ensure new primary is up-to-date? ¤ Only promote a previous backup ¤ This is why view service needs to pick backups
¨ How does view service know if a backup is up-to-date?
¨ Two scenarios for ill-timed primary failure: ¤ Primary applies operation but fails before syncing with
backup ¤ Primary fails before new backup is initialized
31
Transitioning between views
¨ View service broadcasts view change to all
¨ Primary must ACK new view once backup is up-to-date
¨ Two implications: ¤ Liveness detection timeout > State transfer time ¤ Cannot change view if primary fails during sync
32
View service
¨ View change has three steps: ¤ View service announces new view ¤ Primary syncs with new backup if there is one ¤ Primary acknowledges new view
¨ Stuck if primary fails in the midst of this process
33
Scalability of View service
¨ Does every client need to contact view service before any operation?
¨ Clients can cache view across operations
¨ When to invalidate cached view? ¨ Client invalidates cache when no response or
negative response from primary
34
Split Brain
Client
S1
S2
View service (1,S1, _) (2,S1,S2) (3,S2, _)
(1,S1, _) (2,S1,S2)
(2,S1,S2) (3,S2, _)
(2,S1,S2)
35
Avoiding Split Brain
¨ Primary must forward all operations to backups ¤ Goal: Get ACKs from backups that they too recognize
primary
¨ Why can’t backups be mistaken about who is primary? ¤ Only a backup can be promoted as primary
36
View service
¨ Valid sequence of views: ¤ (1, S1, _) à (2, S1, S2) à (3, S1, S3) à (4, S3, S4) à (5, S4, _)
¨ Examples of invalid transitions between views? ¤ (1, S1, S2) à (2, S3, S4) ¤ (1, S1, S2) à (2, _, S2) ¤ (1, S1, _) à (2, S2, S1)
37
Summary of view service
¨ Monitors primary and backups to detect when to change view ¤ Can change only after primary has ACKed view ¤ Primary ACKs only after syncing with backups
¨ Clients cache view for scalability ¨ To address split brain, primary must check with
backup before serving client
38
¨ One copy in SF (primary), one in NY (backup)
Replicating Bank Database
“Deposit $100”
“Pay 1% interest”
$1,000 $1,000
39
Primary-Backup Sync 40
C1
C2
P
B
“Deposit $100”
“Pay 1% interest”
$1,000
$1,000 $1,110
$1,111
Ordering of Updates
¨ All updates must be applied in the same order at all replicas
¨ External view: Total ordering of writes
¨ Primary effectively serializes all writes ¤ Order of events made known to replicas
41
Serving Reads
¨ Can backups serve reads? ¤ Assume no split brain
¨ What if primary’s state is ahead of backup? ¤ Updates to primary not yet externally visible ¤ Effect of read equivalent to if primary fails at this point
¨ What if backup’s state is ahead of primary? ¤ Different backups may not be in sync ¤ Primary may get replaced before it applies update
42
Reads: Primary vs. Backup 43
C1
P
C2
B1
“Deposit $100”
$1100 $1000
B2
Desired Properties
¨ All writes are totally ordered
¨ Once read returns particular value, all later reads should return that value or value of later write
¨ Once a write completes, all later reads should return value of that write or value of later write
44
Reads relative to Writes 45
C1
P
C2
B
“Deposit $100”
“Pay 1% interest”
$1100 $1111
Consistency Spectrum
¨ Consistency: What are the properties of externally visible effects?
46
Linearizability Sequential Causal Eventual
Consistency
Read-after- write
Ease of programming
Linearizability
¨ Total ordering of writes ¨ Read returns last completed write
¨ Single copy semantics ¤ Externally visible effects of writes and reads are
equivalent to if there existed a single copy
¨ Users oblivious to replication
47
Why weaken consistency?
¨ Shouldn’t we always strive for single copy semantics? ¤ Comes at the expense of lower performance
¨ Latency vs. consistency tradeoff
48
Consistency Spectrum 49
Linearizability Sequential Causal Eventual
Consistency
Read-after- write
Ease of programming
Latency
Sequential Consistency
¨ Results of any execution same as that when the operations by all processes on the data store were executed in some sequential order.
¨ And … the operations of each individual process appear in this sequence in the order specified by its program.
Example
¨ (a) is sequentially consistent. All processes see the same interleaving of the operations (x takes value b before a)
¨ (b) is not sequentially consistent. P3 sees x taking on value b before a while P4 sees the opposite.
Causal Consistency
¨ Order of causally related writes must be preserved in values returned to reads ¤ If W1 à W2, then if a read sees effect of W2, it must
see effect of W1
¨ Example: Facebook News Feed ¤ Okay to not see all completed posts ¤ But, if you see a comment, you must see the post on
which the comment is made ¨ Main utility: Lazy sync between replicas
52
Example
¨ The operations above do not result in sequential consistency (see reads of P3 and P4)
¨ But there is causal consistency. This is because the writes W1(x)c and W2(x)b are concurrent (there are no dependencies and thus no causal relationship).
¨ But note that W1(x)a and W2(x)b are causally related since R2(x)a may be the reason for W2(x)b.
Example
¨ (a) is not causally consistent. R3(x)b cannot follow R3(x)a since the the write of b to x is dependent on P2 reading x’s value to be a.
¨ (b) however is causally consistent since W1(x) a and W2(x) b are concurrent.
Linearizability example
¨ Every operation should appear to take effect instantaneously at some moment before its start and completion (sometime in the shaded area)
¨ Thus, the result is consistent regardless of ordering of operations
Linearizability with Locks
Client Replica 2
Replica 1
Replica 3
Lock service
56
Problems?
Client failures!
Lease
¨ Lock with timeout ¨ If lease holder fails, not a problem because lease
will expire
¨ How to pick lease timeout value? ¤ Short timeout à Client needs to renew lease ¤ Long timeout à Unnecessarily block operations
57
Discrepancy in Lease Validity
Client Replica 2
Replica 1
Replica 3
Lease service
58
Scenario in which lease server and client
differ about lease validity?
Discrepancy in Lease Validity
¨ Message that grants lease may have high delay
¨ Clock at lease holder and lease service may have different skew
¨ How to account for potential discrepancy?
59
Discrepancy in Lease Validity
Client Replica 2
Replica 1
Replica 3
Lease service
60
Replica must check with lease service to confirm
lease validity