PyCon Ukraine 2017: Operational Transformation
-
Upload
max-klymyshyn -
Category
Software
-
view
263 -
download
4
Transcript of PyCon Ukraine 2017: Operational Transformation
Operational transformation for fun with python
Max Klymyshyn CTO at CartFresh
What is Operational Transformation
It’s gonna be HUGE.
What is Operational Transformation
Technology or family of algorithms for advanced
collaborative software systems
OT is one of approaches in overall data
synchronization domain
OT is backbone for popular products like Google
Docs and famous failures like Google Wave
Why Operational Transformation
High-latency environments like cellular networks,
bad WIFI connections, packet losses etc.
Real-time collaboration systems like Google
Docs, collaborative text editors
The Idea
Assumption
OT optimisticly assumes that whatever operations
are currently being applied to the document on any given client will not conflict with any operations that might be applied at this same moment by one of the other clients.
Variables
G is groupware system S is set of sites (S1, S2 ..) O is set of parametrized operators:
O1 = insert[X; P] insert X at position P O2 = delete[P]
p request priority
Naïve data flow schema
SITE ATIME
SITE B
T1 T1
O[A] = insert(0, “T2”) O[B] = insert(2, “T3”)
T2,T1 T1,T3
T2,T3,T1 T2,T1,T3RECONCILIATION
LOCAL OPS
O[A] = insert(0, “T2”)O[B] = insert(2, “T3”)
Properties
The Precedence property –execution of
operations should be in same order on all sites
The Convergence Property – all objects are
identical at all sites at quiescence
In order to achieve convergence property order of the operations on all sites should be the same.
OT Functions (Matrix)
Given operations oi and oj with priorities pi and
pj respectively
We’re defining transformation function that o’i = T(oj, oi, pj, pi)
o’j = T(oi, oj, pi, pj)
T(oi, oj) = (o’i, o’j)
where
oj’ ∘ oi = oj ∘ o’i
Data flow schema with OT
SITE ATIME
SITE B
T1 T1
O[A] = insert(0, “T2”) O[B] = insert(2, “T3”)
T2,T1 T1,T3
RECONCILIATION
LOCAL OPS
T2,T1,T3 T2,T1,T3
O[A] = insert(0, “T2”)O[B] = insert(2, “T3”)
O[A]’ = T(O[A] , O[B]) = insert(0, “T2”)
O[B]’ = T(O[B], O[A]) = insert(4, “T3”) TRANSFORMATION
Existing Algorithms
dOPT (GROVE)
GOT
GOTO
STD
SOCT
ABT
But here is the problem
1 2
3 4
Let’s build some stuff
CHANNEL
SITE 1..N
BROADCASTRECEIVE
def execute(self): """dOPT implementation"""
for site_id, state, op, priority in self.queue: # remote less than local, transformation required
… APPLY TRANSFORMATION …
self.apply_commands.append(op) self.log.append([site_id, state, op, priority]) if not site_id in self.state: self.state[site_id] = 0 self.state[site_id] += 1
self.q = []
if self.state.get(site_id, 0) > state.get(site_id, 0): entry = self.log.pop()
while entry and op: k_site_id, k_state, k_op, k_priority = entry
# we don't need to transform operations from same site if k_site_id == site_id: entry = self.log.pop() if self.log else [] continue
if state.get(k_site_id, 0) <= k_state.get(k_site_id, 0): op = self.transform(op, k_op, priority, k_priority)
entry = self.log.pop() if self.log else []
def demo(args): conn = Channel() site1 = Site(1, conn) site2 = Site(2, conn) site3 = Site(3, conn) sites = [site1, site2, site3] conn.register(1, site1) conn.register(2, site2) conn.register(3, site3)
site1.generate(["T1", 0]) # T1 site2.generate(["T3", 2]) # T1T3 site2.generate(["T4", 2]) # T1T4T3 site3.generate(["T2", 2]) # T1T2T4T3 site1.generate(["T0", 0]) # T0T1T2T4T3
[s.execute() for s in sites] [s.show_state() for s in sites]
[CHANNEL] Site 1 registered [CHANNEL] Site 2 registered [CHANNEL] Site 3 registered [EXEC] Site 1 [TRANSFORM] (site 2, 1) ==> ['T4', 2] ['T1', 0] [TRANSFORM] [RESULT] ['T4', 2] [EXEC] Site 2 [EXEC] Site 3 [FINAL STATE] Site #1:
T0T1T2T4T3
[FINAL STATE] Site #2:
T0T1T2T4T3
[FINAL STATE] Site #3:
T0T1T2T4T3
Operational Transformation today
Operations Composition –commands with same
length or other similarities merged into one command
Wave protocol, Jupiter realtime messaging etc.
ShareJS (javascript) etc.
Conclusion
Operational Transformation drawbacks
Huge complexity of transformation functions
I suspect it might be easier using prover with code generation but this idea should be verified
There’s A LOT of papers about OT and it’s hard
to find the shiny one
Joseph Gentle who is a former Google Wave engineer
and an author of the Share.JS library wrote,
"Unfortunately, implementing OT sucks. There's a million algorithms with different tradeoffs, mostly trapped in academic papers. The algorithms are really
hard and time consuming to implement correctly. […] Wave took 2 years to write and if we rewrote it today,
it would take almost as long to write a second time."
Data Synchronization problem
CRDT - Commutative or Convergent Replicated Data Types (and whole family called conflict-free replicated data types)
Differential Synchronization
RAFT, PAXOS and other quorum-based protocols
Thanks!
@maxmaxmaxmax
https://github.com/joymax/uapycon2017-op-demo
Stuffhttps://www.lri.fr/~mbl/ENS/CSCW/2012/papers/Ellis-SIGMOD89.pdf
http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation
https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type#Eventual_consistency
http://fitzgeraldnick.com/2011/03/26/operational-transformation-an-introduction.html
https://github.com/jvanveen/py-infinote
https://pdfs.semanticscholar.org/8112/803b7a72ba8fcbe7de3a2a3f3cec00fa9e80.pdf
https://en.wikipedia.org/wiki/Collaborative_software
https://en.wikipedia.org/wiki/Optimistic_replication
https://en.wikipedia.org/wiki/Data_synchronization