Server side data sync for mobile apps with silex
-
Upload
michele-orselli -
Category
Software
-
view
1.447 -
download
0
Transcript of Server side data sync for mobile apps with silex
![Page 1: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/1.jpg)
Implementing data synchronization API for mobile apps with Silex
![Page 2: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/2.jpg)
Michele OrselliCTO@Ideato
_orso_
micheleorselli / ideatosrl
![Page 3: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/3.jpg)
Agenda
scenario design choices
implementation alternative approaches
![Page 4: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/4.jpg)
![Page 5: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/5.jpg)
![Page 6: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/6.jpg)
Sync scenario
A
B
C
![Page 7: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/7.jpg)
Sync scenario
ABC
ABC
ABC
![Page 8: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/8.jpg)
Dealing with conflicts
A1
A2
?
![Page 9: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/9.jpg)
Brownfield project
several mobile apps for tracking user generated data (calendar, notes, bio data)
iOS & Android
~10 K users steadily growing at 1.2 K/month
Scenario
![Page 10: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/10.jpg)
MongoDB
Legacy App based on Codeigniter
Existing RPC-wannabe-REST API for data sync
Scenario
![Page 11: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/11.jpg)
For every resource
get updates:
POST /m/:app/get/:user_id/:res/:updated_from
send updates:
POST /m/:app/update/:user_id/:res_id/:dev_id/:res
Scenario
![Page 12: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/12.jpg)
api
![Page 13: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/13.jpg)
~6 different resources, ~12 calls per sync
apps sync by polling every 30 sec
every call sync little data
Scenario
![Page 14: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/14.jpg)
Rebuild sync API for old apps + 2 incoming
Enable image synchronization
More efficient than previous API
Challenge
![Page 15: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/15.jpg)
![Page 16: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/16.jpg)
Existing Solutions
Tstamps, Vector clocks,
CRDTs
syncML, syncano
Azure Data sync
Algorithms Protocols/API
Platform
couchDB, riak
Storage
![Page 17: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/17.jpg)
Not Invented Here?
Don't Reinvent The Wheel,Unless You Plan on Learning More About Wheels
J. Atwood
![Page 18: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/18.jpg)
2 different mobile platforms
Several teams with different skill level
Changing storage wasn’t an option
Forcing a particular technology client side wasn’t an option
Architecture
![Page 19: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/19.jpg)
Architecture
c1
server
c2
c3
sync logicconflicts resolution
thin clients
![Page 20: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/20.jpg)
In the sync domain all resources are managed in the same way
Implementation
![Page 21: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/21.jpg)
For every app:
one endpoint for getting new data
one endpoint for pushing changes
one endpoint for uploading images
Implementation
![Page 22: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/22.jpg)
GET /apps/:app/users/:user_id/changes[?from=:from]
POST /apps/:app/users/:user_id/merge
POST /upload/:res_id/images
The new APIs
![Page 23: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/23.jpg)
Silex Implementation
![Page 24: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/24.jpg)
Silex Implementation
Col 1
Col 2
Col 3
![Page 25: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/25.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 26: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/26.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 27: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/27.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 28: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/28.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 29: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/29.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 30: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/30.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 31: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/31.jpg)
Silex Implementation
Col 1
Col 2
Col 3
Sync Service
![Page 32: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/32.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/changes”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null); $syncService = $app[‘syncService’];$syncService->sync($lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 33: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/33.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/changes”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null); $syncService = $app[‘syncService’];$syncService->sync($lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 34: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/34.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/changes”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null); $syncService = $app[‘syncService’];$syncService->sync($lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 35: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/35.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/changes”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null); $syncService = $app[‘syncService’];$syncService->sync($lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 36: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/36.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null);$data = $request->get(‘data’, false);$syncService = $app[‘syncService’];$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 37: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/37.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null);$data = $request->get(‘data’, false);$syncService = $app[‘syncService’];$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 38: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/38.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null);$data = $request->get(‘data’, false);$syncService = $app[‘syncService’];$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 39: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/39.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null);$data = $request->get(‘data’, false);$syncService = $app[‘syncService’];$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 40: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/40.jpg)
Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/merge”,
function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null);$data = $request->get(‘data’, false);$syncService = $app[‘syncService’];$syncService->merge($data, $lastSync, $userId);
$response = new JsonResponse($syncService->getResult()
);
return $response;}
![Page 41: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/41.jpg)
Silex Implementation
$app['mongodb'] = new MongoDb(…);
$app[‘changesRepo’] = new ChangesRepository( $app[‘mongodb’]);
$app[‘syncService’] ? new SyncService( $app[‘changesRepo’]);
![Page 42: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/42.jpg)
GET /apps/:app/users/:user_id/changes?from=:from
Get changes
timestamp?
![Page 43: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/43.jpg)
timestamp are inaccurate
server suggests the “from” parameter to be used in the next request
Server suggest the sync time
![Page 44: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/44.jpg)
Server suggest the sync time
c1 server
GET /changes
{ ‘next’ : 12345, ‘data’: […] }
![Page 45: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/45.jpg)
Server suggest the sync time
c1 server
GET /changes
{ ‘next’ : 12345, ‘data’: […] }
GET /changes?from=12345
{ ‘next’ : 45678, ‘data’: […] }
![Page 46: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/46.jpg)
operations:{‘op’: ’add’, id: ‘1’, ’data’:[…]}{‘op’: ’update’, id: ‘1’, ’data’:[…]}{‘op’: ’delete’, id: ‘1’}{‘op’: ’add’, id: ‘2’, ’data’:[…]}
states:{id: ‘1’, ’data’:[…]}{id: 2’, ’data’:[…]}{id: ‘3’, ’data’:[…]}
what to transfer
![Page 47: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/47.jpg)
we choose to transfer states {id: ‘1’, ’type’: ‘measure’, ‘_deleted’: true}{id: 2’, ‘type’: ‘note’}{id: ‘3’, ‘type’: ‘note’}
ps: soft delete all the things!
what to transfer
![Page 48: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/48.jpg)
How do we generate an unique id in a distributed system?
unique identifiers
![Page 49: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/49.jpg)
How do we generate an unique id in a distributed system?
UUID (RFC 4122): several implementations in PHP (https://github.com/ramsey/uuid)
unique identifiers
![Page 50: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/50.jpg)
How do we generate an unique id in a distributed system?
Local/Global Id: only the server generates GUIDsclients use local ids to manage their records
unique identifiers
![Page 51: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/51.jpg)
unique identifiers
c1 server
POST /merge{ ‘data’: [ {’lid’: ‘1’, …}, {‘lid’: ‘2’, …}] }
{ ‘data’: [ {‘guid’: ‘58f0bdd7-1400’, ’lid’: ‘1’, …}, {‘guid’: ‘6f9f3ec9-1400’, ‘lid’: ‘2’, …}] }
![Page 52: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/52.jpg)
mobile generated data are “temporary” until sync to server
server handles conflicts resolution
conflict resolution algorithm (plain data)
![Page 53: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/53.jpg)
conflict resolution:
domain indipendent: e.g. last-write wins
domain dipendent: use domain knowledge to resolve
conflict resolution algorithm (plain data)
![Page 54: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/54.jpg)
function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {add($newRecord);send($newRecord);continue;
}
if ($newRecord->updated > $s->updated) {update($s, $newRecord);send($newRecord);continue;
}
updateRemote($newRecord, $s);}
conflict resolution algorithm (plain data)
![Page 55: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/55.jpg)
function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {add($newRecord);send($newRecord);continue;
}
if ($newRecord->updated > $s->updated) {update($s, $newRecord);send($newRecord);continue;
}
updateRemote($newRecord, $s);}
conflict resolution algorithm (plain data)
![Page 56: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/56.jpg)
function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {add($newRecord);send($newRecord);continue;
}
if ($newRecord->updated > $s->updated) {update($s, $newRecord);send($newRecord);continue;
}
updateRemote($newRecord, $s);}
conflict resolution algorithm (plain data)
no conflict
![Page 57: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/57.jpg)
function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {add($newRecord);send($newRecord);continue;
}
if ($newRecord->updated > $s->updated) {update($s, $newRecord);send($newRecord);continue;
}
updateRemote($newRecord, $s);}
conflict resolution algorithm (plain data)
remote wins
![Page 58: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/58.jpg)
function sync($data) {
foreach ($data as $newRecord) {
$s = findByGuid($newRecord->getGuid());
if (!$s) {add($newRecord);send($newRecord);continue;
}
if ($newRecord->updated > $s->updated) {update($s, $newRecord);send($newRecord);continue;
}
updateRemote($newRecord, $s);}
conflict resolution algorithm (plain data)
server wins
![Page 59: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/59.jpg)
conflict resolution algorithm (plain data)
c1
{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
server
{ ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ }
![Page 60: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/60.jpg)
conflict resolution algorithm (plain data)
c1 server
{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge
{ ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ }
![Page 61: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/61.jpg)
conflict resolution algorithm (plain data)
c1 server
{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge
{ ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
{ ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ }
![Page 62: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/62.jpg)
conflict resolution algorithm (plain data)
c1 server
{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge
{ ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
{ ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ }
![Page 63: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/63.jpg)
conflict resolution algorithm (plain data)
c1 server
{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge
{ ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
{ ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ }
![Page 64: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/64.jpg)
conflict resolution algorithm (plain data)
c1 server
{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ }
{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge
{ ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
{‘ok’ : { ’guid’: ‘af54d’ }}
{‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
![Page 65: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/65.jpg)
conflict resolution algorithm (hierarchical data)
How to manage hierarchical data?
{‘lid’ : ‘123456’,‘type’ : ‘baby’, …
}
{‘lid’ : ‘123456’,‘type’ : ‘temperature’, ‘baby_id : ‘123456’
}
![Page 66: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/66.jpg)
conflict resolution algorithm (hierarchical data)
How to manage hierarchical data?1) sync root record2) update ids3) sync child records
{‘lid’ : ‘123456’,‘type’ : ‘baby’, …
}
{‘lid’ : ‘123456’,‘type’ : ‘temperature’, ‘baby_id : ‘123456’
}
![Page 67: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/67.jpg)
function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {add($newRootRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;
}
…
conflict resolution algorithm (hierarchical data)
![Page 68: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/68.jpg)
function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {add($newRootRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;
}
…
conflict resolution algorithm (hierarchical data)
parent records first
![Page 69: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/69.jpg)
function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {add($newRootRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;
}
…
conflict resolution algorithm (hierarchical data)
![Page 70: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/70.jpg)
function syncHierarchical($data) {
sortByHierarchy($data);
foreach ($data as $newRootRecord) {
$s = findByGuid($newRootRecord->getGuid());
if($newRecord->isRoot()) {
if (!$s) {add($newRootRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;
}
…
conflict resolution algorithm (hierarchical data)
no conflict
![Page 71: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/71.jpg)
…
if ($newRootRecord->updated > $s->updated) {update($s, $newRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;
} else {updateRecordIds($s, $data);updateRemote($newRecord, $s);
}} else {
sync($data);}
}
conflict resolution algorithm (hierarchical data)
remote wins
![Page 72: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/72.jpg)
…
if ($newRootRecord->updated > $s->updated) {update($s, $newRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;
} else {updateRecordIds($s, $data);updateRemote($newRecord, $s);
}} else {
sync($data);}
}
conflict resolution algorithm (hierarchical data)
server wins
![Page 73: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/73.jpg)
conflict resolution algorithm (hierarchical data)
{ ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
c1 serverPOST /merge
![Page 74: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/74.jpg)
conflict resolution algorithm (hierarchical data)
c1
{ ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
serverPOST /merge
{ ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
![Page 75: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/75.jpg)
conflict resolution algorithm (hierarchical data)
c1
{ ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
serverPOST /merge
{ ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
![Page 76: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/76.jpg)
conflict resolution algorithm (hierarchical data)
c1
{ ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
serverPOST /merge
{ ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }
{ ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }
{‘update’ : { ‘lid’: ‘1’, ’guid’: ‘af54d’ }}
{‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}
![Page 77: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/77.jpg)
e.g. “only one temperature can be registered in a given day”
how to we enforce domain constraints on data?
enforcing domain constraints
![Page 78: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/78.jpg)
e.g. “only one temperature can be registered in a given day”
how to we enforce domain constraints on data?1) relax constraints
enforcing domain constraints
![Page 79: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/79.jpg)
e.g. “only one temperature can be registered in a given day”
how to we enforce domain constraints on data?1) relax constraints2) integrate constraints in sync algorithm
enforcing domain constraints
![Page 80: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/80.jpg)
from findByGuid to findSimilar
first lookup by GUID then by domain rules
“two measures are similar if are referred to the same date”
enforcing domain constraints
![Page 81: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/81.jpg)
enforcing domain constraints
c1 server
![Page 82: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/82.jpg)
enforcing domain constraints
c1 server
{ ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
![Page 83: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/83.jpg)
enforcing domain constraints
c1 server
{ ‘lid’: ‘1’, ‘when’: ‘20141005’ }
{ ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
![Page 84: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/84.jpg)
enforcing domain constraints
c1 server
{ ‘lid’: ‘1’, ‘when’: ‘20141005’ }
{ ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
POST /merge
![Page 85: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/85.jpg)
enforcing domain constraints
c1 server
{ ‘lid’: ‘1’, ‘when’: ‘20141005’ }
{ ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
POST /merge
![Page 86: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/86.jpg)
enforcing domain constraints
c1 server
{ ‘lid’: ‘1’, ‘when’: ‘20141005’ }
{ ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
POST /merge
{ ’guid’: ‘af54d’, ‘when’: ‘20141005’ }
![Page 87: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/87.jpg)
Binary data uploaded via custom endpoint
Sync data remains small
Uploads can be resumed
dealing with binary data
![Page 88: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/88.jpg)
Two steps*1) data are synchronized2) related images are uploaded
* this means record without file for a given time
dealing with binary data
![Page 89: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/89.jpg)
dealing with binary data
c1 server
POST /merge
POST /upload/ac435-f8345/image
{ ‘lid’ : 1, ‘type’ : ‘baby’, ‘image’ : ‘myimage.jpg’ }
{ ‘lid’ : 1, ‘guid’ : ‘ac435-f8345’ }
![Page 90: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/90.jpg)
Implementing this stuff is tricky
Explore existing solution if you can
Understanding the domain is important
What we learned
![Page 91: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/91.jpg)
vector clocks
![Page 92: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/92.jpg)
vector clocks
![Page 93: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/93.jpg)
Conflict-free Replicated Data Types (CRDTs)
Constraining the types of operations in order to:
- ensure convergence of changes to shared data by uncoordinated, concurrent actors
- eliminate network failure modes as a source of error
CRDT
![Page 94: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/94.jpg)
Gateways handles sync
Data flows through channels
- partition data set
- authorization
- limit the data
Use revision trees
Couchbase Mobile
![Page 95: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/95.jpg)
Distributed DBEventually/Strong Consistency
Data Types
Configurable conflict resolution- db level for built-in data types- application level for custom data
Riak
![Page 97: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/97.jpg)
http://www.objc.io/issue-10/sync-case-study.htmlhttp://www.objc.io/issue-10/data-synchronization.html
https://dev.evernote.com/media/pdf/edam-sync.pdfhttp://blog.helftone.com/clear-in-the-icloud/
http://strongloop.com/strongblog/node-js-replication-mobile-offline-sync-loopback/http://blog.denivip.ru/index.php/2014/04/data-syncing-in-core-data-based-ios-apps/?lang=en
http://inessential.com/2014/02/15/vesper_sync_diary_8_the_problem_of_unhttp://culturedcode.com/things/blog/2010/12/state-of-sync-part-1.htmlhttp://programmers.stackexchange.com/questions/206310/data-synchronization-in-mobile-apps-multiple-devices-multiple-users
http://bricklin.com/offline.htmhttp://blog.couchbase.com/why-mobile-sync
Links
![Page 98: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/98.jpg)
Vector Clockshttp://basho.com/why-vector-clocks-are-easy/http://www.datastax.com/dev/blog/why-cassandra-doesnt-need-vector-clockshttp://basho.com/why-vector-clocks-are-hard/http://blog.8thlight.com/rylan-dirksen/2013/10/04/synchronization-in-a-distributed-system.html
CRDTshttp://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems.htmlhttp://www.infoq.com/presentations/problems-distributed-systemshttps://www.youtube.com/watch?v=qyVNG7fnubQ
Riak http://docs.basho.com/riak/latest/dev/using/conflict-resolution/
Couchbase Sync Gatewayhttp://docs.couchbase.com/sync-gateway/http://www.infoq.com/presentations/sync-mobile-data
APIhttp://developers.amiando.com/index.php/REST_API_DataSynchttps://login.syncano.com/docs/rest/index.html
Links
![Page 99: Server side data sync for mobile apps with silex](https://reader033.fdocuments.us/reader033/viewer/2022052913/55a249851a28ab932b8b4778/html5/thumbnails/99.jpg)
phones https://www.flickr.com/photos/15216811@N06/14504964841wat http://uturncrossfit.com/wp-content/uploads/2014/04/wait-what.jpgdarth http://www.listal.com/viewimage/3825918hblueprint: http://upload.wikimedia.org/wikipedia/commons/5/5e/Joy_Oil_gas_station_blueprints.jpgbuilding: http://s0.geograph.org.uk/geophotos/02/42/74/2427436_96c4cd84.jpgbrownfield: http://s0.geograph.org.uk/geophotos/02/04/54/2045448_03a2fb36.jpgno connection: https://www.flickr.com/photos/77018488@N03/9004800239no internet con https://www.flickr.com/photos/roland/9681237793vector clocks: http://en.wikipedia.org/wiki/Vector_clockcrdts: http://www.infoq.com/presentations/problems-distributed-systems
Credits