Implementing data sync apis for mibile apps @cloudconf

Implementing data synchronization API for

mobile apps

Michele OrselliCTO@Ideato

micheleorselli / ideatosrl

_orso_

mo@ideato.it

Agenda

scenario design choices

implementation alternative approaches

Dealing with conflicts

Brownfield project

several mobile apps for tracking user generated data (calendar, notes, bio data)

iOS & Android

~10 K users steadily growing at 1.2 K/month

Scenario

MongoDB

Legacy App based on Codeigniter

Existing RPC-wannabe-REST API for data sync

Scenario

For every resource

get updates:

POST /m/:app/get/:user_id/:res/:updated_from

create/send updates:

POST /m/:app/update/:user_id/:res_id/:dev_id/:res

Scenario

~6 different resources, ~12 calls per sync

apps sync by polling every 30 sec

every call sync little data

Scenario

Rebuild sync API for old apps + 2 incoming

Enable image synchronization

More efficient than previous API

Challenge

Existing Solutions

Tstamps, Vector clocks,

syncML, syncano

Azure Data sync

Algorithms Protocols/API

Platform

couchDB, riak

Storage

Not Invented Here?

Don't Reinvent The Wheel,Unless You Plan on Learning More About Wheels

J. Atwood

2 different mobile platforms

Several teams with different skill level

Changing storage wasn’t an option

Forcing a particular technology client side wasn’t an option

Architecture

server

sync logicconflicts resolution

thin clients

In the sync domain all resources are managed in the same way

Implementation

For every app:

one endpoint for getting new data

one endpoint for pushing changes

one endpoint for uploading images

Implementation

GET /apps/:app/users/:user_id/changes[?from=:from]

POST /apps/:app/users/:user_id/merge

POST /upload/:res_id/images

The new APIs

Silex Implementation

Sync Service

Silex Implementation$app->get(“/apps/{mApp}/users/{userId}/merge”,

function ($mApp, $userId, $app, $request){$lastSync = $request->get('from', null);$data = $request->get(‘data’, false);$syncService = $app[‘syncService’];$syncService->merge($data, $lastSync, $userId);

$response = new JsonResponse($syncService->getResult()

return $response;}

$app['mongodb'] = new MongoDb(…);

$app[‘changesRepo’] = new ChangesRepository( $app[‘mongodb’]);

$app[‘syncService’] ? new SyncService( $app[‘changesRepo’]);

GET /apps/:app/users/:user_id/changes?from=:from

Get changes

timestamp?

timestamp are inaccurate

server suggests the “from” parameter to be used in the next request

Server suggest the sync time

c1 server

GET /changes

{ ‘next’ : 12345, ‘data’: […] }

Server suggest the sync time

c1 server

GET /changes

{ ‘next’ : 12345, ‘data’: […] }

GET /changes?from=12345

{ ‘next’ : 45678, ‘data’: […] }

data format {id: ‘1’, ’type’: ‘measure’, ‘_deleted’: true}{id: 2’, ‘type’: ‘note’}{id: ‘3’, ‘type’: ‘note’}

ps: soft delete all the things!

what to transfer

How do we generate an unique id in a distributed system?

unique identifiers

UUID (RFC 4122): several implementations in PHP (https://github.com/ramsey/uuid)

unique identifiers

Local/Global Id: only the server generates GUIDsclients use local ids to manage their records

unique identifiers

c1 server

POST /merge{ ‘data’: [ {’lid’: ‘1’, …}, {‘lid’: ‘2’, …}] }

{ ‘data’: [ {‘guid’: ‘58f0bdd7-1400’, ’lid’: ‘1’, …}, {‘guid’: ‘6f9f3ec9-1400’, ‘lid’: ‘2’, …}] }

mobile generated data are “temporary” until sync to server

server handles conflicts resolution

conflict resolution algorithm (plain data)

conflict resolution:

domain indipendent: e.g. last-write wins

domain dipendent: use domain knowledge to resolve

function sync($data) {

foreach ($data as $newRecord) {

$s = findByGuid($newRecord->getGuid());

if (!$s) {add($newRecord);send($newRecord);continue;

if ($newRecord->updated > $s->updated) {update($s, $newRecord);send($newRecord);continue;

updateRemote($newRecord, $s);}

no conflict

remote wins

server wins

{ ‘lid’: ‘1’, ‘guid’: ‘af54d’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }

{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }

server

{ ’guid’: ‘af54d’, ‘data’: ‘BBB’, ‘updated’ : ’20’ }

c1 server

{ ‘lid’: ‘2’, ‘data’ : ‘hello!’, ‘updated’: ’15’ } POST /merge

c1 server

{ ‘guid’: ‘e324f’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }

c1 server

{ ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ }

c1 server

{ ’guid’: ‘af54d’, ‘data’: ‘AAA’, ‘updated’ : ’100’ }

{‘ok’ : { ’guid’: ‘af54d’ }}

{‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}

conflict resolution algorithm (hierarchical data)

How to manage hierarchical data?

{‘lid’ : ‘123456’,‘type’ : ‘baby’, …

{‘lid’ : ‘123456’,‘type’ : ‘temperature’, ‘baby_id : ‘123456’

How to manage hierarchical data?1) sync root record2) update ids3) sync child records

{‘lid’ : ‘123456’,‘type’ : ‘baby’, …

{‘lid’ : ‘123456’,‘type’ : ‘temperature’, ‘baby_id : ‘123456’

function syncHierarchical($data) {

sortByHierarchy($data);

foreach ($data as $newRootRecord) {

$s = findByGuid($newRootRecord->getGuid());

if($newRecord->isRoot()) {

if (!$s) {add($newRootRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;

parent records first

no conflict

if ($newRootRecord->updated > $s->updated) {update($s, $newRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;

} else {updateRecordIds($s, $data);updateRemote($newRecord, $s);

}} else {

sync($data);}

remote wins

if ($newRootRecord->updated > $s->updated) {update($s, $newRecord);updateRecordIds($newRootRecord, $data);send($newRootRecord);continue;

} else {updateRecordIds($s, $data);updateRemote($newRecord, $s);

}} else {

sync($data);}

server wins

{ ‘lid’: ‘1’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }

{ ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }

c1 serverPOST /merge

{ ‘lid’: ‘2’, ‘parent’: ‘1’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }

serverPOST /merge

{ ‘lid’: ‘1’, ‘guid’ : ‘32ead’, ‘data’ : ‘AAA’ ‘updated’: ’100’ }

{ ‘lid’: ‘2’, ‘parent’: ‘32ead’, ‘data’ : ‘hello!’, ‘updated’: ’15’ }

serverPOST /merge

{‘update’ : { ‘lid’: ‘1’, ’guid’: ‘af54d’ }}

{‘update’ : { lid: ‘2’, ’guid’: ‘e324f’ }}

e.g. “only one temperature can be registered in a given day”

how to we enforce domain constraints on data?

enforcing domain constraints

how to we enforce domain constraints on data?1) relax constraints

how to we enforce domain constraints on data?1) relax constraints2) integrate constraints in sync algorithm

from findByGuid to findSimilar

first lookup by GUID then by domain rules

“two measures are similar if are referred to the same date”

c1 server