Data Sync on iOS with Couchbase Mobile

(Or, building better UX / Apps with distributed databases and data synchronisation techniques)

http://ti.eng.br @thiago_sjcThiago Faria Alencar

February 17, 2016HERE Maps @ Berlin

What do you mean with User Experience anyway ?

Combination of technology to delivery the best user experience. That’s why people switch / opt to services: think about Blacklane, to Immobile scout, Soundcloud.. What can

you offer at the end of the day? Is it the best possible?

-Blacklane :)-Amazon-Airbnb

-Facebook-Spotify

-EtcDon’t “own" things, but redefine how user

experience music, transport,..

What do you mean with User Experience anyway ?

A little (real life) story

The coffee sales man in the middle of nowhere

Back in 2007..

Let’s go back in time< 1935 2007

Instant coffee for internal Brazilian

market.. let’s build a system for sales …

Product’s website at: www.cafesaojose.com.br

A website & mini ERP

A website & mini ERP

Basic LAMP Setup(Linux + Apache + Mysql + Php)

Good old times.. right?

Using the systemPedralva - Minas Gerais

Network coverage (as of today)

Oops, no connection? all data in the server miles away..

Back to 2016… (last weekend)

same old problems….

Transit routes (online-only feature)

Clear found/searched route, hop in the train

Ups, inside subway now..

1 min later.. hm.. couldn’t we show the past/found route?

Other examples

Spotify - offline advertising!

‘Hobby' projects (telemetry)

IoT

Even the most ‘connected' applications will experience network issues from time to time -> make them operate offline as much as possible

Network coverage getting better …

but as of today still far from ideal..

A data locality/expiration issue

Can we do better?

hell yes.

Recognise the problem..

“Git-like” for databases?

“A distributed database is a database in which portions of the database are stored on multiple computers

within a network.”

“Users have access to the portion of the database at their location so that they can access the data relevant to their

tasks without interfering with the work of others.”

Turns out, a similar set of features needed on of distributed databases (computer clusters) is desired to have at the end nodes

(mobile devices) > Same fundamental design decisions.

Analysing DB designs

Document / NoSql databases vs. relational databases

scalability / flexibility

CAP Theorem : set of basic requirements that describe any distributed system (not

just storage/database systems).

Consistency - All the servers in the system will have the same data so anyone using the system will get the same copy regardless of which server answers their request.

CAP Theorem

Availability - The system will always respond to a request (even if it's not the latest data or consistent across the

system or just a message saying the system isn't working).

CAP Theorem

Partition Tolerance - The system continues to operate as a whole even if individual servers fail or can't be

reached.

Document / NoSql databases vs. relational databases

Document / NoSql databases vs. relational databases

RDMS are more or less ‘all or nothing’, transaction-based, atomic operations oriented.

Distributed databases relaxes this ‘atomic' requirement in order to provide better scalability. It turns out this feature is useful for offline mobile apps as well as we’ll see when compared to relation traditional databases.

Document / NoSql databases vs. relational databases

‘NoSQL databases' improves scaling by relaxing consistency in favor of availability and partition tolerance.

A bit or market research..

Find the right tool to do the job

Intro to Couchbase • Open source• Cross-platform (really)• Open protocols• Backed by company, but small ‘lock-in’• Easy to get started• Small foot-print on client• Built-in sync handling• No ‘migration-hell’ (JSON docs)

How to solve the data sync problem?

Database per-user doesn’t scale

Instead, share a single database

Compare this to a “repository” in git: each subset of data would be like a “repo”.. not all users will have

access to all data (e.g. ACL with gitolite)

The ‘traditional' way of exchanging data..

Encapsulate all of it in a SSL connection and everything will be fine (that is, while your connection

works).

What is happening here?

Notice how in the request – response model, data is “routed” by the different API end-points: URI path will trigger a specific request handler in the server.

How such problems are solved during sync?

• Authentication• Authorization• Data validation / routing

How such problems are solved during sync?

• Data routing:

When another user connects to the same backend, we want only the documents belonging to him to be synchronised. The term used for this in Couchbase Mobile is data replication.

How such issues are resolved during sync?

That’s when Couchbase data orchestration features kicks in:

Not to depend on an active connection, what each data represents and the respective routing are described in the document’s content itself.

-Optimal solution sits neither at one or the other end of the spectrum - normally you need a mix of things : so normal http reqs still useful!

How such issues are resolved during sync?

For iOS, called ‘Couchbase Lite’

-Available for many other platforms.-Advantages of being a JSON database (no migration hell)-Sync gateway ‘like the git merge'

Document: the primary entity stored in a database is called a document instead of a “row” or “record”.

Views: ‘denormalised tables/indexes ’; dynamically generated from JSON objects

Queries: the action of looking up results from a view’s index

Some terminology in Couchbase Lite land..

(Roots in Map/Reduce programming model)

123456

CBLManager *manager = [CBLManager sharedInstance];        NSError *error;CBLDatabase* _database = [manager databaseNamed: @"conektapp_db" error: &error]; if (error) { NSLog(@"error getting database %@",error); }

NSDictionary* properties = @{@"user_id":      @"joe",                             @"type":         @"contactInfo",                             @"contactData":  @“+49610234192"};

CBLDocument* document = [database documentWithID: [NSString stringWithFormat: @"%@:%@", type, user_id]];NSError* error;

if (![document putProperties: properties error: &error]) {    [self handleError: error];}

Saving an object in Couchbase Lite

( For questions on this see more detailed articles at http://ti.eng.br )

123456

- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user{    //1- createView    CBLView * contactInfoView = [self.database viewNamed: @"contactDataByUsername"];    [contactInfoView setMapBlock: MAPBLOCK({        if ([doc[@"type"] isEqualToString: @"contactInfo"]) {            if (doc[@"user_id"])                emit(doc[@"user_id"], doc[@"contactData"]);        }    }) version: @"2"]; }

Retrieve an object in Couchbase Lite

123456

- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user{    //1- createView    CBLView * contactInfoView = [self.database viewNamed: @"contactDataByUsername"];    [contactInfoView setMapBlock: MAPBLOCK({        if ([doc[@"type"] isEqualToString: @"contactInfo"]) {            if (doc[@"user_id"])                emit(doc[@"user_id"], doc[@"contactData"]);        }    }) version: @"2"];    //2 - make the query    CBLQuery* query = [contactInfoView createQuery];    NSLog(@"Querying username: %@", user);    query.startKey = user;    query.endKey   = user;    return query;}

Retrieve an object in Couchbase Lite

123456

- (CBLQuery*) queryContactInfoFromUsername:(NSString*) user{    //1- createView    CBLView * contactInfoView = [self.database viewNamed: @"contactDataByUsername"];    [contactInfoView setMapBlock: MAPBLOCK({        if ([doc[@"type"] isEqualToString: @"contactInfo"]) {            if (doc[@"user_id"])                emit(doc[@"user_id"], doc[@"contactData"]);        }    }) version: @"2"];    //2 - make the query    CBLQuery* query = [contactInfoView createQuery];    NSLog(@"Querying username: %@", user);    query.startKey = user;    query.endKey   = user;    return query;}//run, enumerateCBLQuery *contactQuery = [myObject queryContactInfoFromUsername: @"thiago"];CBLQueryEnumerator* result = [contactQuery run: &error];for(CBLQueryRow* row in result){    NSLog(@"Found document: %@", row.document);}

123456

NSURL* url = [NSURL URLWithString: @“http://yourapp.com/sync_gateway/"];

CBLReplication *push = [database createPushReplication: url];CBLReplication *pull = [database createPullReplication: url];push.continuous = pull.continuous = YES; [pull setCookieNamed:@"SyncGatewaySession" withValue:sessionValue path:nil expirationDate:nil secure:NO];[push setCookieNamed:@"SyncGatewaySession" withValue:sessionValue path:nil expirationDate:nil secure:NO]; NSLog(@"startSync");[pull start];[push start];

Getting replication running.. client side

The server-side

Sync gateway configuration

Located in the sync gateway, the 'sync function' is responsible for document authorisation, validation, routing.

The server-side

Sync gateway configuration123456789101112

{   "interface":":4984",   "adminInterface":":4985",   "log":["REST"],   "databases":{      "sync_gateway":{         "server":"http://localhost:8091",         "bucket":"app_bucket",         "sync":`function(doc) {channel(doc.channels);}`      }   }}

sync function, not much defined for now.. will grow according to your application’s needs

When the sync function kicks in?

More on the sync function..

function(doc, oldDoc){

requireUser(oldDoc.owner);channel(doc.channel);access(doc.members, doc.roomID);

}

Validation

More on the sync function..

function(doc, oldDoc){

requireUser(oldDoc.owner);channel(doc.channel);access(doc.members, doc.roomID);

}

Routing

More on the sync function..

function(doc, oldDoc){

requireUser(oldDoc.owner);channel(doc.channel);access(doc.members, doc.roomID);

}

AccessControl

About the concept of “Channels”

App server tags Documents on entry; Docs are indexed by channel for replication.

Summary

Data scalability: Replicate only docs relevante to respective user.

Access Control: Replicate only documents visible to this user

Data Validation: Replicate only documents that contains valid information

Syncing events.. (change notifications)

Web Bonus

Web

Others

Bonus

Web

Others

Bonus

Thank you :)

Finally we have a way to automate data synchronisation!! Please use it! :)

And let’s shift our designs / apps away from the old "LAMP style” into something that actually works in real life

scenarios also when 'on the go'..

Q&A

One more slide to show how a more ‘elaborated' sync function may look like..

function (doc, oldDoc) { if (doc._deleted) { // Only editors with write access can delete documents: requireRole("role:editor"); requireUser(oldDoc.writers); // Skip other validation because a deletion has no other properties: return; } // Required properties: if (!doc.title || !doc.creator || !doc.channels || !doc.writers) { throw({forbidden: "Missing required properties"}); } else if (doc.writers.length == 0) { throw({forbidden: "No writers"}); } if (oldDoc == null) { // Only editors can create documents: requireRole("role:editor"); // The 'creator' property must match the user creating the document: requireUser(doc.creator) } else { // Only users in the existing doc's writers list can change a document: requireUser(oldDoc.writers); // The "creator" property is immutable: if (doc.creator != oldDoc.creator) { throw({forbidden: "Can't change creator"}); } } // Finally, assign the document to the channels in the list: channel(doc.channels);}