Data replication in Sling
-
Upload
tommaso-teofili -
Category
Technology
-
view
492 -
download
4
description
Transcript of Data replication in Sling
![Page 1: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/1.jpg)
APACHE SLING & FRIENDS TECH MEETUP BERLIN, 22-24 SEPTEMBER 2014
Data replication in Sling Tommaso Teofili, Adobe Systems
![Page 2: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/2.jpg)
adaptTo() 2014 2
Use cases
![Page 3: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/3.jpg)
Replication in AEM
adaptTo() 2014 3
§ Moving authored content to publish servers
author publish
![Page 4: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/4.jpg)
Replication in AEM
adaptTo() 2014 4
§ Moving user generated content back to author for moderation
author publish
![Page 5: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/5.jpg)
Replication in AEM
adaptTo() 2014 5
§ Moving user generated content to other publish instances
author
publish
publish
![Page 6: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/6.jpg)
Replication module should ...
adaptTo() 2014 6
§ transfer resources between Sling instances § push from server A to server B § pull from server C to server D
![Page 7: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/7.jpg)
Example: replication of /foo/bar
adaptTo() 2014 7
§ HTTP request for pushing /foo/bar § Resources get
§ packaged
§ sent § received
§ persisted
![Page 8: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/8.jpg)
we’re not in 1999 ... anymore
adaptTo() 2014 8
![Page 9: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/9.jpg)
NRT sync on N servers
adaptTo() 2014 9
publish1
publish2
author1
publish3
publish4
publish5
author2
![Page 10: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/10.jpg)
Oak instances served by same MongoDB
adaptTo() 2014 10
publish1
publish2 author
oak
oak
mongo
mongo mongo
mongo
p1 or p2
![Page 11: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/11.jpg)
Cloud based infrastructures
adaptTo() 2014 11
publish1 publish2
author1
P1 to pN
author2
publish3
publish4 publish4 publish4 publish4 publish4 publishN
![Page 12: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/12.jpg)
adaptTo() 2014 12
Overview
![Page 13: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/13.jpg)
Sling Replication
adaptTo() 2014 13
§ Contributed to Sling in November 2013 § First release ? § Main goals
§ Simple § Resilient
§ Fast
![Page 14: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/14.jpg)
Replication agents
adaptTo() 2014 14
§ Execute replication requests by: § exporting replication packages from a (remote)
Sling instance § importing replication packages into a (remote)
Sling instance
![Page 15: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/15.jpg)
Replication agents
adaptTo() 2014 15
§ A “push” agent has § A “local” exporter
§ Creating a package locally for the resources to be replicated (e.g. from the underlying JCR repo)
§ A “remote” importer § Importing the exported package remotely by sending
it to a designated endpoint to persist it
![Page 16: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/16.jpg)
Replication agents
adaptTo() 2014 16
§ A “pull” agent has § A “remote” exporter
§ Pulling a package from a remote Sling instance designated endpoint
§ A “local” importer § Importing the package locally by persisting it into the
Sling instance
![Page 17: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/17.jpg)
Replication agents
adaptTo() 2014 17
§ A “coordinating” agent has § A “remote” exporter
§ Pulling a package from a remote endpoint
§ A “remote” importer § Importing the package remotely into a Sling instance
![Page 18: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/18.jpg)
Replication agents
adaptTo() 2014 18
§ A “queuing” agent has § A “local” exporter
§ Creating a package locally for the resources to be replicated (e.g. from the underlying JCR repo)
§ No importer
![Page 19: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/19.jpg)
Replication package serialization
adaptTo() 2014 19
§ Payload to be sent / received § Package builders for (de)serialization
§ Jackrabbit FileVault based package builder
![Page 20: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/20.jpg)
Replication queues
adaptTo() 2014 20
§ Multiple queue providers § Sling Jobs based (sling.event bundle) § In memory
§ Multiple queue distribution strategies § Single
§ Error aware § Priority
![Page 21: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/21.jpg)
Replication rules
adaptTo() 2014 21
§ Rules can be defined within agents § To trigger replications upon resource changes § To schedule periodic replications
§ To chain replicate § ...
![Page 22: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/22.jpg)
Accessing replication resources
adaptTo() 2014 22
§ Agents as OSGi services § Can be defined via ConfigurationAdmin § Resource providers control
§ Service access
§ CRUD operations on configs
![Page 23: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/23.jpg)
Agent configuration example (1/3)
adaptTo() 2014 23
§ Config for a ‘push agent’ {
"jcr:primaryType" : "sling:OsgiConfig",
"name" : "publish",
"type" : "simple",
"packageExporter": [
"type=local”,
"packageBuilder/type=vlt",
...
],
...
![Page 24: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/24.jpg)
Agent configuration example (2/3)
adaptTo() 2014 24
... "packageImporter" : [
"type=remote",
"endpoints[0]=http://.../replication/services/importers/default”,
"authenticationFactory/type=service",
"authenticationFactory/name=user",
...
"packageBuilder/type=vlt",
...
],
...
![Page 25: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/25.jpg)
Agent configuration example (3/3)
adaptTo() 2014 25
... "queueProvider" : [
"type=service",
"name=sjh"
],
"queueDistributionStrategy" : [
"type=service",
"name=error"
]
}
![Page 26: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/26.jpg)
adaptTo() 2014 26
Anatomy of a forward replication request
![Page 27: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/27.jpg)
Anatomy of a fwd. replication (1/4)
adaptTo() 2014 27
§ HTTP POST on /libs/sling/replication/service/agents/publish with form parameters
§ Action = ADD
§ Path = /content/replication
§ Agent with name ‘publish’ gets picked up by the agent resource provider
![Page 28: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/28.jpg)
Anatomy of a fwd. replication (2/4)
adaptTo() 2014 28
§ Agent ‘publish’ calls its Exporter (local) to create the payload § the ‘local’ exporter calls its Package builder
§ the ‘vlt’ package builder creates a FileVault package for resources under /foo/bar
![Page 29: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/29.jpg)
Anatomy of a fwd. replication (3/4)
adaptTo() 2014 29
§ Agent ‘publish’ dispatches the package to its queue provider and distribution algorithm § The queue provider is asked to provide queues
depending on the distribution strategy § The request gets queued
![Page 30: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/30.jpg)
Anatomy of a fwd. replication (4/4)
adaptTo() 2014 30
§ When queue entry gets processed the (remote) importer is called
§ Package sent over the wire to an endpoint bound to a local importer on the receiving server
§ The local importer on the receiving side deserializes and persists the replication package via its package builder
![Page 31: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/31.jpg)
Reverse replication
adaptTo() 2014 31
§ Pull agent on author § Periodically polling
§ Queuing agent on publish
![Page 32: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/32.jpg)
Event based reverse replication
adaptTo() 2014 32
§ No scheduled polling § The publish instance notifies author when to
pull § Notification through server sent events
![Page 33: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/33.jpg)
Looking forward
adaptTo() 2014 33
§ Get information on other instances § e.g. replicate to all instances with run mode ‘xyz’
§ NRT publish sync § Coordinate agents++
§ Performance
![Page 34: Data replication in Sling](https://reader035.fdocuments.us/reader035/viewer/2022062303/5565476ed8b42ad7648b51f9/html5/thumbnails/34.jpg)
Thanks!
adaptTo() 2014 34
§ Questions?