Content Delivery at Aviary - NYC MUG 11/19/13
-
Upload
mongodb -
Category
Technology
-
view
2.434 -
download
1
Transcript of Content Delivery at Aviary - NYC MUG 11/19/13
![Page 1: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/1.jpg)
Content Delivery At Aviary
MongoDb User Group 11/19/13
![Page 2: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/2.jpg)
Aviary
Fully-Baked UI
Configurable, High-Quality Tools
Over 5,000 Partners
Over 50 Million Monthly Users
Over 4 Billion Photos Edited
iOS, Android, Web, Windows, Server
Photo-Editing SDK & Apps
J
![Page 3: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/3.jpg)
Who Are We?
JackDirector of Engineering
Nir Lead Serverside Engineer
● Automated deployment● Big-O notation● Brainteasers
Likes:
● Cilantro
Hates:
● Parallelizing processes● DRY code● Seltzer
Likes:
● Food after the sell-by date
Hates:
![Page 4: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/4.jpg)
ContentEffects Frames Stickers Messages
J
![Page 5: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/5.jpg)
The “Fat Tiny” Situation
We want to dynamically deliver the Fat Tiny
stickers to our users
How can we do that?
J
![Page 6: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/6.jpg)
The CDSAviary’s Content Delivery System
![Page 7: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/7.jpg)
Version 1
● Static files on S3
● Excess data sent
● Data stored in MySQL
● New features meant new code
N
![Page 8: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/8.jpg)
The “Fat Tiny” Situation
We want to display the Fat Tiny sticker pack only
to Picstitch users in Japan who use iOS 7
How can we do that?
N
![Page 9: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/9.jpg)
CDS V2Version 2 Overview
![Page 10: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/10.jpg)
Stack
Load Balancer
APIServers
DatabaseCluster
Content Delivery Network
Management System
GlobalUsers
Akamai
AWS Cloud FormationAWS Elastic LoadBalancer (ELB)
Node.jsAWS Elastic Cloud Computing (EC2)UbuntuBash
MongoDbJSONJSON Schema
N
![Page 11: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/11.jpg)
Why MongoDB?
● Great with Node (JSON-based)
● Schema-less is easy to change/query
● Read-heavy
● Relatively small data set
N
![Page 12: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/12.jpg)
How It WorksA Behind-the-Scenes Look
![Page 13: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/13.jpg)
Delivered Types
● Manifest JSON○ Content Set○ Content Versions
● Content JSON○ Content Metadata○ All Assets
N
![Page 14: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/14.jpg)
The “Fat Tiny” Situation
We want to manage the Fat Tiny sticker pack
as a single entity, but we want to deliver it to
each device in its own
optimal format.
How can we do that?N
![Page 15: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/15.jpg)
Response Formatting ModelContent Entry Response Formats Responses
JSON document describing content item
JSON documents defining mappings from entry to responses
Actual JSON responses delivered to devices J
![Page 16: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/16.jpg)
Content Formats
● JSON schema
● Added properties
● Custom types
● Validates entries
{
"type":"object",
"properties":{
"metadata": {
"type": "object",
"properties":{
"displayName": {
"type": "string",
}
}
},
"icon": {
"type": "object",
"customType": "image"
"properties":{
"path": {
"type": "string",
"required": true, J
![Page 17: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/17.jpg)
Response Formats
● JSON schema
● dataKey property
● Defines response
structure
● Maps content
{
"type":"object",
"properties":{
"identifier": {
"type": "string",
"dataKey": "identifier"
},
"name": {
"type": "string",
"dataKey": "metadata.displayName"
},
"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"
},
"items": {
"type": "array",
"dataKey": "items" J
![Page 18: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/18.jpg)
Content Deployment
"type":"object",
"properties":{
"id": {
"type": "string",
"dataKey": "identifier"
},
"name": {
"type": "string",
"dataKey": "metadata.displayName"
},
"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"
},
"stickers": {
"type": "array",
"dataKey": "items"
"identifier": "com.aviary.stickers.234fe"
"metadata": {
"displayName": "Hats"
},
"icon": {
"path": "cds/hats/icon.png"
"path-100": "cds/hats/icon100.png"
},
"items": [
{
"identifier": "1"
"imageUrl": "cds/hats/1.png"
}
]
"id": "com.aviary.stickers.234fe",
"name": "Hats",
"iconImagePath": "cds/hats/icon100.png"
"stickers": [
{
"identifier": "1"
"imageUrl": "cds/hats/1.png"
}
],
"versionKey": "e4532fd342"
1. Insert/Update CMS Entry 2. Find Response Formats 3. Generate+Insert Responses
J
![Page 19: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/19.jpg)
Manifests "stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
] J
![Page 20: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/20.jpg)
Manifest Deployment
{
$match:{
formatId:{$in:formatIds},
identifier:{$in:identifiers}
}
},
{
$sort: { _id: -1 }
},
{
$group: {
_id: "$identifier",
versionKey:{$first:"$versionKey"}
}
}
Using Aggregate to Find the Newest Versions Manifest with Correct Version Keys
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
}
],
"effects": [
{
"id": "com.aviary.effects.25435",
"versionKey": "fd4324323"
}
]
J
![Page 21: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/21.jpg)
Scopes and Targeting
Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},
"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}
Manifest 2
"targetingScope": {
"apiKey": "def",
},
"formattingScope": {
"platform": "android",
"minOsVersion": "6.0.0"
}
Deployed Manifests Have Scopes
/manifest?
apiKey=abc&
country=JP&
language=ja&
platform=ios&
osVersion=7.2.0
End Users Have Scope Parameters
N
![Page 22: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/22.jpg)
API ServersScope parameters are converted into queries
/manifest?
apiKey=abc&
country=JP&
language=ja&
platform=ios&
osVersion=7.2.0
db.manifest.find({
"apiKey": {$in: ["abc", null]},
"country": {$in: ["JP", null]},
"language": {$in: ["ja", null]},
"platform": {$in: ["ios", null]},
"minOsVersion": {$lte: 7002000}
}).sort({
"apiKey": -1,
"language": -1,
"country": -1,
"minOsVersion": -1,
"platform": -1,
"_id": -1
}).limit(1)
7002000
Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},
"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}
Manifest 2
"targetingScope": {
"apiKey": "abc",
},
"formattingScope": {
"platform": "ios",
"minOsVersion": "6.0.0"
}N
![Page 23: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/23.jpg)
Versioned Content
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]
Received Manifests Contain VersionKeys
/content?
versionKey=e4532fd342
db.content.findOne({
"versionKey": “e4532fd342”
});
N
![Page 24: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/24.jpg)
Response Caching/manifest?
apiKey=abc&
country=JP&
language=ja&
platform=ios&
osVersion=7.2.0
db.manifests.find({
"apiKey": {$in: ["abc", null]},
"country": {$in: ["JP", null]},
"language": {$in: ["ja", null]},
"platform": {$in: ["ios", null]},
"minOsVersion": {$gte: 7002000}
}).sort({
…,
"_id": -1
}).limit(1)
db.cachedManifests.findOne({
"url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios"
})
N
![Page 25: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/25.jpg)
PAULAThe CDS Management Console
![Page 26: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/26.jpg)
Auto-generated UI
J
![Page 27: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/27.jpg)
Other Mongo Usage
● PAULA permissions in user objects
● Integration tests interact with schemaless db willy nilly
users collection
{
"name": "nir",
"email": "[email protected]",
"permissions": [
"content",
"dev",
"admin",
"partying"
]
}
N
![Page 28: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/28.jpg)
ConclusionThe Takeaway
![Page 29: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/29.jpg)
The Facts
● Built and deployed in 3 months
● Very few struggles with MongoDB
● Seamless management
● Graceful scaling from 0 to over 20M MAUs
● Happy serverside engineersJ
![Page 30: Content Delivery at Aviary - NYC MUG 11/19/13](https://reader033.fdocuments.us/reader033/viewer/2022052413/559a9f611a28ab07098b46bb/html5/thumbnails/30.jpg)
The Future
● Targeted Translations
● Granular User Targeting
● PAULA for the masses
N