Wongnai Engineering Story
-
Upload
pattrawoot-suesatayasilp -
Category
Engineering
-
view
27.015 -
download
1
Transcript of Wongnai Engineering Story
![Page 1: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/1.jpg)
Wongnai Engineering Story
Pattrawoot Suesatayasilp @pattrawoots
CTO & Co-founder @ Wongnai
![Page 2: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/2.jpg)
Wongnai• A startup company
• Our mission is to help Thai people discover and connect to great local businesses.
• We believe in trials and errors.
• Time is our most valuable resource.
![Page 3: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/3.jpg)
What It Means To Be Wongnai
![Page 4: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/4.jpg)
WongnaiUser-Generated Content Website
CMS, SEO
Search
Location-Based Service
Social Network
Feed, Follow, Like, Comment
Commerce
Deals, Payment
![Page 5: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/5.jpg)
Web & App
![Page 6: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/6.jpg)
Web & App
![Page 7: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/7.jpg)
Statistics
300k sessions/day 53M page views/month
20k requests/min (3x during holidays)
![Page 8: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/8.jpg)
Statistics
80%
20%
DesktopMobile (App/Web)
34%
66%
Mobile AppMobile Web
![Page 9: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/9.jpg)
• During meal — Lunch, Dinner
• Push Notifications
• Social Post / Dramas
High Traffic
DinnerLunch
Push
![Page 10: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/10.jpg)
• Weekends especially the long ones
• Special Occasion (Valentine’s)
• Holidays
High Traffic
Chinese New Year Valentine’s
Long WeekendSat - Sun
![Page 11: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/11.jpg)
Our 1st dev server
Apple iMac Core 2 Duo 2.0GHz + DynDns
![Page 12: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/12.jpg)
Our 1st prod server
Dell PowerEdge R210 ฿30,xxx
![Page 13: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/13.jpg)
Move To The Cloud
![Page 14: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/14.jpg)
2010-2013 User Growth
0
250000
500000
750000
1000000
Oct '10 Mar '11 Aug '11 Jan '12 Jun '12 Nov '12 Apr '13 Sep '13
Sep 2012 200k subscribers
![Page 15: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/15.jpg)
Email Blast• MailChimp until it became too
expensive. (almost 200k subscribers)
• Set Up our own mail server
• OpenEMM
• No additional cost but many problems
![Page 16: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/16.jpg)
Email Blast• Slow send rate — Takes too long to send all
• Consume too much bandwidth
• Poor open rate — Some went to Junk
![Page 17: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/17.jpg)
Amazon SES
• Send Rate (emails/second)
• Up to 90 emails/second
• Whitelist IP Addresses
• Migrating to SES wasn’t quite smooth.
Amazon SES
![Page 18: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/18.jpg)
SEO & Scalability• Large number of web pages (> 100k pages)
expecting Google bot to crawl
• Poor latency and not quite reliable (e.g. power outage)
• 2013’s goal is to reach 1M users. Be ready to scale.
• Cloud computing is the way to go.
![Page 19: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/19.jpg)
Migrate to AWS• Million of photos to
move to the cloud
• Near-zero downtime
• Installed mobile apps must continue seamlessly.
![Page 20: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/20.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
www.wongnai.com photo.wongnai.com
Static Content Servers
Web Server
Database
![Page 21: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/21.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Upload photos gradually
Database
Update ‘synced’ flag
Static Content Servers
![Page 22: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/22.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
www.wongnai.co
m
1
![Page 23: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/23.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
photo.wongnai.
com
Identify photo’s location
2
3
![Page 24: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/24.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
Redirect
to th
e old host
4
![Page 25: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/25.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
pull not-synced-yet
images
Web Server
Database
Static Content Servers
5
![Page 26: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/26.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
www.wongnai.co
m
1
![Page 27: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/27.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
photo.wongnai.
com
2
3
Identify photo’s location
![Page 28: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/28.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
Static Content Servers
Redirect
to S3
4
![Page 29: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/29.jpg)
Migrate to AWSAmazon Web ServicesHost in TH
Web Server
Database
pull already-synced images
Static Content Servers
5
![Page 30: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/30.jpg)
Database & ORM
![Page 31: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/31.jpg)
When a list of reviews are displayed
a review • review’s details
• user (reviewer) • location • photos • likes
• preview comments • and many more …
![Page 32: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/32.jpg)
Select N+1 Problem
ReviewerReview Details
Review
PhotoPhotoPhotoPhotoPhotoPreview Comments
Location
Many tables needs to be joined to get enough info for displaying a review.
![Page 33: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/33.jpg)
Large Table JoiningReview
500,000 rows
User
2,000,000 rows
Location
200,000 rows
Photo
3,000,000 rows
![Page 34: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/34.jpg)
Large Table Joining
• As a result of experiments, selecting rows separately without table joining performs much better. (No bottleneck at the database) — Feel the NoSQL?
Single SQL command
Multiple SQL commandVS
![Page 35: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/35.jpg)
Schema Changes
• Mid 2015 — We decided to combine Food and Beauty into a single app
• Tables need to be merged. Columns need to be added. Millions of rows are affected.
![Page 36: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/36.jpg)
What we expected
large table
![Page 37: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/37.jpg)
What we expected
large table
![Page 38: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/38.jpg)
What actually happens
create new tablelarge table
![Page 39: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/39.jpg)
What actually happens
create new tablelarge table
copy
• A very large number of I/O operations need to perform to update the schema
![Page 40: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/40.jpg)
I/O Speed• Storage for Amazon RDS (Amazon EBS)
• IOPS — The number of I/O operations completed per second)
• Base Perf: 300 IOPS (for 100GB storage)
• Burst 3,000 IOPS Maximum — Non-provisioned IOPS Storage
• Burst duration is limited.
• Error 404 — almost 24hrs
• Liquibase and many other schema migration tools
• Only one column can be added at a time!
![Page 41: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/41.jpg)
Why still SQL?• ACID Transaction
• NoSQL is not good at searching. (Compared to RDBMS)
• NoSQL was not popular yet, and also still lacked many key features that SQL has. (MongoDB was just one-year old. Cassandra is still in beta.)
![Page 42: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/42.jpg)
Search
![Page 43: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/43.jpg)
Types of Search
Any good restaurants near me?
Show me all branches of Sushi Hiro
Show me all restaurants in Central World
Search for “Bonchon”
![Page 44: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/44.jpg)
Indexing• Apache Solr
• Fast read but slow write
• Our indexes get updated very often, and they affects ranking — e.g. new review/photo/check-in, create new place, edit place’s info, etc.
• Tried updating indexes real-time and Solr’s soft-commit but the performance was not satisfied.
• So, we add another layer to do batch processing.
![Page 45: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/45.jpg)
Indexing
• Why not Amazon Elasticsearch Service?
• Too difficult for testing
• We don’t want to stick with AWS.
• Hybrid cloud computing?
![Page 46: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/46.jpg)
Cache• EHCache
• Most of the content are location-based.
• What should be cached? article, guide
• Content keeps updating
• e.g. Your review may get liked right after posted.
• Most of our caches have very short life. (< 1 min) and never get invalidated. Shared cache is not needed.
![Page 47: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/47.jpg)
Scheduled Task• Quartz (by Terracotta) — Job Scheduler
• Score Updater for calculating ranking
• Business, User, Review, Photo, Activity, etc.
• Leaderboard
• Statistical Data Updater
• Number of Photos/Reviews/Check-ins/…
• etc.
![Page 48: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/48.jpg)
Web Stack
เร็วส์Java@roofimon
“ ”
![Page 49: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/49.jpg)
Web Stack
Java Spring SolrMySQL Hibernate Cassandra
RequireJSjQuery Backbone Less Bootstrap
![Page 50: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/50.jpg)
Infra/DevOps
Tomcat
Git
Jenkins
DockerMaven
ZookeeperHTTP Server
![Page 51: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/51.jpg)
Architecture
Solr Cloud
HTTP/HTTPS Load Balancer
Amazon RDS
ZooKeeper Cluster
Cassandra Cluster
Static Content Servers
Monolithic Application
Servers
Amazon SES
![Page 52: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/52.jpg)
• 6 year ago, we were new to web development.
• Default Templating System — JSP
• No pre-processor was there back then, so we
wrote our own pre-processors in Maven Lifecycle
(Phase: Prepare package).
• Minify JS, CSS files — 1 file per page.
• YUI Compressor
Web Stack (Frontend)
![Page 53: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/53.jpg)
• Asynchronous Module Definition • RequireJS
• Why not Angular? • SEO is one of the most important thing for a
content website like Wongnai.com • JavaScript SEO (prerender.io / …)
• More code to maintain • Code refactoring takes time. Not worth for
now
Web Stack (Frontend)
![Page 54: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/54.jpg)
• Sonar (now SonarQube) • No longer use it as we found it doesn’t help
us that much. Our dev team is relative small. Manual code review is far better.
• Jenkins • No distributed build yet. • Only used on dev environment
DevOps
![Page 55: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/55.jpg)
What’s Next• Personalization
• Machine Learning / Scala / Spark
• NoSQL for data that doesn’t require searchability
• e.g. Feed, Notifications
• Advanced Message Queuing Protocol (AMQP)
• RabbitMQ
• Microservices
• Image Server
![Page 56: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/56.jpg)
Process & Tools• Asana, Slack, Google Docs
• Agile
• Manual First
• TDD
• Daily Scrum
• 2-week Sprint (1-month Sprint before)
• Retrospect
![Page 57: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/57.jpg)
Dev Team 6 Full-Stack
Web Developers
3 iOS Developers + 1 Part-timer
2 Android Developers
1 QA Engineer 1 UI/UX Designer
![Page 58: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/58.jpg)
Dev Team
We are Hiring!
6 Full-Stack Web Developers
3 iOS Developers + 1 Part-timer
2 Android Developers
1 QA Engineer 1 UI/UX Designer
![Page 59: Wongnai Engineering Story](https://reader034.fdocuments.us/reader034/viewer/2022051706/58f9ad91760da3da068b99ad/html5/thumbnails/59.jpg)
Q&A
Q&AJoin Us!
wongnai.com/careers