Scaling Solr in the Cloud - assets.en.oreilly.comassets.en.oreilly.com/1/event/61/Scaling Solr...
Transcript of Scaling Solr in the Cloud - assets.en.oreilly.comassets.en.oreilly.com/1/event/61/Scaling Solr...
www.nutshell.com
Scaling Solr in the CloudBy @ablyler & @LindsaySnider
F R O M
@NutshellCRM
Agenda
About the Speakers
About Nutshell
Nutshell and Solr
Solr Resource Usage
Scaling Methods
Questions and Answers
About the Speakers
BitLeap / Barracuda NetworksDevelopers on the Backup Appliance / Cloud
Scaled databases and storage systems
Used SugarCRM & Salesforce.com
About Nutshell
Web and Mobile CRM application
Heavy use of OpenSource technologies:Gentoo
Nginx / PHP / ZendFramework / jQuery
MySQL / Solr / Gearman
Jenkins / Redmine / Cacti / Nagios
Nutshell and Solr
Heavy use of Solr for searching, table views, and de-duplication
Used for searching / display:Accounts, Competitors, Contacts, Leads, Products, Sources, Teams, and Users
Used for de-duplication:Accounts, Contacts
Reads vs Writes
0
1750
3500
5250
7000
Jul 20, 2011 3:00 AM Jul 20, 2011 6:00 AM Jul 20, 2011 9:00 AM Jul 20, 2011 12:00 PM Jul 20, 2011 3:00 PM Jul 20, 2011 6:00 PM Jul 20, 2011 9:00 PM Jul 21, 2011 12:00 AM
Writes Reads Time
Average Read Query Time: 2.11ms
Auto Provisioning
Setup new Solr Jetty app
Create MySQL database
Populate MySQL / Solr with demo data
Send welcome email
Dark Ages Age of Enlightenment TodayModern Era
Separate Jetty per Customer
Uses a ton of memory
Separate schema / Solr for each customer
Ran into upper limit morning before launch
Dark Ages Age of Enlightenment TodayModern Era
Auto provisioning
Separate Jetty app for each customer
Dark Ages Age of Enlightenment TodayModern Era
Auto provisioning
Separate Jetty app for each customer
Dark Ages Age of Enlightenment TodayModern Era
0
25
50
Solr Core per Customer
Allows for management of Solr on a per customer basis: creating / stopping
Contained within a single Jetty app
Shared schema between all cores
Easily managed via simple HTTP API
Age of EnlightenmentDark Ages TodayModern Era
Fallback to MySQL
Landing page of application
Allows for graceful handling when Solr is down
Abstracted within the application library
Age of EnlightenmentDark Ages TodayModern Era
Sun JVM to IcedTea JVM
IcedTea JVM uses less memory than Sun JVM
Age of EnlightenmentDark Ages TodayModern Era
Separate Solr core per customer
Fallback to MySQL for table data
Migrated from Sun JVM to IcedTea JVM
Age of EnlightenmentDark Ages TodayModern Era
Separate Solr core per customer
Fallback to MySQL for table data
Migrated from Sun JVM to IcedTea JVM
0
150
300
Age of EnlightenmentDark Ages TodayModern Era
Shared Schema Across Cores
Decreases initialization time for each core
Decreases memory usage
Age of EnlightenmentDark Ages TodayModern Era
Solr Index Field Selection
More indexed fields = more used memory
Only index fields that are searched
Store other non-indexed fields for display
Age of EnlightenmentDark Ages TodayModern Era
Splitting of Reader / Writer
Index building is CPU / disk intensive
Writer = Solr with caching disabled
Reader = Solr slave, that doesn’t build indexes
Age of EnlightenmentDark Ages TodayModern Era
MySQL(master-master)
Web
Solr Slave
Solr Master
Range indexerRange indexerRange indexerRange indexerDocument indexers
Gearman workers
Age of EnlightenmentDark Ages TodayModern Era
MySQL(master-master)
Web
Solr Slave
Solr Master
Range indexerRange indexerRange indexerRange indexerDocument indexers
Gearman workers
Age of EnlightenmentDark Ages TodayModern Era
MySQL(master-master)
Web
Solr Slave
Solr Master
Range indexerRange indexerRange indexerRange indexerDocument indexers
Gearman workers
Age of EnlightenmentDark Ages TodayModern Era
MySQL(master-master)
Web
Solr Slave
Solr Master
Reindex ManagerRange indexerRange indexer
Range indexerRange indexerRange indexers
Age of EnlightenmentDark Ages TodayModern Era
MySQL(master-master)
Web
Solr Slave
Solr Master
Reindex ManagerRange indexer
Range indexerRange indexerRange indexerRange indexers
Age of EnlightenmentDark Ages TodayModern Era
MySQL(master-master)
Web
Solr Slave
Solr Master
Reindex ManagerRange indexer
Range indexerRange indexerRange indexerRange indexers
Age of EnlightenmentDark Ages TodayModern Era
Shared schema across cores
Solr index field selection
Splitting of reader / writer roles
Intelligent indexing / reindexing
Age of EnlightenmentDark Ages TodayModern Era
Shared schema across cores
Solr index field selection
Splitting of reader / writer roles
Intelligent indexing / reindexing
Age of EnlightenmentDark Ages TodayModern Era
0
500
1000
Intelligent Core Management
Least recently used Solr cores are spun down
Solr cores started on login
Reindexing is a database flag, and happens on the next login
Age of EnlightenmentDark Ages TodayModern Era
Partitioning
Pairs of readers and writers
Partitioned based on account idLeft pad the account id w/ zeros to length of two
Reverse the account id
Take last two digits of the account id
Age of EnlightenmentDark Ages TodayModern Era
Intelligent core spin down / up
Partitioning of customers to separate Solr servers
Age of EnlightenmentDark Ages TodayModern Era
Intelligent core spin down / up
Partitioning of customers to separate Solr servers
Age of EnlightenmentDark Ages TodayModern Era
0
4000
8000
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
1 2 3 4 5 6
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
1 2 3 4 5 6
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
65
1 2 3 4 5 6
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
65
01
02
03
04
65
66
96
97
98
99
1 2 3 4 5 6
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
65
01
02
03
04
65
66
96
97
98
99
65
1 2 3 4 5 6
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
123456
1 2 3 4 5 6
65
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
SOLR Server
8000 User
01
02
03
04
65
66
96
97
98
99
SOLR Server
8000 User
SOLR Server
8000 User
7000 Users
600 Users
300 Users
50 Usersauto provisioning
separate Jetty app for each customer
separate Solr core per customer
fall back to MySQL for table data
migrated from Sun JVM to IcedTea JVM
shared schema across cores
solr index field selection
splitting reader / writer roles
intelligent core spin down / up
parititioning of customer to separate Solr servers
Scaling Solr in the Cloud
By @ablyler & @LindsaySniderF R O M
@NutshellCRM