Mistakes Made and Lessons Learnt Scaling Plone post-Launch
-
Upload
matt-hamilton -
Category
Technology
-
view
1.718 -
download
0
description
Transcript of Mistakes Made and Lessons Learnt Scaling Plone post-Launch
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Matt Hamilton&
Matt Sital-Singh
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Background0
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• Learning Management-type system
• A lot of content
• Users ‘editing’ content, but not necessarily aware of it
• Lots of ‘clustering’ of load on resources
Use Case
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Scale
• Plone 3
• User accounts: 30,000
• Messages: 160,000
• Course questions: 260,000
• Other: 60,000
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Don’t Create Ghost Content1
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• User perception of speed
• If they expect a ‘read’ they want a ‘read’- user signups- course starting
• Will accept a delay if you are actually processing something
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Use Optimization Products2
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• experimental.catalogqueryplan
• experimental.contentcreation
• experimental.daterangeindexoptimisations
• experimental.aggressiveopaquespeedup
• collective.indexing
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
One Plone site or Multiple?3
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• One Plone site per ‘site’
• Multiple ‘sites’ per Plone site
• Common content?
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Large Catalogs are Slow4
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• We had 260,000 questions
• each had 2-4 answers
• = 520,000 - 1,400,000 answers
All in portal_catalog!
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Optimize your Catalogs5
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• Pre-load Catalog Queryplan
• Balance your BTrees
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
1319817170
1319817198
1319730857
1319126057
1308585257
effective date index:
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
1308585257
1319126057
1319730857
1319817170
1319817198
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• Used catalog multiplexer to assign ‘Answer’ content type to dummy catalog
• Moved messages to separate message_catalog
• Monkey-patch items out of the catalog?
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
If You Know Where Something Is - Don’t
Search for it6
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• Got into bad habit of using the catalog ALL the time
• Don’t just blindly use catalog when unrestrictedTraverse / contentValues will do
• RAM is cheap, user patience is NOT
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Use unrestrictedSearchResults
if possible7
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• unrestrictedSearchResults does not check:
• effective index
• allowed_roles_and_users index
➡ A lot less work for the catalog to do
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Is Plone the Right Tool?8
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• “When you have a hammer, everything looks like a nail...”
• A lot of this data is relational...
• Not much of it is CMSish
• Maybe should have used Pyramid?
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Use Event Driven Workflow
(Don’t recalculate)9
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• User rating / Course ranking
• Use workflow event subscribers, don’t re-calculate on view every time
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Some things don’t need to be immediate10
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
• “You have 10 new messages”
• Not everything has to be updated all the time in every location on the site
Matt HamiltonMatt Sital-Singh
Plone Conference 2011San Francisco
Thanks!