Post on 13-Jan-2015
description
Continuous Deployment at kaChingApplied Lean Startup Ideas:
#leanstartup
&
Pascal-Louis Perez, kaChing Group Inc.
Twenty Deployments a Day Keeps the Nasty Bugs Away
10-May
11-May
12-May
13-May
14-May
15-May
16-May
17-May
18-May
19-May
20-May
21-May
0
5
10
15
20
25
30
Pascal-Louis Perez, kaChing Group Inc.
“Connect investors with outstandinginvestment managers”
Pascal-Louis Perez, kaChing Group Inc.
Release is a Marketing Concern
Reducing Code Inventory is what Engineering Focuses On
Pascal-Louis Perez, kaChing Group Inc.
Lean Startup at kaChing
• Methodical approach to eliminating waste• No Quality Assurance engineers, no Operations
team• Empower everybody to drive their own projects
“Human institution designed to create something new under conditions of extreme uncertainty” – Eric Ries
Pascal-Louis Perez, kaChing Group Inc.
What is Continuous Deployment?
• Continuous, successful and repeatable methodology to deploying code
• Automates every steps of taking checked in code and making it run on production servers, in front of customers
Pascal-Louis Perez, kaChing Group Inc.
True Story
• Investment managers calls, comments about unintuitive trading flow.
• Improvements are made, and deployed within the next 20 minutes.
• We call him back “What do you think of the improvement?”
Pascal-Louis Perez, kaChing Group Inc.
Benefits of Continuous Deployment
• It allows quick iterations• Obsoletes processes, e.g. “cutting a release”• Reduces risk• Everyone is aware of production• No one throws code over the wall• Exposes 24x7 operational requirements• Trunk stable
Pascal-Louis Perez, kaChing Group Inc.
ContinuousDeployment
Immune System
Making Continuous Deployment a Reality
Test Driven Development
Culture
Pascal-Louis Perez, kaChing Group Inc.
#architecture
Pascal-Louis Perez, kaChing Group Inc.
Our architecture
• Service oriented system• Vertical sharding• Everything uses the same platform kawala• Coordination using ZooKeeper• Data interchange using JSON and Protobufs
Pascal-Louis Perez, kaChing Group Inc.
Typical Stack
• Clustered services, multiple instances• Replicated databases (e.g. MySQL or NoSQL)• Caching (e.g. memcached)• Denormalized data (e.g. Voldemort)
Pascal-Louis Perez, kaChing Group Inc.
kawala
• Command pattern. A command is a query• Dynamic Inversion of Control• Queries produce a value
Pascal-Louis Perez, kaChing Group Inc.
Sample Query
class GetUser(Id<User> userId) { @Inject UserRepository repository; public User process() { return repository.load(userId); }}
Pascal-Louis Perez, kaChing Group Inc.
What We do with Queries
RPC Persist Queries
Push into MQ
Command Line Tools
CPS Render Pages
Serve BLOBS
Lifecycle
• 1. Commit• 2. Testing• 3. Deployment• 4. Production Validation• 5. Lifetime Monitoring
Development
• Trunk stable• Small, frequent commits• Unreleased features live behind experiments• Forward & Backward compatibility• Responsive code review• Pair difficult problems• Trivial rollbacks
#testing
Testing Philosophy
• Only automated testing matters• If it isn’t tested, it isn’t finished or correct• Write testable code• Embrace abstractions• Testing is cross functional, we all own quality
Pascal-Louis Perez, kaChing Group Inc.
Benefits of TDD
• It allows quick iterations• It empowers engineers to change anything,
and as such helps in scaling the team• It is more cost effective than debugging• It obsoletes the need for functional QA• It facilitates continuous refactoring, allowing
the code to get better with age• It attracts the right kind of engineers
Types of Testing
• Unit Testing - does the code work?• Integration Testing - does the code work
together?• Regression Testing - learn from your
mistakes• Frontend Testing - a whole different
ballgame
Defensive Testing
• Nightmare scenarios• A common conversation at lunch:– Alice: What would happen if X blew up?– Bob: uh... the site would go down.
• Fix it, test for it
Implementation
• Frameworks: junit, dbunit• CI: Hudson• Frontend: Selenium
Production
• Process, Infrastructure, Deployment, Monitoring
Process
• Everyone: writes code and tests, releases to production, adds monitoring– Specialists help out when needed
• New hires push code the first day• Problems, issues, errors, bugs, oversights...
Stability
• Cluster everything• As little global state as possible• Maintain global state through ZooKeeper• Monitor everything
ZooKeeper
• Reliable Distributed Synchronization• For:– Service Discovery– Service Status– Coarse locking
Pascal-Louis Perez, kaChing Group Inc.
Life of a Deployment
DM
ZK
PMPM PM
Pascal-Louis Perez, kaChing Group Inc.
Life of a Deployment
DM
ZK
PMPM PM
Pascal-Louis Perez, kaChing Group Inc.
Life of a Deployment
DM
ZK
PMPM
Pascal-Louis Perez, kaChing Group Inc.
Life of a Deployment
DM
ZK
PMPMPM
Pascal-Louis Perez, kaChing Group Inc.
Life of a Deployment
DM
ZK
PMPMPM
Deployment
• Exponentially increasing group sizes• Increased monitoring during deployment• Self Test• Automated rollbacks
Implementation
• Building: ant• Deploying: rpm, yum• Monitoring: nagios, jcollectd, RRDtool, jmx• Custom automation
#monitoring
Monitoring Philosophy
• Prefer business metrics• Monitor statistical deviations not absolute
values• Automatically annotate graphs
Monitoring Errors
• False negatives - errors of omission• False positives - errors of implementation
End to End Monitoring
• Run Selenium on production• Accessibility, Speed• Ad-hoc Keynote - customized for our flows• Must control data creation, analytics impact
Pascal-Louis Perez, kaChing Group Inc.
#future-developments
Pascal-Louis Perez, kaChing Group Inc.
Quarantining
• Isolate new releases• Flexible partitioning of requests• Gradually shift load to fresh services
Pascal-Louis Perez, kaChing Group Inc.
Describing Infrastructure
• Many moving parts– nagios, collectd, backups, services, databases, …
• Consistency is key• Adding new tools should be easy and
thorough• Standardize best practices
Pascal-Louis Perez, kaChing Group Inc.
What we Covered Today
• Lean Startup and Continuous Deployment• Anatomy of a Release– Commit, Testing, Deployment, Monitoring
• Future Developments– Quarantining, DSL for Infrastructure
Pascal-Louis Perez, kaChing Group Inc.
References• We’re Recruiting jobs@kaching.com• kaChing’s blog http://eng.kaching.com• kawala http://bit.ly/kawala• Deployment Infrastructure
http://eng.kaching.com/2010/05/deployment-infrastructure-for.html
• Extreme Testing http://bit.ly/9bOFaA• Writing Testable Code
http://googletesting.blogspot.com/2008/08/by-miko-hevery-so-you-decided-to.html
• ZooKeeper http://bit.ly/kc-zookeeper• Eric Ries http://startuplessonslearned.com