Netflix Open Source Meetup Season 4 Episode 1
-
Upload
aspyker -
Category
Technology
-
view
15.049 -
download
2
Transcript of Netflix Open Source Meetup Season 4 Episode 1
Netflix Open SourceSeason 4, Episode 1
Feb 2016
Agenda
● Welcome● Netflix Open Source Evolution
○ @aspyker● Spinnaker
○ @ajordens, @mjduftler, @rcarunmsft, @dstengle● Demo stations and reception
Open Source since 2010● Architecture has shaped public cloud usage
○ Immutability, red/black deploys, chaos, regional and worldwide high availability
● 102 Open Source Projects in Netflix org
Showing some age● Across our projects
● Open issues○ Astyanax (148), Asgard (106), Ice (74),
Total (1631)
● Open pull requests○ Astyanax (44), Ice (23), Asgard (18),
Total (258)
● Some issues & PR’s have been around for a while
Open Issues Per Project
Open PR’s Per Project
What are we doing to evolve?
● Establish ownership and group logically○ Management and engineering aligned with business
● Classify projects into lifecycles○ Clearly identify active vs. retired projects○ Clearly identify experimental projects
● Track active projects○ Actively work to improve project “health”
Establish Ownership and Grouping
● Consistency across related projects
● Document how area fits together
● Focus on OSS health of functional area
OSS Area Shepherds
Already taking action
● Astyanax, Asgard, ICE are either not used or being transitioned away
● Starting to document this better
● OSSMETADATA tagging fileosslifecycle=active | maintenance | archived
● You can see these today○ Eventually badges in README.md
● Allows for a tool to automate lifecycle verification
How do we automate this?
Our Dashboard: Netflix OSS Tracker
OSS TrackerNightly Jobs
Project Ownership Database
Time SeriesProject Statistics
OSS TrackerConsole
● Collects metrics on every project every day● Allows shepherds to see combined stats for their area● Allows project owners to see daily metrics
Whitepages Service
github.com/Netflix/osstracker
OSS Tracker Future (Want to help?)
● Pluggable whitepages & database systems● Collect more metrics from github
○ Time bucketed (last 60 days vs. forever)○ Internal vs. external contributors○ Tagging to allow for issues as features, etc.
● Create single consumable health score● Emailing reports to shepherds and owners● Better user interface
The Netflix Continuous Delivery PlatformDevelopment began in 2014Open sourced 11/2015
Spinnaker
Evolution of CD at Netflix
Asgard
Open Sourced in 2012
Was the primary deployment tool until 2015
A better AWS Console
Codified Conventions- Red/Black- Naming
Pitfalls
Single Account
No first-class API
Mimir
Internal Netflix
Script Support (Jenkins)
Flexible Workflows
Used the “Asgard API”
Edge Center
Internal Netflix
Canaries
Automation
Spinnaker
Application-centric UI
Flexible Pipelines
RESTful API
Multi Account
Canaries (Netflix-only)
https://www.flickr.com/photos/spidermandragon5/2922128673
● Intuitive UI (Self-service)● Moving Beyond Simple Red/Black● Out-of-the-box Best Practices
!! Community Participation !!
Goals
Abstraction
UI Components
Pipeline Stages
Let’s Take a Tour
Contain
er
Suppor
t
PluggableComponentsUI and Services
DeclarativeContinuousDelivery
Perform
ance
Documentatio
n
Community Site /
Whitepapers /
FAQ
● Netflix and Spinnaker principles align with Google’s
○ Massive infrastructure, services at scale
○ Immutable infrastructure○ Continuous delivery○ Traceability○ Self-service tools
Why is Google investing in Spinnaker?
● >30,000 developers in 40+ offices● 13,000+ projects under active development● 30k submissions per day (1 every 3 seconds)● Single monolithic code tree with mixed language code● Development on one branch - submissions at head● All builds from source● 30+ sustained code changes per minute with 90+ peaks● 50% of code changes monthly● 150+ million test cases / day, > 150 years of test / day● Supports continuous deployment for all Google teams!
Internal Google Development
● Developing out GCP toolset● Our internal platform and tooling is container
oriented● Spinnaker is a great starting point for VM
oriented tooling● Build out functionality by contributing to OSS
that aligns with our practices and goals○ Bazel, Kubernetes, ...
Google Cloud Platform
Let’s See it in Action
MAKENEXTPOSSIBLE
● Professional Services NetflixOSS○ Microservices and Cloud
● Spinnaker Launch Contribs○ Installation and VM Images○ Docs
Kenzan & Spinnaker
● Reduce Complexity of a true E2E Spinnaker CD Setup
● Support AWS and GCP
Terraform Spinnaker
What you get● Spinnaker● Jenkins● Apt repo● Networking● Example App● Jobs and Pipelines
What’s Next● Productionized Terraform
Deployment● More sophisticated continuous
deployment pipeline examples
Demo Stations
● Spinnaker○ Snag Spinnaker stickers○ Netflix Spinnaker demos○ See Google-Spinnaker integration○ Meet the Azure team
● Kenzan○ Spinnaker Terraform demo○ Learn about their NetflixOSS consulting
Demo Stations
Come hear about Dynomite updates● Dynamo layer to make a non-distributed
data store into distributed○ Redis/memcache, HA scale-out, encrypted cross-region replication
● Quorum consistency● Data warm up● Speed improvements in latest release
○ 12 Dynomite nodes >1M OPS
Wrapup
● Thanks for attending!
● Join us for○ Food and drinks○ Demo stations