Post on 28-Jan-2018
Over the wall development
copyright 2015 Len Bass 3
Board has idea
Developers implement
Operators place in production
Time
Where Does the Time Go?
• As Software Engineers our view is that there are the following activities in software development
• Requirements
• Design
• Implementation
• Test
• Code Complete
• Different methodologies will organize these activities in different ways.
• Agile focuses on getting to Code Complete faster than with other methods.
4
Developers
implement
copyright 2015 Len Bass
What is wrong?
• Code Complete Code in Production
• Between the completion of the code and the placing of the code into production is a step called: Deployment
•Deploying completed code can be very time consuming because of concern about errors that could occur.
5copyright 2015 Len Bass
What is the work flow for code from a multiteam development effort
• You develop and test your code in isolation
• Your code is integrated with code developed by other teams to see if an executable can be constructed.
• The built system is tested for correctness
• The built system is tested for performance and other qualities (staging)
• The built system is placed into production
copyright 2015 Len Bass 6
Errors can occur at every stage
• Every error must either be corrected or prevented.
• Preventing errors can be done through some combination of
• Process
• Architecture
• Tooling
• Coordination among teams.
• Coordination takes time.
• Correcting errors takes time
© Len Bass 2015 7
How much time?
• Historically, releases are scheduled for once a quarter or once a year to give time to coordinate and adequately test.
• This means there may be months delay before a new concept or feature is added to a system.
• This delay has become more and more unacceptable.
• Weekly or daily releases are becoming the norm.
copyright 2015 Len Bass 8
Goal of DevOps
• The goal of DevOps is to reduce the time to market without compromising quality by• Reducing errors during deployment
• Speeding up incident handling after deployment
• Speeding up deployment
copyright 2015 Len Bass 9
Example for reducing errors during deployment
• Use microservice architecture
• Each team makes its own choices of technology for the service it is developing
• Avoids inconsistencies in versions and technology choices that are discovered during deployment process
© Len Bass 2015 10
Example for speeding up incident handling
• Developers carry pagers and are the first respondents to an incident
• When an incident occurs developers do not need to wait for the incident to be escalated through a ticketing system
• Gives developers cradle to grave responsibility for a service
© Len Bass 2015 11
Example for speeding up Deployment (Deployment pipeline)
• Developer creates and tests code on local machine.
• Checks code into a version control system
• Continuous integration server (CI) builds the system and runs a series of integration tests.
• After passing the tests, the system is promoted to a staging environment where it undergoes more tests including performance, security, and user acceptance tests.
• After passing those tests, the system is promoted to provisional production where it undergoes even more tests.
copyright 2016 Len Bass 12
Pre-commit tests
XBuild Image and Perform Integration
tests
UAT / staging / performance
tests
Deploy to production
Commit
...Pre-commit
testsCommit
Developers
promote to normal
production
Just to summarize
• Placing a system into production can be error prone and time consuming
• Supporting a system after it gets into production can be time consuming and error prone
• DevOps is a set of practices intended to speed up placing a robust system into production
© Len Bass 2015 13
Context for DevOps course
• I teach a DevOps course as a portion of the CMU MSE professional master’s program
• Students are graduate students at CMU. Some have industrial experience, some right out of undergraduate school
• Currently in the fourth offering of the course
© Len Bass 2015
Course structure
• Half theory
• Lectures on video to be seen prior to class
• Textbook and other readings
• In class discussion about lectures and readings
• Half assignments
• Assignments involve a variety of open source tools
• Evaluation based on theory and assignments
© Len Bass 2015
Discussion examples
• What was your takeaway from the videos and readings?
• Each student is called on
• Generate a design of a relevant tool, e.g. Continuous Integration server
• Done in groups
• Time in discussion matches time in video
• Balances out of class reading/video with inclass time spent discussing
© Len Bass 2015
Assignment
• Sample assignment:
• Create two virtual machines
• Install SSH so that one machine can securely connect to another without being prompted for a password
• One page discussion: Discuss the pluses and minuses of having “known hosts” in SSH
• Each assignment takes roughly half a day
© Len Bass 2015 18
Syllabus
• You can retrieve the syllabus from
https://github.com/devopseducator/2016workshop
© Len Bass 2015
Major topics
• DevOps overview
• Networking
• Cloud
• Deployment pipeline
• Architecture
• Deployment
• Security
• Monitoring
© Len Bass 2015 20
Elaborating on the syllabus - networking
• it is amazing what Masters students in CS at CMU do not know. A reflection of their undergraduate curriculum.
• Virtualization
• Virtual machines
• Virtual machine images
• Networks
• IP addresses
• DNS servers and TTL (time to live)
• Ports
• Load Balancers
© Len Bass 2015
Elaborating on the syllabus –cloud
• Failures in the cloud
• Distributed coordination – Zookeeper, Consul
• Scalability
© Len Bass 2015
Deployment pipeline
• Introduction to a deployment pipeline
• What are the roles and responsibilities of various teams with respect to the pipeline
• Case study in business continuity
• Environments of the pipeline in more detail with types of tests
• Development
• Build
• Staging
• Production
• Case study of company that sets up continuous deployment pipelines
© Len Bass 2015
Elaborating on the syllabus –architecture and advanced
packaging
• Microservice architecture
• Case study in migrating to a microservice architecture
• Containers
© Len Bass 2015
Elaborating on the syllabus –deployment
• Designing for deployment
• Use of feature toggles
• Backward and forward compatability
• Managing schemas
© Len Bass 2015
Elaborating on the syllabus –security
• Basic security
• Credential management
• Securing the network
• Securing the pipeline
• Securing an app
© Len Bass 2015
Elaborating on the syllabus –monitoring
• What to monitor
• How to monitor
• Interpreting monitoring data
© Len Bass 2015
Assignments – all open source except AWS
• VirtualBox - virtualization
• SSH – secure connection between VMs
• Vagrant – tool for creating and provisioning a single VM
• Jenkins – continuous integration server
• Chef – configuration management
• Docker - containers
• AWS EC2 Container Service
• Chef Vault – secure configuration management
• Nagios - monitoring
© Len Bass 2015
Student knowledge shortcomings – 1
• Networking 101
• URL - > IP
• Ports
• TCP/IP
• Networking 102
• Subnets
• Firewalls
© Len Bass 2015 30
Student knowledge shortcomings – 2
• Security 101
• Certificates
• Trust
• Vulnerabilities/patching
© Len Bass 2015 31
Student knowledge shortcomings – 3
• Operations
• Deployment/rollback
• Network management
• Incident handling
© Len Bass 2015 32
Proposal
• Establish an “infrastructure” minor to the Software Engineering degree. It could have the following courses:
• Operating systems
• Networking
• Operations/Security
• The cloud
• DevOps
© Len Bass 2015 33
Summary
• DevOps is a set of practices to
• Shorten time to deployment
• Reduce errors during deployment process
• Shorten time to respond to incidents
• A course on DevOps is attractive to students
• There are gaps in knowledge from standard undergraduate CS or SE curricula
© Len Bass 2015 34
More information
• Textbook
• https://github.com/devopseducator/2016workshop
• https://github.com/devopseducator/2017workshop
© Len Bass 2015 35