Testing in a continuous delivery world - continuous delivery Amsterdam meetup
Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery Meetup
-
Upload
simon-storm -
Category
Technology
-
view
526 -
download
2
description
Transcript of Agile and Continuous Delivery for Audits and Exams - DC Continuous Delivery Meetup
1
Positioning Agile and
Continuous Delivery for
Auditors and Examiners
2
Where to Start
The single most important step in preparing for an audit or
examination is to put yourself in the auditors shoes and
understand their goals:
• Does this entity have a sound development practice?
• Do they have repeatable processes that ensure
consistent results?
• Do they have the appropriate controls in place?
• Does the management team understand the risk they
are exposed to?
3
Taking a Step Back…Let’s Start with the Bible
During an examination, the examiner explained that he
wanted to see our “Bible”, aka our SDLC. He wanted every
step to be documented and auditable so he could be sure
that every project followed the exact process, every time.
Credit: http://www.stpatselkhorn.org/AdultFormation/BibleStudy.aspx
4
How We Responded
1. The Mammoth Waterfall SDLC
2. The Mammoth SDLC & SDLC Lite
3. Agile SDLC
4. Agile & Continuous Delivery
5
Enough about us…
We have turned the corner and are now reaping the
rewards of properly implementing Agile and Continuous
Delivery.
We now find that WE HAVE TIME to automate and
strengthen our processes.
Let’s get to the 25 things you can do to better prepare for
your next audit or exam!
6
Tips and Tricks for Audits and Exams
1 - 6 : Agile Education
7 - 12 : Continuous Delivery Education
13 - 18 : Demonstrating Maturity
19 - 21 : Orchestrate for Improved Quality
22 - 24 : Source Code Control is KEY
25 : Getting Ahead
7
Agile Education
Credit: http://flickfacts.com/movie/4925/back-to-school
8
#1 – Socialize Your Plans
Don’t surprise your auditor with a major change to your
process.
Provide Useful Information:
• Agile Overview:
https://www.youtube.com/watch?v=502ILHjX9EE
• Continuous Delivery Overview:
Continuous Delivery: Reliable Software Releases Through Build, Test
and Deployment Automation by Jez Humble and David Farley
• Continuous Delivery Adoption:
http://www.thoughtworks.com/insights/blog/case-continuous-delivery
http://www.perforce.com/continuous-delivery-report
9
#2 – Don’t Risk the Crown Jewels
If possible, demonstrate the new technologies and
procedures on a lower risk application.
You will thank me later….because there will be bumps
If you do start with a major application, find a way to
segment the implementation to minimize the up front risk
10
#3 – Demonstrate Your Expertise
While many of these technologies and procedures are not
new, they may be new to you or your organization. Make
sure you can demonstrate your expertise:
Certifications - Scrum Alliance, etc.
Training Programs – Learning Tree, Scrum Alliance, etc.
Meetups & User Groups – Continuous Delivery, Agile, etc.
Social Media – LinkedIn Continuous Delivery Group, etc.
11
#4 - Map Agile SDLC to Waterfall SDLC
Design Waterfall Agile
Design The entire application is designed at one
time
The design evolves as the application is
developed
The design is created by technical resources
working from the requirements
The design is created by the developers
working with the key stakeholders
The design is based on the best estimate of
how the application is used
The design is based on customer behavior
Design Review The design is reviewed by technical
resources to ensure completeness and
accuracy
The design is shown as a working solution to
the Product Owner and other stakeholders
Changes to the design may have a major
ripple effect to the rest of the application
The design is continually revisited and
adjusts to customer need
Design Sign Off Specific step where designated parties agree
that the design is complete and accurate
Implicit to the process when everyone
agrees that the work is acceptable to go to
production (Sprint Review)
12
#5 – Explain Benefits of Shorter Cycle Time
When a vulnerability is found, how quickly
can you address it?
When a new OS patch is released, how long
until it is on all of your servers?
13
#6 – Explain How Small Batches Reduces Risk
• Schedule risk
– Feature creep
– Gold plating
• Quality risk
– New bugs
– Instability
• Business risk
– Wrong functionality
– Missed opportunity
14
Continuous Delivery Education
15
#7 – A More Auditable Process
The key takeaway….
An automated process is far more auditable!
16
#8 – Correct Version of the Application
Everyone needs environments and now there are great tools
that make it even easier to enable environment sprawl.
Every developer has a local environment
3 Development environments
4 QA environments
4 Staging environments
4 Production environments
17
#9 – Infrastructure as Code
1. Baseline Image
– The latest patched base server OS, ssh, etc
2. Apply common applications (that require configuration)
– TripWire, Splunk, PostFix, etc
3. Application critical applications
– Java, App server, etc
4. Deploy your software
** Even with configuration management, you still need a tool like TripWire
18
Infrastructure as Code – Benefits
• Environments stay in sync
– Changes are made in development and migrated
– Administrators should not make changes directly to environments
– Changes made manually to an environment are undone with the
next migration through the pipeline
• Environments can be built on demand
– Becomes faster to rebuild an environment than to troubleshoot
– A process to build an environment that took weeks can now be
completed in under an hour
– Environments will no longer be a bottleneck to new functionality
• Environments are documented and version controlled
– Each setting change is a line of code that can be read
– All configurations reside in GIT so that the team can recover or
revert to a prior configuration
19
#10 – Static Code Analysis
20
Sonar – Security Tests
21
Sonar – Test Changelog
22
Sonar – Additional Tracking
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
Number of Issues
Issues
Issues - Blocker
Issues - Critical
Issues - Major
Issues - Minor
Issues - Info
23
#11 – Automated Testing
Automated tests are the answer to MANY questions about
reducing risk….but they open the door to a whole new
world of questions
• Who validated that the automated test worked
correctly?
• How do you know that the test meets the desired
result?
• How can you be sure you have sufficient coverage?
• Where are the tests for specific user stories?
24
User Acceptance Test
25
#12 – Repository Management
Single source for software, binaries & libraries
demonstrates:
• Consistency across environments
• Single, auditable repository of external resources
• Control access to external sites
26
Demonstrating Maturity
Credit: http://ihkstories.com/maturity-is-not-when-we-start-speaking-big-thingsit-is-when-we-start-understanding-small-things/
27
#13 – Go Digital
Online Agile Boards
An Auditor once pulled a sticky off our physical board that
was in the Ready for Test queue. He asked “if I don’t put
this back, how do you know this was tested?”
28
#14 – Automating Sign-Offs
Credit: http://www.polscheit.de/plugins/jira/group-sign-off/images/GroupSignOff-Banner.png
29
#15 – Automating Documentation
Credit: http://jiraxporter.xpand-it.com/download/attachments/327684/Banner.png?version=1&modificationDate=1364461203281&api=v2
30
Bank Assetpoint Agile Implementation
Retrievedfrom Jira
Retrievedfrom Jira
31
#16 – Logging Pipeline Activity
32
#17 – Capturing Meaningful Metrics
0
10
20
30
40
50
60
70
80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Positive Sprint Quality Trend
0
2
4
6
8
10
12
14
16
18
1 2 3 4 5 6 7 8 9 10
Sprint 2014-1
Done QA In Progress Backlog
33
#18 – Add one more meeting
Sprint Planning Review Meeting
• Additional demonstration of oversight
• Shows that we are willing to adapt to meet company
goals
• Great catch-all for interested stakeholders
34
Orchestrate for Improved Quality
Credit: http://accupackmidwest.com/quality-control/
35
#19 – Keep QA Firmly in the Process
• When new code comes into Test Environment
• When new code can be moved to a higher environment
• Perform the deployment to the Staging Environment
• Perform the deployment to Production Environment
36
#20 – Don’t Forget Operations
The System Engineering
Team to controls when
code can enter the
Staging Environment
Application Engineering
Team controls when
code can enter the
Production Environment
37
#21 – When All Else Fails – Email!
Email notifications keep parties informed
Security
Compliance
Management
Operations
Product Owner
38
Source Code Control is KEY
39
#22 – Demonstrate Permissions
Making sure that the appropriate controls are in place in
GIT are critical. You will need to use a management tool on
top of GIT like Stash.
40
#23 –Code Reviews with Pull Requests
41
#24 – Secure Your Pull Requests
Custom GIT Hook
42
Administrator approved pull request alert
43
Getting Ahead
Credit: https://dzihxiql01vk4.cloudfront.net/wp-content/uploads/2013/06/Get-Ahead-with-Repricing.jpg
44
#25 - Be Aware of Outstanding Audit Risks
• Get Ahead of Permission Questions
– Jenkins, Puppet, Nexus, Stash, etc.
• Continuous Improvement means that you are not
following the same process over and over
– Allowing Agile Teams to change their development process to
make themselves more efficient is scary to auditors
• Management (e.g. upgrades) of Pipeline software
• Separation of duties
• Management aware (and approving) work
• Continuous Deployment may be a step too far
– There is a lot of value in ensuring that humans are involved in
the process
45
Questions