Continuous Security Testing in a Devops World #OWASPHelsinki
Enterprise DevOps - Software Testing ConferenceEnterprise DevOps: Testable Business Value (TBV)...
Transcript of Enterprise DevOps - Software Testing ConferenceEnterprise DevOps: Testable Business Value (TBV)...
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs1Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs1
Enterprise DevOps: Testable Business Value (TBV) driving
Continuous Delivery & Deployment
Amdocs Testing Services
Harsh Agrawal
Ashish Agrawal
Durgesh Joshi
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs2
Harsh has 23 years of diverse experience in development, infrastructure and testing, and has been
with Amdocs for 17 years. Throughout his career, he has helped Tier-1 and Tier-2 service providers
across the globe achieve successful delivery of testing programs and TEM services.
Harsh Agrawal
A quality thought leader with 18 years of experience in organizational-wide ‘quality function’
deployment and continual improvements, Durgesh has been with Amdocs for nine years. He currently
supports testing pre-sales and leads the Test Centre of Excellence (TCoE) implementation, Test
Assessments and measurements/benchmarking programs.
Durgesh Joshi
Ashish Agrawal
Authors
Ashish has 19+ years of vast experience in development and testing. Ashish has led several testing
programs with the teams spread across multiple global delivery locations. His testing experience
spans across Telecom, Financial Services, Energy Trading and Risk Management, Public Sector,
Airlines domains.
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs3
So everyone develops in Agile. What’s the problem?
Until code is in production, no value is
actually being generated, because it’s merely
WIP stuck in the system.
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs4
Problem Statement
Product Backlog
Story -1
Story -2
Story -3,4
Story -5,6,
Story -8,9
Story -10,11
Story -12-15
E2E Testable ? Value to customer?
E2E Testable ?
E2E Testable ?
Value to customer?
Value to customer?
PI 01
PI 02
PI 03
Vendor-1 Vendor-2 Vendor-3
Large Complex Multi-Vendor SI environment
Continuous Delivery
DevOps
Continuous Integration(Silo agile cycles)
Constraint at Continuous
E2E/UAT
(due to the
Iterations/Sprints getting
queued up)
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs5
Then Each Development Iteration Needs To
Create Something One Can Actually Test & Deploy
Then We Moved To Agile
Need in todays demanding world..
Development and Testing as Waterfall
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs6
Testable Business Value (TBV)
Project PI 2
1-2 Weeks
Project PI 3
1-2 Weeks
Project PI 1
1-2 Weeks
Every PI starts from Scoping in which TBVs are defined
Development & Testing are done continuously per TBV, enabling production deployment of each TBV Separately
TBV
(Testable Business Value)
An entity defined by the Test Architect during scoping, to describe the integrated scope element which the testers must receive in order to certify scope which carries business
value and supports E2E tests
TBV Example A loyal customer with multiple connections, looking for reduced bill amount
due to collective usage via all connections
Continuous Testing • Automated/Exploratory testing activities occur seamlessly in parallel using TBVs
• After each stage acceptance move to the next level and finally to production
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs7
ESTABLISH TBV
PO, BA, Test Architects,
Solution Manager
PO, BA
DEFINE BACKLOG
TBV Creation Process
Feedback
CUSTOMER
Gives High Level
Requirements
Dev
Design
Commit
Code
Test Design
Test
Auto. Build &
Test
Acceptance
Test
Staging
Deploy To
Prod
Production Value!!
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs8
Actual Example
BR-1 : Setup New Family Plan
for Mobile/Data service with
logic as 1st subscriber under
same account will be charge
actual amount, 2nd
subscriber with same Plan will
get 50% discount, 3rd
onwards will be charged with
20% of actual RC rate
BR-2 : Ability to provide this
offer to BYOD(bring on your
device) device
BR-3 : Ability to provide free
voice service & data charges
having Family Plan
TBV01 : As a CSR, I will be able to change Plan to new
Family Plan for subscribers having existing account, so customer will be eligible to avail this offer of Family Plan of Less charge due to multiple subscription
T1F1: Ability to Setup Family PPs in EPC system
Testable Deliverable Deployable
A loyal customer with multiple connections, looking for
reduced bill amount due to collective usage via all
connections
TBV02 : As a CSR, I will be able to provide this offer to
New customer opting for Family Plan with New device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV03 : As a CSR, I will be able to Provide Family offer
to new Customer opting of Family Plan having their own
device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV04 : As a Family Plan Customer, I will be able to
make unlimited voice Call under same account & all data usage free for such subscribers, so Customer has to pay only for Flat Plan charges for all subscribers under ONE account
T1F2: Ability to show all family Plans on CRM
screen, once Agent select Family Plan button for all existing Customers
T1F3: Ability to upgrade subscriber service with
Family Plan.
T1F4: AR routine should be enhanced to calculate
RC for each subscriber such a way, if its 1st subscriber with Family Plan then 100% RC, 2nd sub
having 50% RC & 3rd PP onwards 20% RC
TB01 - FEATURES
“1”
DEFINE
BUSINESS
REQMTS.
“2”
DEFINE
TESTABLE
BUSINESS
VALUE
(TBV)
“3”
DEFINE
FEATURES
FOR
RELEVANT
TBV
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs9
Actual Example
BR-1 : Setup New Family Plan
for Mobile/Data service with
logic as 1st subscriber under
same account will be charge
actual amount, 2nd
subscriber with same Plan will
get 50% discount, 3rd
onwards will be charged with
20% of actual RC rate
BR-2 : Ability to provide this
offer to BYOD(bring on your
device) device
BR-3 : Ability to provide free
voice service & data charges
having Family Plan
TBV01 : As a CSR, I will be able to change Plan to new
Family Plan for subscribers having existing account, so customer will be eligible to avail this offer of Family Plan of Less charge due to multiple subscription
T2F1: Ability to Setup Family PPs in EPC system
Testable Deliverable Deployable
A loyal customer with multiple connection, looking for a
solution as reduced bill by leveraging benefit due to
collective usage via all connections
TBV02 : As a CSR, I will be able to provide this offer to
New customer opting for Family Plan with New device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV03 : As a CSR, I will be able to Provide Family offer
to new Customer opting of Family Plan having their own
device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV04 : As a Family Plan Customer, I will be able to
make unlimited voice Call under same account & all data usage free for such subscribers, so Customer has to pay only for Flat Plan charges for all subscribers under ONE account
T2F2: Ability to purchase device & select Family
Plan on CRM screen for new Customer
T2F3: Ability to attached Family Plan with device &
do activation
T2F4: AR routine should be enhanced to calculate
RC for each subscriber such a way, if its 1st subscriber with Family Plan then 100% RC, 2nd sub
having 50% RC & 3rd PP onwards 20% RC
TB02 - FEATURES
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs10
Actual Example
BR-1 : Setup New Family Plan
for Mobile/Data service with
logic as 1st subscriber under
same account will be charge
actual amount, 2nd
subscriber with same Plan will
get 50% discount, 3rd
onwards will be charged with
20% of actual RC rate
BR-2 : Ability to provide this
offer to BYOD(bring on your
device) device
BR-3 : Ability to provide free
voice service & data charges
having Family Plan
TBV01 : As a CSR, I will be able to change Plan to new
Family Plan for subscribers having existing account, so customer will be eligible to avail this offer of Family Plan of Less charge due to multiple subscription
T3F1: Ability to Setup Family PPs in EPC system
Testable Deliverable Deployable
A loyal customer with multiple connection, looking for a
solution as reduced bill by leveraging benefit due to
collective usage via all connections
TBV02 : As a CSR, I will be able to provide this offer to
New customer opting for Family Plan with New device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV03 : As a CSR, I will be able to Provide Family offer
to new Customer opting of Family Plan having their own
device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV04 : As a Family Plan Customer, I will be able to
make unlimited voice Call under same account & all data usage free for such subscribers, so Customer has to pay only for Flat Plan charges for all subscribers under ONE account
T3F2: Ability to validate BYOD into Provisioning
system if Customer comes with this own device
T3F3: Ability to allow Customer opting for Family
Plan for BYOD device & do activation
T3F4: AR routine should be enhanced to calculate
RC for each subscriber such a way, if its 1st subscriber with Family Plan then 100% RC, 2nd sub
having 50% RC & 3rd PP onwards 20% RC
TB03 - FEATURES
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs11
Actual Example
BR-1 : Setup New Family Plan
for Mobile/Data service with
logic as 1st subscriber under
same account will be charge
actual amount, 2nd
subscriber with same Plan will
get 50% discount, 3rd
onwards will be charged with
20% of actual RC rate
BR-2 : Ability to provide this
offer to BYOD(bring on your
device) device
BR-3 : Ability to provide free
voice service & data charges
having Family Plan
TBV01 : As a CSR, I will be able to change Plan to new
Family Plan for subscribers having existing account, so customer will be eligible to avail this offer of Family Plan of Less charge due to multiple subscription
T4F1: Ability to make voice call & record in Rating
table with free charges if subscribers are belongs to Single Customer
Testable Deliverable Deployable
A loyal customer with multiple connection, looking for a
solution as reduced bill by leveraging benefit due to
collective usage via all connections
TBV02 : As a CSR, I will be able to provide this offer to
New customer opting for Family Plan with New device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV03 : As a CSR, I will be able to Provide Family offer
to new Customer opting of Family Plan having their own
device, so Customer will be charged as per family Plan setup & he will need to pay less for multiple subscription
TBV04 : As a Family Plan Customer, I will be able to
make unlimited voice Call under same account & all data usage free for such subscribers, so Customer has to pay only for Flat Plan charges for all subscribers under ONE account
T4F2: Ability to capture free RC for all data usage if
customer belongs to Family Plan
T4F3: Ability to allow Customer opting for Family Plan for BYOD device & do activation
T4F4: Billing Routine to enhanced to Calculate Flat
charges for all customer belongs to Family Plan and actual bill to show the impact on the change
TB04 - FEATURES
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs12
Test Architect Role
• Customer Engagement
• TBV Creation
• 1st handshake with
Testing Team
• Automation
Recommendation
• Proactive Ownership
Scoping
• Post Scrum testing
review
• Participate in
Customer demos
• Teamwork
Testing
• TBV driven iteration planning
• Define Scrum testing ability
• Scrum testing coverage mapping
• Participate in Scrum demos
• 2nd handshake with testing teams
Construct
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs13
Results
Value to customer
Product Backlog
Story -1
Story -2
Story -3,4
Story -5,6,
Story -8,9
Story -10,11
Story -12-15
E2E Testable!! Value to customer
E2E Testable !!
E2E Testable !!
Value to customer
PI 01
PI 02
PI 03
Vendor-1 Vendor-2 Vendor-3
Large Complex Multi-Vendor SI environment
Continuous Delivery
DevOps
Continuous Integration(Silo agile cycles)
Continuous E2E/UAT
Sprints getting moved to
next level)
Information Security Level 2 – Sensitive
© 2017 – Proprietary & Confidential Information of Amdocs14
Using BEATTM to enable continuous testing and shift-left
Enabling DevOps
Test-driven
Projects
GINGER Automation
Mobile Reports
Amdocs BEATTM Analytics Analytics
for quality,
speed and
efficiency, dev
and
test metrics
Immediate
feedback;
real-time
test status
DevOps
automation
console:
requires
no coding
knowledge
Amdocs BEATTM Cloud
Design Console
A-TEAM
Methodology-
driven,
cross-site project
management
Business flow
creation broken
into testable
units
Dynamic
environment
configuration
and
management
DevOps - Test-Driven Delivery
Test Data & Environment ManagementDynamic definition and supply in real time
AutomationEliminate Manual Testing
Test Driven DeliverySynchronized Dev-Test Assembly line
Thank you