Test Paradigms:- applying test integrity under challenges › wp-content › uploads ›...
Transcript of Test Paradigms:- applying test integrity under challenges › wp-content › uploads ›...
25 May, 2017
Sudeepta Guchhait
Test Paradigms:- Applying test integrity under challenges
2
Schema
3
Exchange - Big Rock Implementation
Mega test project
Unique challenges
Our journey – approach towards continuous testing
Critical success factor
Special about project
Guide you through the challenges faced during test life cycle and how to master the mitigation approach. Speak about how to succeed in continuous test improvement
Exchange - Big Rock Implementation
4
… to stay in the game, we needed a single multi-asset platform with new features to support end-to-end workflows of derivatives communities
Then culture - Why were we upgrading? Applications, Technology & Hardware end of life. Lower performance, lesser interoperability, Lack of efficiency & interoperability btw platforms
Adoption - Changes brought! New platform for Listed derivatives, OTC derivatives. Integrated Clearing & collateral mgt systems Physical gateways 2 new low latency protocols Operation & process improvements
Derivatives Platform OTC
FIX 5.0 GUI OMNet OUCH GUI
Interfaces
Functionalities
Orders
Quotes
Trade Reports
Market Data
Risk Controls
Clearing & CMS
OMNet API
OUCH
ITCH
Pre Trade Risk Control (API + GUI)
OMNet API
FIX 5.0 API
OTC GUI
Trade Reports / Market Data / Risk Controls
Clearing Workstation
Other Benefits
5
+ Embedded modern tools to improve delivery efficiency at Xchange.
+ 22 hours trading longest trading hours in Asia (6:45AM – 4:45AM)
+ Improved co-location culture.
+ Lower latency and higher throughput with New application
+ A reduction in operational costs and headcount for BAU activities
+ Processes that enable frequent releases with higher confidence.
What is a Mega - testing project?
6
Scale
Size
Scope
• Complex business
• Business impact
• Critical enterprise applications
• Multi application vendors
• Massive budget
• Large team
• Multiple test environments
• High number of applications/downstream
• Usage of many test tools
• Extensive business functions
• Millions of transactions
• Business process
• User stories (requirements)
• Hybrid SDLC
What Business Demands?
7
Strive towards “0” prod incidents
Waterfall + Agile SDLC
Driving automation
Continuous Integration
Project outcomes & benefits
BUSINESS FACING
TECHNOLOGY FACING
MEM
BER
FA
CIN
G
Trading & Clearing ITCH/OUCH protocols Glimpse for data vendors Trade handler Risk Groups
Collateral Management Risk Management External system
Reporting Dashboard Heat maps
Market data Data Analytics Algo System Central Data source Market Surveillance
Unique Challenges
8
Platform transformation + multiple applications
Cultural issues
Multiple application vendors
Hybrid SDLC
Imposing different types of test
Testing – “Design for failures”
End to end application test is difficult to follow
Reliability & Stability of application
Availability/Maintenance of test infrastructure
Not enough time – (tight deadline)
Managing complexities - (missing requirements, embracing CRs)
Pressure to deliver on time - (KPI project, reputational risk)
Cross integration test – (application + data + diversified api)
Our Journey – “Mega testing” – Are you ready?
9
Application Delivery Challenges
10
Design & Development
Constrained Environments
Less time, tool time
Manual testing
Too many disparate tools
Automation (F /NF)
Production Roll Out
Waiting time Delays Delays Delays
Build Test/Deploy Implement Test/Deploy Test/Deploy
Developers Testers Release Team Prod Support /
Operations
Code | Integrate | Build Functional / API / Automation
Automation + NF testing Deploy to Prod defects defects defects
Re-execute Re-execute Re-execute Re-execute
SLOW & INEFFICIENT EXCESSIVE IDLE TIME, DELAYS AND LOT OF REWORK
Hurdles We Had To Overcome
11
Manual testing was comfortable to everyone
Hybrid life cycle
Downstream test automation
API / FIX protocol testing
Skill transformation & Capability uplift
Data Migration testing
NF testing (Perf, Capacity, Failover & Latency)
One business day simulation
Critical path/Pattern testing
Embraced Test Harness (Risk based & Combinatorial test)
Bi-Directional traceability
Timeline – Improvement Journey
Re-defining strategy 1. Standardise test process 2. Identify quick wins 3. Define strategic
solution 4. Define tactical solution
Launch Derivatives Platform.
2015 / Q3
2016 / Q1
2016/ Q2
2016 / Q3
2016 / Q4
12
Building Champions 1. Cross training 2. Developing automation
suite 3. Driving sense of
competition 4. Development of
platform focused tactical automation
5. Realise benefits from early automation
6. Engage members
Platform test 1. Automated test (API) 2. User Acceptance test 3. Automated downstream 4. Non functional test 5. Drive upstream testing
Continuous improvement 1. Infuse DevOps 2. One business day simulation 3. Behaviour driven testing 4. Industry wide testing 5. Member test
Business Impact Driven Test Approach
Assess the business impact of each activity Assess the probability for failure Determine the risk based on impact and probability
Application 1
Main Functionality 1.1
Function 1.1.1
Requirement
Requirement
Requirement
Requirement
Requirement
Requirement
Function 1.1.2
Main Functionality 1.2
Function 1.2.1
Requirement
Structuring Test Requirements
Determining the Risk
Analysing Business Impact
Analysing Probability to Fail
13
Criteria \ Results A
High Risk B
Medium Risk C
Low Risk
Types of Transaction / Algo
Calculation/Validation
Change of assignment
Display
Business impact Legal Wrong
information None
Frequency of usage Very Often Often Rare
Number of members affected
Platinum Gold Few
Criteria \ Results 1
Most likely 2
Possible 3
Unlikely
Change in API New
messages Changed message
Unchanged
Functional impact Upstream Down
Stream Support
Impact to member Critical Manageable Internal
HIGH
MEDIUM LOW
Impact \ Probability III
Unlikely II
Possible I
Most Likely
Impact to Business Least Recovery Financial
Impact to Reputation Least Manageable Severe
MAS Reportable None May be Yes
Tactical Solution
Application 1
Main Functionality 1.1
Function 1.1.1
Requirement
Requirement
Requirement
Requirement
Requirement
Requirement
Function 1.1.2
Main Functionality 1.2
Function 1.2.1
Requirement
Functional Complexity Determines Test Effort
Analyse the functional complexity together with technical architect Determine test effort by combining risk & complexity
Structuring Test Requirements Analysing Functional Impact
Analysing Probability
14
Criteria \ Complexity I
High Complex
II Medium Complex
III Low
Complex
Number of Functional application pretentious
> 11 6 – 11 < 6
Number of Calculus application affected
> 04 2 – 4 Just 1
Number of Dashboard affected > 05 1 – 5 None
Impact \ Complexity I
High Complex
II Medium Complex
III Low
Complex
Critical 10, 5 8, 2 6, 3
Important 6, 9 6, 3 5, 6
Nice to have 4, 2 3, 5 2, 8
Tactical Solution
The Risk Determines the Test Approach
15
Technique: Systematic testing using critical component testing, API messages and root-cause analysis
Approach: Automated: 30% Manual: 70%
Tactical Solution
HIGH Risk
A MEDIUM
Risk B
LOW Risk
B
Technique: Systematic testing with patterned & combinatorial data points testing, and root-cause analysis
Approach: Automated: 20% Manual: 80%
Technique: Systematic or ad hoc testing
Approach: Automated: 10% Manual: 90%
Shift Left - Automation Facts
16
Tactical Solution
Requirements Design Development Testing Production
/Maintenance
Trad
itio
nal
Sh
ift
Left
Test Planning Automation Scripting / Execution
Test Planning
Scripting & execution Scripting & execution
API / Message Automating 30%
FIX Automating 20-30%
Behavior / Pattern Automating 20%
Combinatorial Automating 50%
GUI related Automating 30%
Regression Automating 40%
Better utilization of resources Early detection of issues Cost effective Fast time to market
32
Combinatorial Automation
17
Tactical Solution
Multiple test data combination Patterned behavior (Futures/Options/SWAP) Different Markets Month / Product Category / Commodity type Different session state (T or T+1)
Mapping Requirements Different workflow
Profiles Approval Schemas Algorithm validation
18
12
46
0
10
20
30
40
50
Volumes handled / Human Volumes handled / Robot
# o
f T
ran
sact
ion
s/d
ay
Volumes handled
2.4X Faster
1:3 Auto :
Resource
180% Productivity
Gain
40% Resource Reduction
35
12
0
10
20
30
40
Before Automation After Automation
Tim
e (
Min
s)
Average Handling Time
100%
40%
0%
20%
40%
60%
80%
100%
120%
Before RPA After RPA
# o
f F
TE
s
Resource Reduction
Before Automation After Automation
Tactical Solution
Shift Left - Automation Facts
Reactive to Proactive - Non Functional testing
19
Tactical Solution
Challenges • Workflow dependency - (multiple applications)
• Application speed • Application stability • Mimic Production scenario • Design for failure - (H/w, Process, Node)
Reactive to Proactive - Non Functional testing
20
Tactical Solution
Integrated Non Functional testing in Continuous Delivery
NF Testing Solutions
Detect H/w Failure & root cause analysis
Application profiling & root cause analysis
Response time
Throughput
Processor-utilization
Endurance
Interoperability
Node failure
Processor failure
Stability
Recovery
Achieving Resiliency - One Production Day Simulation
Member Test & Industry Wide Testing
21
Strategic Solution
Trading & Clearing APIs
Simulation with member/broking firm products
Replication of Critical path scenarios
Business Continuity scenarios
Performance / Failover & Fallback test
Dashboard & Reports to Regulatory body
Real time reference & Market Data
Test & Measurements
Infuse DevOps & Continuous Integration
22
Requirements/ User Stories
Code development Code review /
check in Build Package
Refresh (Data/Envi)
Functional Test
API test / FIX
Test Automation
Non Functional test
Defect Management
Reports / Dashboard
Deploy in Test Environment
Project Mgt, Collaboration, KM and
Reference docs
Tactical Solution
QEM
23
Strategic Solution
Effectively manage the quality (test) environment to achieve: Improved stability & availability Improved Quality Automation batch runs & easy maintenance Team/Member satisfaction
Automated provisioning of code deployment and easy maintenance of test environments in DevOps
Challenges • Tracking multiple changes in multi environments • Environment sharing with multiple teams for multiple verifications • Environment outage • Environment health check report
20-25% time loss
Blockers or surprises during testing
Quality Environment Management
Test env
QA env
Memb env
IWT env
Paradigm shift for test team
24
• Testers will need to understand the business goals & vision
• Testers become quality leaders within a multi-disciplined team
• Cross-skill and encourage flexibility
• Integrated test planning deliverables covering all teams & types of testing
• Utilize functional understanding to focus only on tests that deliver value
• Automated test execution in place from day 1
• Set targets, push for increased maturity & remove barriers
• Drive for continuous automation / integration
The role of testing becomes focused on technology and achieving business outcomes
Successful test teams will have to embrace continuous integration
Keys for success in a Mega-Testing Project
25
1. Commitment to Quality Culture (QA / QC)
2. Defect prevention and early detection
3. Understanding business and technical risks
4. Well defined ETVOX criteria – Entry Test Validation Output Exit
5. Continuous process improvement
6. Automate repeatable tasks
7. Standardize and simplify processes, tools, and methods
8. Manage by facts
9. End to End Customer Experience Testing
10. Build and maintain a high performing QA team
Paradigm Shift – Prevention over Detection
26
It needs a driver
Harmonize test planning to target testing to be effective & efficient
Collaboration amongst all teams & increased ownership by respective teams
Track in-flight quality & make value driven
Agility with continuous integration & Demonstrate reusability with test automation
Implement deep root cause analysis with a ‘one team’ mind-set
Skilled/champion staff
Pioneered New Processes and Tools
27
Thank you
Key Differentiators
29
Reduce test creation time by 30%
Improves functional coverage by 25% to 30%
Improves fault detection by up to 30%
Reduces maintenance work by 20%
Reduces end-to-end release cycle time by 50% or more
Standardizes test automation framework by stabilizing automation harness calls
Enables easier test framework migration
Helps reduce testing budget by up to 30%
Advantages
Modeling approach enables better system design archive and maintenance Shift Left approach, assures a considerable increase in software quality by capturing 100% requirements
coverage and reducing the cycle time The platform is ready and available to procure immediately – no perpetual software procurement delays
Requirements management • JIRA • IBM Door (RDNG) • HP ALM
Requirements Modeling
Test Execution
AUT
Repository Defect Tracking
Test Design
Test Harness
Code fixes Issues
Maintenance
Execution Reports
Test Coverage Reqs
Test execution • TOSCA • Selenium • OMX Batch runner
Modeling tool • IBM Door (RDNG)
Models Framework
Configuration
Code
Test Execution Test Design
Tactical Solution
18
Implementation Methodology
30
Manual Processes
Automation Workforce
Define Design Develop Test Deploy
Delivery Management Delivery Management Support & Change Mgmt.
Technical Infrastructure, Security and Governance
Ops SOP Build Feasibility
report FD/FS Config
Test Verification
Member Support
Security Policies.
Governance
Business Case
FD
FD/BRD UAT Ops Ready
Process Assessment
Tech Infra
Delivery Management
Program Governance, Security & Technical Infrastructure
= Business = Ops /Support = Test /
BA / SME
= Service Management
Our Improvement Approach
31
Test Governance PEOPLE PROCESSES TOOLS
Sanitize Stabilize Optimize
Driving a need to transform testing
• Transforming skills set • Building relationship • Making it competitive • Testing culture shift • Strong automation
team • Increase awareness
• Process Assessments • Early testing • Everyone’s
responsibility • Broader test coverage • Frequent/apt
communication
• New infrastructure • Making it accessible • Usage of tools • Reusable codes • Dashboard reporting • One click automation
(code deployment)