0 ©2011, Cognizant End to End testing of a complex ERP footprint for Logistics Service provider in...

1

©2011, Cognizant

End to End testing of a complex ERP footprint for

Logistics Service provider in US

By Sathish Rajamani and Jayanta Kumar Rudra

8th Dec 2011

2 | ©2011, Cognizant 2

Agenda

2

Client Overview & Voice of Customer

Key Challenges & Mitigation

5

6

Cognizant Solution & Key Metrics

Benefits & Value Adds

1

3

4


Client Overview

Leading Logistics service provider in the US

Expanding their operations across Canada, Mexico and China

A Premium service provider catering to more than two-thirds of the FORTUNE 500® companies

Operates 14,000 tractors, 15,500 drivers, and 48,000 trailers with 21,500 associates operating in 28 countries

Client had embarked on a major IT transformation project of moving from legacy to ERP.

They wanted to improve the QA process as lacking on below points

No dedicated QA setup

Lack of sufficient Performance benchmarks & SLA metrics

Testing in silos by Business users

No automation of existing regression test cases


I need an accelerated Testing Solution which meets my GTM and Quality benchmarks

I need someone to guide me on

how to Improve my Oracle ERP

Testing Organization and

Processes

I want to Automate my Oracle ERP Testing

Solution

My Oracle ERP Testing processes are

not aligned to Industry Standards

Testers do not have adequate

functional knowledge of the Oracle ERPI want to Automate my Oracle ERP

Testing Solution

I need a Testing Partner with sound experience in

Oracle ERP Testing with strong association with

Oracle

I need assurance on the quality of my Oracle ERP Implementation and that it meets the requirements set out

I want various kinds of Testing for my

application’s Upgrades, Sustainment,

New Release, Refresh/ Rebuild and E2E

scenarios

I need an accelerated Testing Solution which meets Quality benchmarks

I need someone to guide me on how to

extend and enhance my Oracle ERP

Testing Organization and Processes I want to reduce the Total Cost of

Ownership for QA activities

I need someone to guide me on creating

an On Demand Test Model with

optimizations while maintaining

consistency in processes and metrics


CRMOrder Management

ClaimsBilling

Network ManagementRate Server

PricingGeography

CRMOrder Management

ClaimsBilling

Network ManagementRate Server

PricingGeography

Vendor Managed Application/ Legacy

Quote, OrderQuote, Order Pricing decisionPricing

decision Order Management

Order Management

Revenue & Network Mgmt

Revenue & Network Mgmt

Shipment AssignmentShipment

Assignment Order ExecutionOrder Execution Back officeBack office Support Support

Creating customer orders

Releasing orders to OTM (Order Status – Released)

Creating customer orders

Releasing orders to OTM (Order Status – Released)

Assigning driver/SP to the shipments

Completion of special servicesAgreement to quote OrderPricing Decision Support

Assigning driver/SP to the shipments

Completion of special servicesAgreement to quote OrderPricing Decision Support

Financial validationsFinancial validations

HROLMOICGLARAP

HROLMOICGLARAP

Order updates flowing from OTM to SiebelOrder updates flowing from OTM to Siebel

Dem

and Generation

Dem

and Generation

SO

A (B

PE

L/ES

B)

SO

A (B

PE

L/ES

B)

EDIPortal (Web center)

Email

EDIPortal (Web center)

Email

Maptuit

TMG

RAIR

M5

CRA

Drop & Swap

End-to-end Multi-Platform Testing Path

Transportation Management

Qualcomm


Our Solution through MTC

Delivery Management &

Reporting


Reportingwith

Development services

Organization

with Development

services OrganizationScheduling &

Release Management

Scheduling & Release

Management

Benchmark & Metrics

Publication

Benchmark & Metrics

Publication

Process Standardization & Compliance

Process Standardization & Compliance

Infrastructure & Environment

Infrastructure & Environment

TEST MANAGEMENT OFFICE

TEST CENTERFunctional

TestingRegression

TestingAutomation

TestingPerformance

TestingInterface Testing

Simulation with Mobile

Testing

UAT Support

EDI Testing

Data Validation Testing

Smoke Testing

Security Testing

Disaster Recovery Testing

SER

VIC

E

CA

TA

LO

UG

E

End-to-end Test Solution FUNCTIONALNON-

FUNCTIONALAutomation Testing

SER

VIC

E

TO

WER

S

FUNCTIONAL TEST LEADS

AUTOMATION LEADS

SMESs ACROSS LOBs

TEST MANAGER

PROGRAM MANAGER

FUNCTIONAL TEST ANALYST

AUTOMATION TEST

ENGINEERS

METRIC CHAMPIONS

SOA TEST SPECIALIST

DELIV

ER

Y

OFFIC

ECORE TEAM

FLEX TEAM

Oracle EBS-Finance(AP,AR,GL,HRMS,OLM)Oracle Transport Management

Siebel CRM, Analytics, Oracle FusionLegacy & Vendor

Managed Application

BEST-IN-CLASS QA FRAMEWORK

QUALITY ENGINEERING

KNOWLEDGE MANAGEMENT

ENTERPRISE TOOL ALLIANCES

SOLUTION ACCELARATORS

Test Data Management

Consulting on Metrics

Management

NON-FUNCTIONAL

TEST ENGINEERS


Reporting

Tool & Innovation Framework

Scheduling & Release

Management

Benchmark & Metrics Publication

Process Standardization &

Compliance

Test Co-ordination with other Vendor


Key Challenges & Mitigation

MitigationMitigation

• Closely work with Dev or Production Support team

• Leveraging Cognizant’s Partnership with Oracle

• Daily Patch Testing

• Comprehensive

• Regression Testing

• Application

Complexity• End to End Testing

• Non availability of

skilled resources

• Lack of well defined QA Strategy and Build plan

• Intermediate Version Upgrade

• System Instability• Performance Issue

• Cognizant’s BIC (Best In Class Metrics)

• QA Strategy by Efficient PMO

• Involvement of Test Consulting Group & DAG (Delivery Assurance Group)

• Managed Test Service

• Cognizant BA–PA-QA Model

• Core Flex Team composition

• Integrated test approach

• Delivery management with strong TMO

• Domain & product knowledge


Achievents and Key Metrics


MTC Achievements

Dec’09

Jun’10

Jan’11

Jun’11Delivering Enhanced Values

Supporting Business Test Execution Creation of Innovation council Regression Testing for Release 7 Fix pack & Sustainment release for 7.1 Adopted Test Matrix approach for FIT

saving a lot of Test Preparation time Smoke Test for UAT Environment, EDI

Testing , Accessorial Testing were added to scope

200 performance issues identified

System testing Four cycles of R7 System

testing

Business testing Created Test Scenarios ,

Steps & Clicks Introduced Test Manager as a

Test Repository tool & migrated all Excel based test cases to the tool

Automation testing Use of automated data

creation from Cycle 3 System

Performance testing Setup performance testing

process and establish Planning, estimation models and Performance execution standards for the client

TCE Set up & Steady State

TCE Initiation POC for Automation

& Business Test cases

System Test Strategy and testing for R7 for Siebel & EBS

TCE strength : 8 (Onsite 6/ Offshore 2)

TCE in Managed Service Model

Jan’12

Matured QA Organization

Pre-TCE State

No dedicated QA set up

Performance issues due to siloed testing by business users

System testing Extensive test case repository

(SOARS) built across modules Peak team size : 70

Business testing Prepared Test Data Sheets Kick started AIT Business Test

execution Test scheduling and management

Support End-to-end multi Platform testing Better Onsite – Offshore ratio

(25:75)

Automation testing Automation of the end to end

flow starting from Order creation until execution, providing 60 – 70% automation coverage for Business Test scenarios

Automated Data conversion & transition

Performance testing Around 1000 performance tests

were conducted in last 12 months

Till Nov’09

Awaiting New Release

Tools

Client Supplied

SOAP UI, Open Script, Silk Performer, Clarity, Test Manager

In House Accelerators

SOARS, TCP, CRAFT, ADPART, C2, ROI, Script Accelerator, Test Scenario Accelerator, SiFA, Script Sanitizer

10 | ©2011, Cognizant 10

Component testing

Web Service testing (Through SOAP UI)

Load Test

Interface testing

Stress test

Endurance testing

Bandwidth testing

Network device testing

Program Performance Test

Better capacity planning and decision on hardware &

software requirement

Significant cost reduction due to

Early life Performance testing

Zero post-production defects

Executed 1x, 2x and 3x testing

Performance & Automation Testing

Accomplishments

Generated 1650 Siebel Orders using

Automated script in System Test Environment

Automation of 250 Business Components and

400 Scenarios

Automation of initial Prototype E2E Scenario

complete.

Build Automation frame work release based

Testing

Generated 1650 Siebel Orders using

Automated script in System Test Environment

Automation of 250 Business Components and

400 Scenarios

Automation of initial Prototype E2E Scenario

complete.

Build Automation frame work release based

Testing

Highlights

11 | ©2011, Cognizant 11

Structured QA Process

....

....

....

....

LOB1 LOB2 .... ....

Testing Team

PM1 PM2 PM3 ........

DEV1 DEV2 DEV3 .... ....

Pre Engagement

Testing Team

PM2 .... .... ....

LOB1 LOB2 .... .... ....

DEV1 DEV2 .... .... ....

Post EngagementSteady State

TESTINGTEAM /

CHANNELS

Channels Support

TESTINGTEAM /

PRODUCTS

TESTINGTEAM /

SUPPORT

NFT / Automation

NFT / AutomationTEMS

Products

STRATEGY, OPERATIONS, ASSURANCE

Integrated Delivery

BR

OK

ER

AG

E

INTER

MO

DA

L

TR

UC

KLO

AD

FIN

AN

CIA

L

SER

VIC

ES

NFT /Automation

TEMS

COE’s

Delivery Excellence

LO

GIS

TIC

S

Review Efficiency

Schedule Accuracy

Productivity Improvement

Estimation Accuracy

Test Effectiveness

NA

NA

NA

NA

NA

Offshore Leverage

Schedule Accuracy


Estimation Accuracy

Test Effectiveness

35:65

9%

75%

NA

NA

Schedule Accuracy


Estimation Accuracy

Test Effectiveness

95%

95%

95%

Schedule Accuracy


Estimation Accuracy

Test Effectiveness

25%

>95%

>95%

99%

Resource based Requirement based Service based Integrated

Jun 2009 Dec 2009 Jun 2010 Jun 2011

15%Test Design Test Execution

Offshore Leverage

22:78

Test Design Test Execution

Offshore Leverage

18:82

Test Design Test Execution

33%

12 | ©2011, Cognizant 12

Metrics Dashboard

- Test design Coverage- Test design productivity- % Rework Effort- Review Efficiency- % of defective test cases

- Test execution coverage- Defect Density- Test Execution Rate- Defect leakage- Defect Trends

- Application Stability Index( Quantitative Go/No-Go Decision)- Defect Removal Effy.,

Test Metrics DefinitionSetting Bench Mark

Planning

Design

Execution

Closure

- % Effort Variation- % Schedule Variation- % Duration Variation

- % Effort Variation- % Schedule Variation- % Duration Variation- Load Factor

Test StagesApplication Health

CheckProject Health Check

- Productivity trends- Goals Vs Actuals

Test Metrics Definition and Benchmark settings

Key Metrics

Metrics Report – quick glance

• Test Coverage : 100% • Automation Coverage: 76%• Performance Testing (all critical business transaction) : 100% • Schedule Slippage: 0% • Effort Slippage: 0%• Avg Productivity in Test Execution: 9%

14 | ©2011, Cognizant 14

Quarter to Quarter OptimizationOptimization Levers

Process Automation

Solution Accelerators/Tools

Knowledge Repository

Non-Linear Estimation Model

Onsite-Offshore Delivery Model

Right first time

Increase

Productivity Gains

Operational Efficiency

Customer Experience

IT Business Alignment

Impr

ovem

ent O

ppor

tuni

ties

Time to Market

Effort

Cost of Quality

Redundancies

Application Maintenance Cost

Decrease

% of Productivity gain

1st Qtr – 5%

2nd Qtr – 7%

3rd Qtr – 9%

4th Qtr onwards – 12%

Best Practices

Led Oracle ATS Tool Rollout

Created several Tools and Accelerators -

Spider utility and macros

Defined System Test Strategy for R7

Instituted matrix based FIT test design

approach

Built dashboards for test execution tracking

15 | ©2011, Cognizant 15

Innovation Spider Utility: Effort Savings in % compared to Manual Testing

Test Scenario Accelerator:

16 | ©2011, Cognizant 16

Did you know?

Executed 25000+ functional Test Cases and identified closed to 1200 defects

Saved 40% Test Design Effort by

Cognizant’s proprietary tool

7000+ Test Data creation

manually for functional

testing

More than 14000 Hrs

savings through Test

Automation & achieved

ROI ~ 300K

0 ©2011, Cognizant End to End testing of a complex ERP footprint for Logistics Service provider in...

Documents

Transcript of 0 ©2011, Cognizant End to End testing of a complex ERP footprint for Logistics Service provider in...