Deployment Testing in an Internet World

Deployment Testing in an Internet WorldDeployment Testing in an Internet World

Michael Cookson CSQA, Manager, DTTMary Wood, Senior QA Analyst, DTT

April 14, 2023

Deployment TestingDeployment Testing

AGENDA

- Brief overview of MRO Software

- The Challenge

- Specific Issues

- Creation and Role of Deployment Test Team

- Next Steps

If at any time, there are questions,

please shout them out!


MRO Software, Inc.

- MRO = Maintenance, Repairs and Operations

- Office in London Ontario is the operations center for the hosted production site

- We host a series of Internet applications, allowing for the searching and purchasing of operational materials

- Includes workflow, Purchase Order integration to client distribution systems, real time pricing and availability specific to purchasing organizations, order status and integration to different ERP systems for control of inventory


Mike Cookson started with MRO Software in November 1999 to become the test manager

Mary Wood joined MRO Software in June 1997, and moved through a series of positions before becoming the initial member of the test team in 1998

Initially this new test team was responsible for the functional testing of the product and was part of the development organization

Later, in 2000, the test team was moved to the operations area, but maintained its functional testing focus

This created some conflict, as we will discuss

The ChallengeThe Challenge


The Internet – Client Perspective

- Internet environment demands 7x24 availability

- Clients are increasingly dependant on the availability of the system to support their business processes

- In order to be successful in this marketplace, need to be able to provide the highest availability

- Many contracts will have a Service Level Agreement (SLA) attached with penalties for not hitting targets


The Internet – MRO Software Perspective

- We host applications that our clients access

- Multiple teams within MRO Software provide changes to the hosted environment

- These changes usually arrive independently of one another, but could affect other components

- Many of these changes are attached to potential revenue contracts

- The change needs to be implemented in a timely manner, with delivery commitments already made to clients

- Many clients connections can only be tested in the production environment because they do not have test back end systems and firewall issues


Bottom Line:

Change is needed for revenue;

Change can cause loss of revenue!


The Challenge:

Develop a system to ensure the timely implementation

of change into the hosted production environment,

while at the same time ensuring the change has

minimal negative impact in terms of availability of the

systems and client connections.


What does that mean?

- Most companies perform the deployment testing tasks we are about to describe in one form or another

- The question is whether these tasks need to be formalized or more focus put on them

- This presentation will describe the role of the Deployment Test Team at MRO Software and why we have organized it this way


MRO Software was organized with a traditional development organization and functional testing organization

This organization left some ‘gaps’ in terms of coverage

Organizationally, we were not ready to fill these gaps

- For example:

Who is responsible for coordinating changes being made by multiple teams?

Who is responsible for ensuring the changes are made in the change windows? – 6x24 needed by clients currently and shrinking

Who is ensuring changes integrate with the client systems? – not always possible in the development environment

Who is responsible for ensuring the dependancies for environments are taken into account?


There were many other issues that we were trying to deal with

These issues were summarized into four broad categories

- Change Management

- Implementation Management

- Client Management (at a tactical level)

- Test Environment Management

These issues were beginning to have an impact on the reputation and ability of MRO Software to deliver a quality product to the hosted environment in a reasonable time frame


The Test Team at MRO Software had a functional testing focus, although part of the operations group and so functional and operational issues had to be considered simultaneously at each deploy

This was different from all other parts of the organization where the test team was part of the development organization

The Internet world demanded more than just functional testing, but there was no room in the organization for anything but functional testing

The Customer’s business was directly affected by the success or failure of the code deploy


A number of pressure points were converging

Technical and operational issues were mounting

Organizationally we needed to bridge the gap between functional testing and operational issues


The evolution of the Deployment Testing Team was in direct response to the pressure points

Under the Operations umbrella the Functional Test Team was beginning to take ownership of some of the processes

- For example Change Management and Implementation Management

A completely different mind set and conflicting set of priorities were becoming evident

Operational testing tasks and functional testing tasks had different goals, objectives and accountabilities

Staff was having difficulties switching between these

The SolutionThe Solution


The conclusion was that the functional team had to be split and the responsibilities divided

Functional testing would return to the Development organization

Deployment Test Team was created in the Operations area to assume ownership of the Operational processes


The Mission of the Deployment Test Team is to manage all changes to the production hosted environment to ensure minimal negative impact to our customers.

This will be accomplished through the application of a formal, structured approach to every change

to the hosted environment in order to provide the benefits of the change in a timely manner.

The Deployment Test Team will employ:Risk Assessment Process to identify the appropriate verification and validation procedures;

Change Management Process to co-ordinate varied resources to ensure a smooth introduction into the production environment and ensure that all impacted areas have the necessary information

regarding the change;Manual and automated tools and test plans to ensure the change has been properly implemented.

The goal of the Deployment Test Team is not necessarily to apply perfect change, but rather, to

ensure that every change to the production environment is applied perfectly.


Deployment testing, quite simply, is the actions

taken to ensure a deliverable or change

is properly implemented into a production environment.

The word ‘Testing’ in this case, does not adequately

cover the myriad of tasks necessary to ensure the

changes make it to production successfully.


So what are the items that the DTT are responsible for?

- Implementation Management

- Change Management

- Verification of Deployments

- Ongoing Client communication and coordination

- Maintenance of a mirror test environment


Implementation Planning owned by DTT

- For every change, what is needed to successfully implement?

- What dependencies are there?

- How can we test the plan?

- How can this affect the clients with connections to our software?

- The detail needed in the implementation plan is dependent on the risk of the change

- We have developed a very simple risk model to help us identify low, medium and high risk projects and based on that, develop the implementation plan

- Includes back out planning and verification of the plan


Change Management Process owned by DTT

- What changes are expected?

- What is the time frame for these changes?

- What else is planned for that change window?

- Are all the impacted areas aware of this change?

- Do clients need to be notified – who and when?

- Ensures all change is identified before the date of implementation

- If multiple changes needed the same window, ensures the different teams are involved for cross dependencies


Verification of the change in the production environment is owned by the DTT

- Once a change is implemented, how do we know the change was applied correctly?

- What are the indicators to prove all the changes arrived in production correctly?

- What regression tests should be done to ensure client connectivity and critical functions have not been impacted after the change?

- Deploying the code is only half the battle

- Need to ensure all clients are connecting normally, and the application is also functioning

- If things aren’t working, then need to invoke the back out plan and then verify the back out worked


Ongoing Client Communication and Coordination

- What impact does this change have on the client?

- Is there a risk of disrupting their connections to the software?

- How can we verify these changes?

- What change window do we have based on the client’s business needs?

- Involves continual communication with our clients and with the teams that are working with the clients to ensure their needs are understood

- The operations group owns the relationship with the client at a tactical level, the DTT is one tool used to maintain and enhance this relationship


Maintenance of a Mirror test environment

- To ensure deployments happen appropriately, need to have a practice site

- This mirror site needs to be as close to the live environment as possible

- It is not a place to perform functional testing – by the time the code is applied here, it is functionally correct

- Used to test deploys, backouts, connect to client back ends and perform transactions


What is the DTT not responsible for?

- The functional testing of the change

- DTT assumes the change has been tested and is ready to be put into the production environment

- The actual deployment of the change – done by another operations team

- Work closely with the team performing the deployment to ensure their needs are included in the planning

- The Deployment Test Team is not doing any work that another team is responsible for. We do not redo or recheck previously done work.


Current Status

- Change Process has been defined and implemented

- Risk Assessment has been created and is used for each change

- Checklist of key tests for each client has been created and is used to check each change, based on the change and the risk

- Automation begun on these key tests

- Mirror environment in place for key applications and plans in place to create mirror environments for the others

- Skills inventory completed, and self assessments done.


The splitting of the test team allowed the DTT to focus on the operational issues

Due to the way MRO Software splits the development and operational responsibilities, needed to ensure the operations tasks had the focus required

Development needs to focus on the creation and enhancement of new products

Operations needs to focus on the client relationship on a day to day basis and the availability of the hosted production applications

The DTT team can focus on these tasks without the issues of functional testing and project deadlines


Key Success Factors for the DTT

- Marketing – make sure all areas aware of the role of the DTT

- Ensure all change is properly planned for

- All change implemented within the change window

Need to perform all the tasks necessary, but no more – implies a very solid risk assessment process

Need to automate as much of the verification as possible to save time

- External clients are not negatively impacted by change

All connections available when change is done

We minimize impact to world wide clients for down time due to change

Ensure we will not need to perform an emergency back out of upgrades because something was missed in the deployment

What About You?What About You?


Does your organization need focus on these responsibilities?

If, around the water cooler, you hear these questions, it is possible you want to put some focus on these tasks

- The conversion is still going? That’s over 70 hours!

- Why didn’t the change go in? This project has been in the works for months now!

- It worked in development.

- Why is production running a different version of the database? Of course it won’t work


- Why wasn’t our change tested with the other change? Everything has to go at the same time for this to work.

- Why are these two changes scheduled for the same night? They have to be implemented separately.

- This is an emergency. How do we get our change in there now?!?!

- Our change went in and is working. Of course, it broke everything else, but that is not my problem.


If you are hearing conversations like this, then it is probable that some focus needs to be put on the tasks we have discussed

How you implement will depend on your organizational structure and how responsibilities are divided

The creation of the DTT is working at MRO. At your place, you may need a different structure to work, or maybe, the DTT will work for you as well…

Next Steps @ MRONext Steps @ MRO


Next steps for the DTT

- Need to implement a tool to better track Changes

- Finish the Mirror Environment for all applications

- Continue to build the automated test suite

- Enhance and refine the checklist for verification procedures

- Close the gap between the identified skills needed to be successful and our current skill set and level

- Continue to build the visibility of the DTT at MRO Software


Next Steps for the DTT

- Identify the measures for the Deployment Test Team so we can:

Continually improve the Change Process

Continually improve Implementation Process

Continually improve the Risk Assessment Process

Continually improve our client relationships

Continually improve the effectiveness of our test environments

Deployment Testing in an Internet World

Documents

Transcript of Deployment Testing in an Internet World