Fusion Testing

Maximizing Test Execution

By: James Tischart

About Me• Director - Product Delivery SaaS at McAfee • 15+ years experience in testing and engineering

• Multiple certifications in Agile and Testing

• Passionate about Testing as an eng discipline, a science and an art

• Continue to challenge the status quo, test new approaches and always strive to improve

• Support the legitimacy of the testing professional in the broader engineering world

What is Fusion Testing?

An occurrence that involves the production of a union

• Organization of structured testing • Freedom of exploratory testing • Rigors of automated testing • Combined into one test methodology

To achieve

• Maximize code execution• Increased test coverage • Reduced test artifact and documentation• Higher quality for users• Improved data for organization

Fusion

Focus – start your day with 15 minutes of thought

Usage – how will your users work with the system

Scope – decide on the scope of everything

Initiate – just go and explore

Organize – create a plan & be ready to deviate from it

Note – keep track of your exploration to retrace steps

Fusion Testing GuidanceMaximize test execution with Fusion by:

1. Identify Goals for Testing

2. Choose the right mix of methodologies

3. Utilize Tests Lists to guide exploration

4. Automate the Right Things

5. Document at the Right Detail

Fusion Implementation

1. Guide your team through the change

1. Create a Change Message

1. Mentor your team members

2. Train the team in testing and engineering

3. Create Metrics to measure success & failure

Fusion & PlanningRelease Test Plan

• Set the goals for the overall release

• Plan the test environment

• Organize the metrics and performance

• Plan a budget, resources and a time frame

• Plan your automation and tool set

• Plan at a high level

Iterative Test Plan• Very effective in Agile implementations

• Break up the testing into iterations

• Low level details for the time frame• Keep to two weeks or less• Detail the testing to accomplish• Detail the performance tests• List what will not be tested• Identify risks to the plan• Plan what Regression Testing• Identify what test-lists need to be written

Daily Plan• Individually created each morning • Provides guidance and goals for the day

• Details the execution plan

Test Lists• A grouping of test ideas

• No specifics in the test list

• Priority, variations & guide for exploration

The Power of Many• Utilize different users of your system to bring fresh perspectives

- Support personnel - Sales staff- Developers - Training- User groups - Product Management

• Help guide the testing by providing checklists, environments and goals

• Don’t give exact details or specific steps this minimizes innovation

• Make the event fun and you will have many people continuing to help

• Have testing experts available to help the volunteers for problems

Identify Defect Traits• For new products use historical traits from similar projects

• Analyze Severity 1 Customer reported incidents

• Determine the trends to those defects in coding & testing

• Constrain the review to a calendar year of defects

• Next analyze Sev 1 internally found and fixed issues and determine the traits

• Inform the developers & testers of the traits

• Create test lists or test ideas to cover these traits

• Repeat the process with each severity level you use

Fusion & AutomationDetermine your layers for automation:•Server

• Databases, server processes• Machine interaction points • Process interaction with other

layers

•Middleware/API layers

•User Interface

•Optimize your test artifacts • Don’t duplicate test-cases• Use automation for

traceability.

Test Results & Metrics

There will always be interest in the test data no matter the test structure.

Q: How can you accurately assess quality when the testing combinations exceed the particles of the universe?

A: You can’t! We need to present what was tested, what has not been tested and support the assessment used to make this prioritization.

• # of Test Ideas Executed based on Priority

• # of Test Cases passed/failed versus total coverage that the tests represent

• # of Automated Tests Passed/Failed by # of executed times

• Performance/Benchmark comparisons by build/iteration/release • Team Quality Satisfaction Rating – get the gut feeling of the team

• Open defects to highlight potential issues that your customers may find

• Go to fusiontesting.blogspot.com for more details on these metrics

Since you can’t test everything, here are some ideas of results to report:

Traceability Matrices

Positives Negatives

Can trace tests to requirement coverage

Time consuming to create and maintain

Displays what has been executed Often out of date and misleading

Shows relationship between tests and features

Duplicates information from test cases and requirements

Provides defect traceability to features

Dedication to frequent updates needed

Do Don’t

Tie your test lists to requirements or stories

Trace to the test case or test idea

Constrain your traceability to the feature level

Try to trace to the functional level

Build traceability into the automated tests

Duplicate information in many documents

Map automation to test lists or test specs

Over complicate maintenance of multiple sources

Implementation Challenges

Challenge: Management

Identify the Challenges: • Reliance on historical metrics

• Understand the current processes and practices

• Decision-making timelines

• Need traceability to feel confident of data

Respond to the Challenges:

• Provide better metrics

• Show how new process improves quality

• Prove how decision can be made faster • Review traceability needs and support them

Challenge: PMOIdentify the Challenges: • Reliance on historical metrics

• Struggle with the details of testing

• Need predictability for costs and schedules

• Require improved time to market

Respond to the Challenges:

• Debunk misleading metrics

• Show how more testing provides better data

• Prove reduced costs with efficiency gains • Track lost opportunity cost on test artifacts

Challenge: EngineeringIdentify the Challenges: • Takes time away from coding

• Testing can be tedious

• Not their specialization

• Rely on a serial approach to testing

Respond to the Challenges:

• Test first design increases new coding time

• Automated & exploratory tests are less tedious

• Understanding testing improves code writing • Less defects will be logged with up-front tests

Challenge: Regulation

Identify the Challenges: • Documentation Requirements

• Formal or Standards Approval

• Full traceability

• Rigorous Automation & detailed test results

Respond to the Challenges:

• Lean towards more structured testing

• Use exploratory but document results

• Plan for and work in shorter iterations • Automate more, document more

Challenge: Outsourcing

Identify the Challenges: • Require more instruction and detail

• Lack expertise in the product and markets

• Multi-cultural communication

• Formal process & document to support work

Respond to the Challenges:

• Phase in Fusion and train on exploratory

• Provide more details in documentation

• Implement multiple communication paths • Support and train new processes

Ensuring Success

Implement in stages

1.Measure the impact to customers (before & after)2.Collect metrics for a project for test execution time before implementation3.Introduce Fusion to cover about 25% of testing4.Increase Fusion usage to 50% to further improve execution time5.Continue to phase in until 75 to 90% of testing is Fusion

Monitor

1.Increases or decreases in execution time 2.Review documentation size closely reviewing the test artifacts 3.Measure the amount of direct automation4.Review with the team the amount of time spent on maintenance5.Continuously try to improve exploration time and increase efficiency

Adjust based on the needs of the project

1.One project may require more precise documentation 2.Another project may require less structure3.Identify these requirements at the start of projects and adjust4.Reduce the amount of time spent on artifact management5.Inform people of the time impacts of data collection

Five Keys to Fusion1. Detail – the right level of detail at the right time

2. Planning – consistently plan, execute and adjust

3. Automate - spend time on specifics in automation

4. Report – measure what truly represents the team and customer

5. POM – get more people involved to expand the breadth of test

Conclusion