V2.2 | 2017-10-02

Actionable Ideas for Improving Software Quality

2

Software Trends

Actionable Ideas for Improving Software Quality

• Hundreds of files• Thousands of Lines of Code• Lots of “connections”• Lots of safety concerns

SOFTWAREEMBEDDED SOFTWARE

+SIZE

ENTERPRISESOFTWARE

+Connections

• Thousands of files• Millions of Lines of Code• Few “connections”• Few safety concerns

+Safety

3

Management Stress

Larger Code Bases

Longer Life Cycles

Larger and Distributed Teams

Development Stress

Many bugs are found close to release

Code changes break existing features

Full testing takes a long time

Business Stress

Feature Releases are Slow

Customers Report Bugs

Things Could Be Better

Software Stress Points

Actionable Ideas for Improving Software Quality

4

We’ve been trying to improve quality for 30+ years

1982, let’s do a better job defining the problem> Early Modeling and Data Dictionary Tools

1983, let’s build a new language: Ada

1991, let’s build a new language: C++

1996, let’s draw pictures and get the code for free > Object Oriented Design, UML

2001, let’s build and integrate a little at a time> The Agile Manifesto, Continuous Integration

2002, let’s find bugs by modeling the code> Algorithmic Static Analysis

Tool Trends

Actionable Ideas for Improving Software Quality

5

What’s the Next Hot Trend?

Actionable Ideas for Improving Software Quality

Everyone Wants Better Quality Software

A Push-button Tool is probably not going to Solve the Problem

A new Development Paradigm is probably not going to Solve the Problem

Improved Testing is the Best way to Improve Software Quality

How can we Improve our Testing?

6

What’s the goal of testing?

Key Questions to ask

Actionable Ideas for Improving Software Quality

Who Builds and Runs Tests?

How Often do we Test?

Is our Testing Automated?

When are we Done Testing?

7

Common Answer

Application Correctness

How to Improve

Component Correctness

Benefits of Component Testing

Fewer Integration Bugs

Easier Re-Factoring of Modules

Why are we Testing?

What is the Goal of Testing?

8

Test Components in Isolation

Provides a formal check of component behavior, prior to integration

Allows Code to be more easily Re-Used and Re-Factored

Complements Functional Testing> All Component Tests might pass, but the system might not function!

For New Systems, concentrate on

Algorithms (e.g. fuel injection algorithm)

Low-Level building blocks (e.g. a message handler)

For Legacy Systems, concentrate on

Components with frequent changes and/or high bug counts

Components with high complexity

How to Implement Component Testing

What is the Goal of Testing?

9

Common Answer: The QA Group

How to Improve:

Make Everyone a “Tester”> Designers, Developers, Integrators, QA Engineers

Benefits

Spreads the Responsibility for Quality to Everyone

Leverages System Knowledge

Who’s Testing?

Who Builds and Runs Tests?

10

Implement Common Processes and Tools

Make Tests Easy to Build and Run

Deploy a automation server to process test jobs

Jenkins is an ideal platform,

Virtualization allows the infrastructure to scale

Make Tests and Results Easy to Understand

Publish results to the whole team, Track and publish testing trends

Make it possible to Debug Failing Tests

Testing framework must integrate with the debugger

How to Make Everyone A Tester

Who Builds and Runs Tests?

11

Common Answer: Periodically

How To Improve

Test Continuously, on every source change

Benefits of Continuous Test

Changes that break tests are never integrated

Tests are always passing on main branch

QA concentrates on Testing New Features …

When do we Test?

How Often do We Test?

12

Tests must be easy to run

A common platform is key

Tests must be small, atomic, and fast

Must be able to run independently

Parallel Testing must be possible

“We don’t have enough hardware” cannot be an excuse

Virtualization is a key building block

Invest in System-level, Simulation

How to Test Continuously

How Often do We Test?

13

Common Answer: When we hit the release date

How to Improve

Test Requirements not Code

Measure Code Coverage

Benefits

Test are more meaningful

Completeness is easy to measure

Estimates of remaining work are more accurate

Are we Done Yet?

When are we Done Testing?

14

Identify Requirements

What are we doing?

Identify Edge Cases for new Behavior

These are your tests!

Track Tests to Requirements

Ensure that all Requirements have Tests

Track Requirement changes to ensure Tests are Changed

How to Test Requirements

When are we Done Testing?

15

Measure Code Coverage

Coverage is a “free” metric

Measure during all flavors of Testing

Track Progress Over Time

Analyze Results

Develop a plan to improve coverage levels

How to Measure Testing Completeness

When are we Done Testing?

16

Common Answers: No idea, ask Bob

Correct Answer:

Push this button / Run this script

Benefits of Automated Testing

Any team member can run any test

Tests get re-used hundreds of thousands of times

How do I run this Test?

Is our Testing Automated and Repeatable?

17

Test Environment must be easy to setup

Automated Initialization of > Hardware, Software, Test Equipment

Test Purpose must be easily understood

NO code or script reading required

Tests must not be fragile

If a test fails “frequently” it must be improved

Test Results must be easy to interpret

Single Pass or Fail result

Tests must be able to be run manually

With the ability to debug

How to Implement Push Button Testing

Is our Testing Automated and Repeatable?

18

Summary

Actionable Ideas for Improving Software Quality

Unknown Complete

What’s our Code and Requirements Coverage?

No Idea Automatically

How do we Run Tests?

Per Release Per Change

How Often do we Test?

QA Everyone

Who Runs Tests?

19 © 2017. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V6.14 | 2017-04-18

For more information about Vectorand our products please visit

www.vector.com