Benefits of Collecting Code Coverage Metrics during HIL ... · Benefits of Collecting Code Coverage...

V0.1 | 2018-10-30

Jeffrey Fortin – Product Manager VectorCAST

Benefits of Collecting Code Coverage Metrics during HIL/ECU Testing

2

1. How VectorCAST fits into HIL/ECU Testing

2. Demo

3. Questions and Answers

Agenda

3

Vector Testing Solution – Test Tools for all Test Phases

How VectorCAST fits into HIL/ECU Testing

System Validation

System Integration

Test

SWIntegration

Test

SW Unit Test

SWImplementation

So

ftw

are

Syste

m

White-Box testing on host / on target

Vecto

rCAST M

anage a

nd

Analy

tics

Benefits

Full support in the development process, from SW unit test to system validation

Uniform test management, test automation (CI),result analysis and traceability

Lin

k t

o R

equirem

ents

VectorCAST/C++VectorCAST/RSP

CANoe, vTESTstudio vVIRTUALtarget

VectorCAST/C++/QA

CANoe, vTESTstudioVT System

VectorCAST/QA

SW integration testing + code coverage

on PC

System validation+ code coverage

on ECUChange-Based Testing

4

VectorCAST/C++ (VectorCAST/Ada)

Environment to perform software unit and integration tests

Automatically generates an executable test harness including test drivers + stubs

Instrument software to be tested to collect coverage date for coverage analysis purposes

Define test cases to stimulate the functions of the UUT, provide stub results to the UUT and check the expected results

Analyze test results and code coverage

VectorCAST/RSP

Execute test cases in an embedded target or simulator environment

Supports 35+ compiler families and hundreds of chip/board/rtos combinations

Software Unit Test


Benefits

Provide capabilities for white box testing on the host and on the target

5

CANoe and vVIRTUALtarget

Execute ECU software on simulated environment (PC)

Simulate ECU software, e.g. for partly simulated systems

Test simulated ECU software, e.g. for integration tests

Use CANoe as test execution system, apply all CANoe test and analysis features

Design tests with vTESTstudio

VectorCAST/QA

Environment to perform software system tests (functional tests)

Instrument source code of the system for code coverage analysis

Analyze code coverage

Software Integration Test


Service-Emulation

RTE / VFB

SWCA

SWCB

SWCC

Benefits

Enable software integration testing with code coverage on the PC

6

CANoe and VT System

Real-time execution of tests

Quasi industry standard on many test benches

Many open interfaces for testing purposes are available

Design tests with vTESTstudio

Modular system of interface hardware to access the I/O and bus network lines

All necessary functions integrated> I/O and measurement hardware

> Relay switches and fault injection

> Signal conditioning and signal processing

> Dedicated test functions (decade resistor, …)

VectorCAST/QA

Instrument source code for code coverage analysis of embedded software

Allows Change-Based Testing

System Test


Benefits

Get code coverage data during system tests and provide capabilities to perform Change-Based Testing

7

VectorCAST and CANoe Complements Each Other


System Testing

CANoe and VectorCAST/QA

System Testing

CANoe

Unit Testing

VectorCAST/C++

ECU I/O (Bus, A/D) ECU I/O (Bus, A/D)

CANoe is used to perform testing of an ECU via the ECU’s external interfaces.

Test Driver

Adding VectorCAST/QA exposes the completeness of testing, providing a detailed look on the internal structure of the embedded software.

Adding VectorCAST/C++ allows testing of hard-to-reach sections of the application.

8

Improve Test Execution Efficiency with Change-Based Testing


Test Cases

Source code traceability

Code Change

Source CodeInstrumented Source Code

Do we have to execute all test cases if the source code changes?

Without Change-Based Testing

Running all tests take a long time

Many tests are complicated to run

Test results are difficult to interpret


Change-Based Testing

Code coverage provides traceability between source code and test cases: Which source code is covered by which test case


Change-Based Testing

Code coverage provides traceability between source code and test cases: Which source code is covered by which test case

VectorCAST determines which test cases have been affected by a source code change

Only test what has changed to reduce test time, find errors faster and improve testing efficiency

9

Objective

Achieve fast turnaround cycles

Find errors as fast as possible

Support for parallel/virtualized and headless test execution

Automatically identify TCs that have to be executed because:> Code changes

> Previous Test Failures

> Changed and new requirements

> Safety critical requirements

> …

VectorCAST Enterprise Testing

Create suites of atomic tests that can be easily run in multiple configurations

Support massively parallel testing via CI servers such as Jenkins

Support Change-Based Testing to run only the tests affected by code changes

Support for Continuous Testing (CT) and Continuous Integration (CI)


CANoe

COM Script

Start Test

Test Result

CI support already available

In CANoe via the COM interface

In VectorCAST via command line and Python scripting

10

Using VectorCAST with CANoe


Objective: Provide a detailed look on the internal structure of the embedded software during CANoe system tests

Instrument source code of SUT for code coverage with VectorCAST/QA

Automatically execute tests with VectorCAST/QA and CANoe, system test cases are created with vTESTstudio

Analyze code coverage with VectorCAST/QA

ECU

System under Test

Functionality on the granularity of test runs already available

Tight integration to fully support test case oriented coverage planned

11


2. Demo


Agenda

12


Demo

13

Provide a comprehensive test solution based on well established tools for all kind of industries using and developing embedded systems

Enhance system testing with a more detailed look on the software structure including code coverage

Combine the strengths of system testing with unit testing, e.g. combine test results for an overall test analysis

Provide easy-to-use tools for all testing activities

Become the major player for test and validation solutions for embedded software and embedded systems (Automotive, IoT and other industries)

Integrate VI’s system and function testing portfolio with the code based portfolio of VSW

Extend the existing testing solution to

increase efficiency

increase quality

Summary

Demo

14


2. Demo


Agenda

15


Questions and Answers

!?

16 © 2018. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V0.1 | 2018-10-30

Author:Jeffrey FortinVector

More Information!Visit our Website for:> News> Products> Demo Software> Support> Workshops> Contact Addresses

www.vector.com

http://www.vector.com/

Benefits of Collecting Code Coverage Metrics during HIL ... · Benefits of Collecting Code Coverage...

Documents

Transcript of Benefits of Collecting Code Coverage Metrics during HIL ... · Benefits of Collecting Code Coverage...