Static analyzer debugging and quality assuranceapproaches

Maxim Menshikov

28.05.2020

St.Petersburg State University

About the author & the project

Maxim Menshikov —

• PhD student at St.Petersburg State University.• Software engineer.• (ex. security analyst; participated in commercial debuggerproject).

Equid1 — a static analyzer for C/C++/RuC based on ModelChecking and Abstract Interpretation. It verifies contracts andfinds common defects.

1Engine for performing queries on unified intermediate representations ofprogram and domain models 1

What’s special in analyzers?

Not much.

• Many equivalent transformations:input format ̸= intermediate format ̸= output format.

• Intermediate representations are mostly internal.

• The code is usually consistent and has high integrity, butthere are logical mistakes, unprocessed parts → the biggestdefects are logical.

2

The problem

• There are many debugging & quality assurance methods.

• None of them are specialized enough for static analysis.

• Every project brings its own set of hardly formalizedmethods.

What if we find a right specialization of the methods to the staticanalysis field?

3

The paper’s goal

4

Defect sources (observations)

• Missing support for the specific syntax/intermediaterepresentation (IR) construction in submodules.

• Small differences in implementations for repeating parts(classes).

• Transformation and ordering issues.

5

Defect reasons (observations)

• Low visuality of the transformation passes and thedevelopment process.

• Unattainable cross-dependencies between modules.

• Low quality of tests.

6

Proposed solutions

Proposing the solutions of these three groups:

• Code generation:Generated code usage verification.

• Testing:Goal-driven random test case generation.

• Logging:Log fusion and visual representation.

7

Code generation

• One model, several interpretations, many output source files.• Perform a simple integrity check.

8

Code generation: enumeration example

9

Goal-driven random test case generation

The idea is: generate input programs with an integratedverification goal (assertion).

10

Goal-driven random test case generation

1. The tool generates a random goal and asserts it → anexpression.

2. The expression is repeatedly rolled intoif/switch/for/while/... random blocks → a block.

3. The meaningful blocks are shuffled using equivalenttransformations.

11

Goal-driven random test case generation

In result we get:

1. A completely random program.

2. A set of shuffled random programs.

By that, it is possible to verify:

1. Logical issues in transformations.

2. Ordering issues.

3. Runtime failures.

12

Log fusion

Fuse separate logs, set up cross-references, so the final log is atechnical documentation of the run. Allows for easy navigation.

13

Log fusion: reverse recording assistance

The Log Fusion also helps break right after the specific log lineusing reversible debugger like RR2, UndoDB, etc. That is achievedusing logging engine traps and GDB scripts.

2https://rr‐project.org

14

Visual representation: steps

Visualize steps — present all transformations in one window,allow to debug specific transformations.

15

Visual representation: log health

Log health — visualize the time allocation for different modules.

In result, it is possible to determine whether the specific part isunintentially skipped.

16

Random test case generation: discovered issues & theirseverity

The method detected many performance, ordering, logical issues,and even runtime failures.

Defect type Number of issues SeverityPerformance 3 MediumOrdering 5 HighRuntime failure 1 HighLogical issues 1 Medium

17

Log fusion: (rough) time to resolve the issues

Average improvement rate: 2.8.

Defect type Time to resolve before (h) Time to resolve after (h)

Performance 25 13Ordering 5 1Runtime failure 1 0.3Logical issues 1 1

18

Results: code generation and visual representation

The improvement is hard to examine.

In our experiment, developing the same feature twice took 7times less time than on previous iteration - thanks to codegeneration.

Visual representation allowed to discover at least 2 performanceissues, and overall provided an enormous help during defectresolution.

19

Conclusion

• The specialization of the proposed methods helps find realissues in the static analyzer.

• The combination of approaches dramatically decreases thedefect resolution time.

20