Variability-Aware Parsingin the Presence of

Lexical Macros and Conditional Compilation

Christian Kästner

Paolo G. Giarrusso

Tillmann Rendel

Sebastian Erdweg

Klaus Ostermann

Thorsten Berger

Parsing C Codewithout Preprocessing

Christian Kästner

Paolo G. Giarrusso

Tillmann Rendel

Sebastian Erdweg

Klaus Ostermann

Thorsten Berger

Viewer Discretion Is Advised

LinuxKernel

10,000 features, 6 million lines of C code

Parse and Type checkall configurations of the

entire Linux kernel

10,000 features, 6 million lines of C code

AST with

Variability Informationtrue -> (WORLD v BYE)

¬ (WORLD ˄BYE)

greet.c

printf

msg

VWORLD VBYEprintf

msg msg

main

ε ε

…

Variability-Aware AnalysisType SystemStatic AnalysisBug FindingTestingModel CheckingTheorem Proving…

Parsing Cwithout Preprocessing

greet.c

printf

msg

VWORLD VBYE+ printf

+ msg + msg

+ main

ε ε

???

Macro expansion needed for parsing Alternative macros

Undisciplined annotations

Previous Solutions

Disciplined SubsetRequires Code Preparation

Heuristics and Partial AnalysisInaccurate, False Positives

Brute ForceInfeasible Effort

TypeChef

Variability-Aware

Parser-Framework

+

* VA

2 43 5

Variability-Aware

Parser

Variability-Aware

Analysis

https://github.com/ckaestne/TypeChef

Variability-Aware

Lexer

(2

*3

)+

4A5¬A

Variability-Aware Lexer

(2

*3

)

Variability-Aware Lexer

0 included header files per C file

0 macros per C file

0 conditional macros per C file

0 token per C file

0 % conditional

35385901387

33549072

X862.6.33.3

2 * 3 + 4A 5¬A

+

* VA

2 43 5

Variability-Aware Parsing

( )

Undisciplined Annotations

* (A 3 4 )A

+

*

VA

2

4

3

2 +

*

+

3

2

4

Partial Errors

* (A 3 4 )

Parse Error

VA2 +

*

+

3

2

4

Expected + or EOF, but found ) at 1:7

3 + 4A 4¬A B˄ +¬A B˄ 6¬A( (¬A )¬A )

true

3 + 4A 4¬A B˄ +¬A B˄ 6¬A( (¬A )¬A )

true

3 + 4A 4¬A B˄ +¬A B˄ 6¬A( (¬A )¬A )

true3

3 + 4A 4¬A B˄ +¬A B˄ 6¬A( (¬A )¬A )

true3

3 +

4A 4¬A B˄ +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

3

3 +

4A 4¬A B˄ +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

3 +

4A 4¬A B˄ +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

Next Token:

ctx -> tok: Consumectx ->¬tok: Skipelse: Split (ctx ˄pc) and (ctx ˄ ¬pc)

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

3 +

4A 4¬A˄B

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

+¬A B˄ 6¬A3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )¬A4¬A B˄ +¬A B˄ 6¬A )

¬A

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4¬A B˄ +¬A B˄ 6¬A

4¬A B˄ +¬A B˄ 6¬A )¬A )

¬A˄B

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4

4¬A B˄ +¬A B˄ 6¬A

4¬A B˄ +¬A B˄ 6¬A )¬A )

¬A˄B

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4

4¬A B˄ +¬A B˄ 6¬A

4¬A B˄ +¬A B˄ 6¬A )¬A )

¬A˄B

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4¬A B˄ +¬A B˄ 6¬A

4¬A B˄ +¬A B˄ 6¬A )¬A )

¬A˄B

+

64

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A )¬A )

¬A˄B

4

3 +

64

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A )¬A )

¬A˄B

4

3 +

64

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A

)¬A )

¬A˄¬B

4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A )¬A )

¬A˄B

6

4

3 +

64

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A

)¬A

¬A

4

+

64

VB

6

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A )

A

4A (¬A )4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A

)¬A

¬A

+

64

VB

6

4

3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A

4A (¬A

4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A

)¬A

+

64

VB

6

)

4

VA

true3

3 +

4A 4¬A˄B +¬A B˄ 6¬A

(

(¬A )¬A

4A (¬A

4¬A B˄ +¬A B˄ 6¬A

4¬A˄B +¬A B˄ 6¬A

)¬A

+

64

VB

6

)

true

4

VA

+

3

Variability-Aware

Parser Combinator Libraryfor Scala

Variability-Aware Parsers

GNU C + cpp Java + Antenna

X862.6.33.3

Parsing Linux

Variability in Build System (kbuild)and with #ifdef

Variability Model (kconfig)

0 C files (x86)

0 seconds per file (median)

0 hours total

0 % overhead (undiscipl.)

0

0 syntax errors

76653085

4.1

0X862.6.33.3

Variability-AwareParserType SystemModule SystemStatic AnalysisBug FindingTestingModel CheckingTheorem ProvingEditor SupportCode Transform.…

Variability-Aware Parsingin the Presence of

Lexical Macros and Conditional Compilation

Variability-Aware

Parser-Framework

+

* VA

2 43 5

Variability-Aware

Parser

https://github.com/ckaestne/TypeChef

Variability-Aware

Lexer

(2

*3

)+

4A5¬A

(macros, includes) (split & join)