Regular expression that produce parse trees

Efficient Regular Expressions that produce Parse Trees

Aaron Karper Niko Schwarz

University of Bern

January 7, 2014

Aaron Karper, Niko Schwarz (UniBe) Regex Parse Trees January 7, 2014 1 / 38

Regular expressions so far

Regular expressions

https? : // (([a− z ] + \.) + ([a− z ]+))︸︷︷︸domain

((/[a− z0− 9]+)/?)︸︷︷︸path segments

Regular expressions

https? : // (([a− z ] + \.) + ([a− z ]+))︸︷︷︸domain

((/[a− z0− 9]+)/?)︸︷︷︸path segments

http : // www︸︷︷︸domain

. reddit︸︷︷︸domain

. com︸︷︷︸domain

/ r︸︷︷︸path

/ computerscience︸︷︷︸path

/ comments︸︷︷︸path

/ 1sg69d︸︷︷︸path

Regular expressions are greedy by default:(a+)(a?) on "aaa" → "aaa" in group 0 and "" in group 1.

Posix gives only one match.Regular languages are recognized, but parsing with combinatorical parserstakes O(n3).Backtracking implementations (Java, python, perl, . . . ) are exponentiallyslow in the worst case.

Benchmarks

Parsing with https?://(([a-z]+\.)+([a-z]+))((/[a-z0-9]+)/?)

2http:// www. reddit. com /r /computerscience /comments /1sg69d

Figure : Posix

http:// www. reddit. com /r /computerscience /comments /1sg69d2

4 4 4 4

Figure : Our approach

Benchmarks

Matching ((a+b)+c)+ against(a200bc)2000.

Tool Time

JParsec 4,498java.util.regex 1,992

Ours 5,332

Extract all class names from our projectwith complex regular expression1.

Tool Time

java.util.regex 11,319Ours 8,047

1(.*?([a-z]+\.)*([A-Z][a-zA-Z]*))*.*?Aaron Karper, Niko Schwarz (UniBe) Regex Parse Trees January 7, 2014 6 / 38

Benchmarks Optimizations of the algorithm

Benchmarks – Optimizations of the algorithm

Typically most time is spent in long repetitions, we optimize for that case by:Lazily compile deterministic FA.Avoiding to recreate state if seen similar state.Use compressed representation if in static repetition.

Benchmarks NFA interpretation

Example: (a?(a)b)+

Parse(a?(a)b)+

over”a0a1b2a3b4”

a a b a b0 1 2 3 4

Example: (a?(a)b)+

Reading "a0a1b2a3b4"

[[], [], [], []]

q2 q3 q4

q5 q6 q7 q8

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

q5 q6 q7 q8

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

q5 q6 q7 q8

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

[[0], [], [0], []]

q5 q6 q7 q8

Threads

h1h1 h2 h3 h4 h5 h6

State:

Histories:

qCopy of thread is modified.Copy of array of histories makesreading a character O(m2)

Need faster persistent datastructure to get O(m logm).

Optimized thread forking

Set entry 2 to 20:

Optimized thread forking

Set entry 2 to 20:

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

[[0], [], [0], []]

q5 q6 q7 q8

For each character read, threads start hungry and must eat immediately.

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

[[0], [], [0], []]

q6 q7 q8

For each character read, threads start hungry and must eat immediately.Only a hungry thread can eat

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

[[0], [], [0], []]

[[0], [], [0], [0]]

Example: (a?(a)b)+

[[], [], [], []]

[[0], [], [], []]

[[0], [], [0], []]

[[0], [], [0], [0]]

Example: (a?(a)b)+

q1 q2 q3

[[0], [], [], []]

[[0], [], [1], []]

[[0], [], [0], []]

[[0], [], [0], [0]]

Example: (a?(a)b)+

q1 q2 q3 q4

[[0], [], [1], []]

[[0], [], [0], [0]]

Example: (a?(a)b)+

q1 q2 q3 q4

[[0], [], [1], []]

[[0], [], [0], [0]]

Example: (a?(a)b)+

q1 q2 q3 q4

[[0], [], [1], []]

[[0], [], [1], [1]]

Example: (a?(a)b)+

q1 q2 q3 q4

[[0], [], [1], [1]]

Example: (a?(a)b)+

[[0], [2], [1], [1]]

[[0,2], [2], [1], [1]]

[[0,2], [2], [1,3], [1]]

[[0], [2], [1], [1]]

q5 q6 q7

[[0], [], [1], [1]]

[[0], [2], [1], [1]]

Example: (a?(a)b)+

[[0], [2], [1], [1]]

[[0,2], [2], [1], [1]]

[[0,2], [2], [1,3], [1]]

[[0], [2], [1], [1]]

q5 q6 q7

[[0], [], [1], [1]]

[[0], [2], [1], [1]]

Example: (a?(a)b)+

q1 q2 q3

[[0,2], [2], [1], [1]]

[[0,2], [2], [1,4], [1]]

[[0,2], [2], [1,3], [1]]

[[0,2], [2], [1,3], [1,3]]

Example: (a?(a)b)+

q1 q2 q3

[[0,2], [2], [1], [1]]

[[0,2], [2], [1,4], [1]]

[[0,2], [2], [1,3], [1]]

[[0,2], [2], [1,3], [1,3]]

Example: (a?(a)b)+

[[0,2], [2,4], [1,3], [1,3]]

[[0,2,5], [2,4], [1,3], [1,3]]

[[0,2,5], [2,4,5], [1,3], [1,3]]

[[0,2], [2,4], [1,3], [1,3]]

q5 q6 q7

[[0,2], [2], [1,3], [1,3]]

[[0,2], [2,4], [1,3], [1,3]]

Example: (a?(a)b)+

[[0,2], [2,4], [1,3], [1,3]]

a a b a b0 1 2 3 4

Download

https://github.com/nes1983/tree-regex

NFA construction

AlternationS1|S2

OptionalS?

Capture group(S)

Star operationS*?

Backtracking’s nightmare

(a + a+) + b

against”anb”

will backtrack Θ(2n) times.

Backtracking’s nightmare

Extract the first cell in a CSV that starts with "P"1:

∧(.∗?, ) + (P.∗?),

failing against”1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13”

is exponential.

1From http://www.regular-expressions.info/catastrophic.htmlAaron Karper, Niko Schwarz (UniBe) Regex Parse Trees January 7, 2014 34 / 38

Thread execution order matters

.*(a?)

q1start

q3 q4 q5

τ1 ↑ a τ1 ↓

Priority matters

(a)|(a)

q1start

τ1 ↑

τ2 ↑

τ1 ↓

τ2 ↓

Optimization Pipeline

1 Convert to nondeterministic FA2 Interpret nondeterministic FA, building deterministic FA lazily.3 Find similar/mappable states to avoid creating infinite DFA.4 Run on DFA if possible5 Compactify DFA if creation of new states wasn’t necessary for a while.

NFA interpretation

Regular expression that produce parse trees

Education

Transcript of Regular expression that produce parse trees

From SQL parse trees to RA expression trees

11 Syntactic Parsing. Produce the correct syntactic parse tree for a sentence.

Where Syntax Meets Semantics - Webber Labs · 2015. 8. 10. · Why Parse Trees Matter We want the structure of the parse tree to correspond to the semantics of the string it generates

Parse Table Composition · In this paper we introduce an algorithm for parse table composition to support separate compilation of grammars to parse table components. Parse table com-ponents

EE1J2 - Slide 1 EE1J2 – Discrete Maths Lecture 3 Syntax of Propositional Logic Parse trees revised Construction of parse trees Semantics of propositional.

Trees, Pollinators, · Trees in Minnesota Forests and trees are important for people and wildlife. In Minnesota, trees are used to provide ﬁ ber for paper production and produce

Overview Mathematical Induction Derivations Recursive Inference Parse Trees Equivalence of Inference, Derivations, and Parse Trees.

· 2017-06-24 · process. Lexical analysis. Lex package on Unlx system. Context free granunars. Parsing and parse trees. Representation of parse (derivation) trees as rightmost

1 Syntax and Backus Naur Form How BNF and EBNF describe the grammar of a language. Parse trees, abstract syntax trees, and alternatives to BNF.

PARSE JOURNAL

Security Applications of the -Machinecsc.ucdavis.edu/~cmg/papers/SeanWhalen.UCDPhDDissertation2010.pdf · Security Applications of the -Machine By ... Incremental parse trees for

1 Parse Trees Definitions Relationship to Left- and Rightmost Derivations Ambiguity in Grammars.

Trees produce black gold Rich harvest from Jane …...Trees produce black gold Rich harvest from Jane Migao’s garden Newsletter December 2016 Biovision A future for all, naturally

Parse Trees • If w ∈ L(G), for some CFG, then w has a parse tree ...

CS5363 Final Review - College of Engineering and Applied ...qyi/UTSA-classes/cs5363/slides/Review.pdf · cs5363 7 Parse trees and abstract syntax trees Parse tree: graphical representation

READ VERIFY PARSE AUTOMATE READ VERIFY PARSE AUTOMATE READ VERIFY PARSE AUTOMATE Presented by: Todd Liebman SVP Marketing & Operations.

BW PARSE | PARSE ACTIVITIES | WISHTREE TECHNOLOGIES | LEARNING | TIBCO TRAINING |CORPORATE | TRAININ

Capsule Deep Generative Model That Forms Parse Trees

M-ary Trees. m-ary trees Some trees need to be searched efficiently, but have more than two children l parse trees l game trees l genealogical trees,

Department of Computer Science - Columbia Universityaho/Talks/10-03-12_SIGCSE.pdf · only one parse tree for. ... n. has parse trees ... 1.2 * (3.4 + 5.6) 10.8. 33 Al Aho. Added Topics