20130530-PEGjs

PEG.js- Javascript Parser Generator - PEG.js- Javascript Parser Generator -

Hidetomo Suzuki

2013 / 5 / 30

ScopeScope

This MTG Target

Understand What’s PEG.js

Know How to Use PEG.js

Make New DSL and Parser

2013 / 5 / 30

AgendaAgenda

Table of Contents

1.How to make parser

2. What’s PEG.js

3. Try to used to be PEG

4. Let’s make DSL

5.Applications

2013 / 5 / 30

1. How to make parser 1. How to make parser

1. Lexical Analysis(字句解析 )

2. Syntactic Parsing( 構文解析 )

3. Semantic Analysis( 意味解析 )

4. Intermidiate Code Generation( 中間コード生成 )

5. Code Optimization( コード最適化 )

6. Code Generation( コード生成）

General Step of Processing with Compiler

Parser

2013 / 5 / 30


1. Lexical Analysis(字句解析 )

2. Syntactic Parsing( 構文解析 )

3. Semantic Analysis( 意味解析 )

Analysis for Make Parser

Make strings(minimum unit of string has semantic) from characters

Make tree structure from strings which result of Lexical Analysis

Type Check

※ A string of Lexical Analysis is called “Token”.

2013 / 5 / 30


1. Lexical Analysis

position_x = position_x + 2.0 * time

Identifier : position_x Substitution Symbol : = Identifier : position_x Addition Symbol : + Number : 2.0 Multiple Symbol : * Identifier : time

2013 / 5 / 30


2. Syntactic Parsing Identifier : position_x Substitution Symbol : = Identifier : position_x Addition Symbol : + Number : 2.0 Multiple Symbol : * Identifier : time

Substitution Symbol

Identifier

position_x

Expression

ExpressionExpression

Expression ExpressionIdentifier

position_x

Number

2.0

Identifier

time

=

+

*

2013 / 5 / 30


3. Semantic Analysis

Substitution Symbol

Identifier

position_x

Expression

ExpressionExpression

Expression ExpressionIdentifier

position_x

Number Identifier

=

+

Correct Case : Real Number * Integer Number

Wrong Case : Real Number * Function Pointer

2.0 time

*

2013 / 5 / 30

AgendaAgenda

Table of Contents


2. What’s PEG.js


4. Let’s make DSL

5.Applications

2013 / 5 / 30 Here comes your footer

2. What‘s PEG.js?2. What‘s PEG.js?

Parser Generator for JavaScript

Use PEG(Parsing Expression Grammar)

Easy to try with the web page

Outline of PEG.js

http://pegjs.majda.cz/

http://pegjs.majda.cz/



PEG(Parsing Expression Grammar) is one of grammar for artificial language.

Focus point is how input will be analyzed.

What‘s PEG

Ex) Simple Expression Grammar which Has Four Arithmetic Operations

expression <- addexp

addexp <- multiexp (“+” multiexp / “-” multiexp)*

multiexp <- number (“*” number / “/” number)*

number <- [0-9]+

Accept : 2*3+6/2-5*3 -> multiexp + multiexp – multiexp -> ・・・Decline : (1+1)*3+6/2-(7-2)*3 -> ?*3+6/2-?*3 -> can’t recognize…



• Nothing Confusion

• Different : “a b / a”, “a / a b”

• Easy to use compared with CFG

• Don’t need to implement scanner (Lexical Analyzer)

• Don’t Express Left Recursive

PEG vs CFG(Context Free Grammar)

• Confusion Existing

• Same : “a b | a”, “a | a b”

• Need to implement scanner

• Can Express Left Recursive

PEG CFG

2013 / 5 / 30


1. Make Grammar with PEG

2. Generate Parser with “pegjs” Command

3. Use the Parser Loaded as a Library

Step of Make Parser with PEG.js

vim filename.pegjs

Pegjs filename.pegjs parser.js

<script src=“./parser.js” type=“text/javascript”></script><script type=“text/javascript”> parser.parse(“something”);

2013 / 5 / 30


Install PEG.js

# Install Pythonbrew $ curl -kL http://xrl.us/pythonbrewinstall | bash$ vim ~/.bash_profile[[ -s $HOME/.pythonbrew/etc/bashrc ]] && source $HOME/.pythonbrew/etc/bashrc$ source ~/.bashrc$ pythonbrew install 2.7.2$ pythonbrew switch 2.7.2

# Install Node$ git clone git://github.com/creationix/nvm.git ~/.nvm$ nvm install v0.8.7$ source ~/.nvm/nvm.sh$ nvm use v0.8.7$ vim ~/.npmrc $ registry = http://registry.npmjs.org/

2013 / 5 / 30


1. Make Grammar with PEG

2. Generate Parser with “pegjs” Command

3. Use the Parser Loaded as a Library

Let’s use PEG.js a little bit

$ vim ffirst.pegjsstart = addexpaddexp = integer ("+" integer / "-" integer)*integer = [0-9]+

$ ./node_modules/pegjs/bin/pegjs first.pegjs parser.js

$ vim parser.js1st line: module.export = … -> var parser = …$ vim index.html<script type=“text/javascript” src=“./parser.js”></script><script type=“text/javascript”> alert(parser.parse(“1+2+3-5”));</script>

2013 / 5 / 30

AgendaAgenda

Table of Contents


2. What’s PEG.js


4. Let’s make DSL

5.Applications

2013 / 5 / 30

3.Try to used to be PEG3.Try to used to be PEG

PEG.js Online Version (http://pegjs.majda.cz/online)

http://pegjs.majda.cz/online

2013 / 5 / 30

3. Try to used to be PEG3. Try to used to be PEG

Very Simple PEG.js Grammar(calc : integer)

Grammar

Test Input

Test Input Result

start = integerinteger = digits:[0-9]+ { return parseInt(digits.join(“”), 10); }

123456789

123456789

2013 / 5 / 30


Very Simple PEG.js Grammar(calc : add two integer)

Grammar

Test Input

Test Input Result

start = target1:integer "+" target2:integer { return target1+target2; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

81+19

100

2013 / 5 / 30


Very Simple PEG.js Grammar(calc : add and minus two integer)

Grammar

Test Input

Test Input Result

start = target1:integer "+" target2:integer { return target1+target2; } / target1:integer "-" target2:integer { return target1-target2; } integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

100-19

81

2013 / 5 / 30


Very Simple PEG.js Grammar(calc : add and minus integers)

Grammar

Test Input

Test Input Result

start = expressionexpression = ope1:integer "+" ope2:expression { return ope1+ope2; } / ope1:integer "-" ope2:expression { return ope1-ope2; } / ope:integer { return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

1+2+3+4+5-6+7-8+9-10

9

2013 / 5 / 30


Very Simple PEG.js Grammar(calc : four arithmetic)

Grammar

Test Input

Test Input Result

start = expressionexpression = ope1:integer "+" ope2:expression { return ope1+ope2; } / ope1:integer "-" ope2:expression { return ope1-ope2; } / ope1:integer "*" ope2:expression { return ope1*ope2; } / ope1:integer "/" ope2:expression { return ope1/ope2; } / ope:integer { return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

2*3/2

3

2013 / 5 / 30


Very Simple PEG.js Grammar(calc : four arithmetic with priority)

Grammar

Test Input

Test Input Result

start = expressionexpression = ope1:multiple "+" ope2:expression { return ope1+ope2; } / ope1:multiple "-" ope2:expression { return ope1-ope2; } / ope:multiple { return ope; }multiple = ope1:integer "*" ope2:multiple { return ope1*ope2; } / ope1:integer "/" ope2:multiple { return ope1/ope2; } / ope:integer { return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

2*3/2+8/4

5

2013 / 5 / 30


Very Simple PEG.js Grammar(calc : four arithmetic with priority)

Grammar

Test Input

Test Input Result

start = expressionexpression = ope1:multiple "+" ope2:expression { return ope1+ope2; } / ope1:multiple "-" ope2:expression { return ope1-ope2; } / ope:multiple { return ope; }multiple = ope1:bracket "*" ope2:multiple { return ope1*ope2; } / ope1:bracket "/" ope2:multiple { return ope1/ope2; } / ope:bracket { return ope; }bracket = "(" exp:expression ")" {return exp; } / ope:integer {return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }

(7+3)*3/2+8/(6-2)

17

2013 / 5 / 30

AgendaAgenda

Table of Contents


2. What’s PEG.js


4. Let’s make DSL

5.Applications

2013 / 5 / 30

4. Let‘s make DSL4. Let‘s make DSL

Config File Format

Parser Result

Web Page Image

Target

<LogicalExpression?>Y---[function param]N---[function param]

if (isLogicalExpression) function(param);else function(param);

Ex)<Hydea?>Y---[show ‘hydea’]N---[show ‘hydeb’]

Ex)if (isHydea) show(‘hydea’);else show(‘hydeb’);

URL: ex.com/?p=hydea URL: ex.com/?p=hydeb

2013 / 5 / 30


Just write over specification

Make Grammar

start = "<" logic "?>Y---[" function " " param "]\nN---[" function " " param "]"logic = [a-zA-Z]+function = [a-zA-Z]+param = "'" [a-zA-Z]+ "'"

2013 / 5 / 30


Refactoring a little bit

Make Grammar

start = "<" identifier "?>Y---[" identifier " " param "]\nN---[" identifier " " param "]"param = "'" identifier "'"identifier = [a-zA-Z]+

2013 / 5 / 30


Finish to make param and identifier part

Make Grammar

start = "<" identifier "?>Y---[" identifier " " param "]\nN---[" identifier " " param "]"param = "'" id:identifier "'" {return "'" + id + "'"; }identifier = chars:[a-zA-Z]+ {return chars.join(""); }

2013 / 5 / 30


Organize Function Part

Make Grammar

start = "<" identifier "?>Y---[" function "]\nN---[" function "]"function = id:identifier " " param:param {return id + "(" + param + ")"; }param = "'" id:identifier "'" {return "'" + id + "'"; }identifier = chars:[a-zA-Z]+ {return chars.join(""); }

2013 / 5 / 30


Finish to make

Make Grammar

start = "<" id:identifier "?>Y---[" f1:function "]\nN---[" f2:function "]" {return "if (is" + id + ") " + f1 + ";\nelse " + f2 + ";"; }function = id:identifier " " param:param {return id + "(" + param + ")"; }param = "'" id:identifier "'" {return "'" + id + "'"; }identifier = chars:[a-zA-Z]+ {return chars.join(""); }

2013 / 5 / 30

AgendaAgenda

Table of Contents


2. What’s PEG.js


4. Let’s make DSL

5.Applications

2013 / 5 / 30

5. Application5. Application

Show My Demo(Demo’s grammar)start = treetree = stat:statement? { return stat; }statement = stat:if_statement { return stat.toString(); }/ stat:proc_statement { return stat.toString(); }if_statement = fac:if_factor 'Y' edge branch1:proc_statement '\n''N' '\n' edge branch2:statement{ return branch2.toString().match(/if/) ?"if (" + fac + ")\n\t" + branch1 + '\nelse ' + branch2: 'if (' + fac + ')\n\t' + branch1 + '\nelse\n\t' + branch2;}proc_statement = fac1:proc_factor [-|]+ fac2:proc_statement { return fac1 + '.then(' + fac2.slice(0,fac2.length-1) + ');'; }/ factor:proc_factor { return factor + ';'; }proc_factor = '[' procexp:expression ']' { return procexp; }if_factor = '<' ifexp:expression '>' { return ifexp; }expression = elem:element ' ' cdr:arguments { return elem.toString() + '(' +cdr + ')'; }/ elem:element { return elem.toString(); }arguments = elem:element ',' arg:arguments { return elem.toString() + ',' + arg; }/ elem:element { return elem.toString(); }element= characters:[a-zA-Z0-9-_."]+ { return characters.join('') }edge= symbols:[-|\n]+ { return symbols.join('') }

2013 / 5 / 30

5. Application5. Application

Recommendation

Conf1 Conf2 Conf3 ・・・

RecommendationEngine

Choose a config file

2013 / 5 / 30

ReferenceReference

Famous Book for Compiler

2013 / 5 / 30

Thank youThank you

Thank you for your listening

20130530-PEGjs

Software

Transcript of 20130530-PEGjs