Creating own language made easy
Ingvar Stepanyan
@RReverser
Everything sucks
Everything sucks – human version
Everything sucks – developer version
Scary magic
?
?????
Not so scary magic
JS
JS
JS
ParserGenerator
Parsers
Parser generators
jison Bison in javascript, used by Coffeescript PEG.js parser generator for JavaScript based on the parsing expression grammar formalism OMeta/JS (source) metacompiler for many languages to many targets, including js.
languagejs - PEG parser written in JavaScript with first class errors Canopy Self-hosting PEG parser compiler for JavaScript, influenced by Ruby parser generators such as Treetop and Citrus JS/CC LALR(1) parser generator
jsparse PEG by Grandmaster Chris Double ReParse parser combinator library for Javascript like Haskell's Parsec p4js Monadic parser library for JavaScript JSGLR Scannerless, Generalized Left-to-right Rightmost (SGLR) derivation parser for JavaScript
antlr has a javascript target Cruiser.Parse LL(k) parser
Parser generators
Bottom-up (PEG.js)start = additive
additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative
multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary
primary = integer / "(" additive:additive ")" { return additive; }
integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
Top-down (Jison)%left "+"%left "*"
%start program;
%%
program : expression { return $$ } ;
expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;
Choice ordering
Bottom-up (PEG.js)start = additive
additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative
multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary
primary = integer / "(" additive:additive ")" { return additive; }
integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
Top-down (Jison)%left "+"%left "*"
%start program;
%%
program : expression { return $$ } ;
expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;
Ambiguity
Bottom-up (PEG.js)start = additive
additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative
multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary
primary = integer / "(" additive:additive ")" { return additive; }
integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
Top-down (Jison)%left "+"%left "*"
%start program;
%%
program : expression { return $$ } ;
expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;
if a then (if b then s) else s2 orif a then (if b then s else s2)
if a then if b then s else s2:
Left recursion
Bottom-up (PEG.js)start = additive
additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative
multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary
primary = integer / "(" additive:additive ")" { return additive; }
integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
Top-down (Jison)%left "+"%left "*"
%start program;
%%
program : expression { return $$ } ;
expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;
x = 1 - 2 - 3
Left recursion
Bottom-up (PEG.js)[ "x", "=", [ "1", "-", [ "2", "-", "3" ] ]]
Top-down (Jison)[ "x", "=", [ [ "1", "-", "2" ], "-", "3" ]]
x = 1 - 2 - 3
Summary choice
Bottom-up (PEG.js)start = additive
additive = left:multiplicative "+" right:additive { return left + right; } / multiplicative
multiplicative = left:primary "*" right:multiplicative { return left * right; } / primary
primary = integer / "(" additive:additive ")" { return additive; }
integer "integer" = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
Top-down (Jison)%left "+"%left "*"
%start program;
%%
program : expression { return $$ } ;
expression : expression "+" expression -> $1 + $3 | expression "*" expression -> $1 * $3 | "(" expression ")" -> $2 | NUMBER -> $1 ;
Jison syntax: helpers
%{
var scope = {};
%}
…
Jison syntax: lexer
…
%lex
%%
\s+ /* skip whitespace */
[A-Za-z_]\w+ return 'ID';
\d+ return ‘NUMBER’;
[+*;=] return yytext;
<<EOF>> return 'EOF';
/lex
…
Jison syntax: operator precedence
…
%left ';‘
%right ‘=‘
%left ‘+’
%left ‘*’
…
/*
“x=a*b+c”
-> assign(“x”, “a*b+c”)
-> assign(“x”, add(“a*b”, “c”))
-> assign(“x”, add(mul(“a”, “b”), “c”))
*/
Jison syntax: rules
%start program
program
: stmt* EOF { return $1 }
;
stmt
: expr ‘;’ -> $1
;
expr
: expression "+" expression -> $1 + $3
| expression "*" expression -> $1 * $3
| NUMBER -> $1
;
Code generation
Methods
string concatenation
building AST object + escodegen (http://github.com/Constellation/escodegen)
using SourceNode from source-map (https://github.com/mozilla/source-map)
Debugging: source maps
Methods
string concatenation
building AST object + escodegen (http://github.com/Constellation/escodegen)
using SourceNode from source-map (https://github.com/mozilla/source-map)
AST way
Methods
string concatenation
building AST object + escodegen (http://github.com/Constellation/escodegen)
using SourceNode from source-map (https://github.com/mozilla/source-map)
SourceNode
new SourceNode(line, column, filename, jsChunk)
line, column – position in original file
filename – name of original file
jsChunk – JavaScript code string, another SourceNode instance or array of those
Resulting stack
JS
JS
JS
Jisonsource-map
Live demo