Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code...

Review

1. Lexical Analysis

2. Syntax Analysis

3. Semantic Analysis

4. Code Generation

5. Code Optimization

Syntax Analysis

• Often called parsing• Groups tokens of source program into grammatical

phrases that are used by the compiler to check for correct syntax and to help in generating code

• Creation of a hierarchical structure called a syntax tree– Tree helps us determine if program is syntactically correct

– Also aids in the translation of source program to target language

Grammar Example

San Francisco

Seattle Lexical Ogay orthnay eightay undrendhay ilesmay

Syntactical Miles go hundred north eight

Logical Go eight hundred miles(facing south)

Run-Time Go eight hundred miles (facing West)

Grammars – Defining the Language Rules

• Terminals (tokens)• Non-terminals - Syntactic variable. Contains groups

of tokens that define some part of the language– Example: expression, if statement, etc

• Start symbol - A special non-terminal (i.e. a program)• Productions

– The manner in which terminals and non-terminals are combined to form statements

– A non-terminal in LHS and a string of terminals and non-terminals in RHS

Example

• Variable Declaration– A type followed by one or more comma separated

identifiers that end with a semi-colon.<Var> -> <Indent>

<Var> -> <Var> , <Ident>

• <Var> and <Ident> are non-terminals

• The comma is a terminal

• The two lines are called productions

Grammars

• We will be defining a grammar for the entire JO99 programming language.

• We will then have JCUP produce for us a parser that detects if the JO99 program is syntactically correct.

• In addition, the parser will create for us a tree that represents the program and allows us to be able to do other things like semantic checks and code generation.

Example

• Simple arithmetic expressions with + and *– 8.2 + 35.6– 8.32 + 86 * 45.3 – (6.001 + 6.004) * (6.035 * -(6.042 + 6.046))

• Terminals (or tokens)– num for all the numbers– ‘+’, ‘-’, ‘*’, ‘(‘, ‘)’

• What is the grammar for all possible expressions?

Example

<expr> <expr> <op> <expr>

<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> *

Categories of Parsers

( )


– L - parse from left to right– R - parse from right to left

( )


– L - leftmost derivation– R - rightmost derivation

( )


– Number of look ahead characters

( )


– Examples: • LL(0) – Parse Left to Right, Derive the tree using a

leftmost derivation (top down), no look ahead characters• LR(1) – Parse Left to Right, Derive the tree using a

rightmost derivation (bottom up), 1 look ahead character.

– Each category of parsing handles a different type of language.

– We will be learning about LR(k) parsers and will implement an LR(k) parser.

Why LR(k)?

• Virtually all programming language grammars can be parsed using a LR(k) technique

• Most general parsing method for programming grammars

• Can build a very efficient parser engine given just the syntax rules of the language.

• Can detect a syntactic error as soon as it is possible to do so

• Because its so general, programs have been written (JCUP) that produce the parser instead of writing it from scratch.

LR(k) Parser implementation

• Sometimes called a Shift-Reduce Parser

• Parse from left to right (get the tokens from left to right)

• Bottom up parsing (same as rightmost derivation)

Actions of a Shift-Reduce Parser

ParseTree


ParseTreeParseTree


ParseTree


• How do we build this tree?

• As productions are recognized, a portion of the tree is created.

• This portion of the tree will be needed later to build bigger portions of the tree and therefore must be saved for future use.

• This requires a stack.

• The stack plus the next token read from the source program determines the action.

Actions of a Shift-Reduce ParserS

tack

Current Symbol

stack

Parser A

ction

ParserEngine

Actions of a Shift-Reduce Parser• Shift

– Shift the current element into top of the stack– Move the current pointer (next token)

• Reduce– Apply a production (we recognize a part of the

program)– Top of the stack should match the RHS of the

grammar– Remove those symbols from the stack– Add the LHS non-terminal

• Accept– End of stream reached and stack only has the start

symbol• Reject

– End of stream reached but stack has more than the start symbol

Shift-Reduce Parser Example

* ( + num )numnum


* ( + num )numnum


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *


* ( + num )numnum


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *num

SH

IFT


* ( + num )numnum


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *num

RE

DU

CE


* ( + num )num

<expr> <expr> <expr><op> <op>

<expr>

<expr>

<expr>

num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *<expr>

RE

DU

CE


( + num )num


<expr>

<expr>

<expr>

num *


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

*

<expr>

SH

IFT


( + num )num


<expr>

<expr>

<expr>

num *


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

*

<expr>

RE

DU

CE


( + num )num


<expr>

<expr>

<expr>

num *


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

<expr>

RE

DU

CE


+ num )num


<expr>

<expr>

<expr>

num * (


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

SH

IFT


* + num )


<expr>

<expr>

<expr>

num ( num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

num

<expr>

SH

IFT


* + num )


<expr>

<expr>

<expr>

num ( num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

num

<expr>

RE

DU

CE


* + num )


<expr>

<expr>

<expr>

num ( num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

<expr>

RE

DU

CE


* num )


<expr>

<expr>

<expr>

num ( num +


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

+

<expr>

SH

IFT


* num )


<expr>

<expr>

<expr>

num ( num +


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

+

<expr>

RE

DU

CE


* num )


<expr>

<expr>

<expr>

num ( num +


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

<op>

<expr>

RE

DU

CE


* )


<expr>

<expr>

<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

<op>

num

<expr>

SH

IFT


* )


<expr>

<expr>

<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

<op>

num

<expr>

RE

DU

CE


* )


<expr>

<expr>

<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

<op>

<expr>

<expr>

RE

DU

CE


* )


<expr>

<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

<expr>

<expr>

RE

DU

CE


* )


<expr>

<expr>

<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

)

<expr>

SH

IFT


* )


<expr>

<expr>

<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

(

<expr>

)

<expr>

RE

DU

CE


* )


<expr>

num ( num + num


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

<op>

<expr>

<expr>

<expr>

<expr>

RE

DU

CE


* )


<expr>

<expr>

<expr>

num ( num + num

<op>

<expr>

<expr>

RE

DU

CE


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *


* )


<expr>

<expr>

num ( num + num

<expr>

<expr>

RE

DU

CE


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *


* )


<expr>

<expr>

num ( num + num

<expr>

<expr>

AC

CE

PT


<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

What does the parser engine do?

• If the top symbols of the stack match the RHS of a production then do the reduction – Pop the RHS from the top of the stack– Push the LHS symbol onto the stack

• If no production is found do the shift– Push the current input into the stack

• If the input is empty– Accept if only the start symbol is on the stack– Reject otherwise

ParserEngine

This is not that simple!

• Many choices of reductions if there are multiple RHS.

• Which LHS do we put on stack in its place?

• Choice between shift and reduce– Stack matches a RHS– But that may not be the right match– May need to shift an input onto stack and later find

a different reduction



<expr> ( <expr> )

<expr> - <expr>

<expr> num

<op> +

<op> -

<op> *

• Change in the Grammar



<expr> ( <expr> )

<expr> <expr> -<expr> num

<op> +

<op> -

<op> *

• Change in the Grammar


- numnum


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *


- num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

num

num


- num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum

SH

IFT

num


- num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum

RE

DU

CE

<expr>


- num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum

RE

DU

CE

<expr>

-<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

SH

IFT

-<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

We have a choice!!!<op><expr>

RE

DU

CE

-<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

But not the right thing to do!!<op><expr>

RE

DU

CE

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

<expr>

RE

DU

CE

But not the right thing to do!!

num

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

But not the right thing to do!!<expr>

<expr>

SH

IFT

num

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -


<expr>

RE

DU

CE

<expr>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -


<expr>

<expr>

RE

DU

CE

<expr>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

But not the right thing to do!! No more actions!!!

<expr>

<expr>

<expr>

ER

RO

R


• But this is perfectly valid input for the grammar

• We chose the wrong production and thus the wrong LHS

• Lets see what happens when we choose the right production and the right LHS

-<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

We have a choice<op><expr>

RE

DU

CE

The step before we went wrong

-<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

Use the other production

<op><expr>

RE

DU

CE

<op>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr>

RE

DU

CE

<op> Use the other production

num

<op>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr> <op>

SH

IFT

num

<op>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr> <op>

RE

DU

CE

<expr>

<op>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr> <op>

RE

DU

CE

<expr>

<expr>


num

< <expr> <expr> <op> <expr><expr> ( <expr> )<expr> <expr> -

<expr> num<op> +

<op> - <op> *

numnum -

<expr> <op> <expr>

RE

DU

CE

<expr>

<expr>


num


<expr> ( <expr> )

<expr> <expr> -

<expr> num

<op> +

<op> -

<op> *

numnum -

<expr> <op> <expr>

<expr>

AC

CE

PT

Shift-Reduce Parser

• Parser Engine is far more complicated that it appears.

• Requires it to know all possible productions that would match the top of the stack and the given input symbol.

Constructing a LR(k) Parser

• What is in the parse engine– decide between shift and reduce– decide on the right reduction

Constructing a LR(k) Parser

• Create a DFA – Encodes all the possible states that the parser can be in– DFA state transition occurs on terminals and non-

terminals

• Create a Parser Table – From the DFA create a transition table that stores

what action should be taken for the current state and current input character

• Maintain a stack of states in parallel with the stack of symbols

LR(k) Parser Engine

Current Symbol

Parser A

ction

LR(k)ParserEngine

Sym

bol

Sta

ck

Sta

te S

tack

ACTION GotoState ( ) $ Xs0 shift to s2 error error goto s1s1 error error accept s2 shift to s2 shift to s5 error goto s3s3 error shift to s4 error s4 reduce (2) reduce (2) reduce (2) s5 reduce (3) reduce (3) reduce (3)

Parser Tables


• Look-up [top of state stack] [ input symbol] in the parser table

• Carry-out the described action

Parser Tables

• Shift to sn– Push input token into the symbol stack– Push sn into state stack– Advance to next input symbol


Parser Tables

• Reduce (n)– Pop both stacks as many times as the number of symbols

on the RHS of rule n – Push LHS of rule n into symbol stack– Lookup [top of the state stack][top of symbol stack]– Push that state (in goto k) into state stack


Parser Tables

• Accept– Stop parsing and report success


Parser Tables

• Error– Stop parsing and report failure


LR example

• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

Question

• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

Parser Table in Action• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

Parser Table in Action

• Parser Table• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)ACTION Goto

State ( ) $ Xs0 shift to s2 error error goto s1s1 error error accept s2 shift to s2 shift to s5 error goto s3s3 error shift to s4 error s4 reduce (2) reduce (2) reduce (2) s5 reduce (3) reduce (3) reduce (3)



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

$




<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $




<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $




<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( )( $


(

s2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

))( $


s2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

))( ( $


s2 (s2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

))( ( $


s2 (s2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s5 )s2 (s2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s5 )s2 (s2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( )

s5 )s2 (

$


Xs2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s3 Xs2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s4 )s3 Xs2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s4 )s3 Xs2 (



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $

s4 )s3 Xs2 (


X



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s1 X



<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $


s1 X

Parser Table in Action• The grammar

<S> <X> $ (1)

<X> ( <X> ) (2)

<X> ( ) (3)

s0 $

)( ( ) $

Accept


Parser Table

• The table (DFA) and the stacks are called Push Down Automaton (PDA).

• There is an algorithm for converting a grammar to a PDA.

• Parser generators – given a grammar, produce a parser table with a stack (PDA)

• We will not study how this is done because its somewhat complicated.

• JCUP will convert our grammar into a shift-reduce parser.

Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code...

Documents

Transcript of Review 1.Lexical Analysis 2.Syntax Analysis 3.Semantic Analysis 4.Code Generation 5.Code...