org.multijava.mjc
Class ParsingController

java.lang.Object
  org.multijava.mjc.ParsingController

public class ParsingController

extends Object

This class manages the parsing of source code. It supports the use of multiple lexers and parsers to process nested languages (as in javadoc comments in Java programs). It includes most of the functionality of antlr.TokenStreamSelector, though for efficiency it duplicates the necessary code rather than inheriting it. This class buffers the tokens produced by multiple lexers and allows client parsers to accept tokens from just a single lexer.

Nested Class Summary
static class	ParsingController.ConfigurationException           This inner class represents exceptions that can be thrown because of misconfiguration of the parsing controller.
class	ParsingController.FilteredTokenStream           This inner class provides a token stream that accesses one of the lexers managed by the controller.
(package private)  class	ParsingController.IndexedTokenBuffer           This inner class buffers tokens of a nested language and allows selection and discarding of a subset of these buffered tokens based on a reference to an index token of the surrounding language.
(package private) static class	ParsingController.InputState           This nested class is a hack to expose the line and column fields of LexerSharedInputState for reading.
static class	ParsingController.KeyException           This inner class represents exceptions that can be thrown because of bad key values passed as arguments to the outer classes methods.
(package private)  class	ParsingController.StreamBufferPair           This inner class encapsulates a buffer and an input stream so they can be the stored as a single value in a map.
static class	ParsingController.TokenWrapper           This inner class wraps a single token.

 

Field Summary
private  String	activeKey           The key of the currently active input stream.
private  ParsingController.StreamBufferPair	activeStreamBuffer           The currently active input stream and token buffer.
private  boolean	discarding           Indicates whether tokens for the currently active stream are to be discarded.
private  boolean	endOfFileReached           Indicates whether an end-of-file token has been received on some input stream.
private  boolean	fetching           Indicates that a token fetch request is in progress for the active input stream.
static antlr.Token	IGNORE           A token that may be returned by lexers upon detecting a lexical pattern that triggers a switch between lexers (e.g., /** for switching to javadoc).
private  ParsingController.FilteredTokenStream	initialFilteredStream           The output stream for the initially selected lexer.
private  boolean	initialSet           Indicates whether the initial input stream has been selected.
private  ParsingController.InputState	inputState           The input state that should be shared by the lexers bound to this parsing controller.
private  Stack	keyStack           For efficiency, used to track the keys when switching between input streams.
private  Set	keysToDiscard           The set of keys corresponding to lexers whose output are being discarded.
private  Map	keyStreamBufferMap           Maps keys to lexers and buffers
private  int	ltCount
private  ParsingController.IndexedTokenBuffer	nestedLanguageBuffer           The active buffer before the last pop operation.
private static int	NONE           When operationPending equals this, then no operations are pending.
private  int	operationPending           Indicates whether any input stream switching operations (i.e., push, pop) are pending on completion of the current call to bufferNextToken.
private  String	pendingKey           The key of any pending push operation.
private static int	POP           When operationPending equals this, then a pop operation is pending.
private static int	PUSH           When operationPending equals this, then a push operation is pending.
private  Stack	streamBufferStack           Tracks the active input streams.
private  int	unmatchedTypeLT           Number of LT tokens that have not been matched by GT in Type Arguments or Type Parameters
private  Set	usedKeys           The set of keys corresponding to lexers whose output are being discarded or have been bound to output token streams.

 

Constructor Summary

ParsingController(Reader r, File file)           Establish a parsing controller for the given reader.

 

Method Summary
void	addInputStream(antlr.TokenStream stream, String key)           Adds the given input stream to the controller and binds it to the given key.
private  void	bufferNextToken()           Retrieves the next token from the currently active lexer and adds it to the appropriate buffer or discards it if necessary.
int	column()           Returns the current column number in the input.
void	discardAllTokensFor(String key)           Indicates that tokens generated by the lexer bound to the given key should be discarded instead of buffered.
boolean	discardTokensBefore(String key, antlr.Token index)           Discards all tokens from the lexer bound to the given key that precede the given token of the surrounding language.
File	file()           Returns the file name of the input.
antlr.TokenStream	get(String key)           Returns the TokenStream corresponding to a given name.
antlr.TokenStream	getCurrentStream()           Returns the currently active input stream.
antlr.TokenStream	getStream(String sname)           Returns the named input stream (previously added with addInputStream).
int	getUnmatchedTypeLT()           get the value of unmatchedTypeLT
ParsingController.FilteredTokenStream	initialOutputStream()           Returns a token stream that provides tokens generated by the initial lexer.
int	line()           Returns the current line number in the input.
private  ParsingController.IndexedTokenBuffer	newBlockedEmptyBuffer()           Factory method creates a new token buffer that is blocked and empty.
void	pop()           Pops a stream from the stack of input streams.
private  void	processPop()           Pops a stream from the stack of input streams.
private  void	processPush(String key)           Switches to a new input stream, maintaining a stack of the old input streams.
void	push(String key)           Switches to a new input stream for future token requests, maintaining a stack of the old input streams.
void	selectInitial(String key)           Makes the lexer bound to the given key the initial lexer.
private  void	setActive(String key, ParsingController.StreamBufferPair sbp)           Sets the given key, input stream, and token buffer to be active.
void	setUnmatchedTypeLT(int count)           set the value of unmatchedTypeLT
antlr.LexerSharedInputState	sharedInputState()           Returns an input state that can be shared by the lexers attached to this controller.
ParsingController.FilteredTokenStream	streamForBefore(String key, antlr.Token index)           Returns a token stream from the lexer bound to the given key and containing all the tokens preceding the given token of the surrounding language.
private  void	useKey(String key)           Registers that tokens generated by the lexer bound to the given key will be used or discarded.
private  void	verifyHasKey(String key)           Verifies that the given key is bound to a lexer.

 

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

 

Field Detail

inputState

private ParsingController.InputState inputState

The input state that should be shared by the lexers bound to this parsing controller.

initialSet

private boolean initialSet

Indicates whether the initial input stream has been selected.

initialFilteredStream

private ParsingController.FilteredTokenStream initialFilteredStream

The output stream for the initially selected lexer.

keyStreamBufferMap

private Map keyStreamBufferMap

Maps keys to lexers and buffers

usedKeys

private Set usedKeys

The set of keys corresponding to lexers whose output are being discarded or have been bound to output token streams.

keysToDiscard

private Set keysToDiscard

The set of keys corresponding to lexers whose output are being discarded.

discarding

private boolean discarding

Indicates whether tokens for the currently active stream are to be discarded.

keyStack

private Stack keyStack

For efficiency, used to track the keys when switching between input streams. The top of the the stack is the previous active key, not the current active key.

streamBufferStack

private Stack streamBufferStack

Tracks the active input streams. The top of the the stack is the previous active input stream and token buffer, not the current.

activeKey

private String activeKey

The key of the currently active input stream.

activeStreamBuffer

private ParsingController.StreamBufferPair activeStreamBuffer

The currently active input stream and token buffer.

nestedLanguageBuffer

private ParsingController.IndexedTokenBuffer nestedLanguageBuffer

The active buffer before the last pop operation. Used to store index tokens into the buffer of a nested language.

endOfFileReached

private boolean endOfFileReached

Indicates whether an end-of-file token has been received on some input stream.

fetching

private boolean fetching

Indicates that a token fetch request is in progress for the active input stream.

operationPending

private int operationPending

Indicates whether any input stream switching operations (i.e., push, pop) are pending on completion of the current call to bufferNextToken.

ltCount

private int ltCount

pendingKey

private String pendingKey

The key of any pending push operation.

See Also:

operationPending

unmatchedTypeLT

private int unmatchedTypeLT

Number of LT tokens that have not been matched by GT in Type Arguments or Type Parameters

NONE

private static final int NONE

When operationPending equals this, then no operations are pending.

See Also:

operationPending

POP

private static final int POP

When operationPending equals this, then a pop operation is pending.

See Also:

operationPending

PUSH

private static final int PUSH

When operationPending equals this, then a push operation is pending.

See Also:

operationPending

IGNORE

public static final antlr.Token IGNORE

A token that may be returned by lexers upon detecting a lexical pattern that triggers a switch between lexers (e.g., /** for switching to javadoc). This token is not passed to any client parsers but is swallowed by the parsing controller. Using this token in a lexer allows the associated parser to ignore the lexical triggers. Note that using a "skip" token in ANTLR doesn't work for this purpose because the generated lexer code will not return from a nextToken call until a non-skip token is found. Thus using a skip token causes the lexer switch to be delayed.

ParsingController

public ParsingController(Reader r,
                         File file)

Establish a parsing controller for the given reader. The reader is used to generate shared lexer state for the various lexers tied to this. A lexer tied to this parser should be instantiated by passing it this. The lexer's constructor will call a super constructor passing it the result of calling sharedInputState() on this.

See Also:

sharedInputState()

sharedInputState

public antlr.LexerSharedInputState sharedInputState()

Returns an input state that can be shared by the lexers attached to this controller.

addInputStream

public void addInputStream(antlr.TokenStream stream,
                           String key)
                    throws ParsingController.KeyException

Adds the given input stream to the controller and binds it to the given key.

Throws:

ParsingController.KeyException - the given key is already bound or if either argument is null

getCurrentStream

public antlr.TokenStream getCurrentStream()

Returns the currently active input stream.

getStream

public antlr.TokenStream getStream(String sname)

Returns the named input stream (previously added with addInputStream).

selectInitial

public void selectInitial(String key)
                   throws ParsingController.KeyException,
                          ParsingController.ConfigurationException

Makes the lexer bound to the given key the initial lexer.

Throws:

ParsingController.KeyException - if the given key is not bound

ParsingController.ConfigurationException - if the initial lexer has already been selected

initialOutputStream

public ParsingController.FilteredTokenStream initialOutputStream()
                                                          throws ParsingController.ConfigurationException

Returns a token stream that provides tokens generated by the initial lexer.

Throws:

ParsingController.ConfigurationException - if the initial lexer has not been selected.

See Also:

selectInitial(String)

discardAllTokensFor

public void discardAllTokensFor(String key)
                         throws ParsingController.KeyException

Indicates that tokens generated by the lexer bound to the given key should be discarded instead of buffered. This allows the controller to make more efficient use of memory.

Throws:

ParsingController.KeyException - if the given key is not bound to a lexer or if a TokenStream for the key has already been exposed

streamForBefore

public ParsingController.FilteredTokenStream streamForBefore(String key,
                                                             antlr.Token index)
                                                      throws ParsingController.KeyException

Returns a token stream from the lexer bound to the given key and containing all the tokens preceding the given token of the surrounding language.

Throws:

ParsingController.KeyException - if the given key is not bound to a lexer

discardTokensBefore

public boolean discardTokensBefore(String key,
                                   antlr.Token index)
                            throws ParsingController.KeyException

Discards all tokens from the lexer bound to the given key that precede the given token of the surrounding language.

Returns:

true if any tokens are actually discarded

Throws:

ParsingController.KeyException - if the given key is not bound to a lexer

pop

public void pop()
         throws antlr.TokenStreamException

Pops a stream from the stack of input streams. Called when exiting a nested language block.

Throws:

antlr.TokenStreamException - if a pending operation prevents popping

get

public antlr.TokenStream get(String key)
                      throws ParsingController.KeyException,
                             antlr.TokenStreamException

Returns the TokenStream corresponding to a given name.

Parameters:

key - the name of the TokenStream to return

Returns:

the matching TokenStream

Throws:

ParsingController.KeyException

antlr.TokenStreamException

push

public void push(String key)
          throws ParsingController.KeyException,
                 antlr.TokenStreamException

Switches to a new input stream for future token requests, maintaining a stack of the old input streams. Called when entering a nested language block.

Parameters:

key - the unique identifier of the lexer to switch to

Throws:

antlr.TokenStreamException - if a pending operation prevents pushing

ParsingController.KeyException - if the given key is not bound to a lexer

file

public File file()

Returns the file name of the input.

line

public int line()

Returns the current line number in the input.

column

public int column()

Returns the current column number in the input.

bufferNextToken

private void bufferNextToken()
                      throws antlr.TokenStreamException

Retrieves the next token from the currently active lexer and adds it to the appropriate buffer or discards it if necessary.

Throws:

antlr.TokenStreamException - passed on from underlying lexer

processPush

private void processPush(String key)

Switches to a new input stream, maintaining a stack of the old input streams. Called when entering a nested language block.

Parameters:

key - the unique identifier of the lexer to switch to

processPop

private void processPop()

Pops a stream from the stack of input streams. Called when exiting a nested language block.

useKey

private void useKey(String key)
             throws ParsingController.KeyException

Registers that tokens generated by the lexer bound to the given key will be used or discarded. Handles checks to prevent assigning a lexer to multiple output streams or to both discard and output.

Parameters:

key - the unique identifier of the lexer whose tokens are to be discarded or passed to an output stream

Throws:

ParsingController.KeyException - if the given key is not bound to a lexer or if a TokenStream for the key has already been exposed

verifyHasKey

private void verifyHasKey(String key)
                   throws ParsingController.KeyException

Verifies that the given key is bound to a lexer.

Throws:

ParsingController.KeyException - if the given key is not bound

setActive

private void setActive(String key,
                       ParsingController.StreamBufferPair sbp)

Sets the given key, input stream, and token buffer to be active.

newBlockedEmptyBuffer

private ParsingController.IndexedTokenBuffer newBlockedEmptyBuffer()

Factory method creates a new token buffer that is blocked and empty.

getUnmatchedTypeLT

public int getUnmatchedTypeLT()

get the value of unmatchedTypeLT

setUnmatchedTypeLT

public void setUnmatchedTypeLT(int count)

set the value of unmatchedTypeLT