org.eclipse.jdt.internal.compiler.parser
Class Scanner

java.lang.Object
  org.eclipse.jdt.internal.compiler.parser.Scanner

All Implemented Interfaces:

TerminalTokens

Direct Known Subclasses:

CommentRecorderScanner, CompletionScanner, RecoveryScanner, SelectionScanner

public class Scanner
extends java.lang.Object
implements TerminalTokens

IMPORTANT NOTE: Internal Scanner implementation. It is mirrored in org.eclipse.jdt.core.compiler public package where it is API. The mirror implementation is using the backward compatible ITerminalSymbols constant definitions (stable with 2.0), whereas the internal implementation uses TerminalTokens which constant values reflect the latest parser generation state.

Field Summary

static int	BracketKinds
char[][][][]	charArray_length
boolean	checkNonExternalizedStringLiterals
static int	COMMENT_ARRAYS_SIZE
int	commentPtr
int[]	commentStarts
int[]	commentStops
int[]	commentTagStarts
long	complianceLevel
boolean	containsAssertKeyword
static int	CurlyBracket
char	currentCharacter
int	currentPosition
boolean	diet
static java.lang.String	END_OF_SOURCE
int	eofPosition
int	foundTaskCount
char[][]	foundTaskMessages
int[][]	foundTaskPositions
char[][]	foundTaskPriorities
char[][]	foundTaskTags
static int	HIGH_SURROGATE_MAX_VALUE
static int	HIGH_SURROGATE_MIN_VALUE
int	initialPosition
boolean	insideRecovery
static java.lang.String	INVALID_CHAR_IN_STRING
static java.lang.String	INVALID_CHARACTER_CONSTANT
static java.lang.String	INVALID_DIGIT
static java.lang.String	INVALID_ESCAPE
static java.lang.String	INVALID_FLOAT
static java.lang.String	INVALID_HEXA
static java.lang.String	INVALID_HIGH_SURROGATE
static java.lang.String	INVALID_INPUT
static java.lang.String	INVALID_LOW_SURROGATE
static java.lang.String	INVALID_OCTAL
static java.lang.String	INVALID_UNICODE_ESCAPE
boolean	isTaskCaseSensitive
int[]	lineEnds
int	linePtr
static int	LOW_SURROGATE_MAX_VALUE
static int	LOW_SURROGATE_MIN_VALUE
static java.lang.String	NULL_SOURCE_STRING
static int	OptimizedLength
boolean	recordLineSeparator
boolean	returnOnlyGreater
static int	RoundBracket
boolean	scanningFloatLiteral
boolean	skipComments
char[]	source
long	sourceLevel
static int	SquareBracket
int	startPosition
static char	TAG_POSTFIX
static int	TAG_POSTFIX_LENGTH
static char[]	TAG_PREFIX
static int	TAG_PREFIX_LENGTH
char[][]	taskPriorities
char[][]	taskTags
boolean	tokenizeComments
boolean	tokenizeWhiteSpace
boolean	unicodeAsBackSlash
static java.lang.String	UNTERMINATED_COMMENT
static java.lang.String	UNTERMINATED_STRING
boolean	useAssertAsAnIndentifier
boolean	useEnumAsAnIndentifier
boolean	wasAcr
char[]	withoutUnicodeBuffer
int	withoutUnicodePtr

Fields inherited from interface org.eclipse.jdt.internal.compiler.parser.TerminalTokens

TokenNameabstract, TokenNamealso, TokenNameAND, TokenNameAND_AND, TokenNameAND_EQUAL, TokenNameassert, TokenNameAssignableOrSynonym, TokenNameAssignableRedundantlyOrSynonym, TokenNameassume, TokenNameassume_redundantly, TokenNameAT, TokenNameaxiom, TokenNameBehaviorOrSynonym, TokenNameboolean, TokenNamebreak, TokenNamebyte, TokenNamecase, TokenNamecatch, TokenNamechar, TokenNameCharacterLiteral, TokenNameclass, TokenNamecode_bigint_math, TokenNamecode_java_math, TokenNamecode_safe_math, TokenNameCOLON, TokenNameCOMMA, TokenNameCOMMENT_BLOCK, TokenNameCOMMENT_JAVADOC, TokenNameCOMMENT_LINE, TokenNameconst, TokenNameconstraint, TokenNameconstraint_redundantly, TokenNamecontinue, TokenNamedecreases, TokenNamedecreases_redundantly, TokenNamedefault, TokenNamediverges, TokenNamediverges_redundantly, TokenNameDIVIDE, TokenNameDIVIDE_EQUAL, TokenNamedo, TokenNameDOT, TokenNameDOTDOT, TokenNamedouble, TokenNameDoubleLiteral, TokenNameELLIPSIS, TokenNameelse, TokenNameEnsuresOrSynonym, TokenNameEnsuresRedundantlyOrSynonym, TokenNameenum, TokenNameEOF, TokenNameEQUAL, TokenNameEQUAL_EQUAL, TokenNameEQUIV, TokenNameERROR, TokenNameextends, TokenNamefalse, TokenNamefinal, TokenNamefinally, TokenNamefloat, TokenNameFloatingPointLiteral, TokenNamefor, TokenNameforall, TokenNameghost, TokenNamegoto, TokenNameGREATER, TokenNameGREATER_EQUAL, TokenNamehelper, TokenNameIdentifier, TokenNameif, TokenNameimplements, TokenNameIMPLIES, TokenNameimplies_that, TokenNameimport, TokenNamein, TokenNamein_redundantly, TokenNameInformalDescription, TokenNameinitially, TokenNameinstance, TokenNameinstanceof, TokenNameint, TokenNameIntegerLiteral, TokenNameinterface, TokenNameinvariant, TokenNameinvariant_redundantly, TokenNamejml_assert, TokenNamejml_assert_redundantly, TokenNameLBRACE, TokenNameLBRACE_OR, TokenNameLBRACKET, TokenNameLEFT_SHIFT, TokenNameLEFT_SHIFT_EQUAL, TokenNameLESS, TokenNameLESS_EQUAL, TokenNamelong, TokenNameLongLiteral, TokenNameloop_invariant, TokenNameloop_invariant_redundantly, TokenNameLPAREN, TokenNamemaps, TokenNamemaps_redundantly, TokenNameMINUS, TokenNameMINUS_EQUAL, TokenNameMINUS_MINUS, TokenNamemodel, TokenNamemono_non_null, TokenNameMULTIPLY, TokenNameMULTIPLY_EQUAL, TokenNamenative, TokenNamenew, TokenNamenon_null, TokenNamenon_null_by_default, TokenNameNOT, TokenNameNOT_EQUAL, TokenNameNOT_EQUIV, TokenNamenowarn, TokenNamenull, TokenNamenullable, TokenNamenullable_by_default, TokenNameold, TokenNameOR, TokenNameOR_EQUAL, TokenNameOR_OR, TokenNameOR_RBRACE, TokenNamepackage, TokenNamepeer, TokenNamePLUS, TokenNamePLUS_EQUAL, TokenNamePLUS_PLUS, TokenNameprivate, TokenNameprotected, TokenNamepublic, TokenNamepure, TokenNameQUESTION, TokenNameRBRACE, TokenNameRBRACKET, TokenNamereadonly, TokenNameREMAINDER, TokenNameREMAINDER_EQUAL, TokenNamerep, TokenNamerepresents, TokenNameREPRESENTS, TokenNamerepresents_redundantly, TokenNameRequiresOrSynonym, TokenNameRequiresRedundantlyOrSynonym, TokenNamereturn, TokenNameREV_IMPLIES, TokenNameRIGHT_SHIFT, TokenNameRIGHT_SHIFT_EQUAL, TokenNameRPAREN, TokenNameSEMICOLON, TokenNameset, TokenNameshort, TokenNamesignals_only, TokenNamesignals_only_redundantly, TokenNameSignalsOrSynonym, TokenNameSignalsRedundantlyOrSynonym, TokenNameslash_elemtype, TokenNameslash_everything, TokenNameslash_exists, TokenNameslash_forall, TokenNameslash_fresh, TokenNameslash_into, TokenNameslash_max, TokenNameslash_min, TokenNameslash_nonnullelements, TokenNameslash_not_assigned, TokenNameslash_not_modified, TokenNameslash_not_specified, TokenNameslash_nothing, TokenNameslash_num_of, TokenNameslash_old, TokenNameslash_peer, TokenNameslash_pre, TokenNameslash_product, TokenNameslash_readonly, TokenNameslash_rep, TokenNameslash_result, TokenNameslash_same, TokenNameslash_sum, TokenNameslash_type, TokenNameslash_typeof, TokenNamespec_bigint_math, TokenNamespec_java_math, TokenNamespec_protected, TokenNamespec_public, TokenNamespec_safe_math, TokenNamestatic, TokenNamestrictfp, TokenNameStringLiteral, TokenNameSUBTYPE, TokenNamesuper, TokenNameswitch, TokenNamesynchronized, TokenNamethis, TokenNamethrow, TokenNamethrows, TokenNametransient, TokenNametrue, TokenNametry, TokenNameTWIDDLE, TokenNameuninitialized, TokenNameUNSIGNED_RIGHT_SHIFT, TokenNameUNSIGNED_RIGHT_SHIFT_EQUAL, TokenNamevoid, TokenNamevolatile, TokenNamewhile, TokenNameWHITESPACE, TokenNameXOR, TokenNameXOR_EQUAL

Constructor Summary

Scanner()

Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive)

Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, long complianceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive)

Scanner(boolean tokenizeComments, boolean tokenizeWhiteSpace, boolean checkNonExternalizedStringLiterals, long sourceLevel, long complianceLevel, char[][] taskTags, char[][] taskPriorities, boolean isTaskCaseSensitive, long jmlLevel)

Method Summary

boolean	atEnd()
void	checkTaskTag(int commentStart, int commentEnd)
char[]	getCurrentIdentifierSource()
java.lang.String	getCurrentStringLiteral()
int	getCurrentTokenEndPosition()
char[]	getCurrentTokenSource()
char[]	getCurrentTokenSourceString()
int	getCurrentTokenStartPosition()
java.lang.String	getCurrentTokenString()
int	getLineEnd(int lineNumber)
int[]	getLineEnds()
int	getLineNumber(int position) Search the line number corresponding to a specific position
int	getLineStart(int lineNumber) Search the source position corresponding to the beginning of a given line number Line numbers are 1-based, and relative to the scanner initialPosition.
int	getNextChar()
boolean	getNextChar(char testedChar)
int	getNextChar(char testedChar1, char testedChar2)
boolean	getNextCharAsDigit()
boolean	getNextCharAsDigit(int radix)
boolean	getNextCharAsJavaIdentifierPart()
boolean	getNextCharAsJavaIdentifierPartWithBoundCheck()
int	getNextCharWithBoundChecks()
int	getNextToken()
void	getNextUnicodeChar()
NLSTag[]	getNLSTags()
char[]	getRawTokenSource()
char[]	getRawTokenSourceEnd()
char[]	getSource()
void	jumpOverMethodBody()
boolean	jumpOverUnicodeWhiteSpace()
void	pushLineSeparator()
void	pushUnicodeLineSeparator()
void	recordComment(int token)
void	resetInExpressionPartOfJmlClause()
void	resetJmlRelatedAttributes(int pos)
void	resetTo(int begin, int end) Reposition the scanner on some portion of the original source.
void	scanEscapeCharacter()
int	scanIdentifier()
int	scanIdentifierOrKeyword()
int	scanIdentifierOrKeywordWithBoundCheck()
int	scanNumber(boolean dotPrefix)
void	setInExpressionPartOfJmlClause()
void	setSource(char[] sourceString)
void	setSource(char[] contents, CompilationResult compilationResult)
void	setSource(CompilationResult compilationResult)
void	startJmlAnnotation(boolean isSingleLine)
java.lang.String	toString()
java.lang.String	toStringAction(int act)
void	unicodeInitializeBuffer(int length)
void	unicodeStore()

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

sourceLevel

public long sourceLevel

complianceLevel

public long complianceLevel

useAssertAsAnIndentifier

public boolean useAssertAsAnIndentifier

containsAssertKeyword

public boolean containsAssertKeyword

useEnumAsAnIndentifier

public boolean useEnumAsAnIndentifier

recordLineSeparator

public boolean recordLineSeparator

currentCharacter

public char currentCharacter

startPosition

public int startPosition

currentPosition

public int currentPosition

initialPosition

public int initialPosition

eofPosition

public int eofPosition

skipComments

public boolean skipComments

tokenizeComments

public boolean tokenizeComments

tokenizeWhiteSpace

public boolean tokenizeWhiteSpace

source

public char[] source

withoutUnicodeBuffer

public char[] withoutUnicodeBuffer

withoutUnicodePtr

public int withoutUnicodePtr

unicodeAsBackSlash

public boolean unicodeAsBackSlash

scanningFloatLiteral

public boolean scanningFloatLiteral

COMMENT_ARRAYS_SIZE

public static final int COMMENT_ARRAYS_SIZE

See Also:

Constant Field Values

commentStops

public int[] commentStops

commentStarts

public int[] commentStarts

commentTagStarts

public int[] commentTagStarts

commentPtr

public int commentPtr

foundTaskTags

public char[][] foundTaskTags

foundTaskMessages

public char[][] foundTaskMessages

foundTaskPriorities

public char[][] foundTaskPriorities

foundTaskPositions

public int[][] foundTaskPositions

foundTaskCount

public int foundTaskCount

taskTags

public char[][] taskTags

taskPriorities

public char[][] taskPriorities

isTaskCaseSensitive

public boolean isTaskCaseSensitive

diet

public boolean diet

lineEnds

public int[] lineEnds

linePtr

public int linePtr

wasAcr

public boolean wasAcr

END_OF_SOURCE

public static final java.lang.String END_OF_SOURCE

See Also:

Constant Field Values

INVALID_HEXA

public static final java.lang.String INVALID_HEXA

See Also:

Constant Field Values

INVALID_OCTAL

public static final java.lang.String INVALID_OCTAL

See Also:

Constant Field Values

INVALID_CHARACTER_CONSTANT

public static final java.lang.String INVALID_CHARACTER_CONSTANT

See Also:

Constant Field Values

INVALID_ESCAPE

public static final java.lang.String INVALID_ESCAPE

See Also:

Constant Field Values

INVALID_INPUT

public static final java.lang.String INVALID_INPUT

See Also:

Constant Field Values

INVALID_UNICODE_ESCAPE

public static final java.lang.String INVALID_UNICODE_ESCAPE

See Also:

Constant Field Values

INVALID_FLOAT

public static final java.lang.String INVALID_FLOAT

See Also:

Constant Field Values

INVALID_LOW_SURROGATE

public static final java.lang.String INVALID_LOW_SURROGATE

See Also:

Constant Field Values

INVALID_HIGH_SURROGATE

public static final java.lang.String INVALID_HIGH_SURROGATE

See Also:

Constant Field Values

NULL_SOURCE_STRING

public static final java.lang.String NULL_SOURCE_STRING

See Also:

Constant Field Values

UNTERMINATED_STRING

public static final java.lang.String UNTERMINATED_STRING

See Also:

Constant Field Values

UNTERMINATED_COMMENT

public static final java.lang.String UNTERMINATED_COMMENT

See Also:

Constant Field Values

INVALID_CHAR_IN_STRING

public static final java.lang.String INVALID_CHAR_IN_STRING

See Also:

Constant Field Values

INVALID_DIGIT

public static final java.lang.String INVALID_DIGIT

See Also:

Constant Field Values

OptimizedLength

public static final int OptimizedLength

See Also:

Constant Field Values

charArray_length

public final char[][][][] charArray_length

TAG_PREFIX

public static final char[] TAG_PREFIX

TAG_PREFIX_LENGTH

public static final int TAG_PREFIX_LENGTH

TAG_POSTFIX

public static final char TAG_POSTFIX

See Also:

Constant Field Values

TAG_POSTFIX_LENGTH

public static final int TAG_POSTFIX_LENGTH

See Also:

Constant Field Values

checkNonExternalizedStringLiterals

public boolean checkNonExternalizedStringLiterals

returnOnlyGreater

public boolean returnOnlyGreater

insideRecovery

public boolean insideRecovery

RoundBracket

public static final int RoundBracket

See Also:

Constant Field Values

SquareBracket

public static final int SquareBracket

See Also:

Constant Field Values

CurlyBracket

public static final int CurlyBracket

See Also:

Constant Field Values

BracketKinds

public static final int BracketKinds

See Also:

Constant Field Values

LOW_SURROGATE_MIN_VALUE

public static final int LOW_SURROGATE_MIN_VALUE

See Also:

Constant Field Values

HIGH_SURROGATE_MIN_VALUE

public static final int HIGH_SURROGATE_MIN_VALUE

See Also:

Constant Field Values

HIGH_SURROGATE_MAX_VALUE

public static final int HIGH_SURROGATE_MAX_VALUE

See Also:

Constant Field Values

LOW_SURROGATE_MAX_VALUE

public static final int LOW_SURROGATE_MAX_VALUE

See Also:

Constant Field Values

Constructor Detail

Scanner

public Scanner()

Scanner

public Scanner(boolean tokenizeComments,
               boolean tokenizeWhiteSpace,
               boolean checkNonExternalizedStringLiterals,
               long sourceLevel,
               long complianceLevel,
               char[][] taskTags,
               char[][] taskPriorities,
               boolean isTaskCaseSensitive)

Scanner

public Scanner(boolean tokenizeComments,
               boolean tokenizeWhiteSpace,
               boolean checkNonExternalizedStringLiterals,
               long sourceLevel,
               char[][] taskTags,
               char[][] taskPriorities,
               boolean isTaskCaseSensitive)

Scanner

public Scanner(boolean tokenizeComments,
               boolean tokenizeWhiteSpace,
               boolean checkNonExternalizedStringLiterals,
               long sourceLevel,
               long complianceLevel,
               char[][] taskTags,
               char[][] taskPriorities,
               boolean isTaskCaseSensitive,
               long jmlLevel)

Method Detail

atEnd

public final boolean atEnd()

checkTaskTag

public void checkTaskTag(int commentStart,
                         int commentEnd)
                  throws InvalidInputException

Throws:

InvalidInputException

getCurrentIdentifierSource

public char[] getCurrentIdentifierSource()

getCurrentTokenEndPosition

public int getCurrentTokenEndPosition()

getCurrentTokenSource

public char[] getCurrentTokenSource()

getCurrentTokenString

public final java.lang.String getCurrentTokenString()

getCurrentTokenSourceString

public char[] getCurrentTokenSourceString()

getCurrentStringLiteral

public final java.lang.String getCurrentStringLiteral()

getRawTokenSource

public final char[] getRawTokenSource()

getRawTokenSourceEnd

public final char[] getRawTokenSourceEnd()

getCurrentTokenStartPosition

public int getCurrentTokenStartPosition()

getLineEnd

public final int getLineEnd(int lineNumber)

getLineEnds

public final int[] getLineEnds()

getLineStart

public final int getLineStart(int lineNumber)

Search the source position corresponding to the beginning of a given line number Line numbers are 1-based, and relative to the scanner initialPosition. Character positions are 0-based. e.g. getLineStart(1) --> 0 indicates that the first line starts at character 0. In case the given line number is inconsistent, answers -1.

Parameters:

lineNumber - int

Returns:

int

getNextChar

public final int getNextChar()

getNextCharWithBoundChecks

public final int getNextCharWithBoundChecks()

getNextChar

public final boolean getNextChar(char testedChar)

getNextChar

public final int getNextChar(char testedChar1,
                             char testedChar2)

getNextCharAsDigit

public final boolean getNextCharAsDigit()
                                 throws InvalidInputException

Throws:

InvalidInputException

getNextCharAsDigit

public final boolean getNextCharAsDigit(int radix)

getNextCharAsJavaIdentifierPartWithBoundCheck

public boolean getNextCharAsJavaIdentifierPartWithBoundCheck()

getNextCharAsJavaIdentifierPart

public boolean getNextCharAsJavaIdentifierPart()

scanIdentifier

public int scanIdentifier()
                   throws InvalidInputException

Throws:

InvalidInputException

getNextToken

public int getNextToken()
                 throws InvalidInputException

Throws:

InvalidInputException

getNextUnicodeChar

public void getNextUnicodeChar()
                        throws InvalidInputException

Throws:

InvalidInputException

getNLSTags

public NLSTag[] getNLSTags()

getSource

public char[] getSource()

jumpOverMethodBody

public final void jumpOverMethodBody()

jumpOverUnicodeWhiteSpace

public final boolean jumpOverUnicodeWhiteSpace()
                                        throws InvalidInputException

Throws:

InvalidInputException

pushLineSeparator

public final void pushLineSeparator()

pushUnicodeLineSeparator

public final void pushUnicodeLineSeparator()

recordComment

public void recordComment(int token)

resetTo

public void resetTo(int begin,
                    int end)

Reposition the scanner on some portion of the original source. The given endPosition is the last valid position. Beyond this position, the scanner will answer EOF tokens (ITerminalSymbols.TokenNameEOF).

Parameters:

begin - the given start position

end - the given end position

scanEscapeCharacter

public final void scanEscapeCharacter()
                               throws InvalidInputException

Throws:

InvalidInputException

scanIdentifierOrKeywordWithBoundCheck

public int scanIdentifierOrKeywordWithBoundCheck()

scanIdentifierOrKeyword

public int scanIdentifierOrKeyword()

scanNumber

public int scanNumber(boolean dotPrefix)
               throws InvalidInputException

Throws:

InvalidInputException

getLineNumber

public final int getLineNumber(int position)

Search the line number corresponding to a specific position

Parameters:

position - int

Returns:

int

setSource

public final void setSource(char[] sourceString)

setSource

public final void setSource(char[] contents,
                            CompilationResult compilationResult)

setSource

public final void setSource(CompilationResult compilationResult)

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object

toStringAction

public java.lang.String toStringAction(int act)

unicodeInitializeBuffer

public void unicodeInitializeBuffer(int length)

unicodeStore

public void unicodeStore()

startJmlAnnotation

public void startJmlAnnotation(boolean isSingleLine)

setInExpressionPartOfJmlClause

public void setInExpressionPartOfJmlClause()

resetInExpressionPartOfJmlClause

public void resetInExpressionPartOfJmlClause()

resetJmlRelatedAttributes

public void resetJmlRelatedAttributes(int pos)