Token

This struct represents a token, a fundamental building block of all scripts. The code of a script is first separated by token so that the parser can analyze each token.

struct Token {

enum Type;

LiteralFlag literalFlag;

string toString();

string symbol();

bool isKeyword(string keyword);

bool isIdentifier(string id);

bool isAssignmentOperator();

static Token createInvalidToken(Position pos, string text);

static Token createFakeToken(Type t, string txt);

}

Members

Enums

LiteralFlag enum LiteralFlag: This enum is for literal value tokens that require special handling by the parser
Type enum Type: The type of a token.

Functions

isAssignmentOperator bool isAssignmentOperator(): Returns true if the token is an assignment operator such as =, +=, or -=, etc.
isIdentifier bool isIdentifier(string id): Checks for a specific identifier
isKeyword bool isKeyword(string keyword): Returns true if a token is both a keyword and a specific keyword.
symbol string symbol(): Returns a textual representation of the token as it was found in the original script source code.
toString string toString(): Returns a string representing the type of the token and the optional text if present.

Static functions

createFakeToken Token createFakeToken(Type t, string txt): Used by the parser and compiler
createInvalidToken Token createInvalidToken(Position pos, string text): Generates an invalid token at the given position. This is used by the Lexer to throw an exception that requires a token.

Variables

literalFlag LiteralFlag literalFlag;: Optional flag for integer literals.
position Position position;: Position where token occurs
text string text;: Optional text for keywords and identifiers
type Type type;: Type of token

Token

Members

Enums

Functions

Static functions

Variables

Meta

Source