G3D::TextInput Class Reference

A simple tokenizer for parsing text files. More...

#include <TextInput.h>

Classes
class	BadMSVCSpecial
class	Settings
class	TokenException
class	WrongString
class	WrongSymbol
class	WrongTokenType

Public Types
enum	FS { FROM_STRING }

Public Member Functions
	TextInput (const std::string &filename, const Settings &settings=Settings())
	TextInput (FS fs, const std::string &str, const Settings &settings=Settings())
	TextInput (FS fs, const char *str, size_t strLen, const Settings &settings=Settings())
bool	hasMore ()
void	pushSettings (const Settings &settings)
void	popSettings ()
Token	read ()
void	read (Token &t)
Token	readSignificant ()
double	readNumber ()
int	readInteger ()
bool	readBoolean ()
Token	readStringToken ()
std::string	readString ()
void	readString (const std::string &s)
std::string	readUntilNewlineAsString ()
std::string	readUntilDelimiterAsString (const char delimiter1, const char delimiter2= '\0')
Token	readCommentToken ()
std::string	readComment ()
void	readComment (const std::string &s)
Token	readNewlineToken ()
std::string	readNewline ()
void	readNewline (const std::string &s)
Token	readSymbolToken ()
void	readSymbolToken (Token &t)
std::string	readSymbol ()
void	readSymbol (const std::string &symbol)
void	readSymbols (const std::string &s1, const std::string &s2)
void	readSymbols (const std::string &s1, const std::string &s2, const std::string &s3)
void	readSymbols (const std::string &s1, const std::string &s2, const std::string &s3, const std::string &s4)
Token	peek ()
int	peekLineNumber ()
int	peekCharacterNumber ()
void	push (const Token &t)
const std::string &	filename () const

Static Public Member Functions
static double	parseNumber (const std::string &_string)
static bool	parseBoolean (const std::string &_string)

Private Member Functions
void	init ()
int	eatInputChar ()
int	peekInputChar (int distance=0)
int	eatAndPeekInputChar ()
void	nextToken (Token &t)
void	parseQuotedString (unsigned char delimiter, Token &t)
void	initFromString (const char *str, int len, const Settings &settings)

Private Attributes
Array< Settings >	settingsStack
std::deque< Token >	stack
Array< char >	buffer
int	currentCharOffset
int	lineNumber
int	charNumber
Settings	options

Detailed Description

A simple tokenizer for parsing text files.

TextInput handles a superset of C++,Java, Matlab, and Bash code text including single line comments, block comments, quoted strings with escape sequences, and operators. TextInput recognizes several categories of tokens, which are separated by white space, quotation marks, or the end of a recognized operator:

• Token::SINGLE_QUOTED_TYPE string of characters surrounded by single quotes, e.g., 'x', '\0', 'foo'.
• Token::DOUBLE_QUOTED_TYPE string of characters surrounded by double quotes, e.g., "x", "abc\txyz", "b o b".
• Token::SYMBOL_TYPE legal C++ operators, keywords, and identifiers. e.g., >=, Foo, _X, class, {
• Token::INTEGER_TYPE numbers without decimal places or exponential notation. e.g., 10, 0x17F, 32, 0, -155
• Token::FLOATING_POINT_TYPE numbers with decimal places or exponential notation. e.g., 1e3, -1.2, .4, 0.5
• Token::BOOLEAN_TYPE special symbols like "true" and "false"; the exact details can be configured in TextInput::Settings
• Token::LINE_COMMENT_TYPE (disabled by default); generated for line comments as specified by TextInput::Settings
• Token::BLOCK_COMMENT_TYPE (disabled by default); generated for c-style block comments as specified by TextInput::Settings
• Token::NEWLINE_TYPE (disabled by default); generated for any of "\\r", "\\n" or "\\r\\n"

The special ".." and "..." tokens are always recognized in addition to normal C++ operators. Additional tokens can be made available by changing the Settings.

Negative numbers are handled specially because of the ambiguity between unary minus and negative numbers– see the note on TextInput::read.

TextInput does not have helper functions for types with non-obvious formatting, or helpers that would be redundant. Use the serialize methods instead for parsing specific types like int, Vector3, and Color3.

Inside quoted strings escape sequences are converted. Thus the string token for ["a\\nb"] is 'a', followed by a newline, followed by 'b'. Outside of quoted strings, escape sequences are not converted, so the token sequence for [a\nb] is symbol 'a', symbol '\', symbol 'nb' (this matches what a C++ parser would do). The exception is that a specified TextInput::Settings::otherCommentCharacter preceeded by a backslash is assumed to be an escaped comment character and is returned as a symbol token instead of being parsed as a comment (this is what a LaTex or VRML parser would do).

Examples

TextInput ti(TextInput::FROM_STRING, "name = \"Max\", height = 6");

Token t;

t = ti.read(); 
debugAssert(t.type == Token::SYMBOL);
debugAssert(t.sval == "name");

ti.read();
debugAssert(t.type == Token::SYMBOL);
debugAssert(t.sval == "=");

std::string name = ti.read().sval;
ti.read();

TextInput ti(TextInput::FROM_STRING, "name = \"Max\", height = 6");
ti.readSymbols("name", "=");
std::string name = ti.readString();
ti.readSymbols(",", "height", "=");
double height = ti. readNumber();

Assumes that the file is not modified once opened.

Member Enumeration Documentation

enum G3D::TextInput::FS

Enumerator
FROM_STRING

{FROM_STRING};

Constructor & Destructor Documentation

G3D::TextInput::TextInput	(	const std::string &	filename,
		const Settings &	settings = Settings()
	)

 : options(opt) {
 init();
 if (options.sourceFileName.empty()) {
 options.sourceFileName = filename;
 }

 std::string zipfile;
 if (FileSystem::inZipfile(filename, zipfile)) {
 // TODO: this could be faster if we directly read the zipfile
 const std::string& input = readWholeFile(filename);
 size_t n = input.size();
 buffer.resize(n);
 System::memcpy(buffer.getCArray(), input.c_str(), n);
 } else {
 // Read directly into the array
 const uint64 n = FileSystem::size(filename);
 alwaysAssertM(n != uint64(-1), std::string("File does not exist: ") + filename);
 buffer.resize(size_t(n));
 FILE* f = FileSystem::fopen(filename.c_str(), "rb");
 fread(buffer.getCArray(), 1, size_t(n), f);
 FileSystem::fclose(f);
 }
}

Here is the call graph for this function:

G3D::TextInput::TextInput	(	FS	fs,
		const std::string &	str,
		const Settings &	settings = Settings()
	)

Creates input directly from a string. The first argument must be TextInput::FROM_STRING.

 {
 (void)fs;
 initFromString(str.c_str(), (int)str.size(), opt);
}

Here is the call graph for this function:

G3D::TextInput::TextInput	(	FS	fs,
		const char *	str,
		size_t	strLen,
		const Settings &	settings = Settings()
	)

Creates input directly from a fixed-length, non-NULL terminated string. The first argument must be TextInput::FROM_STRING.

 : options(opt) {
 (void)fs;
 initFromString(str, (int)len, opt);
}

Here is the call graph for this function:

Member Function Documentation

int G3D::TextInput::eatAndPeekInputChar ( )

inlineprivate

Helper function to consume the next character in the input buffer and peek at the one following (without consuming it).

 {
 eatInputChar();
 return peekInputChar(0);
 }

Here is the call graph for this function:

Here is the caller graph for this function:

int G3D::TextInput::eatInputChar ( )

private

Consumes the next character from the input buffer, and returns that character. Updates lineNumber and charNumber to reflect the location of the next character in the input buffer.

Note: you shouldn't be using the return value of this function in most cases. In general, you should peekInputChar() to get the next character, determine what to do with it, then consume it with this function (or with eatAndPeekInputChar()). Given that usage, in most instances you already know what this function would return!

 {
 // Don't go off the end
 if (currentCharOffset >= buffer.length()) {
 return EOF;
 }

 unsigned char c = buffer[currentCharOffset];
 ++currentCharOffset;

 // update lineNumber and charNumber to reflect the location of the *next*
 // character which will be read.

 // increment line number for \r, \n and \r\n which matches Token::NEWLINE parsing
 if (c == '\r') {
 ++lineNumber;
 charNumber = 1;

 // check for \r\n
 if (currentCharOffset < buffer.length()) {
 unsigned char c2 = buffer[currentCharOffset];
 if (c2 == '\n') {
 c = c2;
 ++currentCharOffset;
 }
 }
 } else if (c == '\n') {
 ++lineNumber;
 charNumber = 1;
 } else {
 ++charNumber;
 }

 return c;
}

Here is the call graph for this function:

Here is the caller graph for this function:

const std::string & G3D::TextInput::filename

(

)

const

Returns the filename from which this input is drawn, or the first few characters of the string if created from a string. If settings::filename is non-empty that will replace the true filename.

 {
 return options.sourceFileName;
}

Here is the caller graph for this function:

bool G3D::TextInput::hasMore

(

)

Returns true while there are tokens remaining.

 {
 return (peek()._type != Token::END);
}

Here is the call graph for this function:

Here is the caller graph for this function:

void G3D::TextInput::init ( )

private

 {
 currentCharOffset = 0;
 charNumber = 1;
 lineNumber = 1 + options.startingLineNumberOffset;

 if (! options.caseSensitive) {
 // Convert true and false symbols to all uppercase for fast comparisons
 toUpper(options.trueSymbols);
 toUpper(options.falseSymbols);
 }
}

Here is the call graph for this function:

Here is the caller graph for this function:

void G3D::TextInput::initFromString ( const char *  str, int  len, const Settings &  settings  )

private

 {
 options = settings;
 init();
 if (options.sourceFileName.empty()) {
 if (len < 14) {
 options.sourceFileName = format("\"%.*s\"", len, str);
 } else {
 options.sourceFileName = format("\"%.*s...\"", 10, str);
 }
 }
 buffer.resize(len);
 System::memcpy(buffer.getCArray(), str, buffer.size());
}

Here is the call graph for this function:

Here is the caller graph for this function:

void G3D::TextInput::nextToken ( Token &  t )

private

Read the next token, returning an END token if no more input is available.

 {

 t._bytePosition = currentCharOffset;
 t._line = lineNumber;
 t._character = charNumber;
 t._type = Token::END;
 t._extendedType = Token::END_TYPE;

 int c = peekInputChar();
 if (c == EOF) {
 return;
 }

 // loop through white space, newlines and comments
 // found before other tokens
 bool whitespaceDone = false;
 while (! whitespaceDone) {
 whitespaceDone = true;

 // generate newlines tokens for '\n' and '\r' and '\r\n'
 while (isWhiteSpace(c)) {
 if (options.generateNewlineTokens && isNewline(c)) {
 t._type = Token::NEWLINE;
 t._extendedType = Token::NEWLINE_TYPE;
 t._bytePosition = currentCharOffset;
 t._line = lineNumber;
 t._character = charNumber;
 t._string = c;

 int c2 = peekInputChar(1);
 if (c == '\r' && c2 == '\n') {
 t._string += c2;
 }

 eatInputChar();
 return;
 } else {
 // Consume the single whitespace
 c = eatAndPeekInputChar();
 }
 }

 // update line and character number to include discarded whitespace
 t._line = lineNumber;
 t._character = charNumber;
 t._bytePosition = currentCharOffset;

 if (isDigit(c)) {
 // This is an unsigned number. Jump ahead for fast number reading.
 goto numLabel;
 }

 int c2 = peekInputChar(1);

 if ((c == '-') && isDigit(c2) && options.signedNumbers) {
 // This is a simple number. Jump ahead for fast number reading.
 // We treat this case specially because large (i.e., slow) files
 // are usually large because they are full of numbers.
 t._string = "-";
 c = c2;
 // Consume the minus sign
 eatInputChar();
 goto numLabel;
 }

 // parse comments and generate tokens if enabled
 std::string commentString;

 // check for line comments first
 bool isLineComment = false;
 if (options.cppLineComments && (c == '/' && c2 == '/')) {
 // set start of line comment and eat markers
 isLineComment = true;
 eatInputChar();
 eatInputChar();
 } else if ( options.otherCommentCharacter && 
 (options.otherCommentCharacter != '\0' && c == options.otherCommentCharacter) ) {
 // set start of line comment and eat markers
 isLineComment = true;
 eatInputChar();
 } else if ( options.otherCommentCharacter && 
 (options.otherCommentCharacter2 != '\0' && c == options.otherCommentCharacter2) ) {
 // set start of line comment and eat markers
 isLineComment = true;
 eatInputChar();
 }

 if (isLineComment) {

 // consume line comment to newline or EOF
 c = peekInputChar();
 while (! isNewline(c) && c != EOF) {
 // build comment string for token
 commentString += c;

 c = eatAndPeekInputChar();
 }

 if (options.generateCommentTokens) {
 t._type = Token::COMMENT;
 t._extendedType = Token::LINE_COMMENT_TYPE;
 t._string = commentString;
 return;
 } else {
 // There is whitespace after the comment (in particular, the
 // newline that terminates the comment). There might also be
 // whitespace at the start of the next line.
 whitespaceDone = false;
 }

 } else if (options.cppBlockComments && (c == '/' && c2 == '*')) {
 // consume block comment to end-marker or EOF

 // consume both start-comment chars, can't let the trailing one
 // help close the comment.
 eatInputChar();
 eatInputChar();

 // c is the next character we'll read, c2 is the one after *that*
 c = peekInputChar();
 c2 = peekInputChar(1);
 while (! ((c == '*') && (c2 == '/')) && (c != EOF)) {
 commentString += c;

 // Eat input char may consume more than one character if there is a newline
 eatInputChar();

 c = peekInputChar();
 c2 = peekInputChar(1);
 }
 eatInputChar(); // eat closing '*'
 eatInputChar(); // eat closing '/'

 c = peekInputChar();

 if (options.generateCommentTokens) {
 t._type = Token::COMMENT;
 t._extendedType = Token::BLOCK_COMMENT_TYPE;
 t._string = commentString;
 return;
 } else {
 // There is whitespace after the comment (in particular, the
 // newline that terminates the comment). There might also be
 // whitespace at the start of the next line.
 whitespaceDone = false;
 }
 }

 } // while (! whitespaceDone)

 t._line = lineNumber;
 t._character = charNumber;
 t._bytePosition = currentCharOffset;

 // handle EOF
 if (c == EOF) {
 return;
 }

 // Extended ASCII parses as itself, except for EOF
 if (c > 127 && c < 255) {
 t._type = Token::SYMBOL; 
 t._extendedType = Token::SYMBOL_TYPE; 
 t._string = c; 
 c = eatAndPeekInputChar(); 
 }


 // Perform appropriate setup for a symbol (including setting up the token
 // string to start with c), eat the input character, and overwrite
 // 'c' with the peeked next input character.
#define SETUP_SYMBOL(c) \
 { \
 t._type = Token::SYMBOL; \
 t._extendedType = Token::SYMBOL_TYPE; \
 t._string = c; \
 c = eatAndPeekInputChar(); \
 }

 switch (c) {

 case '@': // Simple symbols -> just themselves.
 case '(': 
 case ')':
 case ',':
 case ';':
 case '{':
 case '}':
 case '[':
 case ']':
 case '#':
 case '$':
 case '?':
 case '%':
 SETUP_SYMBOL(c);
 return;

 case '-': // negative number, -, --, -=, or ->
 SETUP_SYMBOL(c);

 switch (c) {
 case '>': // ->
 case '-': // --
 case '=': // -=
 t._string += c;
 eatInputChar();
 return;
 }

 if (options.signedNumbers) {
 if (isDigit(c) || (c == '.' && isDigit(peekInputChar(1)))) {
 // Negative number. 'c' is still the first digit, and is
 // the next input char.

 goto numLabel;
 } else {
 char terminal = peekInputChar(3);
 if (options.simpleFloatSpecials && (c == 'i') && (peekInputChar(1) == 'n') && (peekInputChar(2) == 'f') && 
 ! isLetter(terminal) && (terminal != '_')) {
 // negative infinity
 t._type = Token::NUMBER;
 t._extendedType = Token::FLOATING_POINT_TYPE;
 t._string = "-inf";
 eatInputChar(); // i
 eatInputChar(); // n
 eatInputChar(); // f
 return;
 }
 }
 }


 // plain -
 return;

 case '+': // positive number, +, ++, or +=
 SETUP_SYMBOL(c);

 switch (c) {
 case '+': // ++
 case '=': // +=
 t._string += c;
 eatInputChar();
 return;
 }

 if (options.signedNumbers) {
 if (isDigit(c) || (c == '.' && isDigit(peekInputChar(1)))) {
 // Positive number. 'c' is still the first digit, and is
 // the next input char.

 goto numLabel;
 } else {
 char terminal = peekInputChar(3);
 if (options.simpleFloatSpecials && (c == 'i') && (peekInputChar(1) == 'n') && (peekInputChar(2) == 'f') && 
 ! isLetter(terminal) && (terminal != '_')) {
 // positive infinity
 t._type = Token::NUMBER;
 t._extendedType = Token::FLOATING_POINT_TYPE;
 t._string = "+inf";
 eatInputChar(); // i
 eatInputChar(); // n
 eatInputChar(); // f
 return;
 }
 }
 }

 return;

 case ':': // : or :: or ::> or ::= or := or :>
 SETUP_SYMBOL(c);
 
 if (c == ':') {
 t._string += c;
 eatInputChar();

 if (options.proofSymbols) {
 c = peekInputChar(0);
 
 if ((c == '>') || (c == '=')) {
 t._string += c;
 eatInputChar();
 }
 }
 } else if (options.proofSymbols && (c == '=' || c == '>')) {
 t._string += c;
 eatInputChar();
 }
 return;

 case '=': // = or == or =>
 SETUP_SYMBOL(c);

 if (c == '=') {
 t._string += c;
 eatInputChar();
 } else if (options.proofSymbols && (c == '>')) {
 t._string += c;
 eatInputChar();
 }
 return;

 case '*': // * or *=
 case '/': // / or /=
 case '!': // ! or !=
 case '~': // ~ or ~=
 case '^': // ^ or ^=
 SETUP_SYMBOL(c);
 
 if (c == '=') {
 t._string += c;
 eatInputChar();
 }
 return;

 case '>': // >, >>,or >=
 case '<': // <<, <<, or <= or <- or <:
 case '|': // ||, ||, or |= or |-
 case '&': // &, &&, or &=
 {
 int orig_c = c;
 SETUP_SYMBOL(c);

 if ((c == '=') || (orig_c == c)) {
 t._string += c;
 eatInputChar();
 } else if (options.proofSymbols) {
 if ((orig_c == '<') && (c == '-')) {
 t._string += c;
 eatInputChar();
 } else if ((orig_c == '|') && (c == '-')) {
 t._string += c;
 eatInputChar();
 } else if ((orig_c == '<') && (c == ':')) {
 t._string += c;

 c = eatAndPeekInputChar();

 if (c == ':') {
 t._string += c;
 eatInputChar();
 }
 }
 }
 }
 return;
 
 case '\\': // backslash or escaped comment char.
 SETUP_SYMBOL(c);

 if ((options.otherCommentCharacter != '\0'
 && c == options.otherCommentCharacter)
 || (options.otherCommentCharacter2 != '\0'
 && c == options.otherCommentCharacter2)) {
 
 // escaped comment character. Return the raw comment
 // char (no backslash).

 t._string = c;
 eatInputChar();
 return;
 }
 return;

 case '.': // number, ., .., or ...
 if (isDigit(peekInputChar(1))) {
 // We're parsing a float that began without a leading zero
 goto numLabel;
 }

 SETUP_SYMBOL(c);

 if (c == '.') { // .. or ...
 t._string += c;
 c = eatAndPeekInputChar();

 if (c == '.') { // ...
 t._string += c;
 eatInputChar();
 }
 return;
 }

 return;

 } // switch (c)

#undef SETUP_SYMBOL

numLabel:
 if (isDigit(c) || (c == '.')) {

 // A number. Note-- single dots have been
 // parsed already, so a . indicates a number
 // less than 1 in floating point form.
 
 // [0-9]*(\.[0-9][f]) or [0-9]+ or 0x[0-9,A-F]+

 if (t._string != "-") {
 // If we picked up a leading "-" sign above, keep it,
 // otherwise drop the string parsed thus far
 t._string = "";
 }
 t._type = Token::NUMBER;
 if (c == '.') {
 t._extendedType = Token::FLOATING_POINT_TYPE;
 } else {
 t._extendedType = Token::INTEGER_TYPE;
 }

 if ((c == '0') && (peekInputChar(1) == 'x')) {
 // Hex number
 t._string += "0x";

 // skip the 0x
 eatInputChar();
 eatInputChar();

 c = peekInputChar();
 while (isDigit(c) || ((c >= 'A') && (c <= 'F')) || ((c >= 'a') && (c <= 'f'))) {
 t._string += c;
 c = eatAndPeekInputChar();
 }

 } else {
 // Non-hex number

 // Read the part before the decimal.
 while (isDigit(c)) {
 t._string += c;
 c = eatAndPeekInputChar();
 }
 
 // True if we are reading a floating-point special type
 bool isSpecial = false;

 // Read the decimal, if one exists
 if (c == '.') {
 t._extendedType = Token::FLOATING_POINT_TYPE;

 // The '.' character was a decimal point, not the start of a
 // method or range operator
 t._string += c;
 c = eatAndPeekInputChar();

 // Floating point specials (msvc format only)
 if (options.msvcFloatSpecials && (c == '#')) {
 isSpecial = true;
 // We are reading a floating point special value
 // of the form -1.#IND00, -1.#INF00, or 1.#INF00
 // (with or without the trailing 00
 c = eatAndPeekInputChar();
 char test = c;
 if (! options.caseSensitive) {
 test = toupper(c);
 }
 if (test != 'I') {
 throw BadMSVCSpecial
 ("Incorrect floating-point special (inf or nan) "
 "format.",
 t.line(), charNumber);
 }
 c = eatAndPeekInputChar();
 test = c;
 if (! options.caseSensitive) {
 test = toupper(c);
 }
 if (test != 'N') {
 throw BadMSVCSpecial
 (
 "Incorrect floating-point special (inf or nan) "
 "format.",
 t.line(), charNumber);
 }
 t._string += "#IN";
 c = eatAndPeekInputChar();
 test = c;
 if (! options.caseSensitive) {
 test = toupper(c);
 }
 if ((test != 'F') && (test != 'D')) {
 throw BadMSVCSpecial
 (
 "Incorrect floating-point special (inf or nan) "
 "format.",
 t.line(), charNumber);
 }
 t._string += c;

 // On older systems, there may be an extra 00 tacked on.
 for (int j = 0; j < 2; ++j) {
 c = eatAndPeekInputChar();
 if (c == '0') {
 c = eatAndPeekInputChar();
 if (c != '0') {
 throw BadMSVCSpecial
 (
 "Incorrect floating-point special (inf or nan) "
 "format.",
 t.line(), charNumber);
 } else {
 eatInputChar();
 t._string += "00";
 }
 } else {
 break;
 }
 }

 } else {

 // Read the part after the decimal
 while (isDigit((char)c)) {
 t._string += (char)c;
 c = eatAndPeekInputChar();
 }
 }
 }

 if (! isSpecial && ((c == 'e') || (c == 'E'))) {
 // Read exponent
 t._extendedType = Token::FLOATING_POINT_TYPE;
 t._string += c;

 c = eatAndPeekInputChar();
 if ((c == '-') || (c == '+')) {
 t._string += c;
 c = eatAndPeekInputChar(); 
 }

 while (isDigit(c)) {
 t._string += c;
 c = eatAndPeekInputChar();
 }
 }

 if (! isSpecial && (t._extendedType == Token::FLOATING_POINT_TYPE) && (c == 'f')) {
 // Trailing f on a float
 t._string += c;
 c = eatAndPeekInputChar();
 }
 }
 return;

 } else if (isLetter(c) || (c == '_')) {
 // Identifier or keyword
 // [A-Za-z_][A-Za-z_0-9]*

 t._type = Token::SYMBOL;
 t._extendedType = Token::SYMBOL_TYPE;
 t._string = "";
 do {
 t._string += c;
 c = eatAndPeekInputChar();
 } while (isLetter(c) || isDigit(c) || (c == '_'));

 // See if this symbol is actually a boolean
 if ((options.trueSymbols.size() > 0) || (options.falseSymbols.size() > 0)) {
 std::string str = t._string;
 if (! options.caseSensitive) {
 str = toUpper(str);
 }
 if (options.trueSymbols.contains(str)) {
 t._type = Token::BOOLEAN;
 t._extendedType = Token::BOOLEAN_TYPE;
 t._bool = true;
 } else if (options.falseSymbols.contains(str)) {
 t._type = Token::BOOLEAN;
 t._extendedType = Token::BOOLEAN_TYPE;
 t._bool = false;
 }
 }

 if (options.simpleFloatSpecials && ((t._string == "nan") || (t._string == "inf"))) {
 t._type = Token::NUMBER;
 t._extendedType = Token::FLOATING_POINT_TYPE;
 }
 return;

 } else if (c == '\"') {

 // Discard the double-quote.
 eatInputChar();

 // Double quoted string
 parseQuotedString('\"', t);
 return;

 } else if (c == options.singleQuoteCharacter) {

 // Discard the single-quote.
 eatInputChar();

 if (options.singleQuotedStrings) {
 // Single quoted string
 parseQuotedString(options.singleQuoteCharacter, t);
 } else {
 t._string = c;
 t._type = Token::SYMBOL;
 t._extendedType = Token::SYMBOL_TYPE;
 }
 return;

 } // end of special case tokens

 if ((c == EOF) || (c == '\0')) {
 t._type = Token::END;
 t._extendedType = Token::END_TYPE;
 t._string = "";
 return;
 }

 // Some unknown token
 throw format("Unrecognized token type beginning with character '%c' (ASCII %d)", c, c);
 return;
}

Here is the call graph for this function:

Here is the caller graph for this function:

bool G3D::TextInput::parseBoolean ( const std::string &  _string )

static

toLower(_string) == "true"

 {
 return toLower(_string) == "true";
}

Here is the call graph for this function:

Here is the caller graph for this function:

double G3D::TextInput::parseNumber ( const std::string &  _string )

static

Includes MSVC specials parsing

 {
 std::string s = toLower(_string);
 if (s == "-1.#ind00" || s == "-1.#ind" || s == "nan" || s == "NaN") {
 return nan();
 }
 
 if (s == "1.#inf00" || s == "1.#inf" || s == "inf" || s == "+inf" || s == "Infinity") {
 return inf();
 }
 
 if (s == "-1.#inf00" || s == "-1.#inf" || s == "-inf" || s == "-Infinity") {
 return -inf();
 }
 
 double n;
 if ((_string.length() > 2) &&
 (_string[0] == '0') &&
 (_string[1] == 'x')) {
 // Hex
 uint32 i;
 sscanf(_string.c_str(), "%x", &i);
 n = i;
 } else {
 sscanf(_string.c_str(), "%lg", &n);
 }

 return n;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void G3D::TextInput::parseQuotedString ( unsigned char  delimiter, Token &  t  )

private

Helper for nextToken. Appends characters to t._string until the end delimiter is reached.

When called, the next character in the input buffer should be first the first character after the opening delimiter character.

 {

 t._type = Token::STRING;

 if (delimiter == options.singleQuoteCharacter) {
 t._extendedType = Token::SINGLE_QUOTED_TYPE;
 } else {
 t._extendedType = Token::DOUBLE_QUOTED_TYPE;
 }

 while (true) {
 // We're definitely going to consume the next input char, so we get
 // it right now. This makes the condition handling below a bit easier.
 int c = eatInputChar();

 if (c == EOF) {
 // END inside a quoted string. (We finish the string.)
 break;
 }

 if (options.escapeSequencesInStrings && (c == '\\')) {
 // An escaped character. We're definitely going to consume it,
 // so we get it (and consume it) now.

 c = eatInputChar();

 switch (c) {
 case 'r':
 t._string += '\r';
 break;
 case 'n':
 t._string += '\n';
 break;
 case 't':
 t._string += '\t';
 break;
 case '0':
 t._string += '\0';
 break;

 case '\\':
 case '\"':
 t._string += (char)c;
 break;

 default:
 if (c == options.singleQuoteCharacter) {
 t._string += (char)c;
 break;
 }

 if (((c == options.otherCommentCharacter) && 
 (options.otherCommentCharacter != '\0')) ||
 ((c == options.otherCommentCharacter2) && 
 (options.otherCommentCharacter2 != '\0'))) {
 t._string += c;
 } 
 // otherwise, some illegal escape sequence; skip it.
 break;

 } // switch

 } else if (c == delimiter) {
 // End of the string. Already consumed the character.
 break;
 } else {
 // All other chars, go on to the string. Already consumed the
 // character.
 t._string += (char)c;
 }

 }
}

Here is the call graph for this function:

Here is the caller graph for this function:

Token G3D::TextInput::peek

(

)

Return a copy of the next token in the input stream, but don't remove it from the input stream.

 {
 if (stack.size() == 0) {
 Token t;
 nextToken(t);
 push(t);
 }

 return stack.front();
}

Here is the call graph for this function:

Here is the caller graph for this function:

int G3D::TextInput::peekCharacterNumber

(

)

Returns the character number (relative to the line) for the next token in the input stream. See also peek.

 {
 return peek().character();
}

Here is the call graph for this function:

int G3D::TextInput::peekInputChar ( int  distance = 0 )

private

Returns the next character from the input buffer, without consuming any characters. Can also be used to look deeper into the input buffer. Does not modify lineNumber or charNumber.

Parameters

distance

Index of the character in the input buffer to peek at, relative to the next character. Default is 0, for the next character in the input buffer.

 {
 // Don't go off the end
 if ((currentCharOffset + distance) >= buffer.length()) {
 return EOF;
 }

 unsigned char c = buffer[currentCharOffset + distance];
 return c;
}

Here is the call graph for this function:

Here is the caller graph for this function:

int G3D::TextInput::peekLineNumber

(

)

Returns the line number for the next token. See also peek.

 {
 return peek().line();
}

Here is the call graph for this function:

void G3D::TextInput::popSettings ( )

inline

 {
 options = settingsStack.pop();
 }

Here is the call graph for this function:

void G3D::TextInput::push

(

const Token &

t

)

Take a previously read token and push it back at the front of the input stream.

Can be used in the case where more than one token of read-ahead is needed (i.e., when peek doesn't suffice).

 {
 stack.push_front(t);
}

Here is the caller graph for this function:

void G3D::TextInput::pushSettings ( const Settings &  settings )

inline

Temporarily switch parsing to use settings. Note that this will override the currently recorded sourceFilename unless you explicitly set it back.

See also

popSettings

 {
 settingsStack.push(options);
 options = settings;
 }

Here is the call graph for this function:

Token G3D::TextInput::read

(

)

Read the next token (which will be the END token if ! hasMore()).

Signed numbers can be handled in one of two modes. If the option TextInput::Settings::signedNumbers is true, A '+' or '-' immediately before a number is prepended onto that number and if there is intervening whitespace, it is read as a separate symbol.

If TextInput::Settings::signedNumbers is false, read() does not distinguish between a plus or minus symbol next to a number and a positive/negative number itself. For example, "x - 1" and "x -1" will be parsed the same way by read().

In both cases, readNumber() will contract a leading "-" or "+" onto a number.

 {
 Token t;
 read(t);
 return t;
}

Here is the caller graph for this function:

void G3D::TextInput::read

(

Token &

t

)

Avoids the copy of read()

                             {
    if (stack.size() > 0) {
        t = stack.front();
        stack.pop_front();
    } else {
        nextToken(t);
    }
}

Here is the call graph for this function:

bool G3D::TextInput::readBoolean

(

)

                            {
    const Token& t = read();

    if (t._type == Token::BOOLEAN) {
        return t.boolean();
    }

    // Push initial token back, and throw an error.  We intentionally
    // indicate that the wrong type is the type of the initial token.
    // Logically, the number started there.
    push(t);
    throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                         Token::BOOLEAN, t._type); 
}

Here is the call graph for this function:

std::string G3D::TextInput::readComment

(

)

Like readCommentToken, but returns the token's string.

Use this method (rather than readCommentToken) if you want the token's value but don't really care about its location in the input. Use of readCommentToken is encouraged for better error reporting.

                                 {
    return readCommentToken()._string;
}

Here is the call graph for this function:

void G3D::TextInput::readComment

(

const std::string &

s

)

Reads a specific comment token or throws either WrongTokenType or WrongString. If the next token in the input is a comment matching s, it will be consumed.

Use this method if you want to match a specific comment from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a comment. WrongString will be thrown if the next token in the input stream is a comment but does not match the s parameter. When an exception is thrown, no tokens are consumed.

                                              {
    const Token& t = readCommentToken();

    if (t._string == s) {                         // fast path
        return;
    }

    push(t);
    throw WrongString(options.sourceFileName, t.line(), t.character(),
                      s, t._string);
}

Here is the call graph for this function:

Token G3D::TextInput::readCommentToken

(

)

Reads a comment token or throws WrongTokenType, and returns the token.

Use this method (rather than readComment) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a comment. When an exception is thrown, no tokens are consumed.

                                  {
    Token t;
    read(t);

    if (t._type == Token::COMMENT) {               // fast path
        return t;
    }

    push(t);
    throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                         Token::COMMENT, t._type);
}

Here is the call graph for this function:

Here is the caller graph for this function:

int G3D::TextInput::readInteger

(

)

Reads a number that must be in C integer format: [ '+' | '-' ] #+ | '0x'#+

                           {
    Token t;
    read(t);

    if (t._extendedType == Token::INTEGER_TYPE) {  // common case
        return int(t.number());
    } else {
        // Even if signedNumbers is disabled, readInteger attempts to
        // read a signed number, so we handle that case here.
        if (! options.signedNumbers
            && (t._type == Token::SYMBOL)
            && ((t._string == "-") 
                 || (t._string == "+"))) {

            Token t2;
            read(t2);

            if ((t2._extendedType == Token::INTEGER_TYPE)
                && (t2._character == t._character + 1)) {

                if (t._string == "-") {
                    return (int)-t2.number();
                } else {
                    return (int)t2.number();
                }
            }

            // push back the second token.
            push(t2);
        }

        // Push initial token back, and throw an error.  We intentionally
        // indicate that the wrong type is the type of the initial token.
        // Logically, the number started there.
        push(t);
        throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                             Token::NUMBER, t._type); 
    }    
}

Here is the call graph for this function:

std::string G3D::TextInput::readNewline

(

)

Like readNewlineToken, but returns the token's string.

Use this method (rather than readNewlineToken) if you want the token's value but don't really care about its location in the input. Use of readNewlineToken is encouraged for better error reporting.

                                 {
    return readNewlineToken()._string;
}

Here is the call graph for this function:

void G3D::TextInput::readNewline

(

const std::string &

s

)

Reads a specific newline token or throws either WrongTokenType or WrongString. If the next token in the input is a newline matching s, it will be consumed.

Use this method if you want to match a specific newline from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a newline. WrongString will be thrown if the next token in the input stream is a newlin but does not match the s parameter. When an exception is thrown, no tokens are consumed.

                                              {
    const Token& t = readNewlineToken();

    if (t._string == s) {                         // fast path
        return;
    }

    push(t);
    throw WrongString(options.sourceFileName, t.line(), t.character(),
                      s, t._string);
}

Here is the call graph for this function:

Token G3D::TextInput::readNewlineToken

(

)

Reads a newline token or throws WrongTokenType, and returns the token.

Use this method (rather than readNewline) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a newline. When an exception is thrown, no tokens are consumed.

                                  {
    Token t;
    read(t);

    if (t._type == Token::NEWLINE) {               // fast path
        return t;
    }

    push(t);
    throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                         Token::NEWLINE, t._type);
}

Here is the call graph for this function:

Here is the caller graph for this function:

double G3D::TextInput::readNumber

(

)

Read one token (or possibly two, for minus sign) as a number or throws WrongTokenType, and returns the number.

If the first token in the input is a number, it is returned directly.

If TextInput::Settings::signedNumbers is false and the input stream contains a '+' or '-' symbol token immediately followed by a number token, both tokens will be consumed and a single token will be returned by this method.

WrongTokenType will be thrown if one of the input conditions described above is not satisfied. When an exception is thrown, no tokens are consumed.

                             {
    Token t;
    read(t);

    if (t._type == Token::NUMBER) {  // common case
        return t.number();
    }

    // Even if signedNumbers is disabled, readNumber attempts to
    // read a signed number, so we handle that case here.
    if (! options.signedNumbers
        && (t._type == Token::SYMBOL)
        && ((t._string == "-") 
             || (t._string == "+"))) {

        Token t2(read());

        if ((t2._type == Token::NUMBER)
            && (t2._character == t._character + 1)) {

            if (t._string == "-") {
                return -t2.number();
            } else {
                return t2.number();
            }
        }

        // push back the second token.
        push(t2);
    }

    // Push initial token back, and throw an error.  We intentionally
    // indicate that the wrong type is the type of the initial token.
    // Logically, the number started there.
    push(t);
    throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                         Token::NUMBER, t._type); 
}

Here is the call graph for this function:

Here is the caller graph for this function:

Token G3D::TextInput::readSignificant

(

)

Calls read() until the result is not a newline or comment

                                 {
    Token t;
    do { 
        t = read();
    } while ((t.type() == Token::COMMENT) || (t.type() == Token::NEWLINE));
    return t;
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::string G3D::TextInput::readString

(

)

Like readStringToken, but returns the token's string.

Use this method (rather than readStringToken) if you want the token's value but don't really care about its location in the input. Use of readStringToken is encouraged for better error reporting.

                                {
    return readStringToken()._string;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void G3D::TextInput::readString

(

const std::string &

s

)

Reads a specific string token or throws either WrongTokenType or WrongString. If the next token in the input is a string matching s, it will be consumed.

Use this method if you want to match a specific string from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a string. WrongString will be thrown if the next token in the input stream is a string but does not match the s parameter. When an exception is thrown, no tokens are consumed.

See also

readString(), readStringToken(), readUntilNewlineAsString(), readUntilDelimiterAsString()

                                             {
    const Token& t = readStringToken();

    if (t._string == s) {                         // fast path
        return;
    }

    push(t);
    throw WrongString(options.sourceFileName, t.line(), t.character(),
                      s, t._string);
}

Here is the call graph for this function:

Token G3D::TextInput::readStringToken

(

)

Reads a string token or throws WrongTokenType, and returns the token.

Use this method (rather than readString) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a string. When an exception is thrown, no tokens are consumed.

                                 {
    Token t;
    read(t);

    if (t._type == Token::STRING) {               // fast path
        return t;
    }

    push(t);
    throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                         Token::STRING, t._type);
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::string G3D::TextInput::readSymbol

(

)

Like readSymbolToken, but returns the token's string.

Use this method (rather than readSymbolToken) if you want the token's value but don't really care about its location in the input. Use of readSymbolToken is encouraged for better error reporting.

                                {
    return readSymbolToken()._string;
}

Here is the call graph for this function:

Here is the caller graph for this function:

void G3D::TextInput::readSymbol

(

const std::string &

symbol

)

Reads a specific symbol token or throws either WrongTokenType or WrongSymbol. If the next token in the input is a symbol matching symbol, it will be consumed.

Use this method if you want to match a specific symbol from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a symbol. WrongSymbol will be thrown if the next token in the input stream is a symbol but does not match the symbol parameter. When an exception is thrown, no tokens are consumed.

                                                  {
    Token t;
    readSymbolToken(t);

    if (t._string == symbol) { // fast path
        return;
    }

    push(t);
    throw WrongSymbol(options.sourceFileName, t.line(), t.character(),
                      symbol, t._string);
}

Here is the call graph for this function:

void G3D::TextInput::readSymbols ( const std::string &  s1, const std::string &  s2  )

inline

Read a series of two specific symbols. See readSymbol.

                                                               {
        readSymbol(s1);
        readSymbol(s2);
    }

Here is the call graph for this function:

void G3D::TextInput::readSymbols ( const std::string &  s1, const std::string &  s2, const std::string &  s3  )

inline

Read a series of three specific symbols. See readSymbol.

                             {
        readSymbol(s1);
        readSymbol(s2);
        readSymbol(s3);
    }

Here is the call graph for this function:

void G3D::TextInput::readSymbols ( const std::string &  s1, const std::string &  s2, const std::string &  s3, const std::string &  s4  )

inline

Read a series of four specific symbols. See readSymbol.

                             {
        readSymbol(s1);
        readSymbol(s2);
        readSymbol(s3);
        readSymbol(s4);
    }

Here is the call graph for this function:

Token G3D::TextInput::readSymbolToken

(

)

Reads a symbol token or throws WrongTokenType, and returns the token.

Use this method (rather than readSymbol) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a symbol. When an exception is thrown, no tokens are consumed.

                                 {
    Token t;
    readSymbolToken(t);
    return t;
}

Here is the caller graph for this function:

void G3D::TextInput::readSymbolToken

(

Token &

t

)

Avoids the copy of readSymbolToken()

                                        {
    read(t);
    
    if (t._type == Token::SYMBOL) {               // fast path
        return;
    }

    push(t);
    throw WrongTokenType(options.sourceFileName, t.line(), t.character(),
                         Token::SYMBOL, t._type);
}

Here is the call graph for this function:

std::string G3D::TextInput::readUntilDelimiterAsString	(	const char	delimiter1,
		const char	delimiter2 = '\0'
	)

Read from the beginning of the next token until the following delimiter character and return the result as a string, ignoring all parsing in between. The delimiter is not returned in the string, and the following token read will begin at the delimiter or end of file token (if they are enabled for parsing).

                                                                                            {
/*
    // Reset the read position back to the start of that token
    currentCharOffset = t.bytePosition();
    lineNumber = t.line();
    charNumber = t.character();
    stack.clear();

    if (currentCharOffset == buffer.size()) {
        // End of file
        return "";
    }
    */
    std::string s;

    if (stack.size() > 0) {
        // Need to back up.  This only works if the stack is actually
        // in proper order reflecting the real file, and doesn't
        // contain incorrectly pushed elements.
        Token t = stack.back();
        stack.clear();
        currentCharOffset = (int)t.bytePosition();
        lineNumber = t.line();
        charNumber = t.character();
    }

    // Read until delimiter or eof
    while (currentCharOffset < buffer.size()) {
        const char c = buffer[currentCharOffset];
        if ((c == delimiter1) || (c == delimiter2)) {
            // Done
            break;
        } else {
            s += c;
            ++currentCharOffset;
            ++charNumber;
        }
    }

    return s;    
}

Here is the call graph for this function:

Here is the caller graph for this function:

std::string G3D::TextInput::readUntilNewlineAsString

(

)

Read from the beginning of the next token until the following newline and return the result as a string, ignoring all parsing in between. The newline is not returned in the string, and the following token read will be a newline or end of file token (if they are enabled for parsing).

                                              {
    return readUntilDelimiterAsString('\r', '\n');
}

Here is the call graph for this function:

Member Data Documentation

Array<char> G3D::TextInput::buffer

private

Characters to be tokenized.

int G3D::TextInput::charNumber

private

Character number (within the line) of the next character to be consumed from the input buffer. (1 indicates first character of the line).

Note that this is the character number of the next character to be consumed from the input, not the character number of the last character consumed!

int G3D::TextInput::currentCharOffset

private

Offset of current character (the next character to consumed) in input buffer.

int G3D::TextInput::lineNumber

private

Line number of next character to be consumed from the input buffer. (1 indicates first line of input.)

Note that this is the line number of the next character to be consumed from the input, not the line number of the last character consumed!

Settings G3D::TextInput::options

private

Configuration options. This includes the file name that will be reported in tokens and exceptions.

Array<Settings> G3D::TextInput::settingsStack

private

See also

pushSettings / popSettings

std::deque<Token> G3D::TextInput::stack

private

---

The documentation for this class was generated from the following files:

• dep/g3dlite/include/G3D/TextInput.h
• dep/g3dlite/source/TextInput.cpp