#include <Lexer.h>

Inheritance diagram for clang::Lexer:

Collaboration diagram for clang::Lexer:

Public Member Functions
	Lexer (FileID FID, const llvm::MemoryBuffer *InputBuffer, Preprocessor &PP)
	Lexer (SourceLocation FileLoc, const LangOptions &LangOpts, const char BufStart, const char BufPtr, const char *BufEnd)
	Lexer (FileID FID, const llvm::MemoryBuffer *InputBuffer, const SourceManager &SM, const LangOptions &LangOpts)
const LangOptions &	getLangOpts () const
SourceLocation	getFileLoc () const
bool	isPragmaLexer () const
	isPragmaLexer - Returns true if this Lexer is being used to lex a pragma.
bool	LexFromRawLexer (Token &Result)
bool	isKeepWhitespaceMode () const
void	SetKeepWhitespaceMode (bool Val)
bool	inKeepCommentMode () const
void	SetCommentRetentionState (bool Mode)
void	resetExtendedTokenMode ()
StringRef	getBuffer () const
	Gets source code buffer.
void	ReadToEndOfLine (SmallVectorImpl< char > *Result=nullptr)
DiagnosticBuilder	Diag (const char *Loc, unsigned DiagID) const
SourceLocation	getSourceLocation (const char *Loc, unsigned TokLen=1) const
SourceLocation	getSourceLocation () override
const char *	getBufferLocation () const
	Return the current location in the buffer.
Static Public Member Functions
static Lexer *	Create_PragmaLexer (SourceLocation SpellingLoc, SourceLocation ExpansionLocStart, SourceLocation ExpansionLocEnd, unsigned TokLen, Preprocessor &PP)
static std::string	Stringify (const std::string &Str, bool Charify=false)
static void	Stringify (SmallVectorImpl< char > &Str)
static unsigned	getSpelling (const Token &Tok, const char &Buffer, const SourceManager &SourceMgr, const LangOptions &LangOpts, bool Invalid=nullptr)
static std::string	getSpelling (const Token &Tok, const SourceManager &SourceMgr, const LangOptions &LangOpts, bool *Invalid=nullptr)
static StringRef	getSpelling (SourceLocation loc, SmallVectorImpl< char > &buffer, const SourceManager &SourceMgr, const LangOptions &LangOpts, bool *invalid=nullptr)
static unsigned	MeasureTokenLength (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
static bool	getRawToken (SourceLocation Loc, Token &Result, const SourceManager &SM, const LangOptions &LangOpts, bool IgnoreWhiteSpace=false)
	Relex the token at the specified location.
static SourceLocation	GetBeginningOfToken (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
	Given a location any where in a source buffer, find the location that corresponds to the beginning of the token in which the original source location lands.
static SourceLocation	AdvanceToTokenCharacter (SourceLocation TokStart, unsigned Character, const SourceManager &SM, const LangOptions &LangOpts)
static SourceLocation	getLocForEndOfToken (SourceLocation Loc, unsigned Offset, const SourceManager &SM, const LangOptions &LangOpts)
	Computes the source location just past the end of the token at this source location.
static bool	isAtStartOfMacroExpansion (SourceLocation loc, const SourceManager &SM, const LangOptions &LangOpts, SourceLocation *MacroBegin=nullptr)
	Returns true if the given MacroID location points at the first token of the macro expansion.
static bool	isAtEndOfMacroExpansion (SourceLocation loc, const SourceManager &SM, const LangOptions &LangOpts, SourceLocation *MacroEnd=nullptr)
	Returns true if the given MacroID location points at the last token of the macro expansion.
static CharSourceRange	makeFileCharRange (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts)
	Accepts a range and returns a character range with file locations.
static StringRef	getSourceText (CharSourceRange Range, const SourceManager &SM, const LangOptions &LangOpts, bool *Invalid=nullptr)
	Returns a string for the source that the range encompasses.
static StringRef	getImmediateMacroName (SourceLocation Loc, const SourceManager &SM, const LangOptions &LangOpts)
	Retrieve the name of the immediate macro expansion.
static std::pair< unsigned, bool >	ComputePreamble (StringRef Buffer, const LangOptions &LangOpts, unsigned MaxLines=0)
	Compute the preamble of the given file.
static SourceLocation	findLocationAfterToken (SourceLocation loc, tok::TokenKind TKind, const SourceManager &SM, const LangOptions &LangOpts, bool SkipTrailingWhitespaceAndNewLine)
	Checks that the given token is the first token that occurs after the given location (this excludes comments and whitespace). Returns the location immediately after the specified token. If the token is not found or the location is inside a macro, the returned source location will be invalid.
static bool	isIdentifierBodyChar (char c, const LangOptions &LangOpts)
	Returns true if the given character could appear in an identifier.
static char	getCharAndSizeNoWarn (const char *Ptr, unsigned &Size, const LangOptions &LangOpts)
Friends
class	Preprocessor

Detailed Description

Lexer - This provides a simple interface that turns a text buffer into a stream of tokens. This provides no support for file reading or buffering, or buffering/seeking of tokens, only forward lexing is supported. It relies on the specified Preprocessor object to handle preprocessor directives, etc.

Definition at line 46 of file Lexer.h.

Constructor & Destructor Documentation

Lexer::Lexer	(	FileID	FID,
		const llvm::MemoryBuffer *	InputFile,
		Preprocessor &	PP
	)

Lexer constructor - Create a new lexer object for the specified buffer with the specified preprocessor managing the lexing process. This lexer assumes that the associated file buffer and Preprocessor objects will outlive it, so it doesn't take ownership of either of them.

Definition at line 109 of file Lexer.cpp.

References resetExtendedTokenMode().

Lexer::Lexer	(	SourceLocation	fileloc,
		const LangOptions &	langOpts,
		const char *	BufStart,
		const char *	BufPtr,
		const char *	BufEnd
	)

Lexer constructor - Create a new raw lexer object. This object is only suitable for calls to 'LexFromRawLexer'. This lexer assumes that the text range will outlive it, so it doesn't take ownership of it.

Definition at line 131 of file Lexer.cpp.

References clang::PreprocessorLexer::LexingRawMode.

Lexer::Lexer	(	FileID	FID,
		const llvm::MemoryBuffer *	FromFile,
		const SourceManager &	SM,
		const LangOptions &	langOpts
	)

Lexer constructor - Create a new raw lexer object. This object is only suitable for calls to 'LexFromRawLexer'. This lexer assumes that the text range will outlive it, so it doesn't take ownership of it.

Definition at line 144 of file Lexer.cpp.

References clang::PreprocessorLexer::LexingRawMode.

Member Function Documentation

SourceLocation Lexer::AdvanceToTokenCharacter	(	SourceLocation	TokStart,
		unsigned	CharNo,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		`[static]`

AdvanceToTokenCharacter - If the current SourceLocation specifies a location at the start of a token, return a new location that specifies a character within the token. This handles trigraphs and escaped newlines.

AdvanceToTokenCharacter - Given a location that specifies the start of a token, return a new location that specifies a character within the token.

Definition at line 706 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), getCharAndSizeNoWarn(), and clang::SourceLocation::getLocWithOffset().

Referenced by Diag(), clang::StringLiteral::getLocationOfByte(), getUDSuffixLoc(), and MakeCharSourceRange().

std::pair< unsigned, bool > Lexer::ComputePreamble	(	StringRef	Buffer,
		const LangOptions &	LangOpts,
		unsigned	MaxLines = `0`
	)		`[static]`

Compute the preamble of the given file.

The preamble of a file contains the initial comments, include directives, and other preprocessor directives that occur before the code in this particular file actually begins. The preamble of the main source file is a potential prefix header.

Parameters:

Buffer	The memory buffer containing the file's contents.
MaxLines	If non-zero, restrict the length of the preamble to fewer than this number of lines.

Returns:: The offset into the file where the preamble ends and the rest of the file begins along with a boolean value indicating whether the preamble ends at the beginning of a new line.

Definition at line 543 of file Lexer.cpp.

References clang::comments::tok::eof, clang::SourceLocation::getFromRawEncoding(), clang::Token::getKind(), clang::Token::getLocation(), clang::SourceLocation::getRawEncoding(), clang::Token::getRawIdentifier(), clang::Token::isAtStartOfLine(), clang::SourceLocation::isInvalid(), clang::SourceLocation::isValid(), clang::Token::needsCleaning(), and SetCommentRetentionState().

Referenced by clang::PrintPreambleAction::ExecuteAction().

Lexer * Lexer::Create_PragmaLexer	(	SourceLocation	SpellingLoc,
		SourceLocation	ExpansionLocStart,
		SourceLocation	ExpansionLocEnd,
		unsigned	TokLen,
		Preprocessor &	PP
	)		`[static]`

Create_PragmaLexer: Lexer constructor - Create a new lexer object for _Pragma expansion. This has a variety of magic semantics that this method sets up. It returns a new'd Lexer that must be delete'd when done.

On entrance to this routine, TokStartLoc is a macro location which has a spelling loc that indicates the bytes to be lexed for the token and an expansion location that indicates where all lexed tokens should be "expanded from".

TODO: It would really be nice to make _Pragma just be a wrapper around a normal lexer that remaps tokens as they fly by. This would require making Preprocessor::Lex virtual. Given that, we could just dump in a magic lexer interface that could handle this stuff. This would pull GetMappedTokenLoc out of the critical path of the lexer!

Definition at line 170 of file Lexer.cpp.

References clang::SourceManager::createExpansionLoc(), clang::SourceManager::getBuffer(), clang::SourceManager::getCharacterData(), clang::SourceManager::getFileID(), clang::SourceManager::getLocForStartOfFile(), clang::Preprocessor::getSourceManager(), clang::PreprocessorLexer::ParsingPreprocessorDirective, and SM.

DiagnosticBuilder Lexer::Diag	(	const char *	Loc,
		unsigned	DiagID
	)		const

Diag - Forwarding function for diagnostics. This translate a source position in the current buffer into a SourceLocation object for rendering.

Definition at line 1068 of file Lexer.cpp.

References clang::Preprocessor::Diag(), getSourceLocation(), and clang::PreprocessorLexer::PP.

Referenced by DecodeTrigraphChar(), and isEndOfBlockCommentWithEscapedNewLine().

SourceLocation Lexer::findLocationAfterToken	(	SourceLocation	loc,
		tok::TokenKind	TKind,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		bool	SkipTrailingWhitespaceAndNewLine
	)		`[static]`

Checks that the given token is the first token that occurs after the given location (this excludes comments and whitespace). Returns the location immediately after the specified token. If the token is not found or the location is inside a macro, the returned source location will be invalid.

Definition at line 1162 of file Lexer.cpp.

References AttributeLangSupport::C, clang::SourceManager::getBufferData(), clang::SourceManager::getCharacterData(), clang::SourceManager::getDecomposedLoc(), getLocForEndOfToken(), clang::SourceManager::getLocForStartOfFile(), clang::SourceLocation::getLocWithOffset(), isAtEndOfMacroExpansion(), clang::isHorizontalWhitespace(), and clang::SourceLocation::isMacroID().

Referenced by GenerateFixForUnusedDecl().

SourceLocation Lexer::GetBeginningOfToken	(	SourceLocation	Loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		`[static]`

Given a location any where in a source buffer, find the location that corresponds to the beginning of the token in which the original source location lands.

Definition at line 515 of file Lexer.cpp.

References getBeginningOfFileToken(), clang::SourceManager::getDecomposedLoc(), clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), and clang::SourceManager::isMacroArgExpansion().

Referenced by adjustRemoval().

StringRef clang::Lexer::getBuffer ( ) const [inline]

Gets source code buffer.

Definition at line 202 of file Lexer.h.

Referenced by clang::html::SyntaxHighlight().

const char* clang::Lexer::getBufferLocation ( ) const [inline]

Return the current location in the buffer.

Definition at line 226 of file Lexer.h.

static char clang::Lexer::getCharAndSizeNoWarn	(	const char *	Ptr,
		unsigned &	Size,
		const LangOptions &	LangOpts
	)		`[inline, static]`

getCharAndSizeNoWarn - Like the getCharAndSize method, but does not ever emit a warning.

Definition at line 428 of file Lexer.h.

Referenced by AdvanceToTokenCharacter(), and getSpellingSlow().

SourceLocation clang::Lexer::getFileLoc ( ) const [inline]

getFileLoc - Return the File Location for the file we are lexing out of. The physical location encodes the location where the characters come from, the virtual location encodes where we should *claim* the characters came from. Currently this is only used by _Pragma handling.

Definition at line 134 of file Lexer.h.

StringRef Lexer::getImmediateMacroName	(	SourceLocation	Loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		`[static]`

Retrieve the name of the immediate macro expansion.

This routine starts from a source location, and finds the name of the macro responsible for its immediate expansion. It looks through any intervening macro argument expansions to compute this. It returns a StringRef which refers to the SourceManager-owned buffer of the source where that macro name is spelled. Thus, the result shouldn't out-live that SourceManager.

Definition at line 962 of file Lexer.cpp.

References clang::PreprocessorLexer::FID, clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SrcMgr::ExpansionInfo::getExpansionLocStart(), clang::SourceManager::getFileID(), clang::SourceManager::getImmediateExpansionRange(), clang::SourceManager::getSLocEntry(), clang::SrcMgr::ExpansionInfo::getSpellingLoc(), clang::SourceManager::getSpellingLoc(), clang::SourceLocation::isFileID(), clang::SourceManager::isInFileID(), clang::SrcMgr::ExpansionInfo::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), and MeasureTokenLength().

Referenced by clang::ento::CheckerContext::getMacroNameOrSpelling().

const LangOptions& clang::Lexer::getLangOpts ( ) const [inline]

getLangOpts - Return the language features currently enabled. NOTE: this lexer modifies features as a file is parsed!

Definition at line 128 of file Lexer.h.

Referenced by DecodeTrigraphChar(), and isEndOfBlockCommentWithEscapedNewLine().

SourceLocation Lexer::getLocForEndOfToken	(	SourceLocation	Loc,
		unsigned	Offset,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		`[static]`

Computes the source location just past the end of the token at this source location.

This routine can be used to produce a source location that points just past the end of the token referenced by Loc, and is generally used when a diagnostic needs to point just after a token where it expected something different that it received. If the returned source location would not be meaningful (e.g., if it points into a macro), this routine returns an invalid source location.

Parameters:

Offset an offset from the end of the token, where the source location should refer to. The default offset (0) produces a source location pointing just past the end of the token; an offset of 1 produces a source location pointing to the last character in the token, etc.

Definition at line 765 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), isAtEndOfMacroExpansion(), clang::SourceLocation::isInvalid(), clang::SourceLocation::isMacroID(), and MeasureTokenLength().

Referenced by clang::Sema::CheckLogicalOperands(), CreateIfFixit(), findLocationAfterToken(), clang::arcmt::trans::findSemiAfterLocation(), clang::Sema::getLocForEndOfToken(), and makeRangeFromFileLocs().

bool Lexer::getRawToken	(	SourceLocation	Loc,
		Token &	Result,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		bool	IgnoreWhiteSpace = `false`
	)		`[static]`

Relex the token at the specified location.

Returns:: true if there was a failure, false on success.

Definition at line 423 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::SourceManager::getExpansionLoc(), clang::SourceManager::getLocForStartOfFile(), clang::isWhitespace(), and SetCommentRetentionState().

Referenced by MeasureTokenLength().

SourceLocation Lexer::getSourceLocation	(	const char *	Loc,
		unsigned	TokLen = `1`
	)		const

getSourceLocation - Return a source location identifier for the specified offset in the current file.

Definition at line 1049 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), GetMappedTokenLoc(), clang::SourceLocation::isFileID(), and clang::PreprocessorLexer::PP.

Referenced by makeCharRange(), and clang::format::reformat().

SourceLocation clang::Lexer::getSourceLocation ( ) [inline, override, virtual]

getSourceLocation - Return a source location for the next character in the current file.

Implements clang::PreprocessorLexer.

Definition at line 221 of file Lexer.h.

Referenced by Diag().

StringRef Lexer::getSourceText	(	CharSourceRange	Range,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		bool *	Invalid = `nullptr`
	)		`[static]`

Returns a string for the source that the range encompasses.

Definition at line 926 of file Lexer.cpp.

References clang::CharSourceRange::getBegin(), clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::CharSourceRange::getEnd(), clang::SourceManager::isInFileID(), clang::CharSourceRange::isInvalid(), and makeFileCharRange().

Referenced by GCRewriteFinalize(), getLiteralInfo(), and clang::tooling::replaceStmtWithStmt().

unsigned Lexer::getSpelling	(	const Token &	Tok,
		const char *&	Buffer,
		const SourceManager &	SourceMgr,
		const LangOptions &	LangOpts,
		bool *	Invalid = `nullptr`
	)		`[static]`

getSpelling - This method is used to get the spelling of a token into a preallocated buffer, instead of as an std::string. The caller is required to allocate enough space for the token, which is guaranteed to be at least Tok.getLength() bytes long. The length of the actual result is returned.

Note that this method may do two possible things: it may either fill in the buffer specified with characters, or it may *change the input pointer* to point to a constant buffer with the data already in it (avoiding a copy). The caller is not allowed to modify the returned buffer pointer if an internal buffer is returned.

getSpelling - This method is used to get the spelling of a token into a preallocated buffer, instead of as an std::string. The caller is required to allocate enough space for the token, which is guaranteed to be at least Tok.getLength() bytes long. The actual length of the token is returned.

Note that this method may do two possible things: it may either fill in the buffer specified with characters, or it may *change the input pointer* to point to a constant buffer with the data already in it (avoiding a copy). The caller is not allowed to modify the returned buffer pointer if an internal buffer is returned.

Definition at line 364 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), clang::Token::getIdentifierInfo(), clang::Token::getLength(), clang::Token::getLiteralData(), clang::Token::getLocation(), clang::Token::getRawIdentifier(), getSpellingSlow(), clang::Token::hasUCN(), clang::Token::is(), clang::Token::isLiteral(), and clang::Token::needsCleaning().

Referenced by findDirectives(), clang::ento::CheckerContext::getMacroNameOrSpelling(), and clang::StringLiteralParser::getOffsetOfStringByte().

std::string Lexer::getSpelling	(	const Token &	Tok,
		const SourceManager &	SourceMgr,
		const LangOptions &	LangOpts,
		bool *	Invalid = `nullptr`
	)		`[static]`

getSpelling() - Return the 'spelling' of the Tok token. The spelling of a token is the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

getSpelling() - Return the 'spelling' of this token. The spelling of a token are the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

Definition at line 332 of file Lexer.cpp.

References clang::SourceManager::getCharacterData(), clang::Token::getLength(), clang::Token::getLocation(), getSpellingSlow(), and clang::Token::needsCleaning().

StringRef Lexer::getSpelling	(	SourceLocation	loc,
		SmallVectorImpl< char > &	buffer,
		const SourceManager &	SM,
		const LangOptions &	options,
		bool *	invalid = `nullptr`
	)		`[static]`

getSpelling - This method is used to get the spelling of the token at the given source location. If, as is usually true, it is not necessary to copy any data, then the returned string may not point into the provided buffer.

This method lexes at the expansion depth of the given location and does not jump to the expansion or spelling location.

getSpelling() - Return the 'spelling' of this token. The spelling of a token are the characters used to represent the token in the source file after trigraph expansion and escaped-newline folding. In particular, this wants to get the true, uncanonicalized, spelling of things like digraphs UCNs, etc.

Definition at line 291 of file Lexer.cpp.

References clang::SourceManager::getBufferData(), clang::SourceManager::getDecomposedLoc(), clang::Token::getLength(), clang::SourceManager::getLocForStartOfFile(), and getSpellingSlow().

bool clang::Lexer::inKeepCommentMode ( ) const [inline]

inKeepCommentMode - Return true if the lexer should return comments as tokens.

Definition at line 180 of file Lexer.h.

bool Lexer::isAtEndOfMacroExpansion	(	SourceLocation	loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		SourceLocation *	MacroEnd = `nullptr`
	)		`[static]`

Returns true if the given MacroID location points at the last token of the macro expansion.

Parameters:

MacroEnd If non-null and function returns true, it is set to end location of the macro.

Definition at line 809 of file Lexer.cpp.

References clang::SourceLocation::getLocWithOffset(), clang::SourceManager::getSpellingLoc(), clang::SourceManager::isAtEndOfImmediateMacroExpansion(), clang::SourceLocation::isFileID(), clang::SourceLocation::isMacroID(), clang::SourceLocation::isValid(), and MeasureTokenLength().

Referenced by findLocationAfterToken(), clang::arcmt::trans::findSemiAfterLocation(), getLocForEndOfToken(), and makeFileCharRange().

bool Lexer::isAtStartOfMacroExpansion	(	SourceLocation	loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts,
		SourceLocation *	MacroBegin = `nullptr`
	)		`[static]`

Returns true if the given MacroID location points at the first token of the macro expansion.

Parameters:

MacroBegin If non-null and function returns true, it is set to begin location of the macro.

Definition at line 787 of file Lexer.cpp.

References clang::SourceManager::isAtStartOfImmediateMacroExpansion(), clang::SourceLocation::isMacroID(), and clang::SourceLocation::isValid().

Referenced by makeFileCharRange().

bool Lexer::isIdentifierBodyChar	(	char	c,
		const LangOptions &	LangOpts
	)		`[static]`

Returns true if the given character could appear in an identifier.

Definition at line 1009 of file Lexer.cpp.

References clang::isIdentifierBody().

Referenced by addFixitForObjCARCConversion(), and canBeJoined().

bool clang::Lexer::isKeepWhitespaceMode ( ) const [inline]

isKeepWhitespaceMode - Return true if the lexer should return tokens for every character in the file, including whitespace and comments. This should only be used in raw mode, as the preprocessor is not prepared to deal with the excess tokens.

Definition at line 166 of file Lexer.h.

Referenced by SetCommentRetentionState().

bool clang::Lexer::isPragmaLexer ( ) const [inline]

isPragmaLexer - Returns true if this Lexer is being used to lex a pragma.

Definition at line 143 of file Lexer.h.

bool clang::Lexer::LexFromRawLexer ( Token & Result ) [inline]

LexFromRawLexer - Lex a token from a designated raw lexer (one with no associated preprocessor object. Return true if the 'next character to read' pointer points at the end of the lexer buffer, false otherwise.

Definition at line 154 of file Lexer.h.

References Lex, and clang::PreprocessorLexer::LexingRawMode.

Referenced by findDirectives(), and clang::TokenRewriter::TokenRewriter().

CharSourceRange Lexer::makeFileCharRange	(	CharSourceRange	Range,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		`[static]`

Accepts a range and returns a character range with file locations.

Returns a null range if a part of the range resides inside a macro expansion or the range does not reside on the same FileID.

This function is trying to deal with macros and return a range based on file locations. The cases where it can successfully handle macros are:

-begin or end range lies at the start or end of a macro expansion, in which case the location will be set to the expansion point, e.g: #define M 1 2 a M If you have a range [a, 2] (where 2 came from the macro), the function will return a range for "a M" if you have range [a, 1], the function will fail because the range overlaps with only a part of the macro

-The macro is a function macro and the range can be mapped to the macro arguments, e.g: #define M 1 2 #define FM(x) x FM(a b M) if you have range [b, 2], the function will return the file range "b M" inside the macro arguments. if you have range [a, 2], the function will return the file range "FM(a b M)" since the range includes all of the macro expansion.

Definition at line 862 of file Lexer.cpp.

References clang::CharSourceRange::getBegin(), clang::CharSourceRange::getEnd(), clang::SrcMgr::SLocEntry::getExpansion(), clang::SrcMgr::ExpansionInfo::getExpansionLocStart(), clang::SourceManager::getFileID(), clang::SourceManager::getImmediateSpellingLoc(), clang::SourceManager::getSLocEntry(), isAtEndOfMacroExpansion(), isAtStartOfMacroExpansion(), clang::CharSourceRange::isCharRange(), clang::SourceLocation::isFileID(), clang::SourceLocation::isInvalid(), clang::SrcMgr::ExpansionInfo::isMacroArgExpansion(), clang::SourceLocation::isMacroID(), clang::CharSourceRange::isTokenRange(), makeRangeFromFileLocs(), clang::CharSourceRange::setBegin(), and clang::CharSourceRange::setEnd().

Referenced by getSourceText(), and makeStandaloneRange().

unsigned Lexer::MeasureTokenLength	(	SourceLocation	Loc,
		const SourceManager &	SM,
		const LangOptions &	LangOpts
	)		`[static]`

MeasureTokenLength - Relex the token at the specified location and return its length in bytes in the input file. If the token needs cleaning (e.g. includes a trigraph or an escaped newline) then this count includes bytes that are part of that.

Definition at line 412 of file Lexer.cpp.

References clang::Token::getLength(), and getRawToken().

Referenced by clang::TextDiagnostic::emitDiagnosticLoc(), clang::markup::EmitLocation(), getImmediateMacroName(), getImmediateMacroName(), getLocForEndOfToken(), clang::tooling::getRangeSize(), clang::Rewriter::getRangeSize(), clang::Rewriter::getRewrittenText(), clang::html::HighlightRange(), highlightRange(), and isAtEndOfMacroExpansion().

void Lexer::ReadToEndOfLine ( SmallVectorImpl< char > * Result = nullptr )

ReadToEndOfLine - Read the rest of the current preprocessor line as an uninterpreted string. This switches the lexer out of directive mode.

Definition at line 2426 of file Lexer.cpp.

References clang::Preprocessor::CodeCompleteNaturalLanguage(), clang::Token::is(), clang::PreprocessorLexer::ParsingFilename, clang::PreprocessorLexer::ParsingPreprocessorDirective, and clang::PreprocessorLexer::PP.

void Lexer::resetExtendedTokenMode ( )

Sets the extended token mode back to its initial value, according to the language options and preprocessor. This controls whether the lexer produces comment and whitespace tokens.

This requires the lexer to have an associated preprocessor. A standalone lexer has nothing to reset to.

Definition at line 120 of file Lexer.cpp.

References clang::Preprocessor::getCommentRetentionState(), clang::PreprocessorLexer::PP, SetCommentRetentionState(), and SetKeepWhitespaceMode().

Referenced by Lexer().

void clang::Lexer::SetCommentRetentionState ( bool Mode ) [inline]

SetCommentRetentionMode - Change the comment retention mode of the lexer to the specified mode. This is really only useful when lexing in raw mode, because otherwise the lexer needs to manage this.

Definition at line 187 of file Lexer.h.

References isKeepWhitespaceMode().

Referenced by ComputePreamble(), findDirectives(), getBeginningOfFileToken(), getRawToken(), LexRawTokensFromMainFile(), and resetExtendedTokenMode().

void clang::Lexer::SetKeepWhitespaceMode ( bool Val ) [inline]

SetKeepWhitespaceMode - This method lets clients enable or disable whitespace retention mode.

Definition at line 172 of file Lexer.h.

References clang::PreprocessorLexer::LexingRawMode.

Referenced by clang::DumpRawTokensAction::ExecuteAction(), resetExtendedTokenMode(), and clang::TokenRewriter::TokenRewriter().

std::string Lexer::Stringify	(	const std::string &	Str,
		bool	Charify = `false`
	)		`[static]`

Stringify - Convert the specified string into a C string by escaping '\' and " characters. This does not add surrounding ""'s to the string. If Charify is true, this escapes the ' character instead of ".

Stringify - Convert the specified string into a C string, with surrounding ""'s, and with escaped \ and " characters.

Definition at line 208 of file Lexer.cpp.

Referenced by clang::MacroArgs::StringifyArgument().

void Lexer::Stringify ( SmallVectorImpl< char > & Str ) [static]

Stringify - Convert the specified string into a C string by escaping '\' and " characters. This does not add surrounding ""'s to the string.

Definition at line 222 of file Lexer.cpp.

Friends And Related Function Documentation

friend class Preprocessor [friend]

Reimplemented from clang::PreprocessorLexer.

Definition at line 94 of file Lexer.h.

The documentation for this class was generated from the following files:

Public Member Functions

Static Public Member Functions

Friends

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Friends And Related Function Documentation