Token extraction and manipulation

Collaboration diagram for Token extraction and manipulation:

Classes
struct	CXToken
	Describes a single preprocessing token. More...
Typedefs
typedef enum CXTokenKind	CXTokenKind
	Describes a kind of token.
Enumerations
enum	CXTokenKind {   CXToken_Punctuation, CXToken_Keyword, CXToken_Identifier, CXToken_Literal,   CXToken_Comment }
	Describes a kind of token. More...
Functions
CINDEX_LINKAGE CXTokenKind	clang_getTokenKind (CXToken)
	Determine the kind of the given token.
CINDEX_LINKAGE CXString	clang_getTokenSpelling (CXTranslationUnit, CXToken)
	Determine the spelling of the given token.
CINDEX_LINKAGE CXSourceLocation	clang_getTokenLocation (CXTranslationUnit, CXToken)
	Retrieve the source location of the given token.
CINDEX_LINKAGE CXSourceRange	clang_getTokenExtent (CXTranslationUnit, CXToken)
	Retrieve a source range that covers the given token.
CINDEX_LINKAGE void	clang_tokenize (CXTranslationUnit TU, CXSourceRange Range, CXToken **Tokens, unsigned *NumTokens)
	Tokenize the source code described by the given range into raw lexical tokens.
CINDEX_LINKAGE void	clang_annotateTokens (CXTranslationUnit TU, CXToken *Tokens, unsigned NumTokens, CXCursor *Cursors)
	Annotate the given set of tokens by providing cursors for each token that can be mapped to a specific entity within the abstract syntax tree.
CINDEX_LINKAGE void	clang_disposeTokens (CXTranslationUnit TU, CXToken *Tokens, unsigned NumTokens)
	Free the given set of tokens.

---

Detailed Description

The routines in this group provide access to the tokens within a translation unit, along with a semantic mapping of those tokens to their corresponding cursors.

---

Typedef Documentation

typedef enum CXTokenKind CXTokenKind

Describes a kind of token.

---

Enumeration Type Documentation

enum CXTokenKind

Describes a kind of token.

Enumerator:

CXToken_Punctuation	A token that contains some kind of punctuation.
CXToken_Keyword	A language keyword.
CXToken_Identifier	An identifier (that is not a keyword).
CXToken_Literal	A numeric, string, or character literal.
CXToken_Comment	A comment.

Definition at line 4050 of file Index.h.

---

Function Documentation

CINDEX_LINKAGE void clang_annotateTokens	(	CXTranslationUnit	TU,
		CXToken *	Tokens,
		unsigned	NumTokens,
		CXCursor *	Cursors
	)

Annotate the given set of tokens by providing cursors for each token that can be mapped to a specific entity within the abstract syntax tree.

This token-annotation routine is equivalent to invoking clang_getCursor() for the source locations of each of the tokens. The cursors provided are filtered, so that only those cursors that have a direct correspondence to the token are accepted. For example, given a function call f(x), clang_getCursor() would provide the following cursors:

* when the cursor is over the 'f', a DeclRefExpr cursor referring to 'f'. * when the cursor is over the '(' or the ')', a CallExpr referring to 'f'. * when the cursor is over the 'x', a DeclRefExpr cursor referring to 'x'.

Only the first and last of these cursors will occur within the annotate, since the tokens "f" and "x' directly refer to a function and a variable, respectively, but the parentheses are just a small part of the full syntax of the function call expression, which is not provided as an annotation.

Parameters:

TU	the translation unit that owns the given tokens.
Tokens	the set of tokens to annotate.
NumTokens	the number of tokens in Tokens.
Cursors	an array of NumTokens cursors, whose contents will be replaced with the cursors corresponding to each token.

CINDEX_LINKAGE void clang_disposeTokens	(	CXTranslationUnit	TU,
		CXToken *	Tokens,
		unsigned	NumTokens
	)

Free the given set of tokens.

CINDEX_LINKAGE CXSourceRange clang_getTokenExtent	(	CXTranslationUnit	,
		CXToken
	)

Retrieve a source range that covers the given token.

CINDEX_LINKAGE CXTokenKind clang_getTokenKind

(

CXToken

)

Determine the kind of the given token.

CINDEX_LINKAGE CXSourceLocation clang_getTokenLocation	(	CXTranslationUnit	,
		CXToken
	)

Retrieve the source location of the given token.

CINDEX_LINKAGE CXString clang_getTokenSpelling	(	CXTranslationUnit	,
		CXToken
	)

Determine the spelling of the given token.

The spelling of a token is the textual representation of that token, e.g., the text of an identifier or keyword.

CINDEX_LINKAGE void clang_tokenize	(	CXTranslationUnit	TU,
		CXSourceRange	Range,
		CXToken **	Tokens,
		unsigned *	NumTokens
	)

Tokenize the source code described by the given range into raw lexical tokens.

Parameters:

TU	the translation unit whose text is being tokenized.
Range	the source range in which text should be tokenized. All of the tokens produced by tokenization will fall within this source range,
Tokens	this pointer will be set to point to the array of tokens that occur within the given source range. The returned pointer must be freed with clang_disposeTokens() before the translation unit is destroyed.
NumTokens	will be set to the number of tokens in the *Tokens array.