C++ Boost

Token Iterator

template <
	class TokenizerFunc = char_delimiters_separator<char>, 
	class Iterator = std::string::const_iterator,
	class Type = std::string
>
class token_iterator_generator

template<class Type, class Iterator, class TokenizerFunc>
typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type 
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun)

The token iterator serves to provide an iterator view of the tokens in a parsed sequence.

Example

/// simple_example_5.cpp
#include<iostream>
#include<boost/token_iterator.hpp>
#include<string>

int main(){
   using namespace std;
   using namespace boost;
   string s = "12252001";
   int offsets[] = {2,2,4};
   offset_separator f(offsets, offsets+3);
   typedef token_iterator_generator<offset_separator>::type Iter;
   Iter beg = make_token_iterator<string>(s.begin(),s.end(),f);
   Iter end = make_token_iterator<string>(s.end(),s.end(),f); 
   // The above statement could also have been what is below
   // Iter end;
   for(;beg!=end;++beg){
     cout << *beg << "\n";
   }
}

Template Parameters

Parameter	Description
`TokenizerFunc`	The TokenizerFunction used to parse the sequence.
`Iterator`	The type of the iterator the specifies the sequence.
`Type`	The type of the token, typically string.

Model of

The category of Iterator, up to and including Forward Iterator. Anything higher will get scaled down to Forward Iterator.

Related Types

Type	Remarks
token_iterator_generator::type	The type of the token iterator.

Creation

template<class Type, class Iterator, class TokenizerFunc>
typename token_iterator_generator<TokenizerFunc,Iterator,Type>::type 
make_token_iterator(Iterator begin, Iterator end,const TokenizerFunc& fun)

Parameter	Description
begin	The beginning of the sequence to be parsed.
end	Past the end of the sequence to be parsed.
fun	A functor that is a model of TokenizerFunction

© Copyright John R. Bandela 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.