The Lexer Interface

The Lexer Iterator Interface

Introduction
Wave Lexer synopsis
Public Typedefs
Member functions

Introduction

Every lexer, which should be used in conjunction with the Wave library, has to return tokens formed from the input stream. These tokens should conform to the synopsis described in the topic The Token Type. The lexer type should expose an interface which conforms at least to a forward_iterator (in the sense defined by the the C++ Standard) returning the token type. The code sample below does not show the definition of this forward iterator interface because this is highly implementation defined.

Wave lexer synopsis (header: wave/cpplexer/cpp_lexer_interface.hpp)

    struct lex_iterator
    {
        typedef boost::wave::lex_token<> token_type;

    // Every lex_iterator should implement at least a complete 
    // forward_iterator interface (not shown here)
        typedef std::forward_iterator_tag iterator_category;

    // additional requirements
        lex_iterator();

        template <typename Iterator>
        lex_iterator(Iterator const &first, Iterator const &last
            typename token_type::position_type const &pos, 
            boost::wave::language_support language)
    };

Please note, that the lex_iterator defined in the library header wave/cpplexer/cpp_lexer_interface.hpp actually is a template class taking the token type to use as its template parameter. This is omitted in the synopsis above because it is an implementation detail of the Re2C lexer provided as part of the Wave library.

If you want to use Wave in conjunction with your own lexing component this will have to conform to the interface described above only.

Public Typedefs

Besides the typedefs mandated for a forward_iterator by the C++ standard every lexer to be used with the Wave library should define the following typedefs:

Public typedef's defined by the `boost::wave::context` class
`token_type`	The `token` type returned by the lexer. This is type is used as the return value of the main iterators provided by the `boost::wave::context` object too.

Member functions

Besides the functions, which should be provided for forward_iterators as mandated by the C++ Standard, every lexer must implement the following functions to be used with the Wave library:

constructor

    lex_iterator();

The default constructor should construct a lexer iterator, which may be used as the end iterator of the provided iterator range.

    template <typename Iterator>
    lex_iterator(Iterator const &first, Iterator const &last,
        typename token_type::position_type const &pos, 
        boost::wave::language_support language);

The second constructor should construct a lexer iterator, which may be used as a iterator traversing over the token sequence, generated by the lexer class.

The pair of iterators first and last should represent the input stream to be tokenized by the given lexer class.

The parameter pos contains the initial position information to be used for token generation.

The parameter language controls the reuqired mode with which the lexer should be initialised.

Last updated: Thursday, December 1, 2005 21:25