A TokenizerFunction is a functor whose purpose is to parse a given sequence until exactly 1 token has been found or the end is reached. It then updates the token, and informs the caller of the location in the sequence of the next element immediately after the last element of the sequence that was parsed for the current token.
Assignable, CopyConstructable
| X | A type that is a model of TokenizerFunction |
| func | Object of type X |
| tok | Object of Token |
| next | iterator that points to the first unparsed element of the sequence being parsed |
| end | iterator that points to the past the end of the sequence being parsed |
A token is the result of parsing a sequence.
In addition to the expression in Assignable and CopyConstructable the following expressions are valid
| Name | Expression | Return type |
|---|---|---|
| Functor | func(next, end, tok) | bool |
| reset | reset() | void |
In addition to the expression semantics in Assignable and CopyConstructable, TokenizerFunction has the following expression semantcs
| Name | Expression | Precondition | Semantics | Postcondition |
|---|---|---|---|---|
| operator() | func(next, end, tok) | next and end are valid iterators to the same sequence. next is a reference the function is free to modify. tok is constructed. | The return value indicates whether a new token was found in the sequence (next,end] | If the return value is true, the new token is assigned to tok. next is always updated to the position where parsing should start on the subsequent call. |
| reset | reset() | None | Clears out all state variables that are used by the object in parsing the current sequence. | A new sequence to parse can be given. |
No guarantees. Models of TokenizerFunction are free to define their own complexity
© Copyright John R. Bandela 2001. Permission to copy, use, modify, sell and distribute this document is granted provided this copyright notice appears in all copies. This document is provided "as is" without express or implied warranty, and with no claim as to its suitability for any purpose.