Module tokenize

Classes

BlanklineTokenizer
LineTokenizer
PunktSentenceTokenizer
PunktWordTokenizer
RegexpTokenizer
SExprTokenizer
SpaceTokenizer
TabTokenizer
WhitespaceTokenizer
WordPunctTokenizer
WordTokenizer

Functions

blankline_tokenize
line_tokenize
punkt_word_tokenize
regexp_tokenize
sexpr_tokenize
word_tokenize
wordpunct_tokenize