Class WhitespaceTokenizer

source code

    object --+    
             |    
api.TokenizerI --+
                 |
                WhitespaceTokenizer

A tokenizer that divides a string into substrings by treating any sequence of whitespace characters as a separator. Whitespace characters are space (' '), tab ('\t'), and newline ('\n'). If you are performing the tokenization yourself (rather than building a tokenizer to pass to some other piece of code), consider using the string split() method instead:

>>> words = s.split()