Vertica Tokenizers

A tokenizer does the following:

• Receives a stream of characters.
• Breaks the stream into individual tokens that usually correspond to individual words.
• Returns a stream of tokens.