NlpTools API
Class

NlpTools\Tokenizers\WhitespaceTokenizer

class WhitespaceTokenizer implements TokenizerInterface

Simple white space tokenizer.

Break on every white space

Constants

PATTERN

Methods

array tokenize(string $str)

Break a character sequence to a token sequence

Details

at line 13
public array tokenize(string $str)

Break a character sequence to a token sequence

Parameters

string $str The text for tokenization

Return Value

array The list of tokens from the string