NlpTools API
Interface

NlpTools\Tokenizers\TokenizerInterface

interface TokenizerInterface

Methods

array tokenize(string $str)

Break a character sequence to a token sequence

Details

at line 13
public array tokenize(string $str)

Break a character sequence to a token sequence

Parameters

string $str The text for tokenization

Return Value

array The list of tokens from the string