class PennTreeBankTokenizer extends WhitespaceTokenizer
PennTreeBank Tokenizer Based on http://www.cis.upenn.edu/~treebank/tokenizer.sed
Constants
| PATTERN |
|
Methods
| array |
tokenize(string $str)
Calls internal functions to handle data processing |
|
| __construct() |
Details
at line 30
public array
tokenize(string $str)
Calls internal functions to handle data processing