Class PreprocessingContext.AllTokens

java.lang.Object
org.carrot2.text.preprocessing.PreprocessingContext.AllTokens
Enclosing class:
PreprocessingContext

public class PreprocessingContext.AllTokens extends Object
Information about all tokens of the input documents. Each element of each of the arrays corresponds to one individual token from the input or a synthetic separator inserted between documents, fields and sentences. Last element of this array is a special terminator entry.

All arrays in this class have the same length and values across different arrays correspond to each other for the same index.