Package org.carrot2.language
Class TokenTypeUtils
java.lang.Object
org.carrot2.language.TokenTypeUtils
-
Method Summary
Modifier and TypeMethodDescriptionstatic boolean
isCommon
(int flag) Returntrue
ifTokenizer.TF_COMMON_WORD
is set.static boolean
isDocumentSeparator
(int type) Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_DOCUMENT
set.static boolean
isFieldSeparator
(int type) Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_FIELD
set.static boolean
isInQuery
(int flag) Returntrue
ifTokenizer.TF_QUERY_WORD
is set.static boolean
isTerminator
(int type) Returnstrue
if the given type hasTokenizer.TF_TERMINATOR
set.static int
maskType
(int rawType) Mask the given raw token type and leave just the token type bits.
-
Method Details
-
maskType
public static int maskType(int rawType) Mask the given raw token type and leave just the token type bits. -
isDocumentSeparator
public static boolean isDocumentSeparator(int type) Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_DOCUMENT
set. -
isFieldSeparator
public static boolean isFieldSeparator(int type) Returnstrue
if the given type hasTokenizer.TF_SEPARATOR_FIELD
set. -
isTerminator
public static boolean isTerminator(int type) Returnstrue
if the given type hasTokenizer.TF_TERMINATOR
set. -
isCommon
public static boolean isCommon(int flag) Returntrue
ifTokenizer.TF_COMMON_WORD
is set. -
isInQuery
public static boolean isInQuery(int flag) Returntrue
ifTokenizer.TF_QUERY_WORD
is set.
-