Package gate.creole.tokeniser

Class Summary
DefaultTokeniser A composed tokeniser containing a SimpleTokeniser and a Transducer.
DFSMState Implements a state of the deterministic finite state machine of the tokeniser.
FSMState A state of the finite state machine that is the kernel tokeniser
SimpleTokeniser Implementation of a Unicode rule based tokeniser.
TestTokeniser  
UnicodeType Used as an object wrapper that holds an Unicode type (the byte value of the static member of java.lang.Character).
 

Exception Summary
InvalidRuleException Fired when an invalid tokeniser rule is found
TokeniserException The top level exception for all the exceptions fired by the tokeniser