The WordTokenization type exposes the following members.

Constructors

  NameDescription
WordTokenization
Construct a new tokenization.

Methods

  NameDescription
AddChangeListenerOverloaded.
Equals
Determines whether the specified Object is equal to the current Object.
(Inherited from Object.)
Finalize
Allows an Object to attempt to free resources and perform other cleanup operations before the Object is reclaimed by garbage collection.
(Inherited from Object.)
GetHashCode
Serves as a hash function for a particular type.
(Inherited from Object.)
GetType
Gets the Type of the current instance.
(Inherited from Object.)
IsUnchanging
A particular ChangeType can never be raised by this Changeable.
(Inherited from Unchangeable.)
MemberwiseClone
Creates a shallow copy of the current Object.
(Inherited from Object.)
ParseString
Parse a string.
ParseToken
Returns the symbol for a single token.
RemoveChangeListenerOverloaded.
SplitString
Split up a string.
TokenizeSymbol
Return a token representing a single symbol.
TokenizeSymbolList
Return a string representation of a list of symbols.
ToString
Returns a String that represents the current Object.
(Inherited from Object.)

Properties

  NameDescription
Alphabet
The alphabet to which this tokenization applies.
Annotation
Should return the associated annotation object.
TokenType
Determine the style of tokenization represented by this object.

See Also