Class Tokenizer

java.lang.Object
weka.core.tokenizers.Tokenizer
All Implemented Interfaces:
Serializable, Enumeration, OptionHandler, RevisionHandler
Direct Known Subclasses:
AlphabeticTokenizer, CharacterDelimitedTokenizer

public abstract class Tokenizer extends Object implements Enumeration, OptionHandler, Serializable, RevisionHandler
A superclass for all tokenizer algorithms.
Version:
$Revision: 1.3 $
Author:
FracPete (fracpete at waikato dot ac dot nz)
See Also:
  • Constructor Summary

    Constructors
    Constructor
    Description
     
  • Method Summary

    Modifier and Type
    Method
    Description
    Gets the current option settings for the OptionHandler.
    abstract String
    Returns a string describing the stemmer
    abstract boolean
    Tests if this enumeration contains more elements.
    Returns an enumeration of all the available options..
    abstract Object
    Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
    static void
    runTokenizer(Tokenizer tokenizer, String[] options)
    initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.
    void
    setOptions(String[] options)
    Sets the OptionHandler's options using the given list.
    abstract void
    Sets the string to tokenize.
    static String[]
    tokenize(Tokenizer tokenizer, String[] options)
    initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface java.util.Enumeration

    asIterator

    Methods inherited from interface weka.core.RevisionHandler

    getRevision
  • Constructor Details

    • Tokenizer

      public Tokenizer()
  • Method Details

    • globalInfo

      public abstract String globalInfo()
      Returns a string describing the stemmer
      Returns:
      a description suitable for displaying in the explorer/experimenter gui
    • listOptions

      public Enumeration listOptions()
      Returns an enumeration of all the available options..
      Specified by:
      listOptions in interface OptionHandler
      Returns:
      an enumeration of all available options.
    • getOptions

      public String[] getOptions()
      Gets the current option settings for the OptionHandler.
      Specified by:
      getOptions in interface OptionHandler
      Returns:
      the list of current option settings as an array of strings
    • setOptions

      public void setOptions(String[] options) throws Exception
      Sets the OptionHandler's options using the given list. All options will be set (or reset) during this call (i.e. incremental setting of options is not possible).
      Specified by:
      setOptions in interface OptionHandler
      Parameters:
      options - the list of options as an array of strings
      Throws:
      Exception - if an option is not supported
    • hasMoreElements

      public abstract boolean hasMoreElements()
      Tests if this enumeration contains more elements.
      Specified by:
      hasMoreElements in interface Enumeration
      Returns:
      true if and only if this enumeration object contains at least one more element to provide; false otherwise.
    • nextElement

      public abstract Object nextElement()
      Returns the next element of this enumeration if this enumeration object has at least one more element to provide.
      Specified by:
      nextElement in interface Enumeration
      Returns:
      the next element of this enumeration.
    • tokenize

      public abstract void tokenize(String s)
      Sets the string to tokenize. Tokenization happens immediately.
      Parameters:
      s - the string to tokenize
    • tokenize

      public static String[] tokenize(Tokenizer tokenizer, String[] options) throws Exception
      initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. If no strings remained in the option string then data is read from stdin, line by line.
      Parameters:
      tokenizer - the tokenizer to use
      options - the options for the tokenizer
      Returns:
      the tokenized strings
      Throws:
      Exception - if setting of options or tokenization fails
    • runTokenizer

      public static void runTokenizer(Tokenizer tokenizer, String[] options)
      initializes the given tokenizer with the given options and runs the tokenizer over all the remaining strings in the options array. The generated tokens are then printed to stdout. If no strings remained in the option string then data is read from stdin, line by line.
      Parameters:
      tokenizer - the tokenizer to use
      options - the options for the tokenizer