Plucene::Analysis::LetterTokenizer - Letter tokenizer


Plucene documentation  | view source Contained in the Plucene distribution.

Index


NAME

Top

Plucene::Analysis::LetterTokenizer - Letter tokenizer

SYNOPSIS

Top

	# isa Plucene::Analysis::CharTokenizer

DESCRIPTION

Top

This is the letter tokenizer class, which divides text at non-letters.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces


Plucene documentation  | view source Contained in the Plucene distribution.