Plucene::Analysis::WhitespaceTokenizer - white space tokenizer

Plucene documentation

Contained in the Plucene distribution.

Index

NAME
SYNOPSIS
DESCRIPTION
METHODS

Code Index:

package Plucene::Analysis::WhitespaceTokenizer
- token_re
EOF

NAME

Plucene::Analysis::WhitespaceTokenizer - white space tokenizer

SYNOPSIS

	# isa Plucene::Analysis::CharTokenizer

DESCRIPTION

A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.

METHODS

Plucene documentation

Contained in the Plucene distribution.

package Plucene::Analysis::WhitespaceTokenizer;

use strict;
use warnings;

use base 'Plucene::Analysis::CharTokenizer';

sub token_re { qr/\S+/ }

1;