Search::Tools::TokenPP - a token object returned from a TokenList

 use Search::Tools::Tokenizer;
 my $tokenizer = Search::Tools::Tokenizer->new();
 my $tokens = $tokenizer->tokenize_pp('quick brown red dog');
 while ( my $token = $tokens->next ) {
     # token isa Search::Tools::TokenPP
     print "token = $token\n";
     printf("str: %s, len = %d, u8len = %d, pos = %d, is_match = %d, is_hot = %d\n",
        $token->str,
        $token->len, 
        $token->u8len, 
        $token->pos, 
        $token->is_match, 
        $token->is_hot
     );
 }

A TokenPP represents one or more characters culled from a string by a Tokenizer.

TokenPP is a pure-Perl version of Token. See the Token docs for more details.

This class inherits from Search::Tools::Object. Only new or overridden methods are documented here.

The characters in the token. Stringifies to the str() value with overloading.

The byte length of str().

The character length of str(). For ASCII, len() == u8len(). For non-ASCII UTF-8, u8len() < len().

The zero-based position in the original string.

Did the token match the re() in the Tokenizer.

Did the token match the heat_seeker in the Tokenizer.

Returns true value if the Token matches common sentence-ending punctuation.

Set the is_hot() value.

Set the is_match() value.

Peter Karman <karman@cpan.org>

Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

You can find documentation for this module with the perldoc command.

    perldoc Search::Tools

You can also look for information at:

* RT: CPAN's request tracker

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Search-Tools

* AnnoCPAN: Annotated CPAN documentation

http://annocpan.org/dist/Search-Tools

* CPAN Ratings

http://cpanratings.perl.org/d/Search-Tools

* Search CPAN

http://search.cpan.org/dist/Search-Tools/

Copyright 2009 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.