Search::Tools::TokenListPP - a bunch of tokens from a Tokenizer

Search-Tools documentation Contained in the Search-Tools distribution.

Index

Code Index:

NAME

Top

Search::Tools::TokenListPP - a bunch of tokens from a Tokenizer

SYNOPSIS

Top

 use Search::Tools::Tokenizer;
 my $tokenizer = Search::Tools::Tokenizer->new();
 my $tokens = $tokenizer->tokenize_pp('quick brown red dog');

 # use like an iterator
 while ( my $token = $tokens->next ) {
    # token isa Search::Tools::Token
 }
 # iterate the other way
 while ( my $token = $tokens->prev ) {
    # ...
 }

 # fetch a particular token
 my $token = $tokens->get_token( $position );

 # reset the iterator
 $tokens->reset;

 # get the current iterator position
 my $pos = $tokens->pos;

 # set the iterator position
 $tokens->set_pos( $pos + 1 );

 # how many tokens originally?
 my $num = $tokens->num;

 # treat like array
 push( @{ $tokens->as_array }, $new_token );

 # now how many tokens?
 my $len = $tokens->len;    # $len != $num

 # get all the hot tokens
 my $heat = $tokens->get_heat;

 # get all the matches to the regex in Tokenizer
 my $matches = $tokens->get_matches;

 # just the number of matches
 my $num_matches = $tokens->num_matches;

DESCRIPTION

Top

A TokenListPP is a pure-Perl version of TokenList. See the docs for TokenList for more details.

METHODS

Top

See Search::Tools::TokenListUtils for other methods available on this class.

This class inherits from Search::Tools::Object. Only new or overridden methods are documented here.

next

Get the next Token.

prev

Get the previous Token.

pos

Get the iterator position.

set_pos

Set the iterator position.

reset

Same as calling:

 $tokens->set_pos(0);

len

The number of Tokens in the TokenList.

num

The number of Tokens initially parsed by the Tokenizer. This is the same value as len() unless you alter the TokenList via as_array().

as_array

Returns an array ref to the internal array of tokens. If you alter the array, it will alter the len() value but not the num() value.

get_heat

Returns an array ref to the internal array of tokens with is_hot() set by the original Tokenizer. This method will return an empty list unless you have passed a heat_seeker to the tokenize_pp() method. See Search::Tools::Tokenizer.

matches

Returns an array ref of all the Tokens with is_match() set. The array is constructed at the time you call the method so if you alter the array it will not affect the TokenListPP object, but if you alter a Token in the array it will affect the Token in the TokenListPP object.

num_matches

Like calling:

 my $num = scalar @{ $tokens->matches };

get_token( position )

Returns the Token at position. If position is invalid returns undef.

AUTHOR

Top

Peter Karman <karman@cpan.org>

BUGS

Top

Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

Top

You can find documentation for this module with the perldoc command.

    perldoc Search::Tools

You can also look for information at:

* RT: CPAN's request tracker

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Search-Tools

* AnnoCPAN: Annotated CPAN documentation

http://annocpan.org/dist/Search-Tools

* CPAN Ratings

http://cpanratings.perl.org/d/Search-Tools

* Search CPAN

http://search.cpan.org/dist/Search-Tools/

COPYRIGHT

Top

Copyright 2009 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

Search-Tools documentation Contained in the Search-Tools distribution. 
package Search::Tools::TokenListPP;
use strict;
use warnings;
use base qw( Search::Tools::Object );
use overload
    '""'     => sub { $_[0]->str; },
    'bool'   => sub { $_[0]->len; },
    fallback => 1;
use Carp;
use base qw( Search::Tools::TokenListUtils );

our $VERSION = '0.59';

__PACKAGE__->mk_accessors(qw( pos num ));

sub len {
    return scalar @{ $_[0]->{tokens} };
}

sub get_heat {
    return $_[0]->{heat};
}

sub next {
    my $self   = shift;
    my $tokens = $self->{tokens};
    my $len    = scalar(@$tokens) - 1;
    if ( $len == -1 ) {
        return;
    }
    elsif ( $self->{pos} > $len ) {
        return; 
    }
    else {
        return $tokens->[ $self->{pos}++ ];
    }
}

sub prev {
    my $self   = shift;
    my $tokens = $self->{tokens};
    my $len    = scalar(@$tokens) - 1;
    if ( $len == -1 ) {
        return;
    }
    elsif ( $self->{pos} < 0 ) {
        return;
    }
    else {
        return $tokens->[ --$self->{pos} ];
    }
}

sub reset {
    $_[0]->{pos} = 0;
}

sub set_pos {
    $_[0]->{pos} = $_[1];
}

sub get_token {
    my $self = shift;
    my $len  = scalar( @{ $self->{tokens} } ) - 1;
    my $i    = shift;
    if ( !defined $i ) {
        croak "index position required";
    }
    if ( !defined $self->{tokens}->[$i] ) {
        return;
    }
    else {
        return $self->{tokens}->[$i];
    }
}

sub as_array {
    return $_[0]->{tokens};
}

sub matches {
    return [ grep { $_->{is_match} } @{ $_[0]->{tokens} } ];
}

sub num_matches {
    return scalar @{ shift->matches };
}

1;

__END__