Algorithm::WordLevelStatistics - Pure Perl implementation of the "words level statistics" algorithm

 use Algorithm::WordLevelStatistics;
 my $wls = Algorithm::WordLevelStatistics->new;

 my %spectra = (); # hash from word to positions

 open IN, "<file.txt" or die "cannot open file.txt: $!";
 my $idx = 0;
 while(<IN>) {
   chomp;
   next if(m/^\s*$/); #skip blank lines

   foreach my $w ( split /\W/, lc( $_ ) ) {
     next if($w =~ m/^\s*$/);
     push @{ $spectra{$w} }, $idx++;
   }
 }
 close IN;

 my $ws = $wls->compute_spectra( \%spectra );

 # sort the words by their C attribute (the deviation of sigma_nor with respect to the expected value in a random text)
 my @sw = sort { $ws->{$b}->{C} <=> $ws->{$a}->{C} } keys( %{ $ws } );

 # print all the words with their scores
 foreach my $i (@sw) {
   print $i, " => { C = ",$ws->{$i}->{C}, ", count = ", $ws->{$i}->{count}, ", sigma_nor = ", $ws->{$i}->{sigma_nor}," }\n";
 }

This module implements the word leval statistics algorithm as described in: P. Carpena, P. Bernaola-Galav, M. Hackenberg, A.V. Coronado and J.L. Oliver, "Level statistics of words: finding keywords in literary texts and DNA", Physical Review E 79, 035102-4 ( DOI: 10.1103/PhysRevE.79.035102 )

new()

Creates a new Algorithm::WordLevelStatistics object and returns it.

compute_spectra()

The return value is a reference to an hash of hashes like the following one:

 {
   universe => {
                  C => 50.2020428972437,
                  count => 47,
                  sigma_nor => 6.16069263723295
               },
   x => {
          C => 47.1009427722911,
          count => 150,
          sigma_nor => 3.71679156784182
        }
  ...
 }

compute_spectrum()

The return value is a reference to an hash like the following one:

 {
   C => 50.2020428972437,
   count => 47,
   sigma_nor => 6.16069263723295
 }

The word level statistics algorithm uses a generalization of the level statistics analysis of quantum disordered systems to extract automatically keywords in literary texts.

The systems takes into account not only the relative frequencies of the words present in the text but also their spatial distribution in the text, and it is based on the consideration that relevant words are naturally clustered by the authors of the documents and irrelevant words are distributed randomly in the text (e.g. in an english text, the word 'the' is used with almost uniform distribution).

The word level statistics does not need a reference corpus but it uses just one document to extract the document's keywords. Moreover it is to be considered "language agnostic", because the algorithm does not need a "a priori" words classification (e.g. stop-words)

0.03

Corrected the test case (added test file to tarball.

0.02

Removed the dependency from Statistics::Lite. The removal of the dependency make this a self contained Perl module. The module is indeed faster too! (~15% speed improvement on large files).

0.01

Initial version of the module

Francesco Nidito

Copyright 2009 Francesco Nidito. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

http://bioinfo2.ugr.es/TextKeywords/.