Parallel::MapReduce::Worker - MapReduce, local worker

  use Parallel::MapReduce::Worker;
  my $w = new Parallel::MapReduce::Worker;

  my @chunks = chunk_n_store ($memd, $A, $job, 1000);
  my $cs = $w->map (\@chunks, 'slice1:', SERVERS, $job);
  ...
  my $ks = $w->reduce (\@cs, SERVERS, $job);

This class implements a local, sequential worker. You will only know about it if you want to subclass it to implement your own worker.

Nothing important to be said.

$w->shutdown

While there is no DESTROY method (for technical reasons), there is a shutdown method. It is supposed to terminate any background processes a worker might have started.

For a single-thread, local version nothing will be done, but individual subclasses might have to do substantial work to tear-down network connections, remote servers, etc.

map

$keys = $w->map ($chunks, $slice, $servers, $job)

The chunks are keys into the memcached servers. They allow to reconstruct the hash slice to be worked on. The slice is a simple id for that hash slice. The list reference to the memcached servers is obviously also necessary and the job id is an identifier for the current MR computation.

reduce

$chunks = $w->reduce ($keys, $servers, $job)

The keys are the keys into the intermediate hash within the memcached servers. The list reference to the memcached servers is obviously also necessary and the job id is an identifier for the current MR computation.

Parallel::MapReduce

Copyright 200[8] by Robert Barta, <drrho@cpan.org>

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.