Parallel::Depend::Util

Kitchen-sink module for configuratin, logging, whatever...

	use Parallel::Depend::Util;

	# generate path relative to the #!'s $Bin directory.
	# these can be abs-pathed without regard to the
	# current working directory. The last path element
	# is looked up in $defaulz->{global}{basename} allowing
	# for simpler shared path names.

	my $path = localpath @path_components, $token;
	my $path = localpath @path_components, 'basename';

	# sanity check a set of paths.

	checkdir @dirlist;

	# send email, to can also be an array referent.
	# from defaults to current user at whatever
	# 'hostname' returns.

	send_mail
		To         => 'someone@someplace',
		From       => 'me@here',
		Subject    => 'Message subject',
		Data       => 'Message body',

		'X-Parallel::Depend' => 'Progress'
	;

	# defaults from $que->{user}->moduleconfig.

	progress_mail @message;

	nastygram @message;

Utility functions for queueing: message logging with standard format, sending email, and generating/checking local file paths.

Messages: log_format, log_message, log_error

The format adds a PID and timestamp, converts referents via Data::Dumper, and returns the result as a string.

log_mesasge prints to STDOUT and returns clean (0), log_error prints to STDERR and STDOUT, returning an error status of -1.

Email-notification: send_mail, progress_mail, nastygram.

send_mail is a generic mail wrapper for MIME::Lite; progress mail is useful for tracking long-running jobs (basically log_message via email); nastygram will send the email and then die with the error message (effectively aborting a queue).

progress_mail and nastygram are que methods and take their to and from values via $que->{user}->{moduleconfig}.

Progress mail is intended for monitorig long-running jobs and sends messages to $config->{notify}{$job}:

	$defaults =
	{
		Foobar =>
		{
			queuename => 'Daily Frobnicate',

			notify =>
			{
				download_stuff =>
				[ qw( user1@somehost user2@anotherhost ) ],
			},
		},

		...
	}

	...

	sub que_job
	{
		...

		$que->progress_mail 'download_foobar';
	}

Will send a standard message with a subject of "[$quename-Progress] $jobname" to the names configured for that job.

nastygram notifies the configured list of a fatal run-time error that is aborting queue execution. It sends out a log_format-ed message and then dies with an error message.

	$defaultz =
	{
		Foobar =>
		{
			notify =>
			{
				job1 => 'user1@someplace',
				job2 => 'user2@someplace',

				fatal => [qw( user1@someplace user2@someplace )],
			},
		},
	},




	$que->nastygram 'Fatal: unable to carry the load', $loadref;

nastygram will call log_format on the arguments, prefix a fatality message, and send the result to everyone listed in the fatal list.

Local files: localpath, checkdir, slurp, splat.

localpath uses $config->{basename} to convert a path plus basename-or-token to a path relative to $FindBin::Bin. Its main use is in tokenizing the basenames of paths used in multiple stages of a job. The code uses hash keys for the paths which can then be more descriptive and changed in standard places.

	$defaultz =>
	{
		Foboar =>
		{
			basenames =>
			{
				name2node => 'name-node-table.dump.gz',
			}
		},
	};

	...

	my $path = localpath '../var/tmp', 'name2node';

This is a subroutine and not a que method, it accesses $Parallel::Depend::Execute::defaults->{global} directly to find the hash of basename-tokens.

It is also specific to *NIX, since it uses a join '/' to generate the final path.

If the input path does not begin with '/' then $FindBin::Bin is prefixed to it. Note that this path may not yet exist where localpath is called to generate a path for new output. In those casese it will carp but still return the requested path.

checkdir is used to santiy-check the log, run, and tmp directories before execution. If the path(s) requested do not exist then it attempts to create them with mods of 02775. If the directory does not exist or is not read+executable+writable by the current user it dies with a specific error message.

	checkir $indir, $outdir, $tmpdir;

	# if you are alive at this point then the directories
	# exist and are fully accessable.

slurp reads the output of Data::Dumper and evals it, returning the result:

	$defaultz =>
	{
		Foobar =>
		{
			basename =>
			{
				name2id => '../var/tmp/entry2id.dump',
			},
		},
	};

	my $data = slurp localpath name2id;

Will reload and eval the output of Data::Dumper (or anything else that can be eval-ed), returning the result as a scalar (i.e., the eval is assigned to a scalar).

If the eval failes slurp calls nastygram with a message of the failed path.

splat writes out files in a way that is consistent with sulrp reading them. Its main use is in avoiding Data::Dumper in cases where the block of data is to large to effectively convert to sourceable text. If it is passed an array or if the number of hash keys is greater than 1000, splat will write data out as a tab-separated-values ('.tsv') file.

Note: slurp and splat default to using ".dump" as the extension for Data::Dumper-ed content, ".tsv" for tab separated. splat may modify the path given in basenames to accomodate the format actually written; slurp looks for both ".tsv" and ".dump" extensions.

Steven Lembark, Workhorse Computing <lembark@wrkhors.com>

Parallel::Depend

(C) 2001-2002 Steven Lembark, Workhorse Computing

This code is released under the same terms as Perl istelf. Please see the Perl-5.8 distribution (or later) for a full description.

In any case, this code is release as-is, with no implied warranty of fitness for a particular purpose or warranty of merchantability.