Log::Parallel::Sql - insert a stream of data into an SQL database

 use Log::Parallel::Sql;

 my $dbf = db_insert_func($db_config, $job, $timeinfo, $mode);

 while ($log = ???) {
	$dbf->($log);
 }
 my $rows = $dbf->(undef);

Log::Parallel::Sql is a somewhat general-purpose database insert module. It knows to retry certain types of transactions. It can perform initialization. It can run more than one SQL statement per input record. It can execute SQL conditionally based on the results of other SQL statements.

It is not tied to the rest of Log::Parallel.

The configuration for Log::Parallel::Sql is compiled into a perl function which is then called once for each input object. When there is no more input data, call the function with undef.

The generate-the-fucntion routine, db_insert_func takes four parameters. The first is the configuration object (defined below) that is expected (but not required) to come from a YAML file. The second and third provide extra information. The forth parameter, $mode is set to testing when the code is just being compiled to check the configuration file.

The configuration object is a set of key/value pairs. The following keys are general parameters:

dsn

This is the DBI DSN needed to connect to the database.

user

This is the username parameter for conncting to the database.

pass

This is the password parameter for conncting to the database.

This sets the debug level for Log::Parallel::Sql.

commit_every

This sets the frequency of calling commit. When a deadlock occurs, the data will be re-processed, committing for each record until that batch of data is done.

debug

Additionally, there are two parameters that are either a single query or a list of queries: initialize and queries. The initialize SQL statements are run once per invocation of Log::Parallel::Sql. There is no current method to automatically install/update the database schemea.

If initialize or queries is an array, then the queries will be run in-order.

The statements defined in initialize and queries will have the following strings interpolated into them:

$log->{KEY_NAME}

This is the easy way to grab data from your input records.

<<< PERL CODE >>>

When you need something a little more complicated, you can embed arbitrary perl code. The perl code will be compiled as a function so you can return the value you want interpolated.

LASTID(TABLE,COLUMN)

When you need to need the last ID assigned on an insert.

The queries can be run conditionally. If the string SKIP-CONDITION: or DO-CONDITION: occurs in the query, then the SQL following the condition marker will be run first as a query. If the return value from that query is a perl true value, then the SQL before the condition marker will be run (assuming DO-CONDITION, the resverse is true for SKIP-CONDITION).

This package may be used and redistributed under the terms of either the Artistic 2.0 or LGPL 2.1 license.