Directory Digests - Version 0.90 - README

CONTENTS

OVERVIEW
FEATURES
EXAMPLES
INSTALLATION
DEPENDENCIES
ISSUES
HISTORY
FUTURE
AUTHOR
RIGHTS

OVERVIEW

Dirgest is a compact suite of development modules and operational scripts that provide a comprehensive ability to work with digests of file and directory sets. The modules provide building blocks for developers, while the scripts are directly useful for local host and web site administration.

Two modules are supplied: Digest::Directory::{BASE,API}.pm Two scripts are supplied: dirgest.{pl,cgi}

Primary applications are security and integrity.

The author uses this suite to verify correct replication of data from local host to non-CLI access remote web site, and then ongoing verification that web site content is consistent. The suite is also used to ensure integrity of local host (OpenBSD) installation.

The most recent version is always available on CPAN or from the author (matthew.gream@pobox.com).

FEATURES

Primary features of the package are:

developer BASE and API modules to provide low and high level perl access to dirgests.
operational CLI and CGI scripts to provide command line and common-gateway-interface access to dirgests.
uses Digest::MD5.
computes a 'dirgest' (ternary of 'digest', 'size', 'name' for given file), and also computes master dirgest across all consitituent dirgests.
allows file/directory sets to be specified as 'include' or 'exclude' entries, either from command line, config file, CGI parameters or method calls.
allows file/directory sets to have leading name components trimmed in compute, fetch and load operations to support cross-host activities.
uses LWP::UserAgent to access files and CGI resources, allowing media transparency (support HTTP basic authentication, and secure mode on cgi resources).
object oriented modules, code level trace information about activities, comprehensive unit tests and documentation.

EXAMPLES

Example 1: save and verify contents of directories (CLI scripts).

Various examples of usage, from the command line:

| ------------------------------------------------ |
| napa:tmp $ dirgest.pl --version
| Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ dirgest.pl --filename /tmp/dirgests.txt create +/etc | Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | including /etc
| CREATING
| computing from /etc
| Can't opendir(/etc/ssl/private): Permission denied | Can't opendir(/etc/isakmpd/private): Permission denied | writing to /tmp/dirgests.txt
| napa:tmp $ wc -l /tmp/dirgests.txt | 138 /tmp/dirgests.txt
| napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ dirgest.pl --quiet show +/etc > /tmp/dirgests2.txt | Can't opendir(/etc/ssl/private): Permission denied | Can't opendir(/etc/isakmpd/private): Permission denied | napa:tmp $ diff /tmp/dirgests2.txt /tmp/dirgests.txt | napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ echo '+/etc' >dirgest.conf | napa:tmp $ cat dirgest.conf
| +/etc
| napa:tmp $ dirgest.pl --quiet --configure=dirgest.conf show \ | >/tmp/dirgests3.txt
| Can't opendir(/etc/ssl/private): Permission denied | Can't opendir(/etc/isakmpd/private): Permission denied | napa:tmp $ diff /tmp/dirgests.txt /tmp/dirgests3.txt | napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ ls /usr/local/share/gnupg | options.skel
| napa:tmp $ dirgest.pl --quiet show +/usr/local/share/gnupg | = FEy7kEGM+dOntCi0Z/0KRg== 000000004492 /usr/local/share/gnupg/options.skel | # jicFVzLgkWo8E9CCmQwX2g==
| napa:tmp $ dirgest.pl --quiet --trim=3 show +/usr/local/share/gnupg | = FEy7kEGM+dOntCi0Z/0KRg== 000000004492 share/gnupg/options.skel | # KnnYjjTVmGImyBUxLdWmBg==
| napa:tmp $ dirgest.pl --quiet --trim=4 show +/usr/local/share/gnupg | = FEy7kEGM+dOntCi0Z/0KRg== 000000004492 gnupg/options.skel | # ZEJoUs+1ZIxgsosL+xR1rg==
| napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ dirgest.pl --quiet show +/usr/local/bin | wc -l | 247
| napa:tmp $ dirgest.pl --quiet --nodetails show +/usr/local/bin | wc -l | 1
| napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ dirgest.pl --filename=dirgests_bin.txt create \ | +/usr/local/bin
| Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | including /usr/local/bin
| CREATING
| computing from /usr/local/bin
| writing to dirgests_bin.txt
| napa:tmp $
|
| ------------------------------------------------ |
| napa:tmp $ mkdir temp
| napa:tmp $ echo 'foo' > temp/a
| napa:tmp $ echo 'bar' > temp/b
| napa:tmp $ echo 'soa' > temp/c
| napa:tmp $ echo '+temp' > dirgest.conf | napa:tmp $ cat dirgest.conf
| +temp
| napa:tmp $
|
| napa:tmp $ dirgest.pl --configure=dirgest.conf --filename=dirgest.txt \ | create
| Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | configuring from dirgest.conf
| including temp
| CREATING
| computing from temp
| writing to dirgest.txt
| napa:tmp $
|
| napa:tmp $ cat dirgest.txt
| = 07BzhNET7exJ6qYjitX/AA== 000000000004 temp/a | = wVenkDHhxA+FkxgpvF/FUg== 000000000004 temp/b | = VByjna3FKqRpCn/pBcTNBA== 000000000004 temp/c | # eDwrWidm1bqhudQSNugM1A==
| napa:tmp $
|
| napa:tmp $ ls -lagt temp
| total 10
| drwxrwxrwt 6 root wheel 512 Oct 21 20:36 .. | drwxr-xr-x 2 matt wheel 512 Oct 21 20:35 . | -rw-r--r-- 1 matt wheel 4 Oct 21 20:35 c | -rw-r--r-- 1 matt wheel 4 Oct 21 20:35 b | -rw-r--r-- 1 matt wheel 4 Oct 21 20:35 a | napa:tmp $
|
| napa:tmp $ echo 'woa' >temp/c
| napa:tmp $ dirgest.pl --configure=dirgest.conf --filename=dirgest.txt \ | compare
| Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | configuring from dirgest.conf
| including temp
| COMPARING
| computing from temp
| reading from dirgest.txt
| comparing digests
| ! Pou4+9WzglcSZ/1fIE4SZA== 000000000004 temp/c | comparing summaries
| ? eDwrWidm1bqhudQSNugM1A==
| comparing differences: 2
| napa:tmp $
|
| napa:tmp $ dirgest.pl --quiet --configure=dirgest.conf \ | --filename=dirgest.txt compare | ! Pou4+9WzglcSZ/1fIE4SZA== 000000000004 temp/c | ? eDwrWidm1bqhudQSNugM1A==
| napa:tmp $
|
| napa:tmp $ dirgest.pl --quiet --show_equal --configure=dirgest.conf \ | --filename=dirgest.txt compare | = 07BzhNET7exJ6qYjitX/AA== 000000000004 temp/a | = wVenkDHhxA+FkxgpvF/FUg== 000000000004 temp/b | ! Pou4+9WzglcSZ/1fIE4SZA== 000000000004 temp/c | ? eDwrWidm1bqhudQSNugM1A==
| napa:tmp $
|
| napa:tmp $ dirgest.pl --quiet --nodetails --configure=dirgest.conf \ | --filename=dirgest.txt update | ? eDwrWidm1bqhudQSNugM1A==
| napa:tmp $
|
| napa:tmp $ dirgest.pl --quiet --nodetails --configure=dirgest.conf \ | --filename=dirgest.txt update | napa:tmp $
|
| napa:tmp $ cat dirgest.txt
| = 07BzhNET7exJ6qYjitX/AA== 000000000004 temp/a | = wVenkDHhxA+FkxgpvF/FUg== 000000000004 temp/b | = Pou4+9WzglcSZ/1fIE4SZA== 000000000004 temp/c | # RG0xlEsyo9W4xLArAbSLlg==
| napa:tmp $
|
| ------------------------------------------------

Example 2: cron-based periodic verification of configuration

files on local OpenBSD host (CLI scripts).

a trusted host is used to mount the host drive, the trusted host has console access only, and contains dirgest and configuration in CVS repository.
weekly cron job executes the following:
- extract DIRGEST from CVS repository
- run dirgest.pl in update mode with show option to update DIRGEST - pipe the output to mail and send to administrator + summary to syslog;
- checkin DIRGEST to the CVS repository
manual script allows operator to update DIRGEST from the console.

(hypothetical example)

Example 3: pre-replication verification of web-site upload, and

ongoing cron-based verification, similar to pre/post remote copy verification (CLI and CGI script interaction).

a build script generates the website using Template Toolkit, which involves creating content in 'build' from a 'source' directory.

| ------------------------------------------------ | napa:www $ ./b
| BUILDING
| updating ...
| building ...
| ------------------------------------------------

b. an update script first updates the dirgests and then uses a replication script to transfer the updated contents of the web site, and then verifies the contents of the web site by fetching from the cgi interface.

| ------------------------------------------------ | napa:www $ cat u
| #!/bin/sh
|
| echo UPDATING
|
| EXE_DIRGEST="/usr/local/bin/dirgest.pl" | EXE_REPLICATE="/usr/local/bin/replicate.pl" |
| [...]
|
| echo updating dirgest ...
| rm -f build/cgi-bin/dirgest/dirgest.txt | $EXE_DIRGEST \
| --filename=build/cgi-bin/dirgest/dirgest.txt --trim=1 \ | create \
| -CVS `sed 's,/.*public_html,build,g' < build/cgi-bin/dirgest/dirgest.conf` |
| [...]
|
| echo uploading ...
| $EXE_REPLICATE \
| --root=public_html \
| --source=/home/www/build \
| --not=CVS \
| ...
|
| [...]
|
| echo verifying ...
| $EXE_DIRGEST \
| --fetch=http://$HOSTNAME/cgi-bin/dirgest/dirgest.cgi \ | --login=$USERNAME:$PASSWORD \
| --trim=4 \
| --filename=build/cgi-bin/dirgest/dirgest.txt \ | compare
|
| [...]
|
| ------------------------------------------------

c. the update process results in the following output:

| ------------------------------------------------ | napa:www $ ./u
| UPDATING
| updating dirgest ...
| Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | including build
| excluding CVS
| excluding build/cgi-bin/dirgest/dirgest.txt | excluding build/cgi-bin/perlfect/search/data/sizes | excluding build/cgi-bin/perlfect/search/data/content | excluding build/cgi-bin/perlfect/search/data/terms | excluding build/cgi-bin/perlfect/search/data/titles | excluding build/cgi-bin/perlfect/search/data/docs | excluding build/cgi-bin/perlfect/search/data/desc | excluding build/cgi-bin/perlfect/search/data/urls | excluding build/cgi-bin/perlfect/search/data/dates | excluding build/cgi-bin/perlfect/search/data/inv_index | excluding build/cgi-bin/perlfect/subscribe/mailing_list.txt | CREATING
| computing from build
| writing to build/cgi-bin/dirgest/dirgest.txt | uploading ...
| [...]
| scanning remote directory: public_html | [...]
| verifying ...
| Directory Digest CLI v0.90 -- Matthew Gream <matthew.gream@pobox.com> | Copyright 2002 Matthew Gream. All Rights Reserved. | COMPARING
| reading from build/cgi-bin/dirgest/dirgest.txt | fetching from http://matthewgream.net/cgi-bin/dirgest/dirgest.cgi | comparing digests
| comparing summaries
| comparing differences: 0
| napa:www $ wc -l build/cgi-bin/dirgest/dirgest.txt | 197 build/cgi-bin/dirgest/dirgest.txt | napa:www $ head -3 build/cgi-bin/dirgest/dirgest.txt | = sxNqlix47RIWoX9sDjC5wA== 000000000002 .cvsignore | = tft3saNku+rCpgI0jnht9w== 000000000041 .htaccess | = ym+NbPCKfVB2zv1UT/AhCw== 000000000146 cgi-bin/accesslog/.htaccess | napa:www $
| ------------------------------------------------

d. a periodic cron job runs to call dirgest.pl against dirgest.cgi (on the server) to verify that the website content has not changed. the output is used in a report to the administrator to check for changed site content.

| ------------------------------------------------ | napa:www $ cat /usr/local/share/site/site_check | #!/bin/ksh
| ################################################################# | # $Id: README,v 0.90 2002/10/21 20:24:06 matt Exp matt $ | ################################################################# |
| [...]
|
| EXE_DIRGEST="/usr/local/bin/dirgest.pl" | DIRGESTOPTS="--quiet"
|
| [...]
|
| $EXE_DIRGEST $DIRGESTOPTS \
| --fetch=http://$HOSTNAME/cgi-bin/dirgest/dirgest.cgi \ | --login=$USERNAME:$PASSWORD \
| --trim=4 \
| --filename=$WWW_BUILD/cgi-bin/dirgest/dirgest.txt \ | compare
|
| [...]
|
| napa:www $
| ------------------------------------------------

Example 4: verification of tar/zip/archive contents, such as

with backup, or distribution, or transfer (CLI scripts). a wrapper script used to provide the operation. Note that this is not cryptographically strong and requires use of some form of digital signature for more robust authentication.

archive_create script does the following:
- stores archive name + files
- uses dirgest.pl across files to create a DIRGEST
- uses tar with DIRGEST and files to form archive
archive_extract script does the following:
- uses tar to extract files and DIRGEST
- uses dirgest.pl to verify DIGEST against files
- removes DIRGEST now that operation is complete

(hypothetical example)

Example 5: use of Digest::Directory::BASE.pm as part of packaging

script for software distribution.

(hypothetical example)

INSTALLATION

To install this module type the following:

perl Makefile.PL
make
make test
make install

The following modules will be created:

Digest::Directory::BASE.pm (low level mechanisms) Digest::Directory::API.pm (higher level interface)

The following scripts will be created:

dirgest.pl (Command Line Interface)
dirgest.cgi (Common Gateway Interface)

DEPENDENCIES

This module requires these other modules and libraries:

Perl 5.004
LWP::UserAgent (for Digest::Directory::BASE.pm) Digest::MD5 (for Digest::Directory::BASE.pm) File::Find (for Digest::Directory::BASE.pm) CGI (for dirgest.cgi)
Getopt::Long (for dirgest.pl)

ISSUES

No known issues are outstanding.

HISTORY

Version 0.10 - 20 Sep 2002 - Alpha release (internal). Version 0.90 - 20 Oct 2002 - Beta release (CPAN).

FUTURE

Candidate improvements are:

increased configuration file flexibility.
use hash for method arguments.
include/exclude does better regexp style matching.
integrate 'load' and 'fetch' via. LWP to be seamless.
remove static's in compute method (is this possible?).
allow per-directory dirgest files (e.g. '.dirgest').
include contrib/example scripts.
support file acl to better support ids style operation.

Suggestions (& patches) warmly welcomed.

AUTHOR

Matthew Gream (MGREAM) <matthew.gream@pobox.com>

RIGHTS

This distribution is made available under the same terms and conditions as the perl distribution. The author offers no express or implied warranties as to fitness for use. Your use of this software implies acceptance of these terms and conditions.