Test::A8N - Storytest Automation Runner

Test-A8N documentation Contained in the Test-A8N distribution.

Index

Code Index:

NAME

Top

Test::A8N - Storytest Automation Runner

SYNOPSIS

Top

    my $a8 = Test::A8N->new({
        filenames => [qw( cases/test1.tc )],
        file_root => 'cases',
    });
    $a8->run_tests();

DESCRIPTION

Top

Test::A8N was created as a mechanism for writing and running automated storytests in a human-readable and reusable fashion. Its storytest files are easily readable, are natural to write, and are easy for non-technical users to read, while still being easy for developers to automate.

It works by leveraging Test::FITesque to describe test fixtures in a list style, while providing syntatic sugar for test authors. The tests themselves are written in YAML, and while they have a specific structure, it doesn't limit test author's flexibility. And many of the features of YAML, most notably its concept of creating pointers between different parts of a document, means reusing parts of your tests for use elsewhere is trivial.

Testcase Syntax

An A8N testcase file can consist of mulitple YAML documents, separated by three dashes ---. But the structure for each testcase within a file is the same. It consists of a name, a summary, an optional ID, test instructions, and optional preconditions and postconditions.

The instructions, preconditions, and postconditions contain a list of steps that are to be run as part of the test.

    ---
    NAME:    Administrator changes their timezone
    ID:      admin_changes_tz
    SUMMARY: Administrators need to have a mechanism for changing their
             timezone.
    PRECONDITIONS:
        - ensure user exists: admin
        - ensure timezone is: America/Vancouver
    INSTRUCTIONS:
        - login:
            username: admin
            password: testpass
        - goto page: Account Settings
        - verify current timezone is: America/Vancouver
        - change timezone to: Australia/Brisbane
        - verify current timezone is: Australia/Brisbane
    POSTCONDITIONS:
        - ensure timezone is: America/Vancouver

Despite the actual order specified in a testcase block, each testcase will run its tests in the order of 1) precondition, 2) instructions, 3) postconditions. If you don't specify an ID, then one will be auto-generated for you based on the testcase name.

Goals of this Testcase Format

When a product manager or some other non-technical business user comes up with a set of initial story tests for some feature, most people will naturally think of writing a list. Even describing how to access some feature, people will fall back to describing a list of steps. We therefore set out to capture that as closely as possible in our tests.

As much as possible, we recommend making your fixture calls, and their arguments, read as much as possible as english phrases, without having unnecessary "no-op" filler that will get in the way.

Testcase Fixtures and Suggestions

From the sample testcase described at the beginning of this section, you can see just how natural this method of writing tests is. The tests don't even need to be automated to be able to follow them, as they're readable enough for a user to navigate a website or command-line to be able to follow the specified steps.

However, once you do automate them, there are some subtleties that we've found work quite well in authoring test cases. Namely in there's a difference between performing an action, and testing the result. For instance:

    - change timezone to: <some timezone>
    - verify timezone is: <some timezone>
    - ensure timezone is: <some timezone>

"change" can be used to set some value using the UI you're testing. "verify" can be used to test that some value is set or is present. "ensure" is a subtle one, but it can be used to change a value only if it isn't already present. It's useful to use in preconditions where you don't care to exercise the UI every time you need to add a user, for instance. If you need a user account simply for testing purposes, then your ensure user exists action can simply drop a user account into a database or onto disk without using the UI. This not only speeds up your tests, but makes sure that you only test user creation within your user account tests, not on every test that requires a user to be present.

METHODS

Top

Accessors

fixture_base

Specifies the base fixture classname to use when running testcases. When test files are found within sub-directories of file_root, the directory names are converted to class names, and appended to this fixture_base value.

file_root

Indicates where your testcase files live. This is important because any directory below this point is assumed to be part of the fixture class name.

filenames

The list of test files you wish Test::A8N to run. If none are specified, the test runner will try to find all files under the file_root that has an extension in the allowed_extensions list.

verbose

Turns on increasing amounts of debugging output. All debug messages are prefixed with a "#" so that it doesn't interfere with TAP output.

Default: 0

allowed_extensions

Specifies what file extensions are valid testcases. In this way you can mix story tests and unit tests within the same directory.

Default: "st", "tc"

testcase_id

Specifies a testcase ID you wish to run. If unset, it will run all testcases in test files.

Object Methods

run_tests

Calls run_tests in Test::A8N::File() in all the Test::A8N::File objects returned by files.

file_paths

Returns a list of paths to all the testcase files that are to be processed. If nothing is specified in filenames, or it contains directories, then file_paths will search in those sub-directories to find any available testcase files.

files

Returns the contents of file_paths as instances of the Test::A8N::File class.

SEE ALSO

Top

Test::A8N::File, Test::FITesque

AUTHORS

Top

Michael Nachbaur <mike@nachbaur.com>, Scott McWhirter <konobi@cpan.org>

LICENSE

Top

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.7 or, at your option, any later version of Perl 5 you may have available.

COPYRIGHT

Top

Copyright (C) 2008 Sophos, Plc.

Test-A8N documentation Contained in the Test-A8N distribution. 
package Test::A8N;
use warnings;
use strict;

# NB: Moose also enforces 'strict' and warnings;
use Moose;
use Test::FITesque::Suite;
use Test::FITesque::Test;
use Test::A8N::File;
use File::Find;
use Storable qw(dclone);

our $VERSION = '0.06';

sub BUILD {
    my $self = shift;
    my %defaults = (
        fixture_base       => "Fixture",
        file_root          => "cases",
        filenames          => [],
        verbose            => 0,
        allowed_extensions => [qw( tc st )],
        tags               => {
            include => [],
            exclude => [],
        },
    );
    foreach my $key (keys %defaults) {
        next if exists $self->config->{$key};
        $self->config->{$key} = $defaults{$key};
    }
}

has config => (
    is          => q{ro},
    required    => 1,
    isa         => q{HashRef}
);

my %default_lazy = (
    required => 1,
    lazy     => 1,
    is       => q{ro},
    default  => sub { die "need to override" },
);

has verbose => (
    %default_lazy,
    isa     => q{Int},
    default => sub { return shift->config->{verbose} },
);

has testcase_id => (
    %default_lazy,
    isa     => q{Str},
    default => sub { return shift->config->{testcase_id} || '' },
);

has filenames => (
    %default_lazy,
    isa     => q{ArrayRef},
    default => sub { return shift->config->{filenames} },
);

has file_root => (
    %default_lazy,
    isa     => q{Str},
    default => sub { return shift->config->{file_root} },
);

has fixture_base => (
    %default_lazy,
    isa     => q{Str},
    default => sub { return shift->config->{fixture_base} },
);

has allowed_extensions => (
    %default_lazy,
    isa     => q{ArrayRef},
    default => sub { return shift->config->{allowed_extensions} },
);

has file_paths => ( 
    is       => q{ro}, 
    required => 1, 
    lazy     => 1, 
    isa      => q{ArrayRef},
    default => sub { 
        my $self = shift;
        my @file_list = ();
        my $wanted = sub {
            my $filename = $File::Find::name;
            for my $extension (@{$self->allowed_extensions}) {
                if (-f and /^[^\.].*\.$extension$/) {
                    push @file_list, $filename;
                }
            }
        };
        my $root = $self->file_root;
        my @files = scalar(@{ $self->filenames() }) ? @{ $self->filenames() } : ($root);
        find($wanted, @files);
        return \@file_list;
    }
);

has files => ( 
    is => q{ro}, 
    required => 1, 
    lazy => 1, 
    default => sub { 
        my $self = shift;
        my @files = ();
        for my $filename ( @{ $self->file_paths } ) {
            push @files, Test::A8N::File->new({
                filename => $filename,
                config   => dclone( $self->config ),
            });
        }
        return \@files;
    }
);

sub run_tests {
    my $self = shift;
    my $id = $self->testcase_id;
    my $suite = Test::FITesque::Suite->new();

    my $test_count = 0;
    foreach my $file (@{ $self->files }) {
        my @cases = @{ $file->filtered_cases( $id ) };
        foreach my $case (@cases) {
            my @data = @{ $case->test_data };
            my $test = Test::FITesque::Test->new({
                data => [ 
                    [$file->fixture_class, { testcase => $case } ], 
                    @data 
                ]
            });
            $suite->add($test);
            $test_count++;
        }
    }
    $suite->run_tests() if $test_count > 0;
}

# unimport moose functions to make immutable
no Moose;
__PACKAGE__->meta->make_immutable();
1;