Text::SmartLinks - connecting test files with pod documentation

  smartlinks.pl t/*/*.t t/*/*/*.t
  smartlinks.pl --dir t
  smartlinks.pl --css foo.css --out-dir=public_html t/syntax/*.t
  smartlinks.pl --check t/*/*.t t/*/*/*.t
  smartlinks.pl --check t/some/test.t
  smartlinks.pl --missing t/*/*.t t/*/*/*.t

If in the root directory of a CPAN package type the following:

  smartlinks.pl --pod-dir lib/ --dir t/ --out-dir html/ --index

In the root of Text::SmartLinks type in the following:

  perl -Ilib script/smartlinks.pl --pod-dir lib/ --dir t/ --out-dir html/ --index

The plan is to change the Text::SmartLinks module and write a new smartlinks.pl script so it will be usable in any Perl 5 or Perl 6 project to generate the HTML pages combining the POD content from the .pod and .pm files and test scripts.

In addition the script should be able to generate further reports in HTML format that help the developers.

The usage should default to parsing the files in lib/ for documentation and the .t files in the t/ subdirectory.

Process both Perl 5 and Perl 6 test files in an arbitraty directory to collect smartlinks. Default should be either the local t/ directory or the t/spec directory of Pugs (for historical reasons).

Process .pod and .pm files (but maybe other files as well) with either Perl 5 or Perl 6 pod in them and with possibly also code in them.

Smartlinks should be able to say the name of the document where they link to.

    L<Smolder/Text of head1>
    L<Smolder::Util/Text o head2>

Map to either Smolder.pm or Smolder.pod and Smolder/Util.pm or Smolder/Util.pod

Need special cases for the Perl 6 documentation so the smartlinks can have the following links pointing to S06-routines.pod and S32-setting-library/Abstraction.pod

    L<S06/Blocks>
    L<S32::Abstraction>

• This script should have as few non-core module dependencies as possible.
• One doesn't have to build pugs so as to run smartlinks.pl. Of course, optional advanced features may require the user to run pugs' "make" or even "make smoke".

Smartlinks are planted in the test file, and are pointed to the appropriate sections of the Synopsis you are using to write the test.

They look like pod links:

    L<S06/Blocks>            # "S06" is synopsis 6, and "Blocks" is the section
    L<S03/"Hyper operators"> # quotes can be used when spaces are in the title,
                             # but is NOT required.
    L<S03/Hyper operators>   # just fine

The section name should be copied verbatim from the POD (usually after =head), including any POD tags like ... and punctuations. The sections, however, are not supposed to be nested. That is, a =head1 won't really contain a =head2; they're disjoint according to the current implementation.

The smartlinks also have a weird (also important) extension: you can specify some keyphrases, to skip forward from the linked section, so the smartlink is put into a more specific place:

    L<S05/"Return values from matches"/"In numeric context" number 'matches:'>

The above smartlink is appropriate next to a test case checking rule application in numeric context, and it will place the backlink appropriately.

All the keyphrases listed after the second slash in a smartlink should appear in a single sentence from the synopsis text, and the order is significant. If there're spaces in a keyphrase, quote it using either double-quotes or signle-quotes.

In contrast with the case of section name, you should never use POD tags like ... in a keyphrase. util/smartlinks.pl will do the right thing. You can use, however, pod directives in the keyphrases, just like this:

    # L<S04/Str/"=item split">

Smartlinks in .t files can be preceded by nothing but spaces or "#", furthermore, there should be no trailing text on the same line, otherwise they can't be recognized by tools. Here're some *invalid* samples:

    # the following smartlink is INVALID!!!
    # Link is L<S04/Str>

    # the following smartlink is INVALID TOO!!!
    # L<S04/Str> # This is a comment

There's also a variant for the smartlink syntax:

   # L<syn/sec/key phrases>>

A smartlink can span at most 2 lines:

   # L<S04/section name/key1
   #   "key2" key3 key4>

Only the keyphrase list part can continue to the next line. So the following example is invalid:

   # L<S04/section
   #   name/blah blah blah>      # WRONG!!!

Please don't put a smartlink in the middle of a group of tests. Put it right *before* the group of tests it is related to.

Multiple adjacent smartlinks can share the same snippet of tests right below them:

    # L<S02/Context/boolean "?">
    # L<S03/Changes to Perl 5 operators/"?" imposes boolean context>
    { ... }

By doing this, one can effectively link one group of tests to multplie places in the Synopses, leading to m-to-n correspondance.

smartlinks.pl can take care of this kind of special cases.

You can put a URL to create a generic link:

  L<"http://groups.google.de/group/perl.perl6.language/msg/07aefb88f5fc8429">

or without quotes:

  L<http://www.nntp.perl.org/group/perl.perl6.language/26071>

To see some examples, or look at the *.t files in the t/ directory of this project.

There were also some legacy smartlinks using the following syntax:

   L<S04/"section name" /regex/>
   L<<S04/"section name" /regex/>>
   L<<S04/"section name">>

They're no longer supported by util/smartlinks.pl. Use the current syntax.

1.

We scan over all the specified .t files; collect smartlinks and positional info about the test code snippets as we go. When all these work have been finished, we obtain a tree structure, which is named $linktree in the source code.

To make this tree minimal, we only store the .t file name and line numbers, rather than the snippets' source code itself.

The structure of $linktree is like this:

    {
      'S12' => {
        'Traits' => [
          [
            undef,
            [
              't/oo/traits/basic.t',
              '13',
              '38'
            ]
          ],
          [
            '/If you say/',
            [
              't/oo/delegation.t',
              '56',
              '69'
            ]
          ],
        ],
      },
      'S02' => {
        'Whitespace and Comments' => [
          [
            '"Embedded comments" "#" plus any bracket',
            [
              't/syntax/comments.t',
              10,
              48
            ]
          ],
        ]
      }
    }

This step is mostly done by sub process_t_file.

2.

We process the synopsis .pod files one by one and generate HTML files integrated with test code snippets using the $linktree structure discussed above.

This is mostly done by sub process_pod_file.

Because it is an enormous step, we can further divide it into several sub steps:

• We parse each .pod into a tree, which is known as $podtree in the source code. (See sub parse_pod.)
  
  The structure of $podtree looks like this:
  
  

      {
        'Names and Variables' => [
          '=over 4' . "\n",
          '=item *' . "\n",
          'The C<$Package\'var> syntax is gone.  Use C<$Package::var> instead.' . "\n",
          '=item *' . "\n",
          'Perl 6 includes a system of B<sigils> to mark the fundamental' . "\n".
              'structural type of a variable:' . "\n",
          ...
        ],
        ...
      }
  
  

• We look up every related smartlink from every $podtree, generate .t code snippets along the way, and insert placeholders (like "_SMART_LINK_3" into the corresponding $podtree. (See subs parse_pattern, process_paragraph, and gen_code_snippet.)
• Now we emit Pod source back from the modified $podtree's. (See sub emit_pod.)
• After that, we generate HTML source from the Pod source with snippet placeholders using Pod::Simple::HTML. (See sub gen_html.)
• At last, we replace every snippet placeholders in the HTML source with the real snippet code (also in HTML format).

• t/README in the Pugs source tree.
• The articles on the Pugs blogs:
  
  http://pugs.blogs.com/pugs/2006/08/integrating_the.html
  
  http://pugs.blogs.com/pugs/2006/09/check_smoke_res.html
  
  http://pugs.blogs.com/pugs/2006/09/the_benefits_of.html
• The synopses in http://perlcabal.org/syn are generated by this script.

Constructor, can get a HASH reference as it is a base class of Class::Accessor

Gets a list of .t test files, calls process_t_file on each on of them.

Gets a path to a .t file, reads line by line and collects the smartlinks in it to a hash structure using the add_link function.

Convert patterns used in 00-smartlinks.to perl 5 regexes

Process paragraphs of the pod file: unwrap lines, strip POD tags, and etc.

Gets a triplet of [file, from, to] and generates an HTML snippet from that section of the given file.

Note that this function has been optimized for space rather than time.

Returns the content of the smartlink.js file. Probably we should just copy the .js file to the html directory and not embed it.

link_count_inc increments the link counter.

link_count returns the current number of links.

Agent Zhang (<agentzh@gmail.com>) wrote the initial implementation, getting help from many others in the Pugs team.

Current maintainer: The Pugs team

Copyright (c) 2006 - 2009 by the Pugs Team.

Text::SmartLinks is free software; you can redistribute it and/or modify it under the terms of the Artistic License 2.0. (Note that, unlike the Artistic License 1.0, version 2.0 is GPL compatible by itself, hence there is no benefit to having an Artistic 2.0 / GPL disjunction.)