Scraper - README

WWW::Search::Scraper
WWW::Search::Sherlock
WWW::Search::Scraper::*
WWW::Search::Scraper::Request
WWW::Search::Scraper::Response

DEPENDENCIES -

        WWW::Search
        Tie::Persistent
        Storable

These modules scrape data from search engines on the WWW (much like Apple's Sherlock, but these are more capable, complete, and accurate.)

Version 2.00 is a major departure from versions 1.xx

Search engies are classified by type
1. Job
2. Apartments
3. Auction
4. etc.
Queries are translated from a single canonical form to each search engine
by that engine's Scraper module. For example,
1. A single location property is translated to the numerous coding and taxonomy systems of the various engines.
2. A single price (or range) is matched to the closest price (range) of each search engine.
Post-filtering base on the results page, and on detail pages, via Perl coding.
Retains ease of framing the results page, and extends that to framing the detail page.
Backward compatible.

Complete documentation can be found in WWW::Search::Scraper and WWW::Search::Scraper::Sherlock.

Special options for each type of search engine are documented in their respective modules.

Examples include:

SearchApartments - illustrates how to easily set up and use one search engine.
Sherlock - illustrates the ease of adapting any Sherlock plugin.
Scraper - illustrates how SearchResult sub-classing can be used to build
a more generalized search engine scraper. You can see how this can be extended to build a multi-engine scraper.

If you want to write new Scraper modules to access new search engines, see Brainpower.pm for the current "best practices".

Happy Hunting!

--------------------------------------------------------------------- $VERSION = '2.27'

See "Changes" file for history of changes.

KNOWN PROBLEMS

theWorksUSA.pm more often than not goes into a loop. techies.pm keeps saying "Please enable your cookies", so it doesn't work at all.