Gungho::Component::RobotsMETA - Automatically Parse Robots META

  components:
    - RobotsMETA

This module automatically parses any text/html document for robots exclusion directies embedded in the document.

Initializes the component.

Overrides Gungho::Component::Core::handle_response()

HTML::RobotsMETA