Index
NAME
Gungho::Component::RobotRules - Respect robots.txt
SYNOPSIS
---
components:
- RobotRules
METHODS
setup
setup_robot_rules_parser
setup_robot_rules_storage
handle_response
send_request
allowed($request)
Returns 1 if request is allowed to be fetched, 0 if not. -1 and -2 are returned when there is a pending request to fetch /robots.txt
dispatch_pending_robots_txt
Dispatches requests that were pending because of a missing robots.txt entry
push_pending_robots_txt
Pushes a request in the wait queue for a robots.txt
parse_robot_rules
Parses the robot rule and stores it