A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker


[Up] [Top]

Documentation for package ‘robotstxt’ version 0.1.2

Help Pages

get_robotstxt downloading robots.txt file
parse_robotstxt function parsing robots.txt
print.robotstxt_text printing robotstxt_text
robotstxt An object representation of robots.txt files