A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker


[Up] [Top]

Documentation for package ‘robotstxt’ version 0.3.2

Help Pages

get_robotstxt downloading robots.txt file
guess_domain function guessing domain from path
parse_robotstxt function parsing robots.txt
paths_allowed check if a bot has permissions to access page(s)
print.robotstxt printing robotstxt
print.robotstxt_text printing robotstxt_text
remove_domain function to remove domain from path
robotstxt Generate a representations of a robots.txt file
rt_cache get_robotstxt() cache