get_robotstxt | downloading robots.txt file |
guess_domain | function guessing domain from path |
parse_robotstxt | function parsing robots.txt |
paths_allowed | check if a bot has permissions to access page(s) |
print.robotstxt | printing robotstxt |
print.robotstxt_text | printing robotstxt_text |
remove_domain | function to remove domain from path |
robotstxt | Generate a representations of a robots.txt file |
rt_cache | get_robotstxt() cache |