A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler' Permissions Checker
Documentation for package ‘robotstxt’ version 0.1.2
DESCRIPTION file
.
User guides, package vignettes and other documentation.
Help Pages
get_robotstxt
downloading robots.txt file
parse_robotstxt
function parsing robots.txt
print.robotstxt_text
printing robotstxt_text
robotstxt
An object representation of robots.txt files