Does the Inspyder Crawler obey Robots.txt?

Question: Does the Inspyder Crawler obey Robots.txt?

Answer: By default, no, the Inspyder Crawler does not obey robots.txt. Our website analysis tools are best run across entire websites. If our software did not crawl entire sites, the results would be erroneous or incomplete.

It is possible to import a robots.txt into the Excluded Pages list. Simply click the “Import Robots.txt…” action on the left hand side of the Excluded Pages window.