When InSite tries to crawl my site I get “Error: 401 – Unauthorized”. What’s wrong?

Question: When InSite tries to crawl my site I get “Error: 401 – Unauthorized (Is this page password protected?): Check the Root URL and configure any password protected pages as necessary.”

Since there is no password protection on my server, I’m stumped.

Answer: If your webserver is Microsoft IIS, try using the “HTTP” authentication method. Let’s assume that your Root URL is www.example.com. Click ‘Edit Protected Pages’ to open the password protected pages dialog. In that window, enter ‘Relative Path’ value of “/”, and set the “Method” to HTTP.

From there, enter your Windows user credentials (username and password) in the username and password fields, and optionally your Windows domain (if necessary). Click ‘Add Page’, then ‘OK’. This rule will force InSite to pass the login credentials when it receives the authorization request from your webserver. Internet Explorer does this “behind the scenes” for you, but InSite does not.

Alternatively, it’s possible to enable “Allow Anonymous Access” in your (virtual) directory settings before InSite will be granted permission to crawl your server.