greglindahl Posted February 3, 2011 Share Posted February 3, 2011 We received a complaint that our crawler was accessing the shopping cart of a PrestaShop website.The issue is that the robots.txt had some blank lines in it. According to the robots.txt standard, a blank line begins a new section, and cancels the previous User-Agent line.The webmaster thought that the robots.txt was auto-generated by PrestaShop.Can this be fixed?Example: User-agent: * # Directories Disallow: /classes/ Disallow: /config/ The comment is OK, but the blank line means that the Disallow lines have no effect on any crawler.-- greg Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now