Andy1 Posted September 9, 2011 Share Posted September 9, 2011 Hi, I have just logged into my Google Webmaster Account and I notice I have the following crawl errors: In Sitemaps = 1 Not Found = 33 Restricted by robots.txt = 21 The 'Not Found' errors I can deal with as these are pages I have removed from the site, but need to use the URL removal tool to removed them from Google. I don't know why I have an error in the sitemap, and I am very concerened about the 21 pages restricted by the robots.txt file. I thought you only had a robot.txt file if you wanted google to ignore certain pages. I haven't set the robots.txt file so not sure why this has happened. Within the 21 pages are some products and also the manufactures list etc Does anyone know how I can remove the crawl errors from 'Restricted by robots.txt' and the 'In Sitemaps' Many Thanks Andy The 'Not Found' errors I can deal with as these are pages I have removed from the site, but need to use the URL removal tool to removed them from Google. I don't know why I have an error in the sitemap, and I am very concerened about the 21 pages restricted by the robots.txt file. I thought you only had a robot.txt file if you wanted google to ignore certain pages. I haven't set the robots.txt file so not sure why this has happened. Within the 21 pages are some products and also the manufactures list etc Does anyone know how I can remove the crawl errors from 'Restricted by robots.txt' and the 'In Sitemaps' Many Thanks Andy Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now