Moinsde2Euros Posted June 8, 2013 Share Posted June 8, 2013 Hi If someone can help me out, that would be just wonderful. I was looking through the pages that people have tried to access but they come across the 404 error. One of the pages that they have tried to access is the robots.txt page. What is this page? Please if some out there knows what it is, and how to fix it, please let me know. Many thanks Link to comment Share on other sites More sharing options...
vekia Posted June 8, 2013 Share Posted June 8, 2013 The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites. more about robots.txt you can read here: https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt Link to comment Share on other sites More sharing options...
Moinsde2Euros Posted June 8, 2013 Author Share Posted June 8, 2013 I'm rather a novice at this. Is this because I don't have a www. at the beginning of my website? moinsde2euros.com I have tried changing it to have the www. but I just can't do it. People have been trying to access http://moinsde2euros.com/robots.txt but I have no idea what it is. Please help! Link to comment Share on other sites More sharing options...
vekia Posted June 8, 2013 Share Posted June 8, 2013 it is not people trying to open the file, but crawlers like google robot. This file is for search robots (crawlers), not for people you have to create it manually in bo. more informations in the link that i pasted before. Link to comment Share on other sites More sharing options...
El Patron Posted June 8, 2013 Share Posted June 8, 2013 try this, back office-->seo&url's scroll to the bottom now create your robot.txt file http://screencast.com/t/gli9eq3m1E Link to comment Share on other sites More sharing options...
vekia Posted June 8, 2013 Share Posted June 8, 2013 that's correct :-) 1 Link to comment Share on other sites More sharing options...
.png.022b5452a8f28f552bc9430097a16da2.png)
Recommended Posts