printerink Posted January 25, 2014 Share Posted January 25, 2014 Hello to all, I am not exactly sure if it is OK or not.., (I assume not) but with my site Google has indexed pages such as "mysite/cache/" and "mysite/cache/smarty/ " and "mysitel/cache/smarty/compile/" If anyone out there can let me know if this is OK for SEO purposes (or not) and if not how to prevent Google indexing these pages. Thank you. Link to comment Share on other sites More sharing options...
prestashopninja Posted January 26, 2014 Share Posted January 26, 2014 (edited) Hi, I'm not sure why and how googlebot indexes the cache files and the most sane way would be to find the reason and fix it, but here's how to stop googlebot: Create a file called "robots.txt" with the following content and upload it to your main Prestashop installation folder: User-agent: Googlebot* Disallow: /cache/ Edited January 26, 2014 by prestashopninja (see edit history) Link to comment Share on other sites More sharing options...
El Patron Posted January 26, 2014 Share Posted January 26, 2014 Hi, have you done this yet? This would build the correct robots.txt file for vanilla ps. http://screencast.com/t/lEvzZmZvP1 Link to comment Share on other sites More sharing options...
printerink Posted January 26, 2014 Author Share Posted January 26, 2014 Hi and thanks for your answers. I originally generated the robots.txt in the back office but for whatever reason it does not include Disallow:/cache/ in it. I have also "a while back" put the below in my robots.txt as prestashopninja has suggested but still weeks later these URLs are still in the Google index. User-agent: Googlebot* Disallow: /cache/ Link to comment Share on other sites More sharing options...
Recommended Posts