Attention, PrestaShop may harm Google rankings - action required

[mountarreat]

I have recently updated from 1.6 to 1.7.4.3. You can read about what went wrong, including solutions to all problems encountered, here: https://www.prestashop.com/forums/topic/912572-problem-with-upgrade-from-1620-to-174/#comment-2989412

Now another problem. Today I got a little scared after an index-update because the Google search console suddenly told me that all my pages are no longer mobile-friendly, which they already were with 1.6. The classic Theme 1.7 looks good already, at least better and more mobile than the old 1.6. The problem is this ... by default, PrestaShop blocks important resources in the robots.txt with the lines

Disallow: */cache/

Disallow: */css/

Disallow: */js/

Unfortunately, this also includes '/themes/classic/assets/cache/', which is the resources that Google needs to have access to evaluate the layout. Otherwise, the page is only partially loaded. If the CSS and JS files, among others, can not be loaded, the pure HTML is assessed without styles. Clickable elements are suddenly too close together because they are too small, and the content may be wider than the screen. Boom, the page in the index is marked as not mobile-friendly, although it is actually. This could now have fatal long-term consequences in the Google Mobile-First-Index and should be fixed immediately.

Adding the lines

Allow: */themes/*/cache/*.css
Allow: */themes/*/cache/*.js
Allow: */themes/*/css/*.svg
Allow: */themes/*/css/*.ttf
Allow: */themes/*/css/*.woff
Allow: */themes/*/css/*.woff2
Allow: */themes/*/js/*.js

in the robots.txt again allows access to these files (theme styles, scripts, icons and fonts) and can prevent worse. For me it has restored the original state: Page is mobile-friendly.

If you're concerned, it's best to check with the Google tool itself: https://search.google.com/test/mobile-friendly

Keep in mind that after you uploaded the new robots.txt, it may take a while for Google to internalize it, so results are not immediately visible using the mobile-friendly tool. Give it 30 minutes, and check again later, to verify that your site now is mobile-friendly.

Edited October 25, 2018 by mountarreat (see edit history)

[vlskrbek]

Hi thanks for these information they are intrusting.

But I would like to ask.

Are you sure that these:

Allow: */themes/*/cache/*.css
Allow: */themes/*/cache/*.js

 

must be also allowed? I thought that the cache files are only for caching system and the search google should not take into account?

 

Best Regards

[mountarreat]

Yes, I am 100% sure, and it's nothing new actually. Google notified about this since 2015, this is why I am so surprised that my robots.txt was overwritten with such rules from version 1.7.4.3. They should know better. See the link below for a reference:

https://searchengineland.com/google-search-console-warnings-issued-for-blocking-javascript-css-226227

The themes' cache folder should be allowed because the minified CSS and JS files sit within that very folder.
Furthermore, I have limited it to allow only these certain types to be crawlable.

Check the official source at Google: https://webmasters.googleblog.com/2014/10/updating-our-technical-webmaster.html

Quote

Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.

 

It's known for about 4 years but didn't make it into the store default robots.txt yet.

Edited October 25, 2018 by mountarreat (see edit history)

[vlskrbek]

Thanks for information. I will try it.

(I was also surprised that our web pages are not mobile friendly)

[philcotton]

Thank you mountarreat for this information, I changed the robots.txt and Google can now read the css files. A very simple solution to a problem that has bugged me for months.

Phil

   

[Benny54]

I noticed that updates within version 1.7 also overwrite the robots.txt file. This iis very bad and Prestashop should repair this bug asap.

[vlskrbek]

I think the best will be to send a defect to prestashop. This is just a forum not defect management for developers of prestashop.

Did someone already send a defect to prestashop? If yes what they say?

[Emmyzdon]

On 10/25/2018 at 2:49 PM, vlskrbek said:

Hi thanks for these information they are intrusting.

But I would like to ask.

Are you sure that these:

Allow: */themes/*/cache/*.css
Allow: */themes/*/cache/*.js

 

must be also allowed? I thought that the cache files are only for caching system and the search google should not take into account?

 

Best Regards

Thanks for these information. Are these solutions same for Prestashop 1.6? If not, what is the right ones for 1.6?

[Benny54]

That's the problem that i faced. I added these 'allow' lines to the robots.txt file but after an update i found that robots.txt was overwritten. When you are aware, you can correct the issue. 

[millien]

you could just add this to your robots.txt , or if you just want google ( User-Agent: Googlebot ) 

User-agent: *
Allow: *.js
Allow: *.css

[kellyriverstone]

Huge THANK YOU for this! Finding a solution to this issue has been driving me crazy since upgrading to 1.7 a couple of years ago. I never dreamed of looking at the robots.txt blocking the style sheets! My god! Such a simple solution to a frustrating issue! Thank you again :)

[Nickz]

Robots texts have very little to no influence on ranking. Some directives are even disregarded.

In the GSC Google plays an open platform theater for us. You get messages with image to wide, and font to small bla bla bla every now and then even if your site is using a 16 px font. It is done to proof that Google works for you.