Prestashop dosn't allow google bots to crawl the page after upgrade to 1.7

[Skafander]

Hello, I did upgraded my application from Prestashop 1.6 on PHP 7.0 to Prestashop 1.7 PHP 7.4. Atter changing from developer webspace to production one, appeared a google crawler error wchich can't crawl items from data feeds.

Reported error is:

Update your robots.txt file to allow user-agents "Googlebot" and "Googlebot-Image" to crawl your site

But robots.txt is "wide open" (attachement).

I've tried many methods, rebuilding robots.txt, leaving it "wide open" for crawler, different nginx rules, plesk repair etc. There are still 404 errors in the log and list of rejected url at google is raising

robots.txt

Edited May 9, 2022 by Skafander (see edit history)

[Prestachamps]

Hello,
Did you tried also the native PrestaShop robots.txt generation from the Shop parameters > Traffic & SEO > Robots file generator? 
After you generate it from the back office, the robots.txt should look like the attached one. 
Please take a look:
https://prnt.sc/qO6Gq0tf1wsL
After that, you can check with google's robots.txt checker if it is ok:
https://www.google.com/webmasters/tools/robots-testing-tool
Kind regards,
Leo

robots.txt

[Skafander]

16 minutes ago, Prestachamps said:

Hello,
Did you tried also the native PrestaShop robots.txt generation from the Shop parameters > Traffic & SEO > Robots file generator? 
After you generate it from the back office, the robots.txt should look like the attached one. 
Please take a look:
https://prnt.sc/qO6Gq0tf1wsL
After that, you can check with google's robots.txt checker if it is ok:
https://www.google.com/webmasters/tools/robots-testing-tool
Kind regards,
Leo

robots.txt 3.95 kB · 0 downloads

Hello, I was at seach console few times, but didn't noticed that option. I'd like to thank you very much, as it looks like google was keeping old version of robots, wchich I've updated now. You saved me a lot of unecessary work. Thank you again!

[Prestachamps]

Hello,

Happy to help

If you find my answer helpful just click the ❤️ above and show some love.

Kind regards,

Leo

[Nickz]

Google hardly notices the robots.txt file any more. Usually a meta tag takes care of some directives.

[Zohaib-fk]

Hi,

You can try below tags, hope it will help.

<meta name='robots' content='index,follow' />
<meta name="robots" content="noarchive">
<meta name="googlebot" content="noarchive">
<meta name="googlebot" content="index,follow">

Google documentation link => https://support.google.com/news/publisher-center/answer/9605477?hl=en

 

[Nickz]

On 5/9/2022 at 5:50 AM, Skafander said:

I've tried many methods, rebuilding robots.txt, leaving it "wide open" for crawler,

what was it in the end any solution forthcomming?

[Skafander]

@Nickz solution of @Prestachamps from 2nd post fixed the issue. Google crawled robots.txt only once, and after I needed to force it to crawl it again.

[Mitconsulting]

Sorry but i have always this problem.

I test in Google Webmaster and it's ok, I test in tester robots and it's ok.

I try to regeneration in backend the file robots.

I don't understand this problem and GMC disapproved all my products

The site is www.fashiongriffe.com

[Zohaib-fk]

Hi Mitconsulting,

You can find your website pages on Google search using keyword "site:www.fashiongriffe.com", attached is a screenshot for your reference,

Also, you can use the inspection tool in the Google Search console.

I have below 2 modules on the addons marketplace for fast indexing. The API services are from Google websites.

[1]. Automatic Ping Services Integration

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/91680-automatic-ping-services-integration.html

[2]. SEO Google Indexing API Integration

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/91461-seo-google-indexing-api.html