Google bots can not crawl my shop (urls)

[Rajeshx]

I am facing issues with Google search console/merchant center.

since 2 weeks google can not crawl my pages and images.

When I add a sitemap in search console I get the message: general HTTP-error.

When I add my robots.txt url manually google can not index the page.

 

In merchant center images and most landingpages can not be validated by google.

All pages are normally live and I can access them via multi  devices.

This all happened after I deleted my products and uploaded them again via a product feed from my supplier. After this the products had different id's, but not sure if this is the reason why google bots can not access my shop for crawling.

 

I use prestashop v 1.7.8.5.

 I do use a paid seo module now, but still the same issues.

attached my robots.txt file.

 

Any suggestions what can be the issue and how to fix it.  I read a lot of old topics but did not find the solution

robots2.txt

[Rajeshx]

Sombody any input?

 

I am getting these errors in search console:

 

Details

ophalen van robots.txt

Hoog falingspercentage afgelopen week

Mislukte crawlverzoeken

 

Serverconnectiviteit

Hoog falingspercentage afgelopen week

 

Seems like google is not able to crawl robots.txt and connectivity with the server is also not possible.

 

All my pages are live and can access them, so it seems like something is blocking the crawler, but no idea how to fix this.

 

 

[Nickz]

Have you tested if that sitemap is actually active? Your-shop.xxx/sitemap

 

[Rajeshx]

Thanks for your reply.

 

Yes, all pages are live (sitemap, robots.txt)

 

Ik can load them in my browser

[@diegofrancesco]

Hello, 

the problem is not in uploading new products, but it is probably because the Google crawler cannot read the site due to a block in the server or in the domain. Ask whoever manages the server or the domain provider.

It seems that the domain is not accessible to everyone, check with this tool how the site is seen in the world: https://geopeeker.com/ 

[Tomi14]

Could you test your page by Screaming Frog SEO Spider?

[Nickz]

On 11/28/2022 at 4:02 AM, Rajeshx said:

Seems like google is not able to crawl robots.txt and connectivity with the server is also not possible.

Google gave a note out that they do not follow the robots txt anymore in some features. So there could be a new method. I think for Europeans the merchant center is pretty useless. Even for US its hard to find a statistic nothing more than so many businesses have a merchant account. Which is quite meaningless. 

https://developers.google.com/search/docs/crawling-indexing/robots/intro

and https://www.searchenginejournal.com/google-robots-txt-noindex/314961/#close

Edited December 6, 2022 by Nickz (see edit history)