Problem robot.txt and index

[Ste-WD]

Hello guys,

We have problem with our robot.txt file and google index.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

problem 5 )

really few of the product images are on google images

Please can you help me ?

Many thanks to comunity

[CedCommerce Team]

On 4/26/2020 at 4:23 PM, Ste-WD said:

Hello guys,

We have problem with our robot.txt file and google index.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

problem 5 )

really few of the product images are on google images

Please can you help me ?

Many thanks to comunity

Hello there,

Hope you are fine.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

Reply: The URL you have given in example is for search query of your website and also it is blocked by Robots.txt as we have checked, but this is not a problem it should be blocked.

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

Reply: What type of pages they are? Can you show me here.

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

Reply: There are some missing fields in your product schema why search console shows a warning. You need to contact your developer regarding this.

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

Reply: Same as reply 4

problem 5 )

really few of the product images are on google images

Please can you help me ?

Reply: Need to check product schema whether property is set for images or not.

Thanks!!!

[Ste-WD]

4 hours ago, CedCommerce Team said:

Hello there,

Hope you are fine.

Problem 1)

sitemap: https://www.wdracing.eu/it/module/lgsitemaps/sitemap?fc=module&name=sitemap_1

Google says that some pages are blocked by robot.txt file

for example link: https://www.wdracing.eu/it/pastiglie-e-pinze-freno/25646-distanziale-braking-honda-crf-19-270mm.html

our robot.txt file: https://www.wdracing.eu/robots.txt

When using googlebot tool, it doen not find errors

what can we do ?

Reply: The URL you have given in example is for search query of your website and also it is blocked by Robots.txt as we have checked, but this is not a problem it should be blocked.

REPLY 2 : you are right. but what if i want that products to be index ? beacause we have made a lot of redirects to that products from old website. beacause they are "general" product that send to a specific product when you select your bike. i can consider to allow the search query in robot.txt ?

problem 2)

some pages are not valid.  we have:

23691 valid pages

4200 pages with errors

8024 exluded pages

Reply: What type of pages they are? Can you show me here.

REPLY 2: yes see below

problem 3)

If from search console we go in --> products--> missing field description, brand and others,  i see 0 valid pages e 20000 valid with warnings. 

we have a lot of link of category and not products. see image below

here i have found a similar problem but is not clear how to fix it: https://support.google.com/webmasters/thread/7159245?hl=en

Reply: There are some missing fields in your product schema why search console shows a warning. You need to contact your developer regarding this.

REPLY 2: i have read that one of these modules can solve the product schema problem: 

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/44883-clean-and-optimize-microdata-and-open-graph-seo.html

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/42397-full-json-ld-microdata-and-open-graph-seo.html

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/39955-seo-schema-markup-structured-data-rich-snippet.html

https://addons.prestashop.com/en/url-redirects/19643-advanced-seo-friendly-urls.html#specifications

 

4 hours ago, CedCommerce Team said:

do you agree ? what do you suggest ?

problem 4)

if i analyze a product link i see a lot of products and not only the one i have analyzed. see below link

https://search.google.com/structured-data/testing-tool#url=https%3A%2F%2Fwww.wdracing.eu%2Fit%2Fcavalletti-trasporto-moto%2F670-260435-cavalletto-alzamoto-cross-enduro-racetech.html%23%2F7120-tipo-motocross%2F30185-aa_colore-giallo

Reply: Same as reply 4

problem 5 )

really few of the product images are on google images

Please can you help me ?

Reply: Need to check product schema whether property is set for images or not.

REPLY 2 I think that they are related to previus errors. if we solve them the images will be index

Thanks!!!

Hello, please see my answers REPLY 2

Edited April 28, 2020 by Ste-WD (see edit history)

[Ste-WD]

About problem 1 ) we have allowed the search query from robot.txt. Please check: https://www.wdracing.eu/robots.txt

About problem 2-3-4-5 ) We are almost sure that they are due product schema.

In which way do you suggest to solve the problem ? which one of this modules can help ?

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/44883-clean-and-optimize-microdata-and-open-graph-seo.html

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/42397-full-json-ld-microdata-and-open-graph-seo.html

https://addons.prestashop.com/en/seo-natural-search-engine-optimization/39955-seo-schema-markup-structured-data-rich-snippet.html

https://addons.prestashop.com/en/url-redirects/19643-advanced-seo-friendly-urls.html#specifications

 

Edited April 29, 2020 by Ste-WD (see edit history)

[Constantin86]

Hello,
I have an online store ( https://www.voluptas.ro/ ) and I think I have a problem indexing some products. Can you tell me what the robots.txt file should contain in order to be ok for Google?
Thanks.