SEO, robot.txt .. issue

[deonderd]

y setup is multistore

14 domains using same catalog, different languages

eg

magasindepieces.fr uses french (mainlanguage and english)

but if i check on google .. it crawls all languages.. also the ones that are not visible in this domain..

This can give me troubles with duplicate content..

for the .fr tdl. french is important not the rest.. (thats why only fenchand english available in the store)

in my case

magasindepieces.fr has two languages activated

I donot want the other languagees to be crawled for magasindepieces.fr

so mainissue i think is .; that we need a rebot.txt file for every domain in multistore

also in sitemap.. givesthis issues..if u create a sitemap.; it always gives the maindomainname link.. not the one in the multistore config

[Elena Ruchko]

Hi!

 

If I understood your question correctly, you want for http://magasindepieces.fr , for example, to be crawled only French and English versions and no others. In this case you should write in robots.txt not /*de/ as you wrote for German, for example, but /de/. And you don't need to specify any additional folders like /*de/adressen because by writing /de/ you will close the whole language folder.

 

Additionally, to avoid duplicate problems, I would advise you to generate a new sitemap.xml where all folders which you don't want to be crawled by robots would be exluded. To achieve this I advise you to make the use of this sitemap generator where you are able to specify which folders you don't want to see in the sitemap. When a new sitemap is ready, make sure to add link to sitemap.xml at the end of the robots.txt file and submit sitemap.xml to Webmaster Tools.

 

With such large multistore you may experience soem other problems, so we would be happy to help you and can offer you free audit of your website - we will tell you what technical points could be improved in order to be ranked better by search engines and how to improve website usabiltiy in order to increase traffic conversion rate. No strings attached, so feel free to apply! http://lp.promodo.com/guides/free-site-audit/

 

Regards, Elena