Google sitemap and indexing

[VIXUS]

I need some help, info, guide etc.

Shop works, sitemap works but in my google webmaster tools my sitemap has like 1137 url's and google is indexing only some 590 of them??? Immage attached.

Any tips on how to resolve this?

[JosefPrado]

Hi,

This is normal. Google do some kind of phased crawling in the websites.

It gets some pages from for site then go to the next one in the list. After some time your website i@ indexed again and more pages appears there.

It's just a mater id time for you curls be fully indexed.

Regards,

Josef

[tomerg3]

I doubt your site will ever get fully indexed, and not because of any problems you have with it.

Google won't index all your pages if they think there are duplicates (within or similar to other sites).

I manage about 10 different domains, and whenever I have more than 300 pages, I usually see google index only about 60-70% of them.

Make sure your sitemap is correct, if you have incorrect URLs in there, it may show up as the totla number or URLs, but they will never be crawled successfully.

[Dahl99]

Hi,

I'm having similar problem.

Before I enabled friendly urls I had about 50 out of 1000 urls indexed. Since then I enabled friendly urls, updated the sitemap with Google Site Map v1.7 and also used 301 Redirect Module 1.1.2 (thank you Tomer). In Google Webmaster Tools it now shows 352 urls submitted but none indexed.

I understand that it has only been about 10-14 days since I did this and it takes Google a while to index a site but before the updates if I resubmited my site map I noticed the indexed increase by a few pages on a regular basis. Is there a reason for no index urls as yet.

On another thread (not sure where) someone suggested that you add "sitemap /sitemap.xml" to the robots.txt Should it not be Sitemap: http://www.example.com/mainsitemap.xml or Sitemap: http://example.com/mainsitemap.xml

Regards

[tomerg3]

Hi,

I'm having similar problem.

Before I enabled friendly urls I had about 50 out of 1000 urls indexed. Since then I enabled friendly urls, updated the sitemap with Google Site Map v1.7 and also used 301 Redirect Module 1.1.2 (thank you Tomer). In Google Webmaster Tools it now shows 352 urls submitted but none indexed.

I understand that it has only been about 10-14 days since I did this and it takes Google a while to index a site but before the updates if I resubmited my site map I noticed the indexed increase by a few pages on a regular basis. Is there a reason for no index urls as yet.

On another thread (not sure where) someone suggested that you add "sitemap /sitemap.xml" to the robots.txt Should it not be Sitemap: http://www.example.com/mainsitemap.xml or Sitemap: http://example.com/mainsitemap.xml

Regards

I just check out your sitemap and it looks fine, as long as you submit your sitemap directly to the webmaster tool, the robot.txt link to it doesn't really matter.

As for the number of pages indexed, it could be 0 for various reason all related to how google crawls sites and give them ranking.

You can check the access log on your server to see how often google visits your site, and what response codes they are getting.

If you host has cpanel, look for "Latest Visitors" in the "Logs" section, click on your domain, and seach for the word "googlebot".

You should see something like

Http Code: 200      Date: Jan 21 11:33:30      Http Version: HTTP/1.1      Size in Bytes: 977
   Referer: -
   Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

If you see only "Code: 301" than something is wrong with redirects on your site.
If you see "Code: 301" for an old URL, followed by a "Code: 200" for the new version of it, than things are working just fine, and google is just going to re-index the pages on your site.

Also, if you google "site:yourdomain.com" you will see that many of your pages are showing up in the index.

[Dahl99]

Hi Tomer,

thanks for the reply. Below are the first entries found in the log, each with a different code, so not sure whats going on. Can you enlighten me. Thank you.

Looking at them it has a url, /4-portables-apple?isolang=fr&id_lang=2&p=2, which I'm not aware of or is it just a default url left over from installing PS.

/4-portables-apple?isolang=fr&id_lang=2&p=2
   Http Code: 200     Date: Jan 21 10:57:28     Http Version: HTTP/1.1     Size in Bytes: 24935
   Referer: -
   Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

/sitemap.xml
   Http Code: 304     Date: Jan 21 12:22:23     Http Version: HTTP/1.1     Size in Bytes: -
   Referer: -
   Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

/lang-en/4-portables-apple?isolang=fr&id_lang=2&p=2
   Http Code: 301     Date: Jan 21 13:24:08     Http Version: HTTP/1.1     Size in Bytes: -
   Referer: -
   Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

These were further down the log.

/18-57-home/unusual-aztec-design-necklace-and-earring-set.jpg
   Http Code: 200     Date: Jan 21 12:36:12     Http Version: HTTP/1.1     Size in Bytes: 6384
   Referer: http://www.funkyfashionjewellery.com/
   Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/531.21.8 (KHTML, like Gecko)Version/4.0.4 Safari/531.21.10

/16-55-home/funky-chunky-charm-necklace.jpg
   Http Code: 200     Date: Jan 21 12:36:12     Http Version: HTTP/1.1     Size in Bytes: 5965
   Referer: http://www.funkyfashionjewellery.com/
   Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/531.21.8 (KHTML, like Gecko)Version/4.0.4 Safari/531.21.10

When I check site:www.funkyfashionjewellery.com most of the entries are still reading as

www.funkyfashionjewellery.com/product.php?id_product=15

Rather than

www.funkyfashionjewellery.com/.../32-silver-and-black-fashion-bracelet.html

Regards

[tomerg3]

I'm not sure where google saw this URL /lang-en/4-portables-apple?isolang=fr&id_lang=2&p=2

But I would just give it time, they sometimes take over a month to update their index to reflect these changes (and convert the old URL to friendly URL).

Keep an eye on the recent visitors, and consider changing the crawl rate in the webmaster tool.

[Dahl99]

Thanks for the advice Tomer.

Regards

[janneymarico]

Be passions... google will crawl....... b'coz google crawls comes any time, there is no time particular time for crawl. so pls. loss hope..