Edit History

5 minutes ago, tomerg3 said:

Looks like your robots.txt is blocking the crawl 

 

Here is the content of the robots.txt file, it doesn't seem to be blocking it :

# robots.txt automatically generated by PrestaShop e-commerce open-source solution
# http://www.prestashop.com - http://www.prestashop.com/forums
# This file is to prevent the crawling and indexing of certain parts
# of your site by web crawlers and spiders run by sites like Yahoo!
# and Google. By telling these "robots" where not to go on your site,
# you save bandwidth and server resources.
# For more information about the robots.txt standard, see:
# http://www.robotstxt.org/robotstxt.html
User-agent: *
# Allow Directives
Allow: */modules/*.css
Allow: */modules/*.js
Allow: */modules/*.png
Allow: */modules/*.jpg
Allow: */themes/*/assets/cache/*.js
Allow: */themes/*/assets/cache/*.css
Allow: */themes/*/assets/css/*
# Private pages
Disallow: /*?order=
Disallow: /*?tag=
Disallow: /*?id_currency=
Disallow: /*?search_query=
Disallow: /*?back=
Disallow: /*?n=
Disallow: /*&order=
Disallow: /*&tag=
Disallow: /*&id_currency=
Disallow: /*&search_query=
Disallow: /*&back=
Disallow: /*&n=
Disallow: /*controller=addresses
Disallow: /*controller=address
Disallow: /*controller=authentication
Disallow: /*controller=cart
Disallow: /*controller=discount
Disallow: /*controller=footer
Disallow: /*controller=get-file
Disallow: /*controller=header
Disallow: /*controller=history
Disallow: /*controller=identity
Disallow: /*controller=images.inc
Disallow: /*controller=init
Disallow: /*controller=my-account
Disallow: /*controller=order
Disallow: /*controller=order-slip
Disallow: /*controller=order-detail
Disallow: /*controller=order-follow
Disallow: /*controller=order-return
Disallow: /*controller=order-confirmation
Disallow: /*controller=pagination
Disallow: /*controller=password
Disallow: /*controller=pdf-invoice
Disallow: /*controller=pdf-order-return
Disallow: /*controller=pdf-order-slip
Disallow: /*controller=product-sort
Disallow: /*controller=search
Disallow: /*controller=statistics
Disallow: /*controller=attachment
Disallow: /*controller=guest-tracking
# Directories
Disallow: */cache/
Disallow: */classes/
Disallow: */config/
Disallow: */controllers/
Disallow: */css/
Disallow: */download/
Disallow: */js/
Disallow: */localization/
Disallow: */log/
Disallow: */mails/
Disallow: */modules/
Disallow: */override/
Disallow: */pdf/
Disallow: */src/
Disallow: */tools/
Disallow: */translations/
Disallow: */upload/
Disallow: */vendor/
Disallow: */web/
Disallow: */webservice/
# Files
Disallow: /*fr/adresse
Disallow: /*fr/adresses
Disallow: /*fr/connexion
Disallow: /*fr/panier
Disallow: /*fr/reduction
Disallow: /*fr/suivi-commande-invite
Disallow: /*fr/historique-commandes
Disallow: /*fr/identite
Disallow: /*fr/mon-compte
Disallow: /*fr/commande
Disallow: /*fr/confirmation-commande
Disallow: /*fr/suivi-commande
Disallow: /*fr/avoirs
Disallow: /*fr/recuperation-mot-de-passe
Disallow: /*fr/recherche
Disallow: /*en/address
Disallow: /*en/addresses
Disallow: /*en/login
Disallow: /*en/cart
Disallow: /*en/discount
Disallow: /*en/guest-tracking
Disallow: /*en/order-history
Disallow: /*en/identity
Disallow: /*en/my-account
Disallow: /*en/order
Disallow: /*en/order-confirmation
Disallow: /*en/order-follow
Disallow: /*en/credit-slip
Disallow: /*en/password-recovery
Disallow: /*en/search

User-agent: MJ12bot
Disallow: /