mathew80 Posted April 28, 2014 Share Posted April 28, 2014 Hi, in my shop I sell some products only to a registered customer. Now google shows all my webpages and these products too. How can I solve this? Can I manually insert these in robot.txt to block the crawler? Many thanks Matteo Link to comment Share on other sites More sharing options...
El Patron Posted April 28, 2014 Share Posted April 28, 2014 Hi, in my shop I sell some products only to a registered customer. Now google shows all my webpages and these products too. How can I solve this? Can I manually insert these in robot.txt to block the crawler? Many thanks Matteo here is some good information. http://www.robotstxt.org/orig.html Please let us know specifically how you solved it using robots.txt file. I think this will work for the major search engines, i.e. those that honor robots.txt Note: you sitemap is most likely generating these urls as well. It would be a bit messy trying to resolve sitemap so I'd start with robots.txt for starters. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now