+4 votes
in Web & Google by (43.2k points)
edited by
I want to prevent all bots/crawlers from accessing my website. How can I block the entire website?

1 Answer

0 votes
by (232k points)

I do not think every bot/crawler obeys "Robots.txt". So, it may not be possible to block all bots. However if you want to block Google/Yahoo/Bing and some others, you need to edit your "Robots.txt" to instruct these bots.

If you do not have "Robots.txt" in the home folder of your website, create a file "Robots.txt" there. If you already have "Robots.txt", edit the file to add the following lines:

User-agent: *
Disallow: /