+3 votes
in Web & Google by (39.4k points)
What code should I use in the robots.txt to allow or disallow all pages of my website to all bots? I am using the Apache web server.

1 Answer

+2 votes
by (281k points)
selected by
 
Best answer

If you want to allow every page to all bots, you can use the following code in the robots.txt file.

User-agent: *
Disallow:

If you want to disallow every page to all bots, you can use the following code in the robots.txt file.

User-agent: *
Disallow: /


...