 Robots Txt Tutorial
Robots.txt is the standard (Robots Exclusion Protocol) that instructs Web Crawlers which file/directory
should not be crawled. It is a text file placed in the root directory with a special name robots.txt.
Almost every search engine spiders or crawlers look for this file and follows the instructions entered in
this file. If this file is not present in the root directory or left blank, search engine crawler assumes
every link is allowed to be downloaded and indexed. Special format is used in making this file. Special
instructions can be given for specific crawler or default settings can be made for all crawlers.
User-agent
Search engine crawler name is specified in this field. For example if you want to give special
instructions to Google search engine crawler "googlebot", it is done as follows:
User-agent: googlebot
To make a default entry for all robots, wildcard character "*" can be used as follows:
User-agent: *
Disallow
This field directs crawler which file/directory is not to be indexed. Every User-agent can have one
or more Disallow fields in separate lines. For example if you want crawler not to index restricted.html
file it is done as follows:
Disallow: restricted.html
If this file is not in a root directory but other directory called "private", you can make following
entry:
Disallow: /private/restricted.html
If you want to disallow whole directory, you can do so as follow:
Disallow: /private/
If Disallow: /private is used, /private.html and /private/restricted.html will be restricted.
If Disallow: is left blank, all the files can be indexed. If you want to restrict whole web site
Disallow: / is used.
Some Examples
The following example specifies that no robots should visit and download any file in the website:
User-agent: *
Disallow: /
The following example specifies that every robot can download every file on the website:
User-agent: *
Disallow:
The following example specifies that no robots should index /private/restricted.html file, temp
directory and restricted.html file on the website:
User-agent: *
Disallow: /private/restricted.html
Disallow: /temp/
Disallow: /restricted.html
The following example specifies that one robot is restricted from indexing and rest are permitted to
index whole site:
User-agent: cybercracker
Disallow: /
User-agent: *
Disallow:
| 
