Selectively Excluding Pages from Being Indexed

There are many times you may with to exclude certain pages from being indexed by certain engines. One way to do this is by utilizing a robots.txt file and uploading it to the root directory of your Web site.

Basically, you just create a text file with Window’s NotePad or any other editor that can save ASCII .txt files.

Use the following syntax:

User-Agent: {SpiderNameHere}
Disallow: {FilenameHere}

For example, to tell Inktomi’s spider, called Slurp, to not index files called orderform.html and junk.html, create a robots.txt file as follows:

User-Agent: ArchitextSpider
Disallow: orderform.html
Disallow: junk.html

You would then upload this robots.txt file to the root directory of your Web site. Although this is a voluntary protocol, most major search engines will honor it.

You can add more lines to exclude pages from other engines by specifying the User-Agent parameter again in the same file, followed by more Disallow lines. Each disallow statement will be applied to the last User-Agent that was specified. If you want to exclude an entire directory, use this syntax:

User-Agent: ArchitextSpider
Disallow: /mydirectory/

Other options are to exclude the page from all spiders with:

User-Agent: *
Disallow: /mydirectory/

Do NOT use the wildcard (*) character in the Disallow line since that’s not supported.

Make sure you use the proper syntax. If you misspell something, it’s not going to work.

About Article

This article is copyrighted and has been reprinted with permission from FirstPlace Software, the makers of WebPosition Gold. FirstPlace Software helped define the SEO industry with the introduction of the first product to track your rankings on the major search engines and to help you improve those rankings. A free trial of WebPosition Gold is available from their Web site.

Ritu Sharma
0 0 votes
Article Rating
Notify of
Inline Feedbacks
View all comments

Related Posts

People Also Search For or People Also Ask
If you frequently use Google, it’s very likely that you’ve...
By Navneet Kaushal 83166 reads
map location
Letting your friends and family know about your real-time and...
By Navneet Kaushal 75060 reads
top 15 seo tools4
The Search Engine Optimization (SEO) industry is worth $65 Billion...
By Navneet Kaushal 44318 reads