A Method for Web Robots Control

Martijn Koster

Citation

Martijn Koster, A Method for Web Robots Control, Internet Draft draft-koster-robots-00, December 1996.

Descriptions

Abstract:

This memo defines a method for administrators of sites on the World- Wide Web to give instructions to visiting Web robots, most importantly what areas of the site are to be avoided. This document provides a more rigid specification of the Standard for Robots Exclusion, which is currently in wide-spread use by the Web community since 1994.

Resources

Google:	Search for `["A Method for Web Robots Control" Martijn Koster]`
URI:	`http://www.robotstxt.org/wc/norobots-rfc.html`

Bibliography Navigation: Reference List; Author Index; Title Index; Keyword Index

Generated by sharef2html on 2008-11-08, 02:14:54.