A Method for Web Robots Control

Martijn Koster

Martijn Koster, A Method for Web Robots Control, Internet Draft draft-koster-robots-00, December 1996.

This memo defines a method for administrators of sites on the World- Wide Web to give instructions to visiting Web robots, most importantly what areas of the site are to be avoided. This document provides a more rigid specification of the Standard for Robots Exclusion, which is currently in wide-spread use by the Web community since 1994.


Bibliography Navigation: Reference List; Author Index; Title Index; Keyword Index

Generated by sharef2html on 2011-04-15, 02:00:41.