Indexing Aids at Corporate Websites: The Use of Robots.txt and META Tags

Sixty corporate websites selected from the Fortune Global 500 companies were examined in 2000 and again in 2001 to see if they provided support for automatic indexing. In particular, use of the robots.txt and Meta tags for "keywords" and "description" was examined. Slightly fewer than half of the sites provided one or both of these aids. Among sites providing indexing aids there was a clear under-representation of Asian sites. Nearly 80% of the sites used Java, suggesting a reasonable level of technical sophistication among website creators. About one-third of the sites used cookies, raising the possibility that repeat visitors might find the navigation of the site customized to their needs. Overall an increase in the use of indexing aids, especially Meta tags, represents one way in which web robots could index sites more quickly and thus improve overall index coverage of the web.