Lead Report / History

A Plain File Put Each Site’s Crawler Rules at the Root: robots.txt, 1994
In 1994, Web robot authors agreed on a modest convention: a server could publish crawler preferences at one predictable address, /robots.txt. The choice made the rule cheap to find and easy to serve, but it also concentrated editing in the site administrator and depended on crawlers choosing to comply.











