Summary

  • In his 1994 WebCrawler paper, Brian Pinkerton described a public full-text index of nearly 50,000 documents on more than 9,000 servers, answering more than 6,000 queries daily and updated weekly.
  • The same system could run a real-time search: use index matches as a starting point, follow unexplored links and re-run the query as new pages arrived. Pinkerton said that mode was not available to the public because client-initiated searching would multiply network work for results used by one person.

One index, thousands of questions

Early Web search had a choice about when to pay for knowledge. A robot could fetch pages in advance, store their words and links, and let many people question the resulting database. Or it could wait for one person's question and then go looking beyond what it already knew. WebCrawler did both in software. Only the first method became the public search page described in Pinkerton's paper.

The public index was incomplete by design. The conference account says WebCrawler traversed links broadly but gave every newly found server a turn before digging much deeper into any one of them. It kept a full-text index and a graph of documents, links and servers. When it retrieved a page, it marked the visit, extracted outbound links and indexed words from the title and body. Its public query server searched that stored material and returned ranked titles and URLs. Pinkerton reported nearly 50,000 indexed documents across more than 9,000 servers, over 6,000 queries a day and a weekly index update at the time of writing. Those were operating figures for his 1994 service, not a count of the whole Web.

The choice to read the body mattered. Pinkerton reported that 20% of the documents his robot visited lacked titles. A search service using titles alone would have been blind to many of them and to the substance hidden behind weak titles. The full-text database consumed more local processing and storage, but it let the same fetched page answer many different questions. That was the shared asset: one set of network retrievals could be used repeatedly by strangers who never had to fetch the whole collection themselves.

The second crawler chased the question

Pinkerton's other mode began where the stored index became thin. It ran a user's query against the index, picked relevant pages and followed unexplored links from them. Each newly retrieved page entered the index; the query ran again; new high-ranking pages could send the robot onward. The search stopped when it had enough similar documents or reached a time limit. Pinkerton also experimented with comparing the words in a link's anchor text to the query, so that the robot could choose a promising branch instead of marching blindly through every outgoing link.

That was a striking division of labor. The public index offered a quick answer from work already done. The real-time robot tried to make the index smarter for a particular request. Pinkerton wrote that the real-time searches were fast and well targeted in his experiments. He also wrote, plainly, that they were not available to public users. The search page that answered thousands of daily queries was the index service. Describing those public queries as launching personal crawlers would erase the very decision that made the service affordable to run.

The costs were different in kind. In indexing mode, a server might see a visit every few hours as WebCrawler spread work among sites, according to Pinkerton's account. The index then answered many users from one stored copy. A client-driven search would launch agents to fetch pages for one person's result set. Multiply that pattern by tens of thousands of users and the same remote servers would face many searches whose results were not shared. Pinkerton said widespread client-based searching was not yet practical on the available network.

He was testing a useful feature and refusing to pretend that a private demonstration proved it could be handed to everyone.

The paper's public-service figures also show why the choice had stakes. A weekly refreshed index could miss a new page or a new link. An on-demand robot could find material beyond the old snapshot. Yet sending every searcher into live retrieval would make freshness a burden on publishers and networks that had not asked to serve a search experiment. Pinkerton wanted a more targeted search, but the public interface stayed with the common database while he considered ways for client findings to feed back into that shared store.