Summary

  • ALIWEB moved the choice of what to describe to each information provider, while keeping harvesting and search in a central service.
  • Its 1994 pilot showed that a small index could be used by other catalogs; it did not prove complete Web coverage or broad adoption.

By late 1993, the Web’s problem was no longer just how to publish a document. It was how to find a useful one without asking a person to browse every server or asking a robot to retrieve every reachable page. Manual lists could supply context, but their keepers could not see every move or new service. A crawler could discover more, yet repeated requests imposed costs on servers and a machine could miss the structure that made a collection understandable.

On 30 November 1993, Martijn Koster announced ALIWEB as an experiment in “distributed” Web indexing. Each server would advertise its contents in a local file; a single site would retrieve and process those files; users could search the combined database through the Web. The pilot had already been running at NEXOR since October, according to Koster’s later conference paper. In his 30 November announcement, Koster called the design a first step, not a final answer, and made a striking promise: when better schemes arrived, the data should be easy to use or convert.

That promise changes how to read the architecture. ALIWEB did not decentralize the search service. It divided control over the index. A site administrator chose which documents and services deserved a record and supplied the description; ALIWEB’s operator kept the registry of participating servers, decided when to retrieve each file, checked its format, combined accepted records and ran the search interface. The provider owned the description of its own site. The service owned the route by which that description became discoverable across sites.

The input was a prepared text file, commonly named /site.idx, built from Internet Anonymous FTP Archives (IAFA) templates. Those attribute-and-value records could describe several objects in one file: a template type, title, URI, description and keywords, for example. The form did not ask ALIWEB to infer every page’s meaning from its markup. It gave the people who knew a site’s structure a way to expose selected entry points while leaving the internal organization intact. Koster’s sample describes NEXOR’s Perl material and its manual, not every file on the server.

Participation still required a small operational contract. An administrator registered the host, port and index-file path, supplied a responsible contact, and could specify a minimum interval between retrievals. The service tested for the file and its format before accepting a registration. A separate Perl harvester checked registered files, retrieved changed copies and rebuilt the combined database on a UNIX cron schedule. The same database could be searched through ALIWEB’s own Web form or incorporated into another catalog, including the CUI W3 Catalog.

The distinction from a Web-walking robot was practical, not absolute. ALIWEB automated retrieval and database assembly, but it fetched one nominated index file rather than walking every link and document on a host. That lowered collection traffic and let providers preserve context. It also made silence meaningful: a page absent from the provider’s index might remain invisible, even if a crawler could have reached it. Search results could be as current as the provider’s file, but freshness was not guaranteed by the central harvester alone.

Koster’s reported pilot numbers show both reach and limits. By the time of the paper, 54 hosts had registered and the database held 310 entries. ALIWEB’s simple search form averaged 80 queries a day; the CUI W3 Catalog drew on its database for more than 8,000 queries a week. Those counts establish a working service and one useful integration. They do not measure the share of the Web indexed, the quality of results, or how many hosts could or would maintain a usable file.

The onboarding problem was visible early. Some administrators registered a host without supplying an index file, apparently because they had reached the form before understanding the model. Koster’s team expanded the documentation and added immediate checks for file presence and format. Errors continued, so the pilot logged them and used a manually run script to email the responsible person. This was not maintenance-free decentralization: it shifted the work from crawling every page to creating, validating and refreshing a curated index.

Nor did “manual” have to mean hand-typed forever. Koster argued that software could generate the file, provided that the result retained enough descriptive metadata to support selection. By 1995, a separate archive project documented IAFA templates being updated from filesystem changes and used to derive text, Gopher and HTML indexes. That later W3C conference paper shows that the template idea could travel across views; it does not establish that ALIWEB’s own corpus migrated or that the service reached broad adoption.

Scale exposed the design’s second boundary. A single database for the entire Web, Koster acknowledged, could become too large. He proposed several ways out: distribute the harvesting work, mirror search access, split databases by subject, or build a hierarchy of ALIWEB services. He also expected the index files to be automatically convertible to future directory systems. These were options on paper, not deployed features. Their importance is that the format was meant to outlive the pilot’s central collector.

In a 1994 survey of networked information retrieval, the field was still described as young, with classification and other capabilities not yet widespread. ALIWEB sat among manual indexes, the CUI W3 Catalog and robot-based approaches such as JumpStation and the RBSE URL database; it was not the first Web search tool. Its narrower contribution was to show how source-selected descriptions could be harvested automatically and reused elsewhere.

Heng Lu’s later Note 65 on Running-Code Primacy offers a retrospective lens, not evidence about Koster’s thinking: a format can propose portability, but only working systems that adopt it make that portability real. The paper documents ALIWEB running and the W3 Catalog using its database. It does not document universal adoption, a mandate over Web publishers or a completed migration. The useful historical distinction is therefore three-part: providers decided what their sites said; ALIWEB decided what entered its shared index; and future systems remained free to reuse the file without inheriting the pilot’s central service.