Summary

  • terms.txt revision 00 proposes origin-enforced checks for signed bot requests, declared purposes, user delegations and payment vouchers before content is served.
  • Its own boundary is decisive: a signature attributes a declaration and a receipt records one delivery; neither proves the declaration true nor controls later use of the content.
  • Daniel Kade proposes an access-obligation register that keeps pre-delivery enforcement, post-delivery audit and contractual duties in separate rows. This is editorial guidance, not an IETF requirement.

A proposed bargain at the origin

The new terms.txt draft starts from a real limitation in the web's existing crawler convention. RFC 9309 standardises robots.txt, but says its rules are not access authorisation. They can request that a crawler avoid paths; they do not authenticate an operator, attach a declared purpose, collect a payment or enforce a decision at the server.

Revision 00 proposes a more active exchange. An origin would publish /.well-known/terms.txt, following the pattern for well-known URIs. Path blocks could allow, charge or deny machine access for registered purposes, set a ceiling on how much content may be reproduced, and require a user delegation for selected resources.

An automated client would identify itself through the separate Web Bot Auth signature draft and cover an Access-Intent field with an HTTP Message Signature. Where the terms demand more, the same request would carry a delegation token or a single-use payment voucher. The origin checks the request before serving the representation and, on success, returns an origin-signed Access-Receipt.

That is a substantial change from a passive preference file. It is also much narrower than a system that can verify downstream compliance.

Three kinds of evidence

The draft usefully separates three tiers. Before delivery, the origin can enforce facts that are available at its boundary: the key that signed the request, freshness and replay protection, the full target URI, integrity of the declared intent, a delegation's audience and scope, the applicable terms, and the presence of a valid unspent voucher.

After delivery, the signed declaration and receipt can support an audit. If an operator said search but its public product reproduces full articles without referrals, the declaration gives a dispute a named and time-bounded starting point. Attribution is more useful than an unauthenticated User-Agent string.

The third tier is contractual. The draft states plainly that HTTP cannot govern content after it leaves the origin. A signature proves that the holder of a key made a declaration and that the covered fields were not altered. It does not prove the declared purpose was honest. A receipt proves that one representation was delivered under stated terms. It does not observe model training, redistribution or any other later act.

This distinction also limits the identity claim. The Web Bot Auth mechanism binds a request to a key published under a resolved agent identifier. It does not automatically establish the real-world company operating that agent. An origin that needs that additional assurance needs another trust process.

One receipt cannot travel with every copy

The proposal's caching rule exposes the granularity of its proof. An access receipt hashes one signed request. If a shared cache reused the response for a second requester, the attached receipt would still refer to the first request, while the origin would have logged no second delivery. Revision 00 therefore requires responses carrying receipts to use Cache-Control: no-store, applying the controls defined by HTTP caching. Unsigned responses can remain cacheable under ordinary rules.

The cost is not incidental. Per-request attribution and settlement remove a common efficiency of the web. A future cacheable design would need a separate receipt path or another way to bind delivery without misrepresenting who received what. Until then, a deployment cannot simultaneously claim request-specific evidence and quietly treat one cached receipt as a crowd certificate.

Refusal is similarly explicit. The draft uses HTTP 403 and Problem Details for invalid signatures, denied purposes, missing delegation or payment. It does not repurpose the reserved status 402, and it avoids 401 because the proposed exchange supplies no WWW-Authenticate challenge. Those choices inherit the semantics of HTTP; they do not create a court-enforceable licensing regime.

The document is not an IETF decision

Process status belongs in the story. The Datatracker record labels terms.txt an individual Internet-Draft with no stream and no formal standing in the IETF standards process. Its history records revision 00 as manually posted on 11 September 2026. There is no working-group adoption, responsible Area Director, IESG approval or RFC.

The header says “Standards Track,” while Datatracker currently shows no intended RFC status. That is a proposal about destination, not evidence that the destination has been reached. The related AI usage vocabulary and HTTP attachment method are separate working-group documents. Their institutional state cannot be borrowed by this draft.

Nor should the implementation note be overstated. Revision 00 reports a JavaScript reference implementation, 24 checks and timing observations. It also says release v0.1 predates the document and differs from it in target binding, status codes, token format, identifier binding, receipt fields, voucher representation, caching and link-relation naming. Running code is evidence of an experiment, not conformance to text that changed around it.

Keep an access-obligation register

Daniel Kade proposes a local access-obligation register. Each row should bind an origin and path scope to the operative terms ID and digest, declared purpose and use level, agent identifier and identity-assurance limit, delegation issuer and scope, settlement method, pre-delivery decision, response and content digest, receipt identifier, later observable evidence, contractual duty, dispute owner, expiry, revocation and superseding terms.

The register is deliberately less ambitious than automated truth. It lets an origin prove what it checked and delivered, lets an operator show what it declared, and leaves later conduct to evidence and institutions capable of judging it. Heng Lu's Policy Mirror makes the boundary between evidence and decision visible. The Minimum Initial Specification favours a small common surface without swallowing local authority. Why BTW Media Exists supplies the reporting rule: the draft is news, and its limits are part of the news.

Sources

  1. terms.txt Datatracker record
  2. terms.txt document history
  3. terms.txt revision 00
  4. Web Bot Auth HTTP Message Signatures draft
  5. AI Usage Preferences vocabulary draft
  6. AI Usage Preferences attachment draft
  7. RFC 9309: Robots Exclusion Protocol
  8. RFC 9421: HTTP Message Signatures
  9. RFC 9457: Problem Details for HTTP APIs
  10. RFC 9110: HTTP Semantics
  11. RFC 9111: HTTP Caching
  12. RFC 8615: Well-Known URIs
  13. RFC 8126: IANA registration guidance
  14. The Policy Mirror
  15. Minimum Initial Specification
  16. Why BTW Media Exists