Summary

  • RFC 2406 mapped ESP protection field by field. Transport mode left the original IP header outside ESP; tunnel mode encrypted the inner datagram but still exposed a new outer header, while SPI and Sequence Number remained clear in both.
  • Confidentiality, authentication and anti-replay were not one service. Authentication was optional, encryption could be NULL, and a receiver could ignore the mandatory Sequence Number. A protected payload therefore did not prove a hidden traffic relationship, an executed replay check or a later application result.

Look at an ESP packet from the outside. The upper-layer header and data may be unreadable, yet a router still needs an IP destination. The receiver still needs an SPI to find a Security Association. A sequence value still occupies four octets. An observer can still count packets, measure their sizes and place them in time. The content has crossed a cryptographic boundary; the entire event has not disappeared.

That distinction was built into RFC 2406, published in November 1998 as the second major specification of the IP Encapsulating Security Payload. Its title invited a shortcut—ESP meant encrypted IP—but the document described a mix of services: confidentiality, data-origin authentication, connectionless integrity, anti-replay and limited traffic-flow confidentiality. The mix depended on the chosen Security Association and on where the implementation sat in the network.

The historical change matters. RFC 1827, from 1995, made the SPI the only mandatory transform-independent field. Separate transform documents filled the shell with algorithms and additional fields. RFC 2406 said that the combinatorial growth of those transforms required a fuller base format. Sequence Number, padding, Next Header, optional Authentication Data and their processing rules moved into the common ESP specification.

The result was not a single switch marked secure. Confidentiality could be selected without ESP authentication. Authentication could be selected with a NULL encryption algorithm. At least one of the two had to be present, but their evidence remained different. Anti-replay was available only when authentication was selected, and its use belonged to the receiver.

The top-level packet made the boundaries visible. An outer IPv4 or IPv6 header came first and identified protocol 50. The ESP header then carried a 32-bit SPI and a 32-bit Sequence Number. Payload Data followed, then Padding, Pad Length and Next Header. Optional Authentication Data came last.

For the ordinary separate-algorithm case, encryption covered Payload Data, Padding, Pad Length and Next Header. It did not cover the outer IP header, SPI, Sequence Number or Authentication Data. If an explicit initialization vector appeared at the beginning of Payload Data, the RFC warned that it was usually not encrypted itself even though engineers often called it part of the ciphertext.

Authentication drew a different span. When selected, the Integrity Check Value covered SPI, Sequence Number, Payload Data, Padding, Pad Length and Next Header, but not the Authentication Data that carried the ICV. Encryption ran first, so the authentication calculation saw the encrypted form of the protected payload and trailer. The outer IP header still lay beyond ESP’s ICV.

Transport mode made that exclusion especially clear. ESP followed the original IP header and protected the next-layer material. The source and destination in that original header remained available to the network. With IPv6, extension headers placed before ESP also remained outside its protection, while a destination option placed after ESP could enter the protected region. Header order was part of the security statement.

Tunnel mode moved the boundary outward but did not erase it. ESP encapsulated the complete original IP datagram, including its inner source and destination. A new outer IP header then addressed the tunnel endpoints. An observer might no longer see the ultimate correspondents, but could still see the gateways, packet cadence and lengths. The tunnel concealed one relationship by exposing another.

RFC 2406 therefore called traffic-flow confidentiality limited. It required tunnel mode and worked best at a security gateway where many subscriber flows could be aggregated. Extra padding could obscure some payload length, at a bandwidth cost. Neither tunnel mode nor padding independently hid timing, volume or the outer endpoint pair. Traffic analysis was a separate problem from reading the payload.

The Sequence Number supplied another frequent source of overstatement. The sender had to include and increment it for every packet. Yet the receiver could enable or disable anti-replay per SA. If the check was disabled, the field still travelled. Its presence proved that a counter value was transmitted, not that a sliding window rejected a duplicate.

When anti-replay was enabled, authentication had to be enabled too. Otherwise an attacker could alter the counter. The receiver could perform a preliminary window test, but it could not advance the window until the ICV succeeded. The useful receipt was therefore a joined result: the correct SA, an authenticated counter and a receiver-side verdict. A packet capture alone could show only part of that chain.

Processing order added more boundaries. On output, another IPsec step first selected an SA that called for ESP. RFC 2406 deferred that policy decision to RFC 2401. ESP then encapsulated, padded and encrypted; it calculated an ICV if authentication was selected; fragmentation followed ESP processing. On input, reassembly came first. A fragment still presented directly to ESP had to be discarded as an auditable event.

The receiver used destination address, ESP and SPI to find a unidirectional SA. The SA said whether to check the Sequence Number and Authentication Data and which algorithms and keys to use. No valid SA meant discard. A matching ICV made the datagram valid at the ESP stage. Decryption then reconstructed the next protocol or inner datagram.

Even that sequence did not promise application acceptance. RFC 2406 noted that malformed output from a wrong SA or corrupted encryption might survive IPsec detection when authentication was absent and be left to later protocol processing. If authentication and decryption ran in parallel, verification had to finish before decrypted material was released. ESP could protect a transition without deciding what the receiving system should do with its result.

Auditing was similarly conditional. A system that supported auditing had to let ESP participate, and the RFC named events such as missing SAs, fragment errors, sequence overflow and ICV failure. But not every system was required to implement auditing, and granularity was largely local. “Auditable” in a specification was not proof that an operator retained a useful log.

RFC 4303 replaced RFC 2406 in 2005. It retained the visible SPI and sequence fields and the separation among transport, tunnel, confidentiality, integrity and receiver-controlled replay, while adding extended sequence numbers, combined-mode algorithms and explicit traffic-flow-confidentiality padding. RFC 2406 should therefore be read as a historical design stage, not present-day algorithm guidance or evidence of current deployment.

Lu Heng’s insistence on separating symbolic labels from running effects helps explain why the old packet diagram still matters. “Encrypted” is a category name. The operational claim requires the chosen mode, service combination, SA, algorithms, receiver checks, outer-flow exposure and downstream disposition. The specification drew those lines. Later shorthand often erased them.

Sources