Summary

  • RFC 1191 made the IPv4 source set Don't Fragment and rely on a constraining router to return an ICMP message carrying the next-hop MTU. The router supplied evidence; the endpoint chose how to reduce its packet size.
  • RFC 2923 documented the resulting black hole when ICMP was filtered or lost: a TCP connection could start and then hang as larger packets disappeared. TCP MSS was not a complete substitute for path knowledge.
  • Cloudflare's 2015 ECMP incident showed another failure. TCP and ICMP were hashed differently, so a valid Packet Too Big message could reach a backend that did not own the connection.
  • PLPMTUD and DPLPMTUD let endpoints probe sizes without depending on delivered ICMP. They restore an exit from silence, but require probe traffic, state, delivery confirmation and a disciplined distinction between an MTU limit and other loss.

The data arrived; the explanation did not

Cloudflare's February 2015 account begins with an internal routing change. TCP traffic was distributed across servers with equal-cost multipath routing. The router hashed the familiar four-tuple, so packets belonging to one connection returned to the same backend.

ICMP did not carry the same information in the form the router used. Cloudflare said its ECMP path hashed the source and destination addresses for those messages. The Packet Too Big warning could therefore land on another backend even though the TCP flow kept reaching the correct one. The company wrote that this was exactly what had happened in its case.

Only a couple of users connecting through IPv6 tunnels had reported the symptom. That number cannot establish worldwide prevalence. What the postmortem does establish is a cleaner point: a control message can reach the right organisation and still miss the state that gives it meaning.

Cloudflare first reduced its IPv6 MTU to 1280 and enabled RFC 4821 probing for IPv4. Its longer-term daemon captured IPv4 fragmentation-needed and IPv6 Packet Too Big messages, then broadcast them across the backend network. The accompanying repository set a one-packet-per-second limit for one source and a ten-packets-per-second limit on one interface.

Broadcast converted connection-specific evidence into shared internal evidence. It also created a new validation, distribution and rate-control obligation. The repair was specific to Cloudflare's architecture; it was not a universal design instruction.

Before discovery, the network paid in fragments or small packets

RFC 1191, published in November 1990, defined a path MTU as the smallest MTU among the hops between a source and destination. The definition was already unstable in two dimensions. A path could contain a hidden constraining link, and routing could later change which link was smallest.

The prior conservative practice used the lesser of 576 octets and the first-hop MTU. It avoided assuming that an unknown path could carry a large datagram. It also wasted capacity on paths able to carry more, while still allowing fragmentation if a later hop was smaller than the source expected.

Routine fragmentation moved work into the path. Routers could split one IPv4 datagram into fragments that the destination had to reassemble. Losing a fragment could invalidate the whole original datagram. Later operational work catalogued more difficulties with filtering, stateful devices, unequal paths and tunnels.

The choice in 1990 was not between a perfect discovery system and irrational fragmentation. Sources could stay small, let routers fragment when permitted, obtain explicit path feedback, or combine these behaviours. RFC 1191 chose feedback because it could approach the useful size of the actual path without making every router fragment every oversized packet.

RFC 1191 assigned knowledge to the path and action to the source

The classic IPv4 mechanism has a short control loop. The source sets the Don't Fragment bit. A router that cannot forward the datagram on its next hop without fragmentation discards it and sends ICMP Destination Unreachable with the fragmentation-needed code.

RFC 1191 requires the router to include the next-hop MTU. The source then lowers its path estimate. It may send smaller packets or stop setting Don't Fragment; the router reports the local constraint but does not choose the source's transport policy.

The asymmetry matters. The router has direct knowledge of one interface. The endpoint owns packetization and retains the estimate. Neither has the other's control surface, so the ICMP message is not optional explanatory decoration. It is the bridge between observation and corrective power.

The standard also distrusted effortless increases. An endpoint must not raise its estimate because an ICMP message advertises a larger value. It has to probe later. RFC 1191 recommended waiting at least five minutes after a reduction, preferably ten, before testing upward again.

Those intervals are historical implementation guidance, not measurements of every stack. They show what the designers feared: route changes and false or stale messages could make a previously learned value unsafe in either direction.

IPv6 removed router fragmentation, not the need to learn

IPv6 sharpened the allocation of responsibility. The current base specification requires every link either to carry a 1280-octet IPv6 packet or to provide link-specific fragmentation and reassembly below IP. Routers do not fragment forwarded IPv6 packets. Only the source can add a Fragment header.

RFC 8200 strongly recommends PMTUD so a source can use sizes above 1280. A minimal implementation may instead send no packet larger than 1280. That is a legitimate exit from full discovery, but it is an efficiency compromise rather than proof that every path has a 1280 MTU.

Under RFC 8201, an IPv6 router that cannot forward a packet returns ICMPv6 Packet Too Big. The endpoint must validate that the quoted packet corresponds to traffic it sent. A valid message can reduce the estimate, but not below the IPv6 minimum of 1280.

A Packet Too Big message must never increase the estimate. The source can later probe upward, no more frequently than every five minutes after a reduction and preferably every ten. Again the rules bind the evidence: a router can report a constraint, but the endpoint must authenticate context and decide how long to retain it.

RFC 4890 placed Packet Too Big among ICMPv6 traffic that firewalls must not drop because communications can be prevented or severely impaired. That is clear operational guidance. It does not prove that every firewall follows it or that every other ICMP message deserves identical treatment.

Silence became a black hole

RFC 2923 gave the feedback failure its durable symptom. If a router, firewall, kernel fault or configuration prevented the ICMP error from returning, the source continued sending packets too large for the path. The path discarded them, and the endpoint did not learn why.

A TCP handshake could succeed because its packets were small. Data could begin moving. Then a larger segment disappeared, retransmission repeated the same mistake and the connection appeared to freeze. The document's FTP example had a small control channel working while bulk transfer failed.

That pattern is evidence worth testing, not a verdict. Congestion, corruption, asymmetric policy and other filtering can also lose packets. A serious diagnosis records sizes, address family, DF or Fragment behaviour, retransmissions and any quoted ICMP message rather than naming an MTU black hole from one failed download.

TCP MSS does not remove this uncertainty. It tells the other endpoint the largest TCP payload a receiver is prepared to accept. It does not reveal every tunnel or link on the forward path, and the return path may be different.

RFC 2923 preferred repairing the ICMP path. It also described endpoint black-hole detection: after retransmission timeouts, send smaller packets. That could restore the session, but it added seconds and concealed a path defect that would continue affecting other traffic.

Firewalls held a veto without holding the constraint

An operator can have legitimate authority to filter traffic on its own network. The historical failure does not prove that such an operator acted illegally, maliciously or without a security rationale. The harder question is what the filter was entitled to conclude.

Dropping the control message did not make the larger packet forwardable. It removed the evidence from the endpoint while leaving the link constraint in force. Local policy therefore acquired a practical veto over adaptation without acquiring the router's knowledge or the endpoint's responsibility.

Tunnels deepened the split. Encapsulation adds headers after an endpoint has chosen a packet size, reducing the effective capacity available to the original packet. Traffic distributors and anycast systems can then give data and error traffic different instance identities. The number called “the MTU” becomes meaningful only with a time, direction, flow and encapsulation context.

This is why PMTU is not a permanent attribute of an address block or prefix. An RIR can allocate number resources; it does not certify the end-to-end packet size on every route to the holder. The path state is produced by routing and operations after allocation.

Packetization-layer discovery moved the experiment to the endpoint

RFC 4821 in 2007 changed the evidence source. Packetization Layer PMTUD sends progressively larger probe packets and learns from the transport or packetization layer's delivery feedback. It can make progress even when ICMP never arrives.

The endpoint gains an exit from silence, not certainty. An isolated lost probe may indicate that the tested size crossed the limit. Loss around the probe makes the result inconclusive because congestion or another impairment may be responsible. The method therefore needs search state, timing and cautious interpretation.

The IETF PMTUD Working Group charter described the weakness of earlier PMTUD as a chronic obstacle for new links and tunnels. Its proposed direction began with a small packet and probed upward without relying on ICMP. The charter records an institutional problem statement; it cannot prove universal consensus, adoption or success.

RFC 8899 later extended the approach to datagram packetization layers. A DPLPMTUD implementation tests sizes, raises the confirmed value after delivery confirmation and reduces it when it detects a black hole. It may use a validated Packet Too Big message, but cannot depend on that message alone and can never use it to raise the size.

UDP supplies no acknowledgement by itself. The transport or application must provide a way to confirm a probe. That is the cost of moving control upward: an application that once just sent datagrams may now need state and feedback to learn a path property.

The old and new mechanisms now coexist

The current TCP specification strongly recommends PMTUD and recommends PLPMTUD. Fixed fallback sizes remain defined, but RFC 9293 warns that sending only small packets can impair performance. The standards record did not replace one universal mechanism with another; it built an endpoint alternative alongside useful router feedback.

Current Linux documentation exposes that coexistence. tcp_mtu_probing=0 disables TCP probing. Mode 1 activates it after the stack detects an ICMP black hole. Mode 2 keeps it enabled using a configured base MSS. The documented probe interval defaults to ten minutes.

These controls prove operator choice, not global settings. They also reveal the lock-in. A host can still benefit from a valid ICMP message, fall back to probing when it suspects silence, or remain configured not to probe. Firewalls, tunnels, kernels and applications did not change in one coordinated event.

RFC 8900 therefore stopped short of declaring fragmentation abolished. It catalogued fragility and advised upper layers to reduce reliance on it. The result is a layered settlement: retain the explicit path signal, validate it, build a probing escape route and keep conservative sizes available when neither can be trusted.