Summary

  • A SYN cookie encodes recoverable provisional TCP state in the server's SYN-ACK sequence number, so the server can wait for a valid third ACK before allocating an explicit connection record.
  • The returned cookie is evidence of recent access to the reply path for one connection tuple; it is not user authentication and does not protect packet processing, bandwidth, established state or application capacity.
  • Cookie mode compresses negotiation into a bounded field. Operators must know its activation trigger, retained options, failure behaviour and downstream capacity rather than treating it as a universal DDoS switch.

The ordinary passive TCP open contains a small asymmetry. A client sends one SYN. The server enters SYN-RECEIVED, prepares a SYN-ACK and remembers enough about the unfinished exchange to recognize the returning acknowledgement. The client has not yet demonstrated that it receives packets at the claimed source address, yet the server has already committed a scarce backlog entry and some memory to that claim.

A SYN flood monetizes this timing gap in packets. Many SYNs create many provisional obligations. If their sources are spoofed or their senders decline to answer, those obligations wait for retransmission and expiry while legitimate clients compete for the same half-open capacity. RFC 4987 carefully limits this model: the classic attack exhausts an endpoint's connection-establishment state; it is not automatically the same as saturating a link or every packet-processing resource.

The obvious remedies allocate loss differently. A larger backlog buys time but also increases memory at risk. A shorter SYN-RECEIVED timer reclaims entries sooner but punishes clients on long or lossy paths. Recycling the oldest entry keeps the table bounded while making age, not legitimacy, decide which attempt disappears. A SYN cache stores a smaller request record. A proxy completes the handshake elsewhere and moves both semantics and exposure to another box.

The SYN cookie makes the more radical move: do not retain an explicit request record at all. Instead, choose the server's initial sequence number so that it carries a compact description of the unfinished connection and a secret-derived check. RFC 4987's archived example combines information from the connection's addresses and ports, the client's initial sequence number, a slowly changing counter, a class of advertised maximum segment size and server-held secrets.

The client does not need to understand this encoding. It receives the SYN-ACK and returns an ACK for the server sequence number plus one, as ordinary TCP requires. When that third packet arrives, the server subtracts the expected offset, checks recent time-counter possibilities and recomputes its secret function over the observed tuple. A match allows it to reconstruct the state and enter ESTABLISHED. A miss can be discarded without discovering an embryonic record that never existed.

This is best understood as a receipt carried by the requester. The server wrote the receipt, sealed it with information only it can validate and handed it out without opening an account. The requester returns it before the account is created. Server memory has been replaced by a time-bounded claim in the wire conversation.

The receipt proves less than casual security language suggests. A valid ACK is strong evidence that someone learned the recent SYN-ACK for that address-and-port tuple, whether by receiving it on the return path or by otherwise observing it. It does not name a human, bind a company, authorize an application request or show that the resulting connection will be cheap. Cryptographic guessing resistance protects the admission token; it does not transform TCP into an identity protocol.

The 32-bit sequence-number field also imposes an information budget. A normal request record may remember exact options presented in the first SYN. A basic cookie must reserve bits for freshness, integrity and at least some transport parameters. Historical schemes encoded a small index into common MSS values. Information that is neither encoded nor inferable from the completing ACK cannot be reconstructed faithfully.

Window Scale exposes the consequence. RFC 7323 says the option is exchanged only in SYN-bearing segments and fixes a scale factor in each direction when the connection opens. If a stateless server forgets the client's offer and has no cookie extension that can recover it, the value cannot simply be negotiated after the third packet. RFC 4987 consequently documented option and performance compromises in basic cookie schemes, while also describing a FreeBSD design that used echoed TCP Timestamp bits to carry additional state, including window scaling and SACK permission.

That history forbids two easy claims. It is wrong to say that every SYN-cookie implementation loses the same options; later and platform-specific encodings recover more information. It is equally wrong to assume that every option survives because one implementation found extra bits. The relevant contract is the running stack, kernel version, offload path and listener configuration in front of the actual service.

Application timing introduces another edge. RFC 4987 describes a server-first protocol such as SMTP: if the client's handshake-completing ACK is lost, the stateless server has no remembered SYN-ACK to retransmit and has not notified the application, while the client may believe the handshake completed and wait for the greeting. Retransmission and implementation behaviour determine recovery. The example matters because protecting memory can change which side first notices loss.

Hybrid operation is therefore more credible than a permanent slogan. Under ordinary conditions, a SYN cache can preserve richer state and normal retransmission behaviour. When the cache or per-listener backlog overflows, the implementation can fall back to cookies instead of evicting an existing attempt. That transition turns cookie mode into an emergency allocation policy rather than a universal replacement for connection setup.

The current Linux kernel documentation makes this boundary unusually explicit. It describes tcp_syncookies as a fallback when a socket's SYN backlog overflows and warns that it must not be used to make a highly loaded server withstand a legitimate connection rate. If ordinary demand creates the warning, the operator should investigate backlog, retransmission and service capacity. A defence that conceals underprovisioning has converted an attack alarm into a capacity alibi.

Even a perfect cookie leaves several budgets untouched. The host still receives and parses SYNs, computes replies, transmits SYN-ACKs and validates returned ACKs. Links, interrupts, queues and CPU can still saturate. Real distributed clients can return valid cookies and then consume established socket memory, TLS handshakes, worker pools, databases and application quotas. The accept queue can fill after TCP establishment. Stateless admission protects one commitment boundary, not the service behind it.

Source-address validation remains complementary. BCP 38 and BCP 84 make spoofed traffic harder to export, reducing an attacker's ability to create replies that disappear toward fabricated addresses. They do not make a server immune to bots using real addresses or to valid clients whose rate exceeds service capacity. The local stack still needs its own bounded admission rule because upstream filtering and endpoint allocation answer different questions.

Observability must follow the same separation. RFC 4898 distinguishes embryonic backlog events, the later commitment of full resources and application acceptance. It notes that a SYN-cookie implementation has no explicit representation of the encoded SYN-RCVD state. A dashboard that watches only current half-open rows can therefore report a calm queue while the server is actively issuing cookies under pressure.

The useful record includes incoming SYN rate, SYN-ACK and retransmission rate, backlog occupancy and overflow, cookie generation and validation, invalid or stale returns, handshake completion, accept-queue occupancy, application accepts, connection latency and the options negotiated on cookie-established connections. Without the transition counter, an empty backlog is ambiguous: it may mean no pressure, or it may mean the server stopped remembering.

Newer handshake extensions sharpen this requirement. TCP Fast Open can carry application data in the SYN under a different cookie system and separate replay assumptions. Operators cannot assume that a conventional SYN-cookie fallback retains SYN data or Fast Open semantics. Interaction among listener options, front-end proxies, load balancers and kernel offload must be verified on the path that receives production traffic.

SCTP shows what changes when the protocol reserves a first-class place for deferred state. RFC 9260's four-way setup carries a variable-length State Cookie containing the information needed to create the association, protected by a MAC and bounded by a timestamp and lifespan. That is not TCP wire compatibility, and it is not the same compact algorithm. It does reveal the design lesson: stateless admission is easier to preserve when the protocol explicitly provides a container for it instead of compressing an evolving negotiation into an initial sequence number.

The institutional lesson is equally bounded. Shared specifications can describe the handshake and document mitigation trade-offs. They cannot know a listener's legitimate arrival rate, memory, latency distribution, negotiated option set or downstream service cost. Heng Lu's minimum-specification principle places the activation threshold and acceptable loss with the operator who bears them. Running-code primacy then supplies the test: the configured knob is only an intention until observed connections show which state, options and clients survived.

Sources