Summary

  • RFC 3511 defined firewall performance tests whose meaning depended on topology, rule sets, NAT, caching, authentication, packet and object sizes, connection lifecycle and reporting conditions; RFC 9411 replaced that method with a security-effectiveness-first contract for modern inline devices.
  • A defensible benchmark record keeps the method revision, exact DUT/SUT configuration, enabled functions, reference testbed, traffic and crypto profile, load phases, validation failures, measurement layer and sustained result together. The detached number is not evidence of production capacity or security.

The number was still precise: a throughput figure with two decimal places. What had vanished was the experiment.

No one could recover whether NAT was enabled, how many rules were loaded, whether authentication crossed an external server, whether responses came from cache, which packet sizes were offered, or how many connections failed. A later slide called the number “firewall capacity,” as if the noun supplied the missing denominator.

RFC 3511 was published in April 2003 as an Informational RFC from the IETF Benchmarking Methodology Working Group. It defined tests for forwarding, connections, latency and filtering. In March 2023, RFC 9411 explicitly made it obsolete and introduced a methodology for modern network-security devices whose work includes encrypted traffic and Layer 7 inspection.

The replacement did not reveal that benchmarking was useless. It revealed that the tested object had changed. A result remains intelligible only when the contract that produced it remains attached.

RFC 3511 measured a configured system

The old method already resisted the idea of one natural performance number. The Device or System Under Test sat inside a topology with virtual clients and servers, protected and unprotected segments, traffic generators and a chosen flow distribution. The report had to identify enough of that environment for another reader to understand what work was done.

NAT is the clearest example. Translation adds processing, so RFC 3511 said tests should run with NAT disabled and enabled to expose the differential. The report had to say which state applied. A NAT-off result and a NAT-on result are not two samples of one undocumented condition. They are different experiments.

Rule sets are equally constitutive. Their entries determine what the firewall forwards and rejects. Their order and match distribution can change the work needed for a packet. The method recommended denying traffic not already defined and said the configured rule sets should appear in the report.

That means configuration is not explanatory prose added after the number. It defines the measured system.

A fast path may be a smaller path

Caching can make an HTTP response arrive from internal memory rather than traverse the intended server path. Authentication can add a remote dependency and its latency. A TCP stack can change timeouts, retransmissions, setup and teardown. A result aggregated across Protected, Unprotected and DMZ paths can conceal different constraints.

Each choice can legitimately describe a useful scenario. The error begins when scenario names are stripped away.

If caching is enabled, the report should distinguish cache hits from origin work. If authentication is external, the dependency and its timing belong in the record. If a result aggregates paths, it must be labelled as aggregate. If a benchmark disables a feature, the result describes the reduced path rather than an abstract full product.

This is why capacity cannot be inferred from the appliance name. The workload is partly created by policy.

Throughput is a tuple, not a scalar

RFC 3511’s IP-throughput reporting separated intended load, offered load, forwarding rate, packet size, packets transmitted and packets forwarded. Its latency method tied a legitimate latency measurement to the highest offered load with zero packet loss.

“Zero loss” therefore has coordinates. It belongs to a packet size, traffic direction, offered rate, duration, rule path and validation procedure. Move one coordinate and the claim must be measured again.

The same applies above the packet layer. HTTP transfer rate depends on object size, requests per connection, completed requests and responses, close behavior, timeouts and retransmitted bytes. Goodput at one layer is not automatically application throughput at another.

A chart that retains only the tallest bar destroys the comparison contract. Its numerical precision cannot compensate for missing provenance.

Connection capacity is not connection work

RFC 3511 treated concurrent TCP connection capacity, maximum connection-establishment rate and maximum teardown rate as separate tests. That separation matters.

A large state table may hold many quiet connections while admitting new ones slowly. A system may open connections quickly and reclaim them poorly. It may sustain TCP handshakes while completing fewer useful HTTP transactions. It may preserve an average while a tail of sessions times out.

To call all of these “connections per second” is to remove the lifecycle. A proper receipt records how sessions were opened, held, exercised, closed and validated; it records unexpected resets and failures rather than counting only successful state insertions.

Capacity is not merely a number of slots. It is a sequence of owned state transitions.

Security did not hide inside the performance result

RFC 3511 included denial-of-service and illegal-traffic handling tests, but its Security Considerations were explicit: assessment of firewall security lay outside the document’s scope.

That boundary prevents two opposite mistakes. A high forwarding rate does not prove that prohibited traffic was blocked. A low rate under a defined attack does not prove that every denial-of-service mechanism was exercised.

The illegal-traffic test required reporting the number and percentage of prohibited connections allowed. Without that outcome, a fast allow path could look like excellent performance. The denial-of-service test measured the effect of a particular attack load on specific connection or transfer rates. It was not a universal resistance certificate.

Performance and security can interact without becoming the same fact.

RFC 9411 changed the object under test

By 2023, a firewall benchmark that ignored application recognition, intrusion prevention, malware controls or encrypted inspection could describe a deliberately smaller object than the system buyers thought they were comparing.

RFC 9411 therefore defines modern inline NGFW and NGIPS methodology. It begins with the chosen security-effectiveness configuration and then measures performance. Recommended functions and deviations must be disclosed and held consistently across tests. A realistic number of ACLs should be used where relevant.

The method also requires an isolated test environment and a reference test without the DUT/SUT. Otherwise a switch, router, virtual function, physical link, competing workload or thermal event can be charged to the security device.

The benchmark envelope grew because the implementation surface grew.

Modern traffic carries cryptographic work

RFC 9411 introduces application traffic mixes and requires their composition to be reported: applications, Layer 7 protocols, encrypted share, direction and object sizes.

Its HTTPS paths can include HTTP/1.1, HTTP/2 or HTTP/3, TCP or QUIC, TLS inspection, SNI, ALPN, certificates, cipher suites, keys and record sizes. The baseline disables TLS session reuse and performs full handshakes; the QUIC profile disables 0-RTT and early data. Those choices intentionally define work that real sessions may perform differently.

An HTTPS throughput number without the TLS version, cipher and key, resumption state, inspection state, protocol, object sizes and traffic mix has no stable comparison target. Even the same box can yield different numbers without contradiction because the experiment asked a different question.

The right response is not to choose the most favourable number. It is to preserve the question.

Sustained validity replaces the instant peak

RFC 9411 separates initialization, ramp-up, sustain and ramp-down. The ramp must not overload the device before measurement. KPIs are taken during the sustain phase.

For application-mix throughput, failed transactions and unexpected TCP resets must remain below specified validation thresholds; HTTP/3 tests add unexpected QUIC-error criteria. A successful HTTP transaction transfers all data and receives a valid response, not merely some wire bytes.

Time-series reporting matters because an average can hide a fall after state tables fill, inspection queues grow, caches cool, certificates rotate or thermal limits arrive. A peak before the sustain window is not sustainable inspected throughput.

The metric needs both the value and the conditions under which it stayed valid.

Obsolescence is a provenance transition

RFC 9411 did not retroactively erase measurements made under RFC 3511. It changed the contract against which a reader should interpret them.

A historical result can still answer a historical question if its configuration survives. It cannot be silently placed in a series with a modern result that adds security-effectiveness validation, application traffic, encryption, QUIC and different failure criteria.

The method version belongs beside the value. So do deviations. If a longitudinal chart crosses the 2023 boundary, it needs an explicit discontinuity rather than an invented trend.

This is the operational meaning of preserving history without granting old labels present authority.

The minimum benchmark receipt

Retain the RFC and revision used, test date, DUT/SUT hardware and software, licences, feature flags and policy export. Record interface and topology maps, reference-test results, routing and switching dependencies, NAT, cache and authentication state, ACLs and rule-hit distribution.

Keep client and server stack settings, address spaces, packet and object sizes, applications, encryption share, TLS/QUIC parameters, certificates, ciphers and resumption policy. Preserve init, ramp, sustain and ramp-down timing, offered load, measurement layer, completed transactions, drops, resets, protocol errors, latency distribution and resource telemetry.

Bind the final number to these hashes and records. A summary may display one KPI, but the KPI must resolve back to the complete envelope.

Then record the separate production decision: which real traffic, topology, policy and failure margin justify applying the laboratory receipt to capacity planning.

Evidence boundary

This article does not identify or compare a firewall vendor, model, software release, laboratory or customer. It reports no benchmark score, exploit result, deployment prevalence, breach or production incident.

RFC 3511 is described as an April 2003 Informational methodology and as obsolete under RFC 9411. RFC 9411 is described as a March 2023 Informational methodology, not a certification programme or guarantee.

The 0.001% validation thresholds in RFC 9411 are test-result criteria, not a general acceptable security-failure rate. Disabling machine-learning or behavioural functions defines the method’s scope; it does not measure those functions.

Heng Lu’s minimum-specification and running-code principles are disclosed editorial lenses. They support a small shared receipt and local responsibility for deployment decisions. They are not firewall test data.

The narrow conclusion is enough: when the configuration disappears, the number retains digits but loses identity.

Sources