Summary

  • 5x9 Networks says its Big VM design put one forwarder and 64,000 subscribers on a 144-core Xeon 6780E, producing 200 Mpps with ACLs and no hierarchical QoS, or 800 Gbps at 500-byte packets. Its 1.6 Tbps figure scales the result across two CPUs.
  • The design removes duplicated cache state from 16 Small VM forwarders. That is a concrete efficiency mechanism, but it also concentrates execution, change and subscriber state unless separate redundancy and recovery are demonstrated.
  • The APRICOT deck publishes no state synchronisation, failure injection, restart, upgrade, session-loss or recovery test. The operator must therefore accept throughput and failure behaviour separately before treating the benchmark path as available service capacity.

A Broadband Network Gateway sits at an expensive seam. It terminates PPPoE or IPoE access sessions, forwards Layer 3 traffic and may enforce per-user QoS, ACLs, authentication, accounting and lawful-intercept obligations. When it fails, the problem is not only that packets stop. Subscriber state has to exist somewhere, remain coherent and recover on a time scale the access service can tolerate.

Branimir Rajtar, CTO and Co-Founder of 5x9 Networks, presented Getting 1+ Tbps from an x86 server in the APRICOT 2026 Network Operations session on 11 February in Jakarta. The 19-page deck documents a real engineering progression in the company's virtualised BNG: from many small forwarders to one large forwarder, with SR-IOV, Intel DPDK, newer CPUs, cache work, PCIe/DMA tuning and code redesign.

It is a vendor record. That does not make its measurements disposable. It defines how far they can be carried.

The starting architecture used two second-generation Xeon Gold processors with 18 cores each and 16 forwarder instances. 5x9 reports 40 Mpps without hierarchical QoS and 26 Mpps with it. At a 500-byte packet size the deck gives 160 Gbps and 100 Gbps. The difference is about 35% in that configuration. It is not a universal QoS tax; it is evidence that the feature mix changes the capacity number materially.

The company then describes several cycles of hardware, application and low-level optimisation. Newer hardware and application work added a stated 30%. Deeper DPDK, CPU, PCIe, BIOS and NIC tuning added another stated 30%. Profiling and code rewriting added 20%. These increments should not be treated as a portable formula. Their value is that they identify the actual dependency chain: performance did not emerge from the label x86. It followed a particular CPU, memory behaviour, device layout, driver support, BIOS, NIC and application.

The main bottleneck, according to the deck, became CPU cache misses and waits. Large numbers of memory objects occupied cache; changing data induced stalls; the processor fetched cache lines even when the useful object was smaller. The team separated structures used for reading from those used for writing, adjusted PCIe and DMA scheduling, changed hash algorithms and reports reducing the routing-table footprint by 90%.

That work led to the architectural decision at the centre of the article. Sixteen Small VMs stored the same information repeatedly in cache. The Big VM combined them, spanning an entire NUMA domain and all CPU cores so that the repeated state could become one organised working set.

This is a physical gain. Cache locality can reduce main-memory traffic and waiting. Fewer forwarders can also reduce duplicated tables, process overhead and server count. None of those effects needs an abstract appeal to software-defined infrastructure.

The current result shown in the deck uses one Xeon 6780E with 144 cores. Intel's product page independently confirms the core count and identifies 108 MB of cache, up to 88 PCIe 5.0 lanes and a 330 W server-mode TDP. Those product specifications do not verify the BNG test; they show why CPU, memory, lanes, slots, NICs and power remain part of the packet path.

On that processor, 5x9 reports one forwarder, 64,000 subscribers, 200 Mpps with ACLs and no hierarchical QoS, 800 Gbps at a 500-byte packet size and 32 GB of memory. The deck reaches 1.6 Tbps by scaling across two CPUs. The distinction matters. One server is not the same unit as one CPU, and the headline must not erase the second processor.

The feature qualifications matter as much. The deck says performance starts falling beyond 100,000 subscribers, although as many as 260,000 are supported. It reports about 30% lower performance when hierarchical QoS is enabled for all users and 30–40% lower performance when NAT is used for all users. The 800 Gbps line therefore describes a bounded packet and feature condition, not the capacity of every service the BNG can terminate.

DPDK and SR-IOV explain part of the mechanism. DPDK Poll Mode Drivers access receive and transmit descriptors directly and ordinarily poll rather than waiting for packet interrupts. SR-IOV lets one PCIe device expose Virtual Functions under a Physical Function. These mechanisms shorten or reshape the data path. They do not answer where subscriber state is copied, what happens when the forwarder process exits, whether a Virtual Function or NIC can be replaced without session loss, or how another server assumes the load.

CUPS also has a precise limit. Separating control and user planes can keep policy and session control distinct from packet forwarding. It does not prove that the user plane will continue when its one Big VM, CPU, PCIe path, NIC or server fails. A controller can remain healthy while the packet and state boundary disappears beneath it.

The public APRICOT material contains no redundancy diagram, state-synchronisation method or N+1 reserve. It publishes no injected forwarder, NIC, CPU or server failure; no restart time; no upgrade and rollback sequence; no subscriber-session loss or recovery measurement; no mixed packet-size and full-feature result; and no customer production outcome. This is not evidence that those controls do not exist. It is evidence that the presentation does not measure them.

The distinction should discipline the strongest claim. The deck demonstrates a performance path reported by its vendor. It does not demonstrate that the larger forwarder is unreliable, nor does it prove available BNG capacity. Availability is a result across both load and interruption.

The shift from 16 processes to one also changes the unit that operations must accept. With Small VMs, repeated cache state costs throughput, but a process or VM can represent a smaller set of subscribers. With one Big VM, the working set becomes more efficient while more session state enters the same software, CPU and maintenance boundary. The relationship is not automatically one-for-one: operators may deploy separate Big VMs, servers or state replication. The deck simply does not show that layer.

That missing layer determines who holds power. 5x9 controls its code, instrumentation, test configuration and claim. Intel and the NIC vendors control silicon, firmware, DPDK compatibility and qualified combinations. The deploying operator chooses topology, spare capacity, feature mix, change window and the acceptance threshold for customer service. APRICOT publishes the discussion; it does not certify a platform.

The costs follow those powers. The operator pays for current processors, supported NICs, PCIe-capable servers, engineering, licences, repeat testing and whatever idle capacity is required for a failure. Reducing the number of active servers may lower space, power and operating cost, as the deck proposes. The public record gives no total-cost calculation and no power measurement. A spare capable of receiving the same state and load cannot be removed from the economics merely because it is idle.

Subscribers bear the other side of concentration. If the large forwarder can be replaced within a bounded time and state remains coherent, they may receive a cheaper, flexible access service. If not, a process or server event can correlate more sessions. The article does not assert that such an outage happened. It identifies the test that separates those two outcomes.

A credible acceptance record would freeze the complete service condition. It would declare packet sizes and directions, active subscribers, IPv4/IPv6 mix, ACLs, hierarchical QoS, NAT, AAA and control-plane transactions. It would state the server, CPU, memory, NUMA, PCIe and NIC layout and the reserve held outside the measured load.

Then it would interrupt the system. Kill the forwarder. Remove a Virtual Function or NIC. Restart the VM. Lose a CPU or server. Upgrade the data plane, roll it back and measure both the packet rate and the established sessions. Record which state was synchronised, how many sessions reset, how long recovery took and what steady-state throughput remained while the spare carried traffic.

The counterfactual need not discard the Big VM. Two or more large forwarders with N+1 reserve and tested state handoff may retain the cache benefit. Small VMs may remain preferable at an edge site where subscriber count and blast radius matter more than maximum packet rate; the 5x9 deck itself keeps that option. ASIC or white-box systems can be compared only if they face the same service and failure record.

Internet number resources sit outside this acceptance boundary. An ASN and its prefixes can identify the operator and show external routing. BGP can demonstrate that a path is announced and reachable. None of those records proves that the BNG behind the route can maintain subscriber state, hierarchical QoS or NAT through a change. An RIR and a conference have no legitimate mechanism to certify that internal capacity.

The important achievement in the 5x9 account is not that software made physical limits disappear. It is that engineers found the limits in cache lines, memory objects, NUMA, PCIe, NICs and code, then moved them. The next boundary is equally concrete: one forwarder, one state domain and the hardware beneath it.

Cache efficiency can reduce server count. It does not by itself reduce the failure domain. The APRICOT result is useful when read at its stated packet, feature and processor boundary. It becomes saleable service capacity only when the same architecture survives the failures and changes the operator is promising subscribers it can absorb.

Sources