Summary

  • The IESG approved a Benchmarking & Performance Measurements working group on 29 October. Its charter covers controlled lab tests and measurements on live, variable networks.
  • BPM will define methods and reporting, but its benchmarking recommendations will not set acceptance criteria or performance requirements. The buyer or operator must write the pass mark.

A test report can contain a beautiful number and still leave the meeting room with its most expensive question unanswered: would you put this box, controller or service into your network? The new IETF Benchmarking & Performance Measurements working group can help everyone take the measurement the same way. Its charter deliberately stops before it tells anyone what result is good enough.

That division is in the document approved by the Internet Engineering Steering Group on 29 October. BPM will develop metrics, methodologies and protocols for the quality, performance, reliability and scale of data-delivery services and applications. It covers routers, firewalls, data-centre fabrics, controllers, virtual systems, containers, network-management software and endpoints. It reaches the forwarding plane, the control plane and the management plane. If a result can travel from a test rig to a purchasing committee or an operations review, the new group wants the method behind it to be intelligible.

The charter joins two kinds of work that used to have different homes. The older Benchmarking Methodology working group confined its tests to controlled laboratory stimuli; its charter expressly excluded live operational-network benchmarks. The IP Performance Measurement group developed metrics and methods for delivery services and applications over IP, including active, passive and hybrid measurement. BPM is named as the successor to both and inherits maintenance of their RFCs.

The two older group pages still display an active status in the Datatracker today; the approved charter establishes the new remit, while the public directory has not yet caught up with its account of succession.

The lab and the running network remain different witnesses. A laboratory can isolate a device, repeat a traffic pattern and calibrate the test equipment. A production network changes under the measurement: traffic mix, routing, endpoint load and neighbouring flows vary. BPM's charter describes both environments and asks for a common body of metrics, procedures and terminology. It also requires methods for production tests to minimize their impact on the data, control and management planes. A common vocabulary can connect the two reports. It cannot make their conditions identical.

That is why the charter insists on the test environment and unambiguous reporting. Consider a procurement file containing a throughput figure from a quiet, calibrated lab and a service report from a busy network. The figures may share a metric name while answering different questions. The first characterizes an implementation under stated stimuli. The second describes a service during a particular operational interval. The missing bridge is a description of configuration, traffic, topology, observation window, calibration and the service obligation against which the result will be judged.

The charter supplies the reporting discipline; the organization deploying the service supplies its obligation.

The IETF has been explicit about this boundary before. BMWG's older charter declined to define acceptance criteria because deployments make different demands. BPM carries that refusal into a broader measurement portfolio. Its benchmarking recommendations are to describe performance characteristics, test environment and clear result reporting while striving for vendor independence. They will not set a universal pass line. That sentence prevents a standard method from quietly becoming a purchasing rule written by the standards committee for every network on earth.

BCP 170, the IETF's guidance for developing performance metrics, shows how a neat average can get in the way of the actual decision. It gives packet loss as an example: a mean loss rate may be useful, yet an application that suffers during short bursts needs the distribution of those losses. Two reports can be numerically comparable on the average and still have different value for the application. The same guidance asks metric authors to identify their audience, method, sampling, topology, accuracy and reporting period. Those are conditions for a usable number, not decoration around it.

BPM also treats measurement traffic as an operating cost. Its charter asks new methods to mitigate harm to flows sharing a path and to endpoints that did not expect the test traffic. A live probe can consume capacity or disturb the thing it claims to observe. That is a practical reason to name the test's owner, rate, schedule and stop condition before a production trial. The buyer who demands a measurement and the operator who runs it should not outsource those choices to the mere existence of a protocol.

The working group can standardize more than a stopwatch. Its remit includes active, passive and hybrid protocols, YANG data models for performance-related configuration and operation, and IPFIX entities for telemetry export. That creates a route for results to move into management systems. A machine-readable field makes a measurement easier to collect at scale. It also makes a poorly specified threshold easier to apply at scale. The organization that turns the field into an alert, a procurement score or a service penalty has made a second decision. BPM's charter does not make it for them.

For a buyer, the useful contract therefore has two documents inside it. The test method names the metric, equipment, configuration, traffic, calibration and reporting format. The acceptance schedule names the service to be delivered, the load and period in which it must work, the result that passes, the exception process and the retest. Write the second part before a vendor hands over a striking lab number. Otherwise the most polished report in the tender becomes a substitute for the one decision the customer needed to make.

The new group gives the industry a chance to stop arguing over incompatible stopwatches. It also makes the remaining argument harder to hide. Once both sides can see how a number was made, they have to say whose service, whose risk and whose pass mark the number serves.

Sources: IESG approval record; approved BPM charter; BPM group page; BMWG charter; IPPM charter; BCP 170 / RFC 6390.