Summary

  • NVM Express, Inc. is the non-profit industry consortium that governs the NVMe specification family; it manufactures neither SSDs, nor controllers, nor storage arrays.
  • NVMe replaced the serial queue assumptions inherited from mechanical disks with many in-memory submission and completion queues suited to multicore processors and flash parallelism.
  • The same controller, subsystem and namespace model can run locally over PCIe or remotely through NVMe over Fabrics, with TCP and RDMA transports at different costs.
  • The NVMe 2.x series is modular. The 2.4 set, published on 4 August 2026 after ratification, comprises the Base, PCIe/RDMA/TCP transports, NVMe-MI, Boot and several specialised command sets.
  • The protocol makes capacity and access more composable. It does not define the filesystem, durability, RAID, erasure coding, networking, operational security or application consistency.

A storage command can leave the server without changing its fundamental language

An application can issue a read or write to a local NVMe namespace attached over PCIe. In a fabric architecture, the same command model can reach a remote subsystem over Ethernet or an RDMA network. The media, controller and path change; the storage grammar remains broadly common.

This continuity is the basis of disaggregation. Operators can pool capacity in shared systems and attach it to hosts on demand, instead of placing every device in the chassis that uses it.

Moving it removes no dependencies. It adds discovery, networking, multipath, authentication, reconnection and failure domains. NVMe turns storage into a composable service, not a topology-free resource.

Flash needed a protocol designed for parallelism

Protocols inherited from hard disks had been shaped by mechanical latency and narrower queues. Flash can process many operations in parallel, while modern servers have many cores.

NVMe uses pairs of in-memory submission queues and completion queues. Applications or drivers place commands, notify the controller through doorbells and retrieve completions by interrupt or polling. This structure reduces shared locks and lets queues be placed close to the cores that use them.

The benefit is not automatic. Firmware, PCIe topology, NUMA memory, queue depth, interrupts and workload determine actual latency and throughput.

The consortium separates protocol governance from competition between products

NVMe work began in the late 2000s; version 1.0 was released in 2011 and NVM Express, Inc. was incorporated in 2014. The consortium brings together manufacturers of platforms, processors, controllers, media, networks and systems.

At the time of research, Amber Huffman of Google was chair, Curtis Ballard of AMD treasurer and David Allen of Microchip secretary. Thirteen promoter member representatives sat on the public board.

The consortium defines the contracts and compliance programmes. Members continue to compete on silicon, firmware, systems, support and performance. Participation gives access to the process; it does not prove that a product implements every feature.

NVMe 2.0 turned a growing document into a modular family

As the protocol expanded to multiple transports and data models, a single document became hard to maintain. The 2.0 restructure in 2021 separated the Base, command sets and transports.

This architecture lets a TCP transport evolve without rewriting the whole command set, or Zoned Namespaces advance without modifying every conventional controller. It does, however, require buyers to track multiple versions.

The NVMe 2.4 set, made public on 4 August 2026, grouped together Base 2.4, PCIe Transport 1.4, RDMA Transport 1.3, TCP Transport 1.3, NVMe-MI 2.2, Boot 1.4 and the NVM, Key Value, Zoned Namespace, Computational Programs and Simple Log Memory command sets. “NVMe 2.4” therefore denotes a coordinated collection, not a single file.

Submission and completion queues bring work closer to the CPU

A submission queue contains commands written by the host; the completion queue receives the controller’s results. Several pairs can be associated with cores or applications, reducing contention.

Doorbells signal new entries. The controller can generate interrupts or the host can poll. Polling sometimes reduces latency at the cost of higher CPU consumption. Interrupt coalescing reduces cost but can add delay.

Queue depth is not a universal performance measure. Queues that are too deep increase waiting and queue latency. Queues that are too short can underuse the device. Operations must choose according to workload.

Controllers, subsystems and namespaces separate the logical endpoint from the media

A controller exposes queues and commands. A subsystem can contain multiple controllers. Namespaces represent logical storage spaces that can be shared, resized or reachable through multiple paths.

This abstraction avoids confusing a physical SSD with the identity seen by the host. A remote subsystem can aggregate many media and present stable namespaces.

The protocol does not decide how data is placed, replicated or protected. RAID, erasure coding, thin provisioning and consistency remain functions of the system surrounding NVMe.

Administrative commands matter as much as I/O

An admin queue manages identification, queue creation, features, logs, firmware, namespaces and security. These commands control the environment in which ordinary reads and writes become possible.

An administrative error can delete a namespace, activate the wrong firmware or change a power configuration. Permissions, audits and change windows are therefore essential.

The admin/I/O separation helps architecture, but does not make the control plane less critical. In a fabric, these commands sometimes travel over the same network as data.

PCIe keeps the local path close to memory and hardware

NVMe over PCIe uses shared memory structures and controller registers on the local bus. It offers a direct path suited to SSDs and cards installed in the server.

Physical proximity does not guarantee a simple topology. A device may sit behind a PCIe switch, on another NUMA socket or share lanes. The placement of process, memory and device influences performance.

Systems must therefore inventory the PCIe path, not just count drives. Unexpected latency can come from server topology before data even reaches the media.

NVMe over Fabrics turns a local relationship into a network service

NVMe-oF, whose first version was released in June 2016 with NVMe-MI 1.0, carries command capsules and data to a remote subsystem. The host establishes queues across a fabric and sees network namespaces.

The separation allows capacity and performance to be pooled. Hosts can be replaced without moving data, and systems can provide resources to several clusters.

It creates new failure points: NIC, switch, route, discovery, controller, authentication and multipath policy. Storage becomes dependent on the network as a data component, not merely as connectivity.

Discovery controllers make fabrics dynamic and concentrate a critical function

A host can query a discovery controller to obtain the available subsystems and addresses. This avoids manually configuring every path and allows services to be added or removed.

Discovery can, however, become a central dependency. Wrong data can direct the host to the wrong target or prevent a connection. Caches, redundancy, validation and identity protection are necessary.

The discovery service must not be confused with the availability of storage itself. An already-connected host can continue, while a new host fails to discover the service.

NVMe/TCP made the fabric accessible over ordinary IP networks

The TCP transport, standardised in 2019, maps NVMe commands and data onto TCP connections. Operators can use routed Ethernet, IP tools, firewalls and familiar networking practices without building an RDMA fabric.

This accessibility has a cost. The TCP stack, copies, interrupts and CPU can add overhead and tail latency. Kernel optimisations, offload and transfer size strongly influence the outcome.

The transport also adds specific framing and digest controls to detect corruption. TLS can protect the session where implementation and policy provide for it.

NVMe/RDMA targets low latency at the cost of fabric discipline

RDMA moves commands and data with less CPU involvement and uses queue pairs, memory registration and NIC capabilities. It suits environments where microseconds and CPU cycles matter.

RDMA is not a single fabric. RoCE, iWARP and other bindings have different behaviours and requirements. Congestion, loss, PFC or ECN, NIC configuration and registered memory must be understood.

A low-latency benchmark does not prove that operations will be simple. Fabric errors can show up as storage timeouts and require cross-disciplinary network-system skills.

Multipath turns redundancy into a host decision

A namespace can be reachable through multiple controllers and paths. The host system chooses how to balance or fail over. Asymmetric Namespace Access indicates which paths are optimised, non-optimised or unavailable.

The presence of two links does not guarantee independence. They can share a switch, power supply, controller or route. Operators must test the loss of each element and verify the host multipath reaction.

A poor policy can send traffic down a slow path or keep a failing route for too long. Redundancy must be observed as behaviour, not inventoried as a number.

Reservations coordinate shared access without replacing cluster consensus

NVMe reservations let hosts register and reserve a namespace to avoid unauthorised concurrent writes. They are useful in clusters and failover scenarios.

They do not replace a consensus protocol or application consistency. A failed host can leave reservation state to be cleaned up, and recovery must prevent an old node from continuing to write.

Recovery requires fencing, generation and identity verification. A poorly managed reservation can turn a protective measure into unavailability.

Authentication and TLS become necessary when storage leaves PCIe

A local device often inherited an implicit physical boundary. On a fabric, an initiator and a target must prove their identity and protect the appropriate channels.

NVMe defines authentication mechanisms, and NVMe/TCP can use TLS. Their value depends on keys, certificates, renewals, algorithms and access policies. A transport that supports TLS is not necessarily configured to use it.

The discovery plane, administrative plane and data plane must be considered together. A correctly authenticated identity can still have overly broad rights.

NVMe-MI provides a management path separate from application I/O

The NVMe Management Interface lets tools discover and manage subsystems, read health, inventory devices and perform certain operations even when the main I/O path is not in use.

This separation helps maintenance and recovery. It also adds a second control plane to secure. A management tool can read sensitive data or modify components at scale.

NVMe-MI can integrate with Redfish systems and other management models. Integration does not remove the boundaries of responsibility between consortia and vendors.

Zoned Namespaces expose media constraints to the host

ZNS divides capacity into zones that are written sequentially. By giving software visibility into how the media is organised, it can reduce some internal garbage collection work and improve endurance or predictability.

The benefit requires the filesystem, database or storage layer to understand zones. An application designed for a conventional block device does not automatically benefit from the model.

ZNS illustrates NVMe’s tension: exposing more behaviour can improve efficiency, but increases portability requirements for software.

Key Value, Simple Log Memory and Computational Programs broaden the namespace

The specialised command sets provide access to key/value pairs, simple log memory or compute programs executed close to storage. They aim to reduce translations and data movement for certain workloads.

Their adoption depends on controllers, drivers, libraries and applications. A command set in the specification is not a universal feature of every SSD or array.

The more models the protocol offers, the more precisely software must discover capabilities and have a fallback path. Technical modularity creates a new commercial matrix.

Compliance provides useful proof without certifying end-to-end performance

Interoperability programmes and workshops verify specific behaviours between hosts and controllers. They detect deviations that reading the specification does not reveal.

A compliance list does not measure latency, endurance, recovery or security in every topology. Optional features can differ and the driver-firmware matrix evolves.

Buyers should treat compliance as a floor and test their workload, failures and upgrade cycle.

NVMe separated capacity from the server and moved responsibility

Local storage often tied together the device, the host and the system team. Disaggregation allows a pool to be shared among many consumers and allocation to be changed by software.

It brings network, storage, platform, security and application teams into the same incident. Fabric degradation can look like a database problem; controller firmware can look like a network loss.

The economic gain therefore depends on capacity utilisation and operational capability, not just price per terabyte.

AI makes storage latency visible as a compute cost

Training runs write checkpoints, load datasets and move state at scale. When thousands of accelerators are waiting, storage variability can waste significant compute investment.

NVMe/TCP, RDMA, multipath and shared namespaces offer several possible architectures. The choice must consider congestion, topology, throughput, metadata, recovery and queue behaviour.

The goal is not only a maximum benchmark, but predictable queue and recovery latency during synchronised operations.

The 2.4 set shows both the scope and the version pressure

NVMe 2.4 coordinates the Base, transports, Boot, Management Interface and specialised command sets. This breadth shows that NVMe has become a stack, not just an SSD connector.

It complicates support statements. A host can support Base 2.4 without supporting every command set or security feature. A controller can offer TCP without the same capabilities as an RDMA product.

Tracking must cover driver, firmware, OS, transport and management tool versions. In practice, this matrix is part of the protocol.

NVMe made storage composable without making the service simple

The common protocol reduces one cost of change: the command model can survive a move from PCIe to a fabric or from one vendor to another. Other costs remain: data migration, namespace identity, security policy, networking, observability and support.

The strategic advantage is a cleaner separation between protocol and product. The risk is hiding a complex architecture behind the same NVMe name.

Storage becomes programmable and distributable. It does not stop being an integrity system whose every layer must be understood under failure.