Scaling Data Centers with NVMe over Fabrics Architectures

In the early days of solid state drives, we treated flash memory as a faster version of a spinning disk. We used the same SATA cables and the same AHCI protocols that were designed for mechanical platters. This approach created a massive performance bottleneck because the software stack could not keep up with the speed of the underlying NAND gates.

The NVMe protocol solved this by utilizing the PCIe bus, allowing for thousands of parallel queues and direct access to the CPU memory. However, this high-performance storage was physically tethered to the server chassis. If a database server ran out of local disk space while a neighboring web server had terabytes of idle flash, that capacity was effectively stranded.

This physical limitation created storage silos where resource utilization was inefficient and scaling required expensive hardware migrations. Engineers were forced to choose between the ultra-low latency of direct-attached storage and the flexibility of traditional network-attached storage. This is the specific gap that NVMe over Fabrics was designed to bridge.

NVMe-oF extends the NVMe protocol across a network fabric, allowing a host to access remote storage as if it were plugged into a local PCIe slot. It preserves the efficiency of the NVMe command set while removing the distance constraints of the motherboard. This enables a truly disaggregated architecture where compute and storage can scale independently of one another.

The primary goal of NVMe-oF is to maintain the sub-microsecond latency of local NVMe while providing the agility of a shared storage pool across the entire data center.

NVMe-oF is transport-agnostic, meaning it can run over different types of network fabrics depending on the performance requirements and existing infrastructure. The two most common choices for developers and infrastructure engineers are Remote Direct Memory Access and Transmission Control Protocol. Each comes with significant trade-offs regarding cost, complexity, and raw performance.

RDMA allows data to be transferred from the memory of one computer to another without involving either computer's operating system. This zero-copy mechanism bypasses the standard kernel networking stack, which dramatically reduces CPU utilization and latency. Popular implementations include InfiniBand and RDMA over Converged Ethernet, commonly known as RoCE.

While RDMA offers the highest possible performance, it often requires specialized network interface cards and switches that support Priority Flow Control. Without a lossless network, RoCE performance can degrade rapidly when congestion occurs. This hardware requirement can be a significant barrier for teams operating in standard cloud environments or smaller on-premises labs.

NVMe over TCP has emerged as a compelling alternative that works on existing standard Ethernet hardware. It encapsulates NVMe commands in standard TCP segments, making it compatible with every modern data center switch and network interface. Although it introduces more CPU overhead than RDMA, it is much easier to deploy and manage at scale.

• RDMA (RoCE/iWARP): Lowest latency and lowest CPU overhead, but requires specialized, lossless network hardware.
• TCP: High compatibility and lower cost, but requires more CPU cycles to process the network stack.
• Fibre Channel: Extremely reliable and optimized for storage, but increasingly viewed as a niche legacy technology compared to Ethernet-based options.

To understand how this works in practice, we can look at a Linux-based implementation using the native kernel modules. In this scenario, one server acts as the target, which shares its physical NVMe drives over the network. The other server acts as the initiator, which discovers and mounts those drives as if they were local devices.

We use the nvmet module on the target side to create a subsystem and export a specific block device. This involves creating a directory structure in the configfs virtual filesystem, which the kernel uses to manage storage exports. Once the subsystem is created, we define an allowed host and a network port to listen for incoming connections.

On the initiator side, we use the nvme-cli utility to connect to the target. The initiator sends a discovery request to the target's IP address to see available subsystems. If authorized, the initiator can then connect, and the operating system will create a new block device entry such as /dev/nvme0n1.

The following script demonstrates the basic steps to set up an NVMe over TCP target on a modern Linux distribution. It assumes you have a spare partition or drive available for export and the necessary kernel modules loaded. This pattern is often automated using configuration management tools like Ansible or integrated into Kubernetes CSI drivers.

While NVMe-oF is incredibly fast, it is not immune to the laws of physics and networking. The introduction of a network switch and cables adds a layer of latency that does not exist in a local PCIe connection. In a well-optimized environment, this overhead is typically in the range of 10 to 100 microseconds.

One major factor that influences performance is the number of I/O queues and the queue depth. NVMe was built to support up to 64,000 queues, each with 64,000 entries, allowing it to leverage the many cores of modern CPUs. When using NVMe-oF, ensure that your initiator is configured to use multiple queues that match the core count of your application server.

Network congestion is another common pitfall that can cause latency spikes, also known as the tail latency problem. Even if the average latency is low, a few slow requests can hang an entire application thread. This is especially problematic in multi-tenant environments where a noisy neighbor might saturate the network bandwidth.

To mitigate these issues, engineers often implement Quality of Service rules on their switches. By prioritizing storage traffic over general background traffic, you can ensure that database writes and reads remain consistent. Monitoring tools should track the p99 latency to identify these intermittent bottlenecks before they impact users.