eBPF
Accelerating Network Throughput with eBPF and XDP Hooks
Discover how to bypass the standard networking stack to process packets at the driver level using the eXpress Data Path (XDP) for ultra-low latency applications.
In this article
The Performance Bottleneck in the Linux Networking Stack
The standard Linux networking stack is a masterpiece of engineering designed for general-purpose compatibility. It supports a vast array of protocols and complex routing rules, but this flexibility introduces significant overhead at high speeds. When a network interface card receives a packet, the kernel traditionally allocates a complex data structure called a socket buffer to manage it.
The creation of the socket buffer, or sk_buff, is one of the most expensive operations in the packet processing pipeline. This structure contains extensive metadata about the packet to ensure it can travel through the IP, TCP, and UDP layers safely. For high-volume environments like load balancers or firewalls, allocating and deallocating these structures millions of times per second creates a massive CPU bottleneck.
Beyond memory allocation, the standard stack involves multiple layers of processing and context switching before a packet ever reaches a user-space application. Each layer performs its own checks and lookups, which adds incremental latency. In scenarios like Distributed Denial of Service mitigation, the system may expend more energy processing a packet to the point of a drop decision than it does handling legitimate traffic.
The Cost of Context Switching and Interrupts
Every time a packet arrives, the hardware triggers an interrupt that forces the CPU to stop its current task and handle the networking event. While features like the New API (NAPI) help by allowing the kernel to poll the driver for packets, the fundamental cost of switching between user space and kernel space remains high. This transition involves saving and restoring CPU registers and flushing various caches, which degrades overall system throughput.
As network speeds have climbed from one gigabit to one hundred gigabits and beyond, the time budget for processing each individual packet has shrunk to nanoseconds. Standard kernel processing simply cannot keep up with these rates without consuming every available CPU core. Developers require a way to make decisions about traffic as early as possible in the software path, ideally before the kernel even knows the packet exists as a full socket buffer.
Why Kernel Bypass is Not Always the Answer
Historically, developers reached for kernel bypass technologies like DPDK to solve these latency issues. DPDK moves the entire networking stack into user space, giving the application direct control over the hardware. While extremely fast, this approach requires specialized hardware and forces the developer to reimplement every networking protocol from scratch within their application.
Bypassing the kernel also means losing access to the robust security models and debugging tools that the Linux kernel provides. Managing your own networking stack is a massive operational burden that increases the risk of bugs and security vulnerabilities. This creates a need for a middle ground that offers the performance of kernel bypass while maintaining the safety and integration of the standard kernel.
The Architecture of the eXpress Data Path
The eXpress Data Path, or XDP, provides that middle ground by allowing custom eBPF programs to run directly within the network driver. This hook point is located at the earliest possible stage of the software stack, occurring as soon as the packet is transferred from the network interface card to main memory. Because the program runs here, it can make a verdict on the packet before the kernel allocates any socket buffer metadata.
XDP programs are written in a restricted version of the C language and compiled into eBPF bytecode. This bytecode is then loaded into the kernel, where it is thoroughly checked by a verifier to ensure it does not crash the system or access unauthorized memory. This safety model allows developers to run high-performance logic in the kernel without the instability risks associated with traditional kernel modules.
XDP effectively transforms the network driver into a programmable pipeline. By shifting logic from the application layer to the driver layer, we can achieve wire-speed performance while still benefiting from the Linux kernel's management and safety features.
The XDP Execution Pipeline
When a packet enters the XDP hook, the program receives a context structure containing pointers to the start and end of the raw packet data. The program can inspect the headers, modify the contents, or even encapsulate the packet with new headers. Once the processing is complete, the program returns one of several action codes to the driver.
The available actions determine the fate of the packet with minimal delay. These include passing the packet to the normal stack, dropping it immediately, transmitting it back out of the same interface, or redirecting it to a different interface or CPU. This immediate feedback loop is what enables XDP to handle millions of packets per second with negligible latency.
The Role of the eBPF Verifier
The verifier is the guardian of the XDP execution environment, ensuring that every program is safe to execute. It performs a static analysis of the code to guarantee that all loops are bounded and that every memory access is within the bounds of the packet data. This is critical because a bug in a driver-level program could otherwise lead to a complete system crash.
While the verifier imposes constraints that can be frustrating for developers, it is the foundation of the eBPF security model. It ensures that custom code cannot stay in an infinite loop, which would lock up a CPU core indefinitely. By adhering to these constraints, developers can build networking logic that is both fast and incredibly stable.
Implementing a High-Performance DDoS Mitigator
To understand the power of XDP, we can look at a practical scenario: protecting a server from a volumetric DDoS attack. In a traditional setup, the firewall would only drop the packet after it has traversed several layers of the networking stack. With XDP, we can write a program that identifies malicious traffic based on IP headers and drops it at the driver level.
This approach prevents the malicious traffic from ever consuming the resources required to build a socket buffer. The CPU remains available for legitimate requests because the energy spent on each dropped packet is nearly zero. This is the gold standard for high-performance security at the edge of a cloud infrastructure.
1#include <linux/bpf.h>
2#include <linux/if_ether.h>
3#include <linux/ip.h>
4#include <bpf/bpf_helpers.h>
5
6// Define a map to store blocked IP addresses
7struct {
8 __uint(type, BPF_MAP_TYPE_HASH);
9 __uint(max_entries, 10240);
10 __type(key, __u32); // IPv4 Address
11 __type(value, __u64); // Hit counter
12} blocklist SEC(".maps");
13
14SEC("xdp")
15int xdp_mitigator(struct xdp_md *ctx) {
16 void *data_end = (void *)(long)ctx->data_end;
17 void *data = (void *)(long)ctx->data;
18
19 struct ethhdr *eth = data;
20 if ((void *)(eth + 1) > data_end)
21 return XDP_PASS;
22
23 if (eth->h_proto != __constant_htons(ETH_P_IP))
24 return XDP_PASS;
25
26 struct iphdr *iph = data + sizeof(struct ethhdr);
27 if ((void *)(iph + 1) > data_end)
28 return XDP_PASS;
29
30 __u32 src_ip = iph->saddr;
31 __u64 *value = bpf_map_lookup_elem(&blocklist, &src_ip);
32
33 if (value) {
34 // Atomically increment the hit counter for visibility
35 __sync_fetch_and_add(value, 1);
36 return XDP_DROP; // Packet discarded at driver level
37 }
38
39 return XDP_PASS; // Legitimate traffic proceeds to stack
40}
41
42char _license[] SEC("license") = "GPL";The code example above demonstrates how to parse an Ethernet frame and extract the IPv4 source address. It uses an eBPF map to check if the source IP is on a blocklist. If a match is found, the program returns the XDP_DROP code, which instructs the network driver to recycle the memory buffer immediately without further processing.
State Management with BPF Maps
In the example, we used a hash map to track malicious IPs. Maps are the primary way that eBPF programs store state and communicate with user-space applications. A control-plane application running in user space can add or remove entries from this map in real time without having to reload the XDP program.
This decoupling of the data plane and the control plane is a core principle of high-performance networking. The XDP program remains simple and fast, while the complex logic of identifying which IPs to block is handled by a sophisticated user-space service. This allows for dynamic and reactive security policies that can adapt to changing attack patterns.
Deployment Modes and Hardware Interaction
XDP can be deployed in three different modes, depending on the capabilities of your hardware and the level of performance required. Choosing the right mode is essential for balancing throughput with compatibility across different server environments. The mode you choose determines where exactly the eBPF bytecode is executed within the system.
- Native XDP: The program is loaded directly into the network driver. This offers the best performance for most standard servers and requires driver-level support.
- Offloaded XDP: The program is loaded onto a SmartNIC and executed on the network card's hardware. This removes all processing load from the host CPU.
- Generic XDP: The program runs in the networking stack's early software path. This is compatible with all drivers but offers lower performance due to early stack overhead.
Native XDP is the most common choice for production environments. It provides a significant performance boost over the standard stack while still running on commodity CPUs. Most modern drivers for Intel, Mellanox, and Broadcom network cards support this mode natively, allowing for seamless integration into existing data centers.
Understanding XDP Return Codes
The return code of an XDP program is the final verdict that dictates the packet's journey. While we have seen the drop and pass actions, the redirect action is particularly powerful for building complex network functions. It allows a packet to bypass the local networking stack entirely and be sent directly to another interface or even a different CPU core for processing.
The transmit action is another key tool, often used for load balancing. It allows an XDP program to modify the MAC addresses or IP headers of a packet and then send it immediately back out of the same interface. This enables one-legged load balancers that can route traffic at massive scales without the overhead of moving data through the kernel stack.
Loading and Attaching the Program
Once compiled, the XDP program must be loaded into the kernel and attached to a specific network interface. This is typically done using tools like the iproute2 suite or custom loaders written in Go or C using the libbpf library. The loader handles the heavy lifting of interacting with the kernel's BPF subsystem and ensuring the program is correctly pinned to the hardware.
1package main
2
3import (
4 "log"
5 "net"
6 "github.com/cilium/ebpf/link"
7)
8
9func main() {
10 // Load the compiled eBPF objects from the ELF file
11 objs := bpfObjects{}
12 if err := loadBpfObjects(&objs, nil); err != nil {
13 log.Fatalf("loading objects: %v", err)
14 }
15 defer objs.Close()
16
17 // Look up the network interface by name
18 iface, err := net.InterfaceByName("eth0")
19 if err != nil {
20 log.Fatalf("interface look up: %v", err)
21 }
22
23 // Attach the XDP program to the interface in native mode
24 l, err := link.AttachXDP(link.XDPOptions{
25 Program: objs.XdpMitigator,
26 Interface: iface.Index,
27 Flags: link.XDPGenericMode, // Use NativeMode for production
28 })
29 if err != nil {
30 log.Fatalf("could not attach XDP: %v", err)
31 }
32 defer l.Close()
33
34 log.Printf("Successfully attached XDP mitigator to %s", iface.Name)
35 // Program remains attached until the process exits or l.Close is called
36}Practical Trade-offs and Best Practices
While XDP is incredibly powerful, it is not a silver bullet for every networking problem. Because XDP runs so early, it does not have access to many of the conveniences of the standard stack, such as TCP stream reassembly or complex fragmentation handling. Developers must be prepared to handle raw packet parsing manually, which increases the complexity of the code.
Another critical consideration is observability. Since XDP drops packets before they reach tools like tcpdump, traditional debugging techniques often fail. You must build your own monitoring into the XDP program using maps or tracepoints to ensure you have visibility into what traffic is being dropped and why. This requirement makes testing and validation a significant part of the development lifecycle.
Optimizing for Cache Performance
When writing XDP programs, every instruction counts. To achieve maximum throughput, you should minimize the number of map lookups and avoid complex logic that could lead to CPU cache misses. Data structures should be kept small and aligned to ensure they fit within the CPU's limited cache lines.
Using per-CPU maps instead of global maps can also significantly improve performance. Per-CPU maps eliminate the need for atomic operations or locking when multiple cores are processing packets simultaneously. This pattern is essential for scaling XDP applications to tens of millions of packets per second on multi-core systems.
The Road to Production
Before deploying XDP to production, it is vital to perform extensive benchmarking using realistic traffic patterns. Synthetically generated traffic often fails to uncover edge cases related to packet headers or timing issues. Tools like the BPF self-test suite and customized packet generators are indispensable for ensuring that your code behaves correctly under pressure.
XDP represents a fundamental shift in how we think about the Linux networking stack. By moving logic into the driver, we can overcome historical performance barriers while maintaining the safety and flexibility of software-defined networking. As eBPF continues to mature, XDP will remain the cornerstone of high-performance cloud infrastructure and security.