A case study in cross-layer failure.
One of the longest-running problems I've dealt with throughout my career was a stubborn control-plane instability: BGP sessions would intermittently flap on individual hosts, causing brief but recurring outages. The issue had existed for years. Multiple people had looked at it, but nobody had managed to make it go away.
The symptom🔗
When it happened, it looked like a straightforward BGP/BFD flap:
- BGP sessions would reset
- BFD timers would expire
- routes would withdraw
- load would shift and second-order effects would ripple through the system
- and then eventually everything might recover
Initially this looked like one of the usual suspects. Maybe a bad cable or a flaky switch port. I decided to pull in the network team early on. As it turned out, from their side, things looked pretty clear-cut. The adjacent switch/router reported the session as being closed from our end, with no corresponding interface events and no signs of instability on the switch itself. As far as the fabric was concerned, the host simply stopped maintaining the session.
The strange part was how localized it was. Entire racks and sites were fine. It was always a small set of machines, and not in a predictable pattern. Sometimes CPU would spike. Sometimes memory pressure would increase. Sometimes nothing obvious happened at all.
Whatever was causing the flaps didn’t line up cleanly with any of the usual signals.
Inconclusive signals🔗
At first I did what seemed reasonable: I dug through metrics and stared at graphs. Everything we had: CPU usage, memory pressure, packet drops, softirq statistics, interrupt rates, and plenty more. There were correlations, but nothing that held up. A CPU spike might appear near a flap, but just as often it wouldn’t.
In a subset of incidents we also saw host-level failures (OOMs or outright crashes) in the same window. I couldn’t establish directionality: sometimes the flap came first, sometimes the host was already degraded. I treated these as signals of contention rather than a root cause. Overall, load looked more like a side effect than a direct cause.
After about a week of getting nowhere, I changed strategy.
Instead of asking “what is the system doing?”
I started asking “what is different about the machines that fail?”
The first real signal🔗
By correlating incidents with host scheduling data, a pattern slowly emerged: machines that flapped tended to run the same small set of customer workloads. So I tried something simple. I moved those workloads to other hosts, and over the next weeks and months, the flaps followed them!
That was the first moment the problem stopped feeling random. It was being triggered by particular workloads!
Looking below BGP🔗
Since I now knew on which hosts we were likely to see the problem again, I'd finally be able to preemptively start a packet capture to see what was actually happening on the wire. I set up captures on both sides of the control plane (Linux host + adjacent network switch), focusing on BGP keepalives and BFD control packets. After successfully capturing a couple of events, I imported the pcap files into Wireshark1 for further analysis.
What I saw was subtle, but important: some packets were arriving late — late enough for timers to expire. This was a real breakthrough! We proved without a doubt a hunch we'd had early on but couldn't prove at the time. This was not a problem specific to BGP, but rather a general packet-delivery problem! It just happened to surface in BGP first!
That pushed the investigation down another layer:
Somewhere in the host receive path, control packets were occasionally delayed enough to trip timers.
Where the delay could be introduced🔗
The control plane here is just packets moving through a receive path. If keepalives or BFD control packets are delayed anywhere along this chain, timers can expire even though the physical link and peer are healthy.+------------------+
| L3 Switch |
| (BGP/BFD peer) |
+---------+--------+
|
| Ethernet frames
v
+------------------+
| NIC (RX queue) |
+---------+--------+
|
| DMA + interrupts
v
+------------------+
| Kernel network |
| stack |
| (softirq, NAPI) |
+---------+--------+
|
| socket buffers
v
+------------------+
| Userspace |
| BGP + BFD |
| daemons |
+------------------+
The important takeaway here is that this path runs through shared CPU time (softirq) and shared queues. Under contention, “late” can happen without drops.
The obvious next step was to reproduce it outside production.
Why reproducing it was so hard🔗
I tried to reproduce the issue with synthetic traffic, stress tests, and replay tools. Nothing triggered it reliably.
What did happen, however, was that I started tripping over other low-level issues — including NIC firmware bugs that were later fixed by the vendor after being reported. That reinforced a suspicion that this wasn’t one clean bug, but a fragile timing path that only broke under very specific real-world load.
What “late” means in timer-driven protocols🔗
When the system is healthy, BFD packets arrive well within the detection window:t0 t1 t2 t3 t4 t5
|------|------|------|------|------|
^ ^ ^ ^ ^
| | | | |
BFD BFD BFD BFD BFD
recv recv recv recv recv
During an event, packets still arrive, but the gap is long enough to trip the timer:t0 t1 t2 t3 t4 t5
|------|------|------|------|------|
^ ^
| |
BFD BFD
recv recv
<--- stall --->
Treating the system, not the symptom🔗
Since I couldn’t isolate a single smoking gun, I focused on reducing the system’s sensitivity to timing jitter by making the control-plane packet path more deterministic.
1) Adjusting NIC interrupt coalescing🔗
Modern NICs can batch interrupts and packet processing for efficiency (fewer interrupts mean less overhead), but batching introduces latency variance: packets can sit in the RX ring longer before the CPU is nudged to service them. NIC CPU 0
┌───────────────────────────┐ ┌────────────────────────────────────────────────────┐
│ │ │ │
│ ring buffers │ │ softnet queue backlog processes │
│ ┌──────────────────────┐ │ │ ┌───────────────────────┐ ┌───┐ ┌───┐ │
│ │ [ ][ ][ ][ ][ ] ... │──┼──────┼──▶│ [ ][ ][ ][ ][ ] ... │ ┌───▶│BGP│ │ │◀──┐ │
│ └──────────────────────┘ │ │ └───────────────────────┘ │ └───┘ └───┘ │ │
│ ▲ │ │ │ │ ┌───┐ ┌───┐ │ │
│ fills while waiting │ │ ┌─────────────┘ ├───▶│ │ │ │◀──┤ │
│ (rx-usecs / rx-frames) │ │ │ socket receive queue │ └───┘ └───┘ │ │
│ ▼ │ │ │ ┌───────────────────────┐ │ │ │
│ ┌──────────────────────┐ │ │ └▶│ [ ][ ][ ][ ] ... │───┘ │ │
│ │ [ ][ ][ ] ... │──┼───┐ │ └──────────┬────────────┘ │ │
│ └──────────────────────┘ │ │ │ └───────────────────────────────────┘ │
└───────────────────────────┘ │ └────────────────────────────────────────────────────┘
│ CPU 1
│ ┌────────────────────────────────────────────────────┐
│ │ │
│ │ softnet queue backlog processes │
│ │ ┌───────────────────────┐ ┌───┐ ┌───┐ │
└──┼──▶│ [ ][ ][ ] ... │ ┌───▶│ │ │ │◀──┐ │
│ └───────────────────────┘ │ └───┘ └───┘ │ │
│ │ │ ┌───┐ ┌───┐ │ │
│ ┌─────────────┘ ├───▶│ │ │ │◀──┤ │
│ │ socket receive queue │ └───┘ └───┘ │ │
│ │ ┌───────────────────────┐ │ │ │
│ └▶│ [ ][ ] ... │───┘ │ │
│ └──────────┬────────────┘ │ │
│ └───────────────────────────────────┘ │
└────────────────────────────────────────────────────┘
That’s usually fine for bulk traffic. It’s not fine for timer-driven control packets with tight detection windows.
We tuned RX coalescing to trade a bit of CPU overhead for tighter latency bounds. The goal was to avoid “long waits, then big bursts”.
Inspect and tune coalescing parameters with ethtool.2:
# Show current settings
ethtool -c eth0
# Example: reduce RX batching (values are NIC/driver/workload dependent)
ethtool -C eth0 rx-usecs 10 rx-frames 5 adaptive-rx off2) Make userspace control-plane scheduling less “best effort”🔗
Even if packets reach the host quickly, they still need to be serviced by softirq/NAPI, delivered to userspace, and processed by the daemon in time. Under contention, the failure mode is usually not “slow on average” — it’s “occasionally not runnable soon enough”.
We attacked this by making the daemons more predictable under load.
2a) CPU affinity (pinning) for control-plane daemons🔗
Pinning is mostly about predictability. If your control plane competes with bursty workloads on the same cores, you will eventually lose a timer race.
With systemd3, you can pin the service via cgroup properties:
systemctl set-property bgp.service AllowedCPUs=0Without systemd, pin the running process using taskset4:
taskset -cp 0 "$(pidof your-bgp-daemon)"2b) Real-time scheduling🔗
If you’ve already carved out a controlled CPU budget, a real-time policy can reduce tail latency further by making scheduling5 more deterministic.
With systemd (example using SCHED_RR):
# /etc/systemd/system/bgp.service.d/50-scheduling.conf
[Service]
CPUSchedulingPolicy=rr
CPUSchedulingPriority=20
# Safety rails (avoid runaway RT starvation)
LimitRTPRIO=20
LimitRTTIME=200msApply:
systemctl daemon-reload
systemctl restart bgp.serviceWithout systemd using chrt6:
# Example: SCHED_RR with priority 20
chrt -r -p 20 "$(pidof your-bgp-daemon)"Real-time scheduling is powerful and can be dangerous if misused; it’s best paired with explicit CPU budgeting and guardrails.
2c) Optional: isolating CPUs at boot🔗
For extreme cases, Linux supports CPU isolation (isolcpus, nohz_full, rcu_nocbs) so the scheduler keeps “clean” cores mostly free of normal workload scheduling.
We decided against this: isolating a whole core purely for BGP/BFD wasn’t worth it for our fleet, and we still wanted normal softirq processing on those CPUs.
3) Keep control-plane flows on the “right” core with packet steering🔗
With the daemon pinned, time can still be lost if the receive-side work for that flow happens on a different CPU. That adds cross-core handoff, cache misses, and queueing in the “wrong” softnet backlog — which is exactly the kind of tail-latency that was hurting us.
Linux has a few ways to decide which CPU ends up doing RX processing7. They all try to answer the same question:
When a packet arrives, which core should run the kernel RX path for it?
There’s a natural progression:
- RSS (Receive Side Scaling) spreads traffic across RX queues (and therefore CPUs) using a hash of packet headers. Since it is primarily a flow based throughput scaling mechanism it is ideal for workloads that have many balanced independent flows.
- RFS (Receive Flow Steering) tries to steer packets toward the CPU where the consuming socket last ran, reducing cross-core bouncing and thus removing latency from the path.
- aRFS (accelerated RFS) keeps the same goal as RFS (“follow the socket”), but uses NIC hardware capabilities to make it cheaper and more consistent under load.
In our case the goal wasn’t maximum throughput but tighter tail latency for a small set of host-terminated control-plane flows. Where supported, we enabled aRFS so that when the kernel can identify the consuming socket those packets are steered toward the same CPU the daemon runs on, reducing cross-CPU queueing and jitter.
These changes went out as a bundle, so I can’t attribute the fix to any single knob. But together they reduced enough jitter in the receive → schedule → userspace path to stabilize the control plane. The flaps didn’t recur over the following months. The failure mode here wasn’t loss — it was jitter. Each layer (NIC optimizations, softirq scheduling, userspace runnability) can be “fine on average” while still producing occasional stalls large enough to exceed protocol timing windows. The mitigations worked because they tightened the worst-case latency, not the mean.