Background

Docker becomes increasingly popular since 2013. It shows many advantages compared with VM on many aspects. But it still has shortcomings still and is under quick evolution. This blog wants to show you the status of network performance for containers across multi-host.

Experiments

The experiment was conducted on two CentOS 7.2 VMs created by Hyper-V. Both of them installed LIS 4.2.1. The performance test driver is ntttcp-for-linux. They are directly connected physically. The performance base line is the perf result of two directly linked VMs, then collect network performance data for containers under bridge network, swarm overlay, and consul overlay.

The final result indicates that docker bridge network performance is the best, there is almost no performance drop. But the other two overlay network have huge performance drop. ksoftirq consumed almost 50%~100% one core CPU.

Client Server Avg Tput (Gbps) Ratio to ‘vm-vm’
VM VM 15.21 1
bridge port mapping bridge port mapping 13.92 0.92
consul overlay consul overlay 1.02 0.07
SWARM overlay SWARM overlay 0.89 0.06
calico calico 1.50 0.10

Performance details information is as follows. The standard deviation for VM to VM is the biggest among all tests.

Connection type Median (Gbps) Avg (Gbps) Stddev (Mbps)
VM-VM 15.70 15.21 2520.56
bridge-bridge 14.00 13.92 751.78
consum overlay 0.89 1.02 295.2
SWARM overlay 0.85 0.89 231.13
calico 1.52 1.50 220.10

Analysis

The CPU usage indicates that “ksoftirqd” is the main overhead.