High Latency Between Two VMs In a L2 Private Guest Network

[fnavidan]

Hi everyone,

I am running Apache CloudStack v4.22.1.0. I have created a private L2 guest network, then added a second NIC to each VM connected to private network. When I ping other VMs in the same private network, I get following results:

[root@da1 ~]# ping -c 100 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
...
100 packets transmitted, 100 received, 0% packet loss, time 100024ms
rtt min/avg/max/mdev = 0.266/2.122/11.104/2.249 ms

[root@da1 ~]# traceroute 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 1.334 ms !X 1.088 ms !X 1.013 ms !X

All VMs are running on the same KVM hosts which should normally give under 1ms ping rtt between guests but it doesn't. My test scenario implements GlusterFS having 2 replicas + 1 arbiter which is sensitive to latency. Currently when I test write performance on the gluster volume, I get very low throughput around 30MB/s while writing to the local filesystem of VMs gives me a 350-400MB/s write speed.

Additionally, all VMs are running on the same primary storage which is mounted as a local NFS on the KVM host, and all nodes connected to the same l2 private network.

As my gluster traffic is passing through L2 network, there must not be any extra networking overheads in a l2 network type.

[kiwiflyer]

Your traceroute seems to be showing ICMP unreachable. Can you provide information on your two networks and as well as the routing table on your VMs?

What happens if you source your ping or traceroute from a specific interface?

[fnavidan]

Here are routing and network info:

[root@da1 ~]# ip route
default via 149.50.127.129 dev ens3 proto dhcp src 149.50.127.163 metric 100
10.0.0.0/24 dev ens9 proto kernel scope link src 10.0.0.1 metric 101
149.50.127.128/26 dev ens3 proto kernel scope link src 149.50.127.163 metric 100
[root@da1 ~]# traceroute -I ens9 10.0.0.2
ens9: Name or service not known
Cannot handle "host" cmdline arg `ens9' on position 1 (argc 2)
[root@da1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 1e:01:2c:00:02:d1 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 149.50.127.163/26 brd 149.50.127.191 scope global noprefixroute ens3
valid_lft forever preferred_lft forever
inet6 fe80::1c01:2cff:fe00:2d1/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:01:00:d4:00:01 brd ff:ff:ff:ff:ff:ff
altname enp0s9
inet 10.0.0.1/24 brd 10.0.0.255 scope global noprefixroute ens9
valid_lft forever preferred_lft forever
inet6 fe80::acc7:f278:d355:ff8f/64 scope link noprefixroute
valid_lft forever preferred_lft forever

Here are 4 samples of traceroute that shows variance in icmp tests:

[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 1.053 ms * *
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 7.838 ms 8.433 ms 8.229 ms
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 0.961 ms * *
[root@da1 ~]# traceroute -I -i ens9 10.0.0.2
traceroute to 10.0.0.2 (10.0.0.2), 30 hops max, 60 byte packets
1 da-node2 (10.0.0.2) 16.925 ms 16.459 ms 13.767 ms

And this is exact route path:

[root@da1 ~]# ip route get 10.0.0.2 from 10.0.0.1
10.0.0.2 from 10.0.0.1 dev ens9 uid 0
cache

And this ping results from exactly same private interface:

[root@da1 ~]# ping -I ens9 -c 100 10.0.0.2
PING 10.0.0.2 (10.0.0.2) from 10.0.0.1 ens9: 56(84) bytes of data.
...
100 packets transmitted, 100 received, 0% packet loss, time 100048ms
rtt min/avg/max/mdev = 0.268/2.195/21.840/2.881 ms

Interestingly even when I ping a VM's public IP inside its own KVM host, I get same high ping result.

[kiwiflyer]

Can you run virsh dumpxml on the host for this vm and post it?

[fnavidan]

We found out that the KVM host is experiencing extremely high CPU utilization so that no cpu time was left for real-time network packet processing. Further investigation revealed that we has a security incident on the host where xmrig, a crypto miner was installed on the host.

Further investigation, showed no sign of successful SSH login from untrusted IPs, So that we suspect that the attacker might have been able to execute remote commands using libvirt remote management port 16514, 8250 or 9090.

We have blocked ports 16514, 5900–6100, 8250, 9090, 49152–49216 in public zone and added to trusted zone. Also, because we only have one NIC on the host, all traffic types are going through cloudbr0 bridge. We needed to manually edit Network Manager config and change zone to trusted.

At this time, all services are working fine and all critical ports are filtered from public Internet. Also, significant reduction of CPU utilization on KVM host, actually fixed the network latency between VMs.