KVM security groups: ingress rules are bypassed for traffic between VMs on the same host (regression from #10594) #14432

problem

Since #10594 ("security groups: Do not use conntrack when it is not required"), a VM can reach any port of another VM that runs on the same KVM host and the same bridge, regardless of the destination VM's ingress rules. Traffic from outside the host is still filtered correctly.
This breaks tenant isolation. The source and destination VMs can belong to different accounts.

Root cause

Each bridge chain evaluates the source VM first (BF-<br>-IN, --physdev-in <src vif>) and the destination VM second (BF-<br>-OUT, --physdev-out <dst vif>):

-A BF-br -m physdev --physdev-is-in  --physdev-is-bridged -j BF-br-IN
-A BF-br -m physdev --physdev-is-out --physdev-is-bridged -j BF-br-OUT

Since #10594, the source VM's egress chain ends in a terminating ACCEPT when the security group has no egress rules (and allowed egress rules also use ACCEPT):

# add_network_rules()
if egressrule_v4 == 0 :
    execute('iptables -A ' + egress_vmchain + ' -j ACCEPT')

The packet is accepted inside BF-<br>-IN, so BF-<br>-OUT is never reached and the destination VM's ingress chain is never evaluated. In 4.20 the egress chain ended with RETURN, which let the packet go on to the destination VM's chains.

When the two VMs run on different hosts, the destination host receives the packet on the uplink and evaluates BF-<br>-OUT, so ingress is enforced there. The bug only affects VMs on the same host and bridge.

versions

CLOUDSTACK VERSION: 4.23.0.0 (cloudstack-agent 4.23.0.0). Also present on current main.
COMPONENT: KVM agent – scripts/vm/network/security_group.py (Security Groups)
OS: KVM host: Ubuntu 24.04, iptables-nft
CONFIGURATION:

• Advanced zone with security groups enabled
• Shared guest network (VLAN), Linux bridge (brXXX-)
• net.bridge.bridge-nf-call-iptables = 1

The steps to reproduce the bug

STEPS TO REPRODUCE

1. In an advanced zone with security groups, create two security groups:
   • SG-A: no rules at all (default egress allow).
   • SG-B: one ingress rule, TCP 22 from 1.1.1.1/32 only.
2. Deploy VM-A with SG-A and VM-B with SG-B on the same shared network, and make sure both run on the same KVM host. For example, use a single host, or migrate one VM to the other's host.
3. From a machine outside the host (not 1.1.1.1), run ssh <VM-B IP>. The connection is blocked, as expected.
4. From inside VM-A, run ssh <VM-B IP>. The connection succeeds.

EXPECTED RESULTS

VM-B's ingress rules are enforced for all sources, including VMs on the same host.
SSH from VM-A to VM-B is dropped.

ACTUAL RESULTS

SSH from VM-A to VM-B succeeds. Only VM-A's egress rules are evaluated;
VM-B's ingress chain is skipped for same-host, same-bridge traffic.

What to do about it?

PROPOSED FIX

Use RETURN instead of ACCEPT for allowed egress, so the packet goes back to BF-<br> and continues to BF-<br>-OUT. This is the same pattern the system VM chains use (-A r-XXX-VM -m physdev --physdev-in vnetX -j RETURN). Traffic to the uplink is still accepted by -A BF-<br> -m physdev --physdev-out <uplink> -j ACCEPT.

Explicit egress denial moves into the egress chain. ESTABLISHED,RELATED is kept there so replies to allowed inbound connections are not dropped by the egress rules.

@@ default_network_rules()
-        execute("iptables -A " + vmchain_default + " -m physdev --physdev-is-bridged --physdev-in " + vif + " -j DROP")
+        execute("iptables -A " + vmchain_default + " -m physdev --physdev-is-bridged --physdev-in " + vif + " -j RETURN")
@@
-        execute('ip6tables -A ' + vmchain_default + ' -m physdev --physdev-is-bridged --physdev-in ' + vif + ' -j DROP')
+        execute('ip6tables -A ' + vmchain_default + ' -m physdev --physdev-is-bridged --physdev-in ' + vif + ' -j RETURN')

@@ add_network_rules()
             if rule['ruletype'] == 'E':
                 vmchain = egress_vmchain
                 direction = "-d"
-                action = "ACCEPT"
+                action = "RETURN"
@@
         if egressrule_v4 == 0 :
-            execute('iptables -A ' + egress_vmchain + ' -j ACCEPT')
+            execute('iptables -A ' + egress_vmchain + ' -j RETURN')
         else:
-            execute('iptables -A ' + egress_vmchain + ' -j RETURN')
+            execute('iptables -A ' + egress_vmchain + ' -m state --state ESTABLISHED,RELATED -j RETURN')
+            execute('iptables -A ' + egress_vmchain + ' -j DROP')

         if egressrule_v6 == 0 :
-            execute('ip6tables -A ' + egress_vmchain + ' -j ACCEPT')
+            execute('ip6tables -A ' + egress_vmchain + ' -j RETURN')
         else:
-            execute('ip6tables -A ' + egress_vmchain + ' -j RETURN')
+            execute('ip6tables -A ' + egress_vmchain + ' -m state --state ESTABLISHED,RELATED -j RETURN')
+            execute('ip6tables -A ' + egress_vmchain + ' -j DROP')

We tested the IPv4 part on 4.23.0.0. After a stop/start of the VMs so the -def chains are rebuilt:

• Same-host traffic from VM-A to VM-B is dropped by VM-B's ingress chain.
• External access that VM-B's ingress rules allow still works.
• VM outbound and internet access is unaffected.

The IPv6 part mirrors the IPv4 change but has not been tested.

Note: VMs added to cs_notrack (any rule with protocol=all from 0.0.0.0/0) have no conntrack state, so the ESTABLISHED,RELATED rule does not match for them. That behaviour is unchanged from 4.23. Also, the conntrack*_not_needed check does not look at the rule type, so an egress all 0.0.0.0/0 rule also disables conntrack for the VM. This may deserve a separate look.