I got a very weird issue with a GNU/Linux firewall used to do NAT for about 40 VoIP phones using the SIP protocol. Very often the phone calls broke unexpectedly, sometimes the call just drop, other times the voice stream stops.
Looking at the syslog I found a clear clue as to what the problem was:
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
So this is not a pure NAT problem, e.g. it is not a problem like a full NAT table. Here the problem is about the expect table, i.e. a table used by some specific helper modules to do connection tracking for specific network protocols. The specific connection tracking in my case is about SIP protocol, handled by kernel module nf_ct_sip.
The SIP protocol does not relay on normal TCP/IP sessions, so the association between the host behind NAT and the public server is not maintained by the normal NAT table. Instead, the kernel maintains the expect table, which can be inspected with:
cat /proc/net/nf_conntrack_expect 34 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=48968 PERMANENT sip/signalling 37 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=35460 PERMANENT sip/signalling 35 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=40949 PERMANENT sip/signalling 56 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=14161 PERMANENT sip/signalling ...
This should be the same information shown by:
conntrack -L expect
The expect table have a size limit, that can be inspected with:
cat /proc/sys/net/netfilter/nf_conntrack_expect_max
This is not the number of the expect table entries, but an unspecified size; I suspect that you have to divide by 4 to obtain the hash table size. It seems that the default nf_conntrack_expect_max on GNU/Linux kernel 5.10 is a very low 256 (maybe 64 entries).
The solution is to enlarge the expect table. You can do it at runtime, after the nf_conntrack module is loaded, writing to the /proc/ pseudo filesystem. Or you can pass the expect_hashsize parameter at module load time.
The last method is better, because you don't know when the kernel module is loaded during the boot process, so you risk to write the /proc/ filesystem too early, when the module is still not present. So the best solution is to add a /etc/modprobe.d/local.conf file with this line:
options nf_conntrack expect_hashsize=2048
You might think of a solution by adding a file /etc/sysctl.d/local.conf with this line:
# This does not work, because module nf_conntrack is not loaded yet. net.netfilter.nf_conntrack_expect_max=8192
But the systemd unit systemd-sysctl.service may enforce that setting too early, when the kernel module is not loaded yet. The error message in syslog will be;
Couldn't write '8192' to 'net/netfilter/nf_conntrack_expect_max',
ignoring: No such file or directory
The option works instead if, after the bootstrap, you run:
systemctl restart systemd-sysctl.service