I got a very weird issue with a GNU/Linux firewall used to do NAT for about 40 VoIP phones using the SIP protocol. Very often the phone calls broke unexpectedly, sometimes the call just drop, other times the voice stream stops.

Looking at the syslog I found a clear clue as to what the problem was:

nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800

So this is not a pure NAT problem, e.g. it is not a problem like a full NAT table. Here the problem is about the expect table, i.e. a table used by some specific helper modules to do connection tracking for specific network protocols. The specific connection tracking in my case is about SIP protocol, handled by kernel module nf_ct_sip.

The SIP protocol does not relay on normal TCP/IP sessions, so the association between the host behind NAT and the public server is not maintained by the normal NAT table. Instead, the kernel maintains the expect table, which can be inspected with:

cat /proc/net/nf_conntrack_expect
34 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=48968 PERMANENT sip/signalling
37 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=35460 PERMANENT sip/signalling
35 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=40949 PERMANENT sip/signalling
56 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=14161 PERMANENT sip/signalling
...

This should be the same information shown by:

conntrack -L expect

If you just want the count of entries:

conntrack --count expect

Statistics about creation/deletion of entries can be obtained with:

conntrack --stats expect

If the kernel is configured with CONFIG_NF_CONNTRACK_PROCFS (which is indeed declared obsolete), there are several procfs entries that can be inspected.

The expect table have a size limit, that can be inspected with:

cat /proc/sys/net/netfilter/nf_conntrack_expect_max

This is not the number of the expect table entries, but an unspecified size; I suspect that you have to divide by 4 to obtain the hash table size. It seems that the default nf_conntrack_expect_max on GNU/Linux kernel 5.10 is a very low 256 (maybe 64 entries).

The solution is to enlarge the expect table. You can do it at runtime, after the nf_conntrack module is loaded, writing to the /proc/ pseudo filesystem. Or you can pass the expect_hashsize parameter at module load time.

The last method is better, because you don't know when the kernel module is loaded during the boot process, so you risk to write the /proc/ filesystem too early, when the module is still not present. So the best solution is to add a /etc/modprobe.d/local.conf file with this line:

options nf_conntrack expect_hashsize=2048

The number that you pass to expect_hashsize results multiplied by 4 when you inspect the /proc/sys/net/netfilter/nf_conntrack_expect_max pseudo file.

You might think of a solution by adding a file /etc/sysctl.d/local.conf with this line:

# This does not work, because module nf_conntrack is not loaded yet.
net.netfilter.nf_conntrack_expect_max=8192

But the systemd unit systemd-sysctl.service may enforce that setting too early, when the kernel module is not loaded yet. The error message in syslog will be;

Couldn't write '8192' to 'net/netfilter/nf_conntrack_expect_max',
    ignoring: No such file or directory

The option works instead if, after the bootstrap, you run:

systemctl restart systemd-sysctl.service

Entries into the conntrack expect table are added by the netfilter helpers, e.g. the sip helper must be addressed by an iptables rule in the raw table, using the relevant port (e.g. the 5060/UDP for the SIP protocol).

Here it is ho to verify that the connection tracking sip helper rule exists into the raw table:

iptables -t raw -L -n 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
...
CT         udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:5060 CT helper sip

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
...
CT         udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:5060 CT helper sip

If the rules are missing, you can add them using the following commands:

iptables -A PREROUTING -t raw -p udp --dport 5060 -j CT --helper sip
iptables -A OUTPUT     -t raw -p udp --dport 5060 -j CT --helper sip

: Shorewall automatically adds that rules? When?

• conntrack expectation table full issues (nf_conntrack)
• Secure use of iptables and connection tracking helpers