rigacci.org

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Linux netfilter nf_conntrack_expect ======

I got a very weird issue with a **GNU/Linux firewall** used to do **NAT** for about 40 VoIP phones using the **SIP protocol**. Very often the phone calls broke unexpectedly, sometimes the call just drop, other times the voice stream stops.

===== Problem: all ports in use =====

Looking at the **syslog** I found a clear clue as to what the problem was:

<file>
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23404 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: all ports in use for SDP media IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
nf_ct_sip: dropping packet: cannot add expectation for voice IN= OUT= SRC=a.a.a.a DST=b.b.b.b
           LEN=820 TOS=0x08 PREC=0x80 TTL=127 ID=23405 PROTO=UDP SPT=13182 DPT=5060 LEN=800
</file>

So this is not a pure NAT problem, e.g. it is not a problem like a full NAT table. Here the problem is about the //expect table//, i.e. a table used by some specific helper modules to do connection tracking for specific network protocols. The specific connection tracking in my case is about **SIP protocol**, handled by kernel module **nf_ct_sip**.

The SIP protocol does not relay on normal TCP/IP sessions, so the association between the host behind NAT and the public server is not maintained by the normal NAT table. Instead, the kernel maintains the expect table, which can be inspected with:

<code>
cat /proc/net/nf_conntrack_expect
34 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=48968 PERMANENT sip/signalling
37 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=35460 PERMANENT sip/signalling
35 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=40949 PERMANENT sip/signalling
56 l3proto = 2 proto=17 src=b.b.b.b dst=a.a.a.a sport=0 dport=14161 PERMANENT sip/signalling
...
</code>

This should be the same information shown by:

<code>
conntrack -L expect
</code>

If you just want the count of entries:

<code>
conntrack --count expect
</code>

Statistics about creation/deletion of entries can be obtained with:

<code>
conntrack --stats expect
</code>

If the kernel is configured with **CONFIG_NF_CONNTRACK_PROCFS** (which is indeed declared obsolete), there are several procfs entries that can be inspected.

The //expect table// have a size limit, that can be inspected with:

<code>
cat /proc/sys/net/netfilter/nf_conntrack_expect_max
</code>

This is not the number of the //expect table// entries, but an unspecified size; I suspect that you have to **divide by 4** to obtain the hash table size. It seems that the default ''nf_conntrack_expect_max'' on GNU/Linux kernel 5.10 is a very low **256** (maybe 64 entries).

===== Solution =====

The solution is to enlarge the expect table. You can do it at runtime, after the **nf_conntrack module** is loaded, writing to the **/proc/** pseudo filesystem. Or you can pass the **expect_hashsize** parameter at module load time.

The last method is better, because you don't know when the kernel module is loaded during the boot process, so you risk to write the ''/proc/'' filesystem too early, when the module is still not present. So the best solution is to add a **/etc/modprobe.d/local.conf** file with this line:

<file>
options nf_conntrack expect_hashsize=2048
</file>

The number that you pass to **expect_hashsize** results **multiplied by 4** when you inspect the **/proc/sys/net/netfilter/nf_conntrack_expect_max** pseudo file.


===== sysctl and systemd =====

You might think of a solution by adding a file **/etc/sysctl.d/local.conf** with this line:

<file>
# This does not work, because module nf_conntrack is not loaded yet.
net.netfilter.nf_conntrack_expect_max=8192
</file>

But the **systemd** unit **systemd-sysctl.service** may enforce that setting too early, when the kernel module is not loaded yet. The error message in syslog will be;

<code>
Couldn't write '8192' to 'net/netfilter/nf_conntrack_expect_max',
    ignoring: No such file or directory
</code>

The option works instead if, after the bootstrap, you run:

<code>
systemctl restart systemd-sysctl.service
</code>

===== Who add entries into the expect table =====

Entries into the **conntrack expect table** are added by the **netfilter helpers**, e.g. the **sip helper** must be addressed by an **iptables** rule in the **raw table**, using the relevant port (e.g. the 5060/UDP for the SIP protocol).

Here it is ho to verify that the connection tracking sip helper rule exists into the raw table:

<code>
iptables -t raw -L -n 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
...
CT         udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:5060 CT helper sip

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
...
CT         udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:5060 CT helper sip
</code>

If the rules are missing, you can add them using the following commands:

<code>
iptables -A PREROUTING -t raw -p udp --dport 5060 -j CT --helper sip
iptables -A OUTPUT     -t raw -p udp --dport 5060 -j CT --helper sip
</code>

===== Shorewall and helpers =====

The **Shorewall** firewall can be configured to install helpers for some particular protocols, e.g. FTP, IRC, SIP, ...

The configuration file **/etc/shorewall/shorewall.conf** may contain the options **AUTOHELPERS** and **HELPERS**. The first option defaults to **AUTOHELPERS=Yes** and the second defaults to **HELPERS=**, but it is recommended to set it to **AUTOHELPERS=No** on modern kernels (>= 3.5) because it is not desiderable to have ruleset with automatically associated helpers for all the applications (the default when HELPERS is empty).

A possible configuration may be the following:

<file>
AUTOHELPERS=Yes
HELPERS=ftp,sip
</file>

in this case helpers will be configured into the iptables raw table for ''tcp dpt:21'' (FTP) and ''udp dpt:5060'' (SIP).

But the **recommended** configuration is the following:

<file>
AUTOHELPERS=No
HELPERS=ftp,sip
</file>

And be more specific with helpers in **/etc/shorewall/rules**, e.g.:

<file>
HELPER    loc    -    udp    5060    ; helper=sip
</file>

This combination will create the following iptable rule into the raw table:

<code>
Chain PREROUTING (policy ACCEPT 12 packets, 792 bytes)
 pkts bytes target  prot opt in     out  source     destination  
    0     0 CT      17   --  lan0   *    0.0.0.0/0  0.0.0.0/0    udp dpt:5060 CT helper sip
</code>

Or you can add this into **/etc/shorewall/conntrack**:

<code>
CT:helper:sip:PO
</code>

In this case the helper is instantiated into the raw table in both PREROUTING and OUTPUT chains.

The default **Debian 12 Bookworm** configuration for Shorewall provides a **conntrack** file where helpers are enabled only if the Shorewall **AUTOHELPERS** option is enabled (in ''shorewall.conf'') and if the **CT_TARGET** iptables/netfilter capability is available (verify the output of ''shorewall show capabilities'').

==== Shorewall upgrade from Debian 11 to 12 ====

In Debian, upgrading to **Shorewall 5.2.8** as per upgrade from **Debian 11 Bullseye** to **Debian 12 Bookworm**, connection tracking protocol helpers are no longer globally enabled by default; use **shorewall-conntrack(5)** or **shorewall-rules(5)** to enable them as appropriate where they are required.

Setting **AUTOHELPERS** to 'Yes' in shorewall.conf restores the previous behavior.

===== Web references =====

  * **[[https://forums.thinkbroadband.com/dslrouter/4443347-conntrack-expectation-table-full-issues-nf-conntrack.html|conntrack expectation table full issues (nf_conntrack)]]**
  * **[[https://home.regit.org/netfilter-en/secure-use-of-helpers/|Secure use of iptables and connection tracking helpers]]**