Firewall stops forwarding traffic and web GUI becomes unreachable after upgrade to 26.7 - ROOT CAUSE FOUND

[ubcnathan]

OPNsense forwarding failure — ROOT CAUSE FOUND

Acknowledgements

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/src/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/src/issues?q=is%3Aissue

The Non-AI Part

It's the human again... After some work today, it looks like having syn cookies enabled causes the firewall to block new TCP connections. Disabling syn cookies got the 26.7 firewall forwarding traffic again. I've updated the report below. Hopefully that's helpful.

This firewall is using virtio network drivers in a Proxmox VM.

Root cause (confirmed 2026-09-25)

pf SYN cookies set to always breaks all new TCP on OPNsense 26.7.x (FreeBSD 15.1 base).

• Config has <syncookies>always</syncookies> (Firewall → Settings → Advanced → SYN cookies). This was already set on 26.1.11_10, where it worked fine — so this is a regression in pf's syncookie handling on 26.7 / FreeBSD 15.1, not a config change from the upgrade.
• Generated ruleset line: set syncookies always (line 7 of /tmp/rules.debug).
• Note: net.inet.tcp.syncookies (a kernel TCP tunable, also in config.xml) is a different mechanism and not involved.

Fix / workaround

• Firewall → Settings → Advanced → SYN cookies → disabled (adaptive untested), then Apply.
• Verify: grep syncookies /tmp/rules.debug and pfctl -vsi | grep -i -A2 syncookie → mode never.
• Emergency CLI-only fix (non-persistent, lost on filter reload/reboot):

sed -E 's/^set syncookies.*/set syncookies never/' /tmp/rules.debug > /tmp/rules.test
pfctl -nf /tmp/rules.test && pfctl -f /tmp/rules.test

Loading this restored the web GUI immediately.

Evidence

• tcpdump on WAN: pf answers every SYN itself with SYN-ACK win 0, MSS-only options — including on behalf of hosts behind the firewall.
• After the client ACK validates, pf's own SYN toward the backend appears in pflog with IP cksum 0 / incorrect TCP cksum, tos 0x10, tiny window — and is blocked by pf's own default block rules (rule 77 on WAN, rule 62 on LAN).
• pfctl -s info: proto-cksum climbing ~25–30/s even with Suricata stopped and hw.vtnet.csum_disable=1; state-insert 0.
• pfctl -vsi: syncookies sent ~55/s, validated ~37/s, mode always.
• Connections to the firewall's own GUI (443): handshake completes, but the TLS ClientHello is never ACKed (ack 1 forever) → GUI unreachable. SSH sessions kept working.
• Symptoms: clients behind the firewall can't reach the internet; web GUI unreachable from WAN-side management host; SSH OK.

Ruled out (tested on fresh 26.7 install with imported config)

• Suricata / netmap IPS (configctl ids stop — no change; zero vtnet0 counters were just netmap owning the rings)
• Traffic Shaper / ipfw (net.inet.ip.fw.enable=0, no dnpipe/dnqueue in pf — no change); config_aqm flowset busy dmesg messages are a side issue
• vtnet checksum offload (hw.vtnet.csum_disable=1 in /boot/loader.conf.local — no change; can be removed)
• Captive Portal (disabled — no change). Note: the portal DOES generate rdr rules on vtnet1 and a default block rule (! <__captiveportal_zone_0>), so the old note "no rdr in this ruleset" was wrong.
• Older kernel on 26.7 userland — no change (unsupported mix; syncookie issue lives in pf, but result was inconclusive)
• Earlier theories (DDoS, php-cgi, Proxmox firewall, pf state-insert bug divert: issues on 15.1 #303 / core#10615, igb driver bug src#329) — not the cause.

Environment

• VM on Proxmox, virtio NICs
• Suricata IPS (netmap) on WAN, Traffic Shaper, Captive Portal zone 0 on LAN, Proxmox Firewall on vNICs
• Last good: 26.1.11_10. Broken: 26.7.1_1 (upgrade) and fresh 26.7.x install.

Bug report

• Firewall stops forwarding traffic and web GUI becomes unreachable after upgrade to 26.7 - ROOT CAUSE FOUND #326 — filed by the user; update with the syncookie finding (draft comment written 2026-09-25) and retitle to mention syncookies.

History (brief)

1. Upgrade 26.6 → 26.7.1_1: forwarding stopped, GUI unreachable, SSH fine. Rolled back to 26.1.11_10 — resolved.
2. Many theories chased (see Ruled out). Filed Firewall stops forwarding traffic and web GUI becomes unreachable after upgrade to 26.7 - ROOT CAUSE FOUND #326 blaming pf state handling with Suricata/shaper — superseded.
3. 2026-09-25: fresh 26.7 install + config import from backup disk (see config-backup-disk.md). Isolated each feature; pflog showed pf-generated SYNs being blocked; tcpdump showed win-0 SYN-ACKs; traced to set syncookies always; never fixed it.

Sources

• Firewall stops forwarding traffic and web GUI becomes unreachable after upgrade to 26.7 - ROOT CAUSE FOUND #326
• divert: issues on 15.1 #303, 26.7.1 regression: permitted traffic dropped with "state-insert,block" core#10615 (state-insert — not the cause)
• igb: DH89xxCC (I347-AT4, 8086:0438) ports never process RX/TX completions after upgrade to 26.7 (Riverbed SteelHead CX-770) #329 (igb driver regression on 26.7 — not the cause)
• https://docs.opnsense.org/releases/CE_26.7.html

[ubcnathan]

I had some more time to work on this today. The short, non-AI version is that having SYN cookies enabled was causing the firewall not to build new TCP connections.

This was turned on and working in 26.1, and the config was carried over to 26.7.

Disabling SYN cookies restores the functionality of the firewall.