sk181996 - Traffic outages may occur because of high utilization of CPU cores that run CoreXL SND instances

Traffic outages may occur because of high utilization of CPU cores that run CoreXL SND instances

Product: CoreXL
Version: R81.20
OS: Gaia
Last Modified: 2025-02-17

Symptoms

Random traffic outage through a Security Gateway / Cluster / VSX Virtual System.
High rate of RX-Drops even though we see low rate of throughput.
Output of the top command on the Security Gateway / Cluster Member during the traffic outage shows very high utilization of CPU cores that run the CoreXL SND instances.
Output of the perf top -C command on the Security Gateway / Cluster Member during the traffic outage shows that the function responsible for the high CPU utilization is fwmultik_do_seq_on_packet.

Example:

61.11% snd          [kernel.kallsyms] [k] fwmultik_do_seq_on_packet
26.99% ksoftirqd/37 [kernel.kallsyms] [k] fwmultik_do_seq_on_packet

The $FWDIR/log/fwk.elg file (in VSX mode) or /var/log/messages file (in Gateway mode) on the Security Gateway / Cluster Member contains this line repeatedly:

fwmultik_do_sequence_accounting_on_entry: sequence xxxxxxxx out of order (last handled sequence 4294967295)

Cause

Utilization of CPU cores that run CoreXL SND instances may reach high levels when there are connections with a long duration that transfer large traffic volume.

Solution

This problem was fixed. The fix is included in:

Check Point R82
Jumbo Hotfix Accumulator for R81.20 starting from Take 70

If you choose not to upgrade, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue.

A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.

For faster resolution and verification, please collect CPinfo files from the Security Management Server and Security Gateways involved in the case.

Hotfix installation instructions:

Refer to sk168597 - How to install a Hotfix.

This immediate workaround is available:

Connect to the command line on the Security Gateway / each Cluster Member / Security Group.
Log in to the Expert mode.
Configure the value of the kernel parameter fwmultik_validate_sequence to 0 (zero):

Note - This kernel parameter controls whether to print additional information during a kernel debug. It has no effect on the performance or security.

Important Note - Before you install the hotfix for this issue, permanently configure the value of the kernel parameter fwmultik_validate_sequence to 1 (one).


Deployment	Temporary Configuration	Permanent Configuration
Security Gateway, ClusterXL	In Gaia Clish, or in the Expert mode: `fw ctl set int fwmultik_validate_sequence 0`	In Gaia Clish, or in the Expert mode: `fw ctl set -f int fwmultik_validate_sequence 0`
Security Group in Maestro, Security Group on Scalable Chassis	In Gaia gClish: `fw ctl set int fwmultik_validate_sequence 0` In the Expert mode: `g_fw ctl set int fwmultik_validate_sequence 0`	In Gaia gClish: `fw ctl set -f int fwmultik_validate_sequence 0` In the Expert mode: `g_update_conf_file $FWDIR/modules/fwkern.conf fwmultik_validate_sequence=0`

NOTE

This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Article Properties

Access Level: General
Status: Approved by TAC
Date Created: 2024-02-15
Last Modified: 2025-02-17