Security Gateways with a large number of CPU cores allocated to CoreXL SND may experience performance issues when the "fwaccel dos deny" feature is configured

Product: Security Gateways
Version: R80.40 (EOS), R81 (EOS), R81.10 (EOS), R81.20
OS: Gaia
Last Modified: 2024-06-05

Symptoms

• Random traffic outage and latency followed by Security Gateway crash.
• No VMcore or process core dump files.
• CPview > CPU > Overview shows that most or all of the CoreXL_SND cores are utilized at around 100% (in the User column).
• CoreXL Dynamic Balancing is increasing the number of SND CPU cores, but it does not resolve the issue.
• Output of the perf top -C <SND CPU ID> command on the Security Gateway / Cluster Member during the traffic outage shows that the function responsible for the high CPU utilization is native_queued_spin_lock_slowpath.

Cause

System with a large number of CPU cores allocated to CoreXL SND may experience performance issues when an IoC Feed and the "fwaccel dos deny" feature are configured.

To confirm the issue, both of these conditions must be true:

1. Output of the perf top -C <SND CPU ID> command on the Security Gateway / Cluster Member during the traffic outage shows that the function responsible for the high CPU utilization is native_queued_spin_lock_slowpath.

2. The function dos_db_read_lock_deny_list appears in the perf report after running these commands in the Expert mode, while the SND CPU cores are highly utilized:

   1. perf record -g -C <CPU_Number> sleep 20
   2. perf report -g flat -sw > /var/log/perf_report.txt
   3. cat perf_report.txt

Solution

This problem was fixed. The fix is included starting from:

• Jumbo Hotfix Accumulator for R81.20 starting from Take 54
• Jumbo Hotfix Accumulator for R81.10 starting from Take 141
• Jumbo Hotfix Accumulator for R81 starting from Take 99
• Jumbo Hotfix Accumulator for R80.40 starting from Take 211

Check Point recommends to always upgrade to the Recommended version ( Security Gateway / VSX).

If you choose not to upgrade, contact Check Point Support to get a Hotfix for your version.

A Support Engineer will make sure the Hotfix is compatible with your environment before providing it.

For faster resolution and verification, collect CPinfo files from the Management Server and Security Gateways / Cluster Members involved in the case.

Hotfix installation instructions: Refer to sk168597 - How to install a Hotfix.

NOTE

This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.