sk181925 - High CPU utilization on a Security Group when traffic undergoes NAT and L4 distribution is enabled

High CPU utilization on a Security Group when traffic undergoes NAT and L4 distribution is enabled

Product

Maestro HyperScale Firewall, Scalable Chassis

Version

R81.20

OS

Gaia

Last Modified

2024-07-01

Symptoms

High CPU utilization on a Security Group when traffic undergoes NAT and L4 distribution is enabled.
The $FWDIR/log/fwk.elg file on the Security Group contains this line:

[spike_detective <XXX>] Warning:cp_timed_blocker_handler: A handler [0x<XXX>] blocked for <XX> seconds

The /var/log/messages or /var/log/spike_detective/spike_detective.log file (sk166454) on the Security Group contains this line:

spike_detective: spike info: type: cpu, cpu core: <ID>, top consumer: fwk0_<ID>, start time: <DATE TIME>, spike duration (sec): <XX>, initial cpu usage: 100, average cpu usage: <XX>, perf taken: 0

The /var/log/messages file on the Security Group contains this line:

kernel:NMI watchdog: BUG: soft lockup - CPU#<ID> stuck for <XX>s! [fwk0_<ID>:<XXX>]

The CPU Spike Detective profiling data (sk166454) on the Security Group shows high CPU utilization in these functions:

Overhead     Command     Shared Object                 Symbol
........     .........   ...........................   .................................................

<XX>.<XX>%  fwk0_<ID>   libfw_kern_64_us_sp_<ID>.so   [.] fwx_alloc_global_atomic_get_next_port_and_inc

<XX>.<XX>%  fwk0_<ID>   libfw_kern_64_us_sp_<ID>.so   [.] kiss_ghtab_bl_insert_impl

Cause

When traffic undergoes NAT and L4 distribution is enabled, individual Security Group Members are restricted from allocating all available NAT ports. Instead, each Security Group Member must identify a NAT port that has a distribution impact similar to the original port. This constraint results in a prolonged duration for locating a free NAT port, leading to occasional soft-lockups.

Solution

This problem was fixed. The fix is included in:

Jumbo Hotfix Accumulator for R81.20 starting from Take 70

If you choose not to upgrade, Check Point can supply a Hotfix. Contact Check Point Support to get a Hotfix for this issue.

A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.

For faster resolution and verification, please collect CPinfo files from the Security Management Server and Security Gateways involved in the case.

Hotfix installation instructions:

Refer to sk168597 - How to install a Hotfix.

Best Practice

Disable the L4 distribution and enable the L3 distribution. In this configuration, every Security Group Member can utilize all available NAT ports.

See the R81.20 Quantum Maestro Administration Guide > Chapter "Managing Security Groups" > Section "Working with the Distribution Mode".

NOTE

This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Article Properties

Access Level: General
Status: Approved by TAC
Date Created: 2024-01-28
Last Modified: 2024-07-01