# High CPU utilization on a Security Group when traffic undergoes NAT and L4 distribution is enabled

## Product
**Maestro HyperScale Firewall, Scalable Chassis**  
## Version  
**R81.20**  
## OS  
**Gaia**  
## Last Modified  
**2024-07-01**

## Symptoms

- High CPU utilization on a Security Group when traffic undergoes NAT and L4 distribution is enabled.
- The _$FWDIR/log/fwk.elg_ file on the Security Group contains this line:

```
[spike_detective <XXX>] Warning:cp_timed_blocker_handler: A handler [0x<XXX>] blocked for <XX> seconds
```
- The `/var/log/messages` or `/var/log/spike_detective/spike_detective.log` file ([sk166454](https://support.checkpoint.com/results/sk/sk166454)) on the Security Group contains this line:

```
spike_detective: spike info: type: cpu, cpu core: <ID>, top consumer: fwk0_<ID>, start time: <DATE TIME>, spike duration (sec): <XX>, initial cpu usage: 100, average cpu usage: <XX>, perf taken: 0
```
- The `/var/log/messages` file on the Security Group contains this line:

```
kernel:NMI watchdog: BUG: soft lockup - CPU#<ID> stuck for <XX>s! [fwk0_<ID>:<XXX>]
```
- The CPU Spike Detective profiling data ([sk166454](https://support.checkpoint.com/results/sk/sk166454)) on the Security Group shows high CPU utilization in these functions:

```
Overhead     Command     Shared Object                 Symbol
........     .........   ...........................   .................................................

<XX>.<XX>%  fwk0_<ID>   libfw_kern_64_us_sp_<ID>.so   [.] fwx_alloc_global_atomic_get_next_port_and_inc

<XX>.<XX>%  fwk0_<ID>   libfw_kern_64_us_sp_<ID>.so   [.] kiss_ghtab_bl_insert_impl
```

## Cause
When traffic undergoes NAT and L4 distribution is enabled, individual Security Group Members are restricted from allocating all available NAT ports. Instead, each Security Group Member must identify a NAT port that has a distribution impact similar to the original port. This constraint results in a prolonged duration for locating a free NAT port, leading to occasional soft-lockups.

## Solution

This problem was fixed. The fix is included in:

- [Jumbo Hotfix Accumulator for R81.20](https://sc1.checkpoint.com/documents/Jumbo_HFA/R81.20/Default.htm) starting from Take 70

If you choose not to upgrade, Check Point can supply a **Hotfix**. [Contact Check Point Support](https://www.checkpoint.com/support-services/.contact-support/) to get a Hotfix for this issue.

A Support Engineer will make sure the Hotfix is compatible with your environment before providing the Hotfix.

For faster resolution and verification, please collect [CPinfo files](http://supportcontent.checkpoint.com/solutions?id=sk92739) from the Security Management Server and Security Gateways involved in the case.

**Hotfix installation instructions:**

Refer to [sk168597 - How to install a Hotfix](https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk168597).

### Best Practice

Disable the L4 distribution and enable the L3 distribution. In this configuration, every Security Group Member can utilize all available NAT ports.

See the [R81.20 Quantum Maestro Administration Guide](https://sc1.checkpoint.com/documents/R81.20/WebAdminGuides/EN/CP_R81.20_Maestro_AdminGuide/Default.htm) > Chapter "Managing Security Groups" > Section "Working with the Distribution Mode".

#### NOTE
This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

## Article Properties

- **Access Level**: General
- **Status**: Approved by TAC
- **Date Created**: 2024-01-28
- **Last Modified**: 2024-07-01