High CPU, high packet drops, soft lockup, loss of neighborships and total traffic outage when multiple Elephant Flows are running in parallel

Product: ClusterXL, Security Gateways
Version: R81.20, R82
OS: Gaia
Last Modified: 2025-02-17

Symptoms

• Customer's monitoring system reported high CPU utilization on a Security Gateway / Cluster Member.
• The /var/log/messages file contains messages like this:

zeco_vm_ops_shinfo_fault: dmd_1_worker_4, bad kernel_address (user_address 0xNNNNNNNNNNNN, vm_start 0xNNNNNNNNNNNN)

• The $FWIDR/log/dsd.elg file contains these messages:

ds_dmd_stop_wt_on_pcpu: ds_dmd_stop_wt failed

ds_dmd_perform_turn_off: Failed to remove WT from DMD 1

ds_change_dmd: Failed to perform DMD action Turn Off DMD

ds_change_do: Failed performing DMD action

ds_reset_dmd_state: Failed to turn off DMD 1

Warning: cp_timed_blocker_handler: A handler [0xNNNNNNN] blocked for NNN seconds.

Warning: cp_timed_blocker_handler: Handler info: Library [dsd], Function offset [0xNNNNN].

Warning: cp_timed_blocker_handler: Handler info: Nearest symbol name [ds_single_cycle], offset [0xNNNNN].

ds_dmd_down_handler: DMD exited

ds_connect_to_dmd_sensor: Could not connect to dmd sensor

• The /var/log/dump/usermode/ directory contains the dsd core dump files.
• The only way to recover the Security Gateway / Cluster Member is a reboot.

Cause

In a rare scenario, when multiple Elephant Flows are running in parallel in the accelerated pipelining path, there may be high CPU utilization. In the reported case, there were network scans and system backups running in parallel.

Solution

This problem was fixed. The fix is included starting from:

• Jumbo Hotfix Accumulator for R82 starting from Take 10
• Jumbo Hotfix Accumulator for R81.20 starting from Take 96

Check Point recommends to always upgrade to the Recommended version ( Security Gateway / VSX / Security Management Server / Multi-Domain Security Management Server / SmartConsole).

If you choose not to upgrade, contact Check Point Support to get a Hotfix for your version. A Support Engineer will make sure the Hotfix is compatible with your environment before providing it.

For faster resolution and verification, collect these files:

1. CPinfo file from the Management Server involved in the case.
2. CPinfo file from the Security Gateway / each Cluster Member involved in the case.

Hotfix installation instructions:

Refer to sk168597 - How to install a Hotfix.

NOTE

This solution has been verified for the specific scenario, described by the combination of Product, Version and Symptoms. It may not work in other scenarios.

Article Properties

Access Level: General
Status: Approved by TAC
Date Created: 2025-01-20
Last Modified: 2025-02-17