RX-OVR drops and 10 gb hardware buffer - Check Point CheckMates

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

8 Replies

JozkoMrkvicka

Authority

‎2018-09-0402:24 PM

https://community.checkpoint.com/message/22354-re-increasing-fifo-buffers-on-firewall-interface

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

One option is to increase RX ring size buffer, BUT as was mentioned by Tim, this is not the solution.

Check this one:

cc Timothy Hall ‌

Kind regards,

Jozko Mrkvicka

Kudos

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Timothy_Hall

MVP Gold

‎2018-09-0507:13 AM

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

10Gbps interfaces definitely have a lot more NIC hardware buffer space and much more processing power to avoid overruns. 1Gbps interfaces tend to start running out of gas somewhere north of 900Mbps of throughput, especially if most frames are near the minimum size. Increasing ring buffer sizes is a last resort and may get you into further trouble.

So a 10Gbps interface would certainly help, but before going down that road there are a few things that you should check:

Is the traffic being reasonably load balanced between the two 1Gbps bonded interfaces? Please post the output of netstat -ni showing both physical interfaces that are part of the bond. The RX-OK and TX-OK values should be roughly equivalent between the two interfaces. If RX-OK is substantially higher on one interface vs. another, check your bond operation mode/hash policy & STP setup on the upstream switch; if TX-OK is substantially higher on one interface vs. another, check your bond operation mode/hash policy on the firewall itself via the Gaia web interface.
Some driver/NIC combinations increment RX-DRP and RX-OVR in lock-step, and it is not possible to conclusively determine what is going on with just the netstat command. Please post the output of ethtool -S (interface) for the two physical interfaces, with this info it will be possible to see if there are overruns or drops/misses occurring on the interfaces. The mitigation strategy for one vs. the other is quite different.
If the adjacent switches are truly losing link integrity that sounds like a physical issue, since no matter how bad RX-DRPs/RX-OVR's get it should never cause a loss of link integrity. The ethtool output will show if there are actual carrier transitions occurring, it is also possible the switches are seeing the firewall NICs sending a flow control XOFF but that situation is quite different from actually losing carrier.

Second Edition of my "Max Power" Firewall Book

Now Available athttp://www.maxpowerfirewalls.com

New Book: "Max Power 2026" Coming Soon

Check Point Firewall Performance Optimization

1\ \ \ Kudo

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Collaborator

‎2018-09-0507:27 AM

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

Thanks! An SE is going on site today and we may get this info you asked about. In the time since I posted this, I found a thread on CPUG for a similar issue to what we are seeing here, and so for the time being I'm running with your ideas from that thread, that the rx-ovr may just be regular drops in lock step as you say here, and that "Keep all connections" should reduce the load on policy push at the time we see connections drop and rx-drops and the link show as down from the switch. The customer understands that is more of a fix for the symptom than the core issue.

Kudos

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Collaborator

‎2018-09-0509:25 AM

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

Below are netstat -ni and ethtool -S (interface) from one of the physical interfaces, and they all look similar. I don't see override errors on the physical. I do see in netstat -ni that TX-OK is about 3 times more for the first interface in the bond than the second. They are running 802.3ad for the bond operational mode. The ClusterXL Admin guide says "All the slave interfaces of a bond must be connected to the same switch. The switch itself must support and be configured for Link Aggregation, by the same standard (for example, 802.3ad, or XOR) as the Security Gateway bond." I do not know that to be the case on the switch, but I can check. Let me know your thoughts:

Kernel Interface table

Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

Mgmt 1500 0 0 0 0 0 0 0 0 0 BMU

Sync 1500 0 139761106 0 4769 0 483359145 0 0 0 BMRU

bond1 1500 0 5871992169 0 45491 45491 7323084565 0 0 0 BMmRU

bond1.5 1500 0 256770036 0 0 0 197671961 0 0 0 BMmRU

bond1.6 1500 0 7 0 0 0 7051 0 0 0 BMmRU

bond1.10 1500 0 5349158116 0 0 0 6813570243 0 0 0 BMmRU

bond1.162 1500 0 7998804 0 0 0 9863117 0 0 0 BMmRU

bond1.164 1500 0 94 0 0 0 302840 0 0 0 BMmRU

bond1.2096 1500 0 5913743 0 0 0 7977585 0 0 0 BMmRU

bond1.2128 1500 0 240366030 0 0 0 287076273 0 0 0 BMmRU

bond2 1500 0 8035743271 0 232231 232231 6498159149 0 0 0 BMmRU

bond2.2 1500 0 417512540 0 0 0 493632825 0 0 0 BMmRU

bond2.3 1500 0 6657176536 0 0 0 5183424132 0 0 0 BMmRU

bond2.4 1500 0 221669412 0 0 0 173697823 0 0 0 BMmRU

bond2.7 1500 0 7 0 0 0 7051 0 0 0 BMmRU

bond2.11 1500 0 715470330 0 0 0 639767094 0 0 0 BMmRU

bond2.12 1500 0 26 0 0 0 16350 0 0 0 BMmRU

bond2.69 1500 0 252108 0 0 0 204379 0 0 0 BMmRU

bond2.209 1500 0 4406029 0 0 0 753373 0 0 0 BMmRU

bond2.2231 1500 0 575132 0 0 0 176454 0 0 0 BMmRU

bond2.2232 1500 0 7 0 0 0 7051 0 0 0 BMmRU

bond2.2233 1500 0 3237699 0 0 0 3268497 0 0 0 BMmRU

eth1-05 1500 0 3207332404 0 15410 15410 5565121913 0 0 0 BMsRU

eth1-06 1500 0 2664661974 0 30081 30081 1757965501 0 0 0 BMsRU

eth1-07 1500 0 4295070957 0 130127 130127 4859459422 0 0 0 BMsRU

eth1-08 1500 0 3740674264 0 102104 102104 1638701316 0 0 0 BMsRU

lo 16436 0 14648647 0 0 0 14648647 0 0 0 LRU

NIC statistics:

rx_packets: 3743909849

tx_packets: 1640140201

rx_bytes: 3340303812889

tx_bytes: 695877051313

rx_broadcast: 9467756

tx_broadcast: 8181805

rx_multicast: 506441

tx_multicast: 14089

multicast: 506441

collisions: 0

rx_crc_errors: 0

rx_no_buffer_count: 83

rx_missed_errors: 102104

tx_aborted_errors: 0

tx_carrier_errors: 0

tx_window_errors: 0

tx_abort_late_coll: 0

tx_deferred_ok: 0

tx_single_coll_ok: 0

tx_multi_coll_ok: 0

tx_timeout_count: 0

rx_long_length_errors: 0

rx_short_length_errors: 0

rx_align_errors: 0

tx_tcp_seg_good: 0

tx_tcp_seg_failed: 0

rx_flow_control_xon: 0

rx_flow_control_xoff: 0

tx_flow_control_xon: 0

tx_flow_control_xoff: 0

rx_long_byte_count: 3340303812889

tx_dma_out_of_sync: 0

lro_aggregated: 0

lro_flushed: 0

lro_recycled: 0

tx_smbus: 0

rx_smbus: 0

dropped_smbus: 0

os2bmc_rx_by_bmc: 0

os2bmc_tx_by_bmc: 0

os2bmc_tx_by_host: 0

os2bmc_rx_by_host: 0

rx_errors: 0

tx_errors: 0

tx_dropped: 0

rx_length_errors: 0

rx_over_errors: 0

rx_frame_errors: 0

rx_fifo_errors: 102104

tx_fifo_errors: 0

tx_heartbeat_errors: 0

tx_queue_0_packets: 196959828

tx_queue_0_bytes: 83040941180

tx_queue_0_restart: 0

tx_queue_1_packets: 201112750

tx_queue_1_bytes: 92708792933

tx_queue_1_restart: 1

tx_queue_2_packets: 213423707

tx_queue_2_bytes: 86340004477

tx_queue_2_restart: 1

tx_queue_3_packets: 216066507

tx_queue_3_bytes: 83047793486

tx_queue_3_restart: 1

tx_queue_4_packets: 204186677

tx_queue_4_bytes: 73263555527

tx_queue_4_restart: 0

tx_queue_5_packets: 204344731

tx_queue_5_bytes: 92081661081

tx_queue_5_restart: 2

tx_queue_6_packets: 213009461

tx_queue_6_bytes: 93208852639

tx_queue_6_restart: 1

tx_queue_7_packets: 191036540

tx_queue_7_bytes: 77855042012

tx_queue_7_restart: 0

rx_queue_0_packets: 453259566

rx_queue_0_bytes: 409861555187

rx_queue_0_drops: 0

rx_queue_0_csum_err: 835

rx_queue_0_alloc_failed: 0

rx_queue_1_packets: 489210382

rx_queue_1_bytes: 429176166020

rx_queue_1_drops: 0

rx_queue_1_csum_err: 782

rx_queue_1_alloc_failed: 0

rx_queue_2_packets: 525554274

rx_queue_2_bytes: 416704083839

rx_queue_2_drops: 0

rx_queue_2_csum_err: 1347

rx_queue_2_alloc_failed: 0

rx_queue_3_packets: 477789928

rx_queue_3_bytes: 449291458631

rx_queue_3_drops: 0

rx_queue_3_csum_err: 1047

rx_queue_3_alloc_failed: 0

rx_queue_4_packets: 464353091

rx_queue_4_bytes: 423139957349

rx_queue_4_drops: 0

rx_queue_4_csum_err: 1394

rx_queue_4_alloc_failed: 0

rx_queue_5_packets: 455349424

rx_queue_5_bytes: 390616991572

rx_queue_5_drops: 0

rx_queue_5_csum_err: 1065

rx_queue_5_alloc_failed: 0

rx_queue_6_packets: 428981359

rx_queue_6_bytes: 384441256460

rx_queue_6_drops: 0

rx_queue_6_csum_err: 493

rx_queue_6_alloc_failed: 0

rx_queue_7_packets: 449411832

rx_queue_7_bytes: 407122763165

rx_queue_7_drops: 0

rx_queue_7_csum_err: 1075

rx_queue_7_alloc_failed: 0

Kudos

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Timothy_Hall

MVP Gold

‎2018-09-0509:57 AM

In response to Daniel_Westlund

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

OK now we are getting somewhere:

No carrier transitions, at least on the eth1-08 interface. Looks like bond1 consists of eth1-05 and eth1-06, while bond2 is eth1-07 and eth1-08. RX-DRP percentage is far less than the target of 0.1% though.
Looks like you only provided the ethtool stats for eth1-08, but is showing 99% misses/drops. There were 83 overruns probably caused by the ring buffer being full thus causing backpressure into the NIC buffer which was then overrun a few times. I would expect the other interfaces are similar, see #3 below...
Inbound RX balancing of the bonds looks good, but TX numbers are far enough apart that you probably should set L3/L4 hash balancing if you haven't already, although you don't seem to be having any problems on the TX side.
Given this is a 13800 with 20 cores, you are almost certainly running with a default split of 2/18 (4/36 if SMT enabled) for CoreXL allocations. So only two physical SND/IRQ cores are emptying the ring buffers of four very busy 1Gbps interfaces and they are not keeping up and causing drops/misses; if a large percentage of traffic is accelerated (use fwaccel stats -s to check) those 2 cores will be getting absolutely killed and seriously crimp the throughput of the box.

Would strongly recommend adjusting CoreXL split via cpconfig. If "Accelerated pkts/Total pkts" is >50% as reported by fwaccel stats -s reduce number of kernel instances from 18 to 14 to allocate 6 SND/IRQ cores; you may also want to disable SMT/Hyperthreading in this instance.

If "Accelerated pkts/Total pkts" is <50% as reported by fwaccel stats -sreduce number of kernel instances from 18/36 to 16/32 to allocate more SND/IRQ cores and leave SMT/Hyperthreading on.

Second Edition of my "Max Power" Firewall Book

Now Available athttp://www.maxpowerfirewalls.com

New Book: "Max Power 2026" Coming Soon

Check Point Firewall Performance Optimization

Kudos

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Collaborator

‎2018-09-0511:11 AM

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

Thanks again. We are actually way below 50% acceleration because they have a ton of blades turned on, and we already have Hyperthreading enabled with 8 SNDs and 32 workers like you suggested. Multiqueue is enabled on these interfaces (igb) and we added all 8 SNDs to multiqueue. Our next plan is to change to Keep all connections on the policy install. So, I don't know how to set L3/L4 hash balancing offhand but am looking into how to do that.

Kudos

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Timothy_Hall

MVP Gold

‎2018-09-0706:14 AM

In response to Daniel_Westlund

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

Right since you have plenty of SND/IRQ instances, automatic interface affinity (assuming SecureXL is enabled) will eventually assign each firewall interface its own dedicated SND/IRQ core to empty its ring buffer. If you are still experiencing RX-DRPs in that situation, that's the time to enable Multi-Queue on your busiest interfaces which you have already done.

You may want to run sar -n EDEV to see exactly when you are piling up those RX-DRPs. If they are only happening around the time of policy installation that is expected to some degree, and setting "Keep all connections" under Connection Persistence will help. If you are slowly accumulating them over time though, I'd argue that is not an actual problem since your RX-DRP rate is well below the generally recommended 0.1% (yours is actually 0.002%). This is a rather deep topic; please see the "RX-DRP Analysis & Discussion" section of my book for a discussion about why we don't necessarily expect RX-DRP to always be zero, even on a well-tuned firewall. RX-ERR and RX-OVR on the other hand are another matter entirely and should be zero or very close to it.

Second Edition of my "Max Power" Firewall Book

Now Available at http://www.maxpowerfirewalls.com

New Book: "Max Power 2026" Coming Soon

Check Point Firewall Performance Optimization

2\ \ \ Kudos

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

5.0

Collaborator

‎2018-09-0703:03 PM

Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Report Inappropriate Content

Thank you. We are sure the drops in question are happening on policy push so I think you're right that Keep all connections is a good next step. 23900s are on the way as well.

Kudos