Check Point VPN-1 & FireWall-1 NG Performance Tuning Guide

• General Performance Considerations
  • Hardware Considerations
• OS & IP Stack Tuning Guidelines
  • Solaris
  • Windows NT/2000
  • Linux
  • Nokia
• VPN-1/FireWall-1 Performance Tuning

This section contains general recommendations that can significantly improve the VPN-1 performance.
These rules are platform independent and do not require any special tuning.

1. Always use the latest versions of Check Point products
2. Place the most commonly accessed rules on top of the rulebase
3. Keep the rulebase small & simple. Reduce the number of rules by combining similar rules together
4. Disable any FireWall-1 implied rules that you do not need
5. If not using VPN (encryption) on the module, make sure the VPN-1 Power product is disabled on that module
6. If using VPN, use AES-128 instead of 3DES or AES-256
7. Avoid using Domain objects
8. Use Networks instead of address ranges in NAT
9. Disable Decrypt on accept property if not using VPN (encryption)
10. Keep logging to a minimum

The hardware spec that affects VPN-1 performance most is the CPU speed. Use a machine with the fastest CPU available. 256 MB of RAM is a minimum for a high performance VPN/firewall system, in most of the cases, 512MB is recommended. Amount of L2 cache has significant positive impact on VPN-1/firewall performance - use large L2 cache whenever possible.

SMP (2 or more CPUs) has near linear performance scalability effect when using SecureXL, Security Servers and VPNx. Other VPN-1 products/policies although not designed to take direct advantage of the SMP, still benefit from additional CPUs, but to lesser extent.

Another very important hardware spec that has to be considered when deploying a high-performance VPN-1 gateway is the PCI bus bandwidth. VPN-1 enforcement software running on most modern CPUs, with lightweight policies, is capable of forwarding over a Gigabit of traffic. Gateways running SecureXL implementations are capable of forwarding over 3.0Gbps of traffic on the right hardware. In order to achieve these multi-Gigabit forwarding rates, multiple 66Mhz/64-bit PCI or 133Mhz/64-bit PCI-X busses must be used on the VPN-1 gateway (when PCI-X busses are used, PCI-X-compliant network interfaces are required to take advantage of the PCI-X 133Mhz standard). Multiple network interfaces must be evenly distributed over the PCI/PCI-X buses to provide for better bandwidth utilization.

Most of the information (unless specifically mentioned) here concerns Solaris version 8. Most of the tunables are also applicable to earlier 2.7 and 2.6 versions. At the same time, if using earlier Solaris versions an upgrade to version 8 is strongly recommended.

1. Always maintain the latest recommended patch level
(http://access1.sun.com/patch.recommended/rec.html)

2. Harden the Solaris for VPN/firewall use

This is not really a performance recommendation, but a rather a very important security issue that requires a dedicated document. Before firewall is installed on a machine, the underlying OS must be secured or 'hardened' for the highest security level possible. Numerous guideline books & documents exist that describe the necessary steps and considerations in detail. A good book to mention is Building Internet Firewalls by D. Brent Chapman and  Elizabeth D. Zwicky (O'Reilly, ISBN: 1565921240)

Nevertheless, a few hardening tips that also have a performance bearing are:

- disable all unnecessary network services, daemons, etc.

Comment out all unneeded service entries in /etc/inetd.conf file, definitely turn off netstat, systat, tfpt and finger services. Turn off rshd, rlogind and rexecd daemons; disable NFS if possible (rename or remove /etc/rc3.d/S15nfs.server); if not running VPN-1 GUI client on the Solaris firewall machine kill and disable dtlogin (run /etc/init.d/dtlogin stop and rename or remove /etc/rc2.d/S99dtlogin);

- modify the following IP stack parameters:

in /etc/rc2.d/S69inet:

ndd -set /dev/ip ip_forward_src_routed 0
ndd -set /dev/ip ip_forward_directed_broadcasts 0
ndd -set /dev/ip ip_respond_to_echo_broadcast 0
ndd -set /dev/ip ip_respond_to_timestamp_broadcast 0
ndd -set /dev/ip ip_send_redirects 0 (do not issue redirects)
ndd -set /dev/ip ip_ignore_redirect 1 (ignore redirects)

3. When multiple network interfaces or multiport NICs (qfe), set the MAC addresses of different interfaces to different values using ifconfig. By default all four interfaces are assigned the same MAC address. Alternatively set the 'local-mac-address?' variable in eeprom to true:

eeprom 'local-mac-address?=true'

4. When using Sun GigaSwift (ce) Gigabit Ethernet network interfaces, force the ce driver to use the 'traditional' STREAMS interface (as opposed to the GigSwift STREAMS interface). Yields significant improvement in throughput:

in /etc/system:

ce:ce_put_cfg = 1

* In general, Sun GigaSwift Gigabit interfaces (ce) achieve significantly better performance (especially with small packets) then standard Sun Gigabit interfaces (ge). It is therefore advisable to use the GigaSwift NICs in high-performance environments.

5. Force all LAN network interfaces to maximal speed and full duplex when applicable. Disable auto-negotiation.

For hme (also qfe) driver:

ndd -set /dev/hme instance 0 (1, 2, 3 etc)
ndd -set /dev/hme adv_autoneg_cap 0
ndd -set /dev/hme adv_100fdx_cap 1

repeat the commands above for every instance of the NIC driver

6. Tuning the STREAMS queues for high-throughput VPN-1 gateways

in /etc/system:

set sq_max_size = 100 (for a Solaris gateway with 256MB RAM)

Can produce noticeable throughput improvement.

7. Tuning the TCP hiwater parameters for maximal throughput - affects security servers and logging performance

ndd -set /dev/tcp tcp_xmit_hiwat 65535 (default 8192)
ndd -set /dev/tcp tcp_recv_hiwat 65535 (default 8192)

These values are essentially maximal send and receive buffer sizes. Small but noticeable HTTP security servers performance improvement.

8. Tuning the TCP Slow Start and TCP queue sizes - affects security servers performance

These changes are also recommended for tuning Solaris for maximal HTTP server
performance when the performance improvement can reach 100%.

In the /etc/system:

set tcp:tcp_conn_hash_size = 16384

and in /etc/rc2.d/S69inet:

ndd -set /dev/tcp tcp_slow_start_initial 2 (default 1)
ndd -set /dev/tcp tcp_conn_req_max_q 1024 (default 128)
ndd -set /dev/tcp tcp_conn_req_max_q0 4096 (dafault 1024)
ndd -set /dev/tcp tcp_time_wait_interval 60000 (default 240000)
    for Solaris 7 and earlier:
    ndd -set /dev/tcp tcp_close_wait_interval 60000 (default 240000)

ndd -set /dev/tcp tcp_fin_wait_2_flush interval 67500 (default 675000)

Small but noticeable HTTP security server performance improvement.

9. Tune the TCP Selective Acknowledgement (SACK) mechanism

In /etc/rc2.d/S69inet:

- turn on for better security servers performance over WAN links

ndd -set /dev/tcp tcp_sack_permitted 1

- turn off for better logging performance over LAN links

ndd -set /dev/tcp tcp_sack_permitted 0

10. Increase the number of open file descriptors - esp. relevant for busy security servers

in /etc/system:

rlim_fd_max = 16384 - 32768 (default 1024, should be at least 2 x tcp_conn_req_max)

11. Change the fsflush behavior
for busy Solaris gateways with more then 128MB RAM, decreases the amount of memory the fsflush scans every time it runs:

in /etc/system:

set autoup = 120 (default 30)

As a result, less time will be spent by the OS on flushing the memory and more on forwarding the packets. (Please note this parameter has been known to cause issues in 3rd Party HA/LB solutions and is currently not supported in those situations)

Recommendations in this section refer to NT 4.0 and/or 2000 versions of the Windows OS. Some of the tunables are common to both OS versions, in these cases no additional comments are made. If a particular tuning technique is only relevant to one of the OS versions, this fact is specifically mentioned in the guide.

1. Set the Windows NT memory strategy to "Maximize Throughput for Network Applications".

NT: By default it is set to "Maximize Throughput for File Sharing" which allocates all available to NT memory to file cache. Set under ControlPanel -> Network -> Server.

2. Disable all unneeded services and drivers

The services that are installed and started by default in Windows NT 4.0 and especially in Windows 2000 are too numerous to list. It is a good practice to go over all of them and disable everything that is not directly relevant to your VPN-1 gateway installation. Examples are DHCP client/server, Remote Registry Service, Distributed File System Service and others. The same concerns all the drivers that are not productively used in your installation, examples are Parallel, ParPort, etc.

3. Disable performance boost for foreground applications

NT: Set in ControlPanel -> System -> Performance. Move the Application Performance slider to None.
Gives the security servers the maximum amount of system resources.
2000: Set in ControlPanel -> System -> Advanced -> Performance Options. Give priority to Background Services

4. Pagefile optimization

Create a fixed size (do not allow it to grow dynamically) page file of at least 2 times the amount of RAM available on a another (preferably dedicated) disk drive.

5. Tuning TCP/IP registry values to improve network performance

Parameters that affect the IP forwarding performance:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\ForwardBufferMemory = 296960
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\MaxForwardBufferMemory = 296960

REG_DWORD, multiple of 256, default 74240. Buffer the IP allocates to store packet data in the router queue. The default value is enough for 50 1480-byte packets.

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\NumForwardPackets = 200
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\MaxNumForwardPackets = 200

REG_DWORD, default 50. Number of IP headers allocated for router queue. Should be at least as large as ForwardBufferMemory / IP data size of the network.

Increasing these four parameters can have significant positive effect on throughput especially with 'slow' policies. Also good for FloodGate-1 tuning.

Other TCP/IP stack parameters:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize

REG_DWORD, default 8760 for Ethernet. Larger TCP receive window size will improve performance over high-speed networks. For highest efficiency should be even multiple of TCP Maximum Segment Size (MSS).

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\MaxFreeTcbs = 0xFA0

REG_DWORD, default 2000, timewait table size

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\MaxHashTableSize = 0x400

REG_DWORD, default 512, TCB hash table size

Increase the number of ephemeral TCP user ports available to applications - prevents the SS from running out of ports:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Tcpip\Parameters\MaxUserPort = 65535

REG_DWORD, default 5000

6. On SMP systems, tune the Network Interface Card interrupt affinity for maximal performance

NT: Change the ProcessorAffinityMask:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NDIS\Parameters\ProcessorAffinityMask = 0

This improves the CPUs servicing of interrupts and DPCs generated by the network interface card(s). Also unties the NIC from a specific CPU.

W2000: Use the IntFiltr (Interrupt Affinity Filter) utility from Microsoft (available for free download) to assign interrupts from different NICs to different CPUs:

ftp://ftp.microsoft.com/bussys/winnt/winnt-public/tools/affinity/intfiltr.zip

7. Tuning the specific vendor's NIC parameters for maximal performance

Tuning the relevant NIC parameters can have significant performance effect.

Although the NIC Setup Interface and the NIC tunable parameters (properties) are proprietary to the specific network adapter most of the basic tunable parameters may be common to other brands/models.

Below are recommended settings for some commonly used Fast Ethernet and Gigabit NICs:

7.1 Intel PRO/1000 Family of Gigabit Ethernet network adapters

in the registry, under the following branch:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\E1000n\Parameters\

(x - number of the Intel PRO/1000 interface installed in the system)

set the following registry values:

NumberOfReceiveBuffers = 768
NumberOfCoalesceBuffers = 512
NumberOfTransmitDescriptors = 512
NumberOfReceiveDescriptors = 256

The actual registry value names may differ slightly depending on the PRO/1000 driver version (NumberOfTransmitDescriptors vs. NumTxDescriptors) but usually the names are close enough to be easily figured out. Most of the same values can also be set via the Intel PROSet utility under ControlPanel->Network->Adapter Properties->Advanced.

7.2 Alteon ACENic Gigabit Ethernet network adapter

TransmitControlBlocks = 64
RecvCoalMax = 20
RecvCoalTicks = 1000
SendCoalMax = 20
SendCoalTicks = 1000

7.2 Intel PRO/100+ PCI Fast Ethernet network adapter

Under ControlPanel->Network->Adapter Properties->Advanced:

Adaptive Performance Tuning set to Maximal Adapter Bandwidth
Receive Buffers = 256
Transmit Control Blocks =  64
Coalesce Buffers = 16

The actual performance effect of these tunables depends on many factors such as traffic type, packet sizes, etc, so it is advised, if such opportunity exists, to experiment with the values for the maximal performance gain. Otherwise use values listed here.

The information provided in this section refers to Linux RedHat 7.2 distribution. At the same time, many of the tuning techniques and tunable parameters are also applicable to earlier 7.0 and 6.2 versions.

1. Use Check Point SecurePlatform Linux for VPN-1 installations

The Check Point SecurePlatform product is Check Point version of Linux based on the Red Hat 7.2 distribution that is specifically adapted (hardened and tuned) for use with VPN-1 software. Current SecurePlatform Edition II is based on Linux kernel 2.4.9-31. It is strongly advised to use Check Point SecurePlatform for Linux-based VPN-1 installations.

The rest of the Linux tuning section is only relevant if you are not using Check Point Secure Platform Linux.

2. Use the latest Red Hat Linux kernels

Use the latest 2.4.x Linux kernels from Red Hat. The earliest 2.4.x kernel version suitable for use with VPN-1 use is 2.4.9-13. Current kernel recommended at the time of this guide's writing is 2.4.9-31.

During the installation process select Custom Installation option. That will allow you to install only the components required for your VPN-1 installation. Install the kernel Development Environment if you are going to build customized kernels (see below).

3. Install the latest recommended security patches

Latest recommended security and other patches can be downloaded from the Red Hat Errata Support Site

4. Recompile the minimal Linux kernel optimized for your particular hardware configuration and optimized for use as IP router

Implementing the recommendations provided below require significant Linux system administration skill & expertise. Kernel misconfiguration may render your system unusable and require a complete re-install. Follow these recommendations at your own risk.

while in your Linux kernel source directory, usually /usr/src/linux, run make config or make menuconfig:

- compile the kernel optimized for the particular CPU type you're using
- leave out support & drivers for all irrelevant stuff that is normally included in the default distribution kernel. Examples are: support for obscure IDE HD and floppy drives, bizarre filesystems, parallel ports, CMD640 chipset fixes, sound, etc, etc. This will produce the kernel with much smaller memory footprint which should & will work faster.
- statically compile in the drivers for hardware you're using on the gateway - NICs, relevant IDE/SCSI controllers, etc.
- during kernel compilation, turn on the Advanced Router (CONFIG_IP_ADVANCED_ROUTER) option. This will trigger some IP router specific questions. Turn on support for large routing tables (CONFIG_IP_ROUTE_LARGE_TABLES) and Optimize for Use as Router (CONFIG_IP_ROUTER) options. Turn on large windows support (CONFIG_SKB_LARGE option). Do not turn on the native IP firewalling options (CONFIG_IP_FIREWALL, CONFIG_IP_FIREWALL_NETLINK, CONFIG_IP_TRANSPARENT_PROXY, CONFIG_IP_MASQUERADE, etc).
- on IDE systems turn on the CONFIG_BLK_DEV_IDEDMA and CONFIG_IDEDMA_AUTO options
- on some systems PCI bridge optimization (CONFIG_PCI_OPTIMIZED option) can improve PCI bus performance. Disable it if you experience any PCI-related problems

For more info on compiling & installing customized Linux kernels refer to Linux Kernel HOWTO.

5. Disable all services & daemons that are not required on your VPN-1 installation

As in case of other OSs, listing all unnecessary services and daemons can take a document of it's own. A good idea is to go over all of them in /etc/xinetd.d/ and /etc/rc3.d/ directories and get rid of everything that is not directly required. Few examples are: netstat, finger, pop-2/3, apmd, NFS (if possible) pcmcia, etc, etc.

Do not install or run any of the X Windows components - XFree86, window managers, etc.

6. Tune the TCPIP stack parameters for maximal security servers and logging performance

Linux kernel versions 2.4.x and later apparently do quite a good job of auto-tuning some the TCP stack parameters, so changing them might not be necessary. On earlier Linux kernels (versions 2.2.x are supported by the VPN-1) most of the info below is still relevant.

edit /etc/rc.local, or create your own /etc/rc3.d/S100vpn1tuning file:

- increase the number of TCP ephemeral (short lived) ports:

echo "32768 65535" > /proc/sys/net/ipv4/ip_local_port_range

- turn off the TCP timestamps:

echo 0 > /proc/sys/net/ipv4/tcp_timestamps

- for better logging performance over fast LAN links, turn off SACKS and windows scaling:

echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling

- for better security servers performance over slow/noisy WAN links, turn SACKS and windows scaling on:

echo 1 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

- increase the amount of memory associated with input and output socket buffers:

echo 262144 > /proc/sys/net/core/rmem_default
echo 262144 > /proc/sys/net/core/rmem_max
echo 262144 > /proc/sys/net/core/wmem_default
echo 262144 > /proc/sys/net/core/wmem_max

7. Increase the number of file descriptors available to security servers

edit /etc/rc.local, or create your own /etc/rc3.d/S100vpn1tuning file:

echo 65536 > /proc/sys/fs/inode-max
echo 32768 > /proc/sys/fs/file-max
ulimit -n 32768

Some of the info below is based on the Nokia Support Performance Tuning recommendations.

1. Make sure FLOWS mechanism is enabled

on Nokia console run:

ipsofwd list     - if switch_mode is not flowpath, then set the switch_mode to FLOWS:
ipsctl -w net:ip:forward:switch_mode "flowpath"

2. For environments with massive numbers of concurrent connections

- along with adjusting the VPN-1 connections table size (see VPN-1 tuning section), expand the FLOWS tables:

by default, Nokia FLOWS tables can hold up to 131,072 connections without NAT or 65536 NAT-ed connections. To adjust the FLOWS table size do the following (in /var/etc/rc.local):

ipsctl -w net:ip:flow:flows_max_nexthops xxx    (xxx < 2,097,151)

- also to support over 100,000 concurrent connections in IPSO

in /var/etc/rc.local:

ipsctl -w kern:nokmalloc:waitokfreelimit 9437184
ipsctl -w kern:nokmalloc:nowaitfreelimit 6291456

2. Tune the IPSO for better security servers performance

run several instances of security servers, in case of HTTP security server, in $FWDIR/conf/fwauthd.conf:

80 in.ahttpd wait -4

along with the change above, increase the per-system open files limit to support 4 HTTP security servers:

ipsctl -w kern:maxfiles 16384    (can be added to the /var/etc/rc.local file)

2. Adjust the TCP MSS parameter for better security servers and logging performance

in /var/etc/rc.local:

ipsctl -w net:ip:tcp:default_mss 1460

This section describes different changes to the VPN-1 environment that can affect various aspects of firewall performance.

Unless specifically mentioned, the techniques and methods described here apply to version NG of the VPN-1.

1. In high-performance environments use Check Point performance acceleration products - Performance Pack, ACII and VPNx

In high-performance environments and on supported platforms, use Check Point Performance Pack to accelerate all aspects of VPN-1 & FireWall-1 performance. If for some reason Performance Pack is not an option, use ACII hardware accelerator to accelerate 3DES/DES encryption operations and VPNx (on SMP systems) to accelerate AES and 3DES/DES operations.

2. Expanding the VPN-1 concurrent connections limits - tuning the table space, hash and memory allocations

In latest NG versions of VPN-1 product, the concurrent connections limits can be tuned per enforcement module via the Check Point GUI Client interface. In Check Point Gateway Properties -> Advanced, set the supported number of concurrent connections to a maximum you foresee for you VPN-1 installation (allow for a sufficient margin). Also specify the size of the connections hash table as well as the default and maximal enforcement module memory pool sizes. Again, allow for sufficient margins. Bear in mind that due to platform limitations, Windows NT versions of VPN-1 cannot utilize more then ~100MB of kernel memory (~200M for Windows 2000 VPN-1 versions).

For VPN-1 installation with 100,000 anticipated concurrent connections, use the following values:

concurrent connections - 100,000
connections table hash size - 524,288
memory pool size, default - 32MB
memory pool size, max - 64MB

3. Adjusting the NAT tables parameters - size and hash

In environments with large (> 25000) number of concurrent connections with address translation increase the NAT tables size and hash size. Insufficient NAT tables size can lead to serious performance degradation.

in $FWDIR/conf/objects_5_0.C file, under props: section:

:nat_limit (xxx)              - to xxx desired value, default 25000
:nat_hashsize (yyy)        - to yyy desired value, power of 2 close to (or over) the table limit

4. Windows encryption performance tuning when handling massive numbers of concurrent encrypted tunnels.

add the following registry values (type DWORD):

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\FW1\Parameters\PacketPoolSize = 3000

(default 1Kbytes)

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\FW1\Parameters\BufferPoolSize = 6000

(PacketPoolSize x 2, default 2Kbytes)

Yields significant performance improvement when working with large numbers of concurrent encrypted tunnels.

5. VPN-1 HTTP security server performance tuning

to accelerate long data transfers, in $FWDIR/conf/objects_5_0.C, under props: section add value:

:http_buffer_size (16384)    (or 32768, default buffer size 4096)

According to our test results, may cause up to 30% proxy throughput improvement.

6. Run multiple instances of security servers on SMP systems

Make sure to run multiple instances of security servers on SMP systems, at least one instance per each CPU:

in case of HTTP proxy, for dual CPU system - in $FWDIR/conf/fwauthd.conf:

80    in.ahttpd    wait    -2

7. When LDAP server as user database minimize the number of groups