TCPIP tuning in Linux

Still under construction

Actions to adequate a linux web server for high load web traffic


Nice place to do network calculations…


  14 bytes    20 bytes    20 bytes       1460 bytes      4 bytes
| Ethernet  |    IP     |    TCP    |                  |     |
|  header   |  header   |  header   |      Payload     | FCS |
|           |           |           |                  |     |
                                    |---- TCP MSS -----|
            |-------------- IP MTU --------------------|
            |-------------- ETHERNET MTU --------------|

The common value of MTU in the internet is 1500 bytes. The MTU is built from payload (also referred as data) and the TCP and the IP header, 20 bytes each. The total value of the IP and the TCP header is 40 bytes and mandatory for each packet, which leaves us 1460 bytes for our data.

Now, imagine that we are using the GRE protocol in our network, encapsulating the original packet and adding 24 bytes for the GRE header.

The total size of this kind of packet will be 1524 bytes, exceeding the 1500 bytes MTU value. The “data” size in this packet is 1460, but we can and should decrease it in order to make sure the total size will be 1500 bytes or less. And this is where TCP MSS comes into the picture.

TCP MSS, the maximum segment size, is a parameter of the options field of the TCP header that specifies the largest amount of data, specified in bytes, that a computer or communications device can receive in a single TCP segment. It does not include the TCP header or the IP header. This value will dictate the maximum size of the “data” part of the packet. In the following case for the GRE tunnel, we will set the tcp mss value to be 1436 or lower, while the default size is 1460.

The MSS announcement (often mistakenly called a negotiation) is sent during the three-way handshake by both sides, saying: “I can accept TCP segments up to size x”. The size (x) may be larger or smaller than the default. The MSS can be used completely independently in each direction of data flow.

Since the end device will not always know about high level protocols that will be added to this packet along the way, like GRE packets for example, it won’t usually adjust the TCP MSS value. As a result the network devices have the option to rewrite the value of TCP MSS packets that are processed through them. For example, in a Cisco Router the command “ip tcp mss-adjust 1436” in the interface level will rewrite the value of the TCP MSS of any SYN packet that will go via this interface.

OS level

As everything in unix is a file or, better, part of the filesystem (impressive answer !!), any TCPIP socket consume file descriptors.

Default settings allow a small number of file descriptors per user/process. This is set in /etc/security/limits.conf

Verify file limits with

cat /proc/sys/fs/file-max   for system wide file limit
cat /proc/<PID>/limits      for file limits for a given process PID

Set file limits in

/etc/security/limits.conf for user level file limits, adding entries like

  *         hard    nofile      600000
  *         soft    nofile      600000
  root      hard    nofile      600000
  root      soft    nofile      600000

/etc/sysctl.conf for system wide file limit adding

  fs.file-max = 2097152  

Change will take effect with ´sysctl -p´


Allowed local port range net.ipv4.ip_local_port_range = 2000 65535

Number of processes