Zabbix 7.0 and Increasing System Limits

What does the message “…the user limit of 1024 file descriptors is insufficient. The maximum number of concurrent checks per worker has been reduced…” mean?

This message indicates that the current file descriptor limit for the user (in this case, the user under which the Zabbix server or proxy runs) is set to 1024. This limit is insufficient for the number of operations that Zabbix performs, leading to an automatic reduction in the maximum number of concurrent checks per worker. In practice, this means that the system cannot open enough files or network connections simultaneously, limiting the efficiency and performance of Zabbix in processing data.

What are File Descriptors?

File descriptors are identifiers used by the operating system to access files and network resources. Each process in the system has its own limit of file descriptors, which restricts the number of files or sockets that can be open simultaneously.

Why Increase the File Descriptor Limit?

With the growing number of devices and data that Zabbix monitors, more file descriptors may be needed. If a Zabbix instance requires more descriptors than the current limit, it can lead to errors or performance limitations, which may result in a reduction of the number of concurrent checks per worker and slow down data processing.

Steps to Increase the File Descriptor Limit

Checking the Current Limit

First, determine the current file descriptor limit for the running zabbix_server process with a single command:

cat /proc/$(pgrep -o zabbix_server)/limits | grep "Max open files"

This command will return the current soft and hard file descriptor limits for the zabbix_server process. For example:

Max open files   1024   524288   files

• The first value is the soft limit.
• The second value is the hard limit.

We need to increase the soft limit from 1024 to 10000. The value should be set according to your specific needs and can be higher than this example. Zabbix will notify you in the log if the setting is insufficient.

Creating a Directory for Override Configuration

Create a directory to store the new system configuration:

mkdir /etc/systemd/system/zabbix-server.service.d

Setting Permissions

Set the appropriate permissions for this directory:

chmod 755 /etc/systemd/system/zabbix-server.service.d

Creating Override Configuration

Create a file with override configuration to increase the file descriptor limit. In this example, we set the limit to 10000:

echo -e "[Service]\nLimitAS=infinity\nLimitRSS=infinity\nLimitNOFILE=10000\nLimitNPROC=1024" \
| sudo tee /etc/systemd/system/zabbix-server.service.d/override.conf

Setting Permissions for Override Configuration

Set the appropriate permissions for this file:

chmod 644 /etc/systemd/system/zabbix-server.service.d/override.conf

Verifying the Override Configuration Content

Ensure that the contents of the file have been correctly saved:

cat /etc/systemd/system/zabbix-server.service.d/override.conf

Activating the Changes

Reload the systemd configuration to apply the new settings:

systemctl daemon-reload

Verifying the Service Status

Check the status of the Zabbix server service:

systemctl status zabbix-server.service

Viewing the Additional Configuration

The image shows that the Zabbix server service is using additional configuration stored in the /etc/systemd/system/zabbix-server.service.d/ directory as the override.conf file. This additional configuration includes the necessary changes to increase the file descriptor limit.

Restarting the Application

To apply the new limits, restart the Zabbix server service:

systemctl restart zabbix-server.service

Verifying the Configuration After Restarting Zabbix Server

Check the current file descriptor limit for the running zabbix_server process again:

cat /proc/$(pgrep -o zabbix_server)/limits | grep "Max open files"

You should see that the process is now running with the new limit, for example:

Max open files   10000   10000   files

Conclusion

Increasing the file descriptor limit for a specific user can help prevent performance and stability issues with the Zabbix server or proxy from version 7.0 onwards. By following the steps outlined above, you can increase the file descriptor limits for a specific user, ensuring that your Zabbix instance can effectively handle more concurrent requests.