Set up APM in Zabbix 8.0 LTS with ClickHouse and OpenTelemetry

Add APM to an existing Zabbix 8.0 LTS installation. Prepare ClickHouse, set the data source in the frontend, and collect traces, metrics, and logs through Zabbix proxy.

What awaits us?

In this guide we configure APM in an existing Zabbix 8.0 LTS installation. We install ClickHouse for telemetry, enable collection on the proxy, and view application metrics, traces, and logs in Zabbix.

Architecture

OTLP is the OpenTelemetry protocol for sending metrics, traces, and logs from applications. Zabbix proxy accepts OTLP over gRPC. If an app exports only over HTTP or HTTPS, the OpenTelemetry Collector receives its telemetry and forwards it to the proxy over gRPC.

Telemetry flow into Zabbix and ClickHouse OTLP/gRPC Write Read App with OTLP/HTTP(S) Export without gRPC App with OTLP/gRPC Direct export OpenTelemetry Collector Forwards telemetry over gRPC Zabbix proxy Receives OTLP/gRPC Zabbix server + frontend APM views and telemetry queries ClickHouse otel database

ClickHouse runs on a dedicated host in a private network. The proxy writes telemetry to it. The frontend reads it for APM views, and Zabbix server reads it for Telemetry query items. The Collector runs alongside the proxy on the same host. The main Zabbix database continues to handle configuration and regular monitoring. Synchronize the clocks on Zabbix server and ClickHouse.

ConnectionPort and protocol
Application → Collector4318 OTLP/HTTP or 8443 OTLP/HTTPS
Collector → proxy4317 OTLP/gRPC on the same host
App with gRPC → proxy4317 OTLP/gRPC when enabled on the proxy
Active proxy → Zabbix server10051 Zabbix protocol
Proxy, server, and frontend → ClickHouse8123 HTTP on the private network

Supported versions

According to the documentation checked on November 19, 2026, Zabbix 8.0 supports ClickHouse 26.8.17.4 through 26.9.x.x for APM. Check the current range before installing or upgrading. Use Zabbix 8.0 for the server, frontend, proxy, and schema package. Current Zabbix 8.0 requirements

Operating system selection

Start with the OS of the ClickHouse host. Replace the sample addresses and names in the commands with your own. For Zabbix server installation, use our installation guide.

Alma/CentOS/RHEL/Rocky 10
Debian 13

Prepare ClickHouse for telemetry

Install ClickHouse

Run the commands in this section as root. On the dedicated ClickHouse host, add the official LTS repository and list the available version.

dnf install dnf-plugins-core curl nano
dnf config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo
dnf config-manager --set-disabled clickhouse-stable
dnf config-manager --set-enabled clickhouse-lts
dnf --refresh list --available clickhouse-server clickhouse-client

Run the next command if the offered version meets the current Zabbix APM requirements. If it is outside that range, choose matching supported server and client versions from the official ClickHouse packages. On a fresh installation via dnf, the default account starts with an empty password. Set it after checking the version.

dnf install clickhouse-server clickhouse-client

Install the Zabbix schema package

On the same ClickHouse host, add the Zabbix 8.0 repository. The zabbix-sql-scripts package provides the script that creates the APM tables.

. /etc/os-release
rpm -Uvh "https://repo.zabbix.com/zabbix/8.0/release/${ID%linux}/${VERSION_ID%%.*}/noarch/zabbix-release-latest.el${VERSION_ID%%.*}.noarch.rpm"
dnf --setopt='epel.excludepkgs=zabbix*' install zabbix-sql-scripts

Configure access to ClickHouse

On the ClickHouse host, allow 8123/TCP for the Zabbix server, frontend, and proxy. Run the firewalld commands as root. The example uses the public zone on the private network interface.

firewall-cmd --permanent --zone=public --add-port=8123/tcp
firewall-cmd --reload

After configuring the firewall, create the following file on the ClickHouse host.

nano /etc/clickhouse-server/config.d/20-otel-listen.xml
<clickhouse>
    <listen_host>0.0.0.0</listen_host>
</clickhouse>

0.0.0.0 listens on every IPv4 interface. It replaces the default local-only listening so the other hosts can reach the database. To listen only on a specific private IP, use that IP instead of 0.0.0.0 and add a second listen_host with 127.0.0.1 for local administration.

clickhouse-client uses port 9000 for its native SQL connection. The built-in SQL console runs at http://127.0.0.1:8123/play and uses the same accounts and permissions. access_management lets the default account manage users and access through SQL commands such as CREATE USER and GRANT. We restrict that account to local connections. ClickHouse web console

nano /etc/clickhouse-server/users.d/20-local-admin.xml
<clickhouse>
    <users>
        <default>
            <networks replace="replace">
                <ip>127.0.0.1</ip>
                <ip>::1</ip>
            </networks>
            <access_management>1</access_management>
        </default>
    </users>
</clickhouse>

Start ClickHouse and check its version. The client connects locally as default. If the account already has a password, add --ask-password to the query.

systemctl enable clickhouse-server
systemctl restart clickhouse-server
clickhouse-client --host 127.0.0.1 --query 'SELECT version()'

Set the password for the default account

If the default account still has an empty password, run the following block in Bash and enter a new password.

read -rsp 'New password for the ClickHouse default account: ' CH_PASSWORD
printf '\n'
printf '%s' "$CH_PASSWORD" | sha256sum | cut -d ' ' -f 1
unset CH_PASSWORD

Copy the printed hash into a temporary note. Then open the configuration file as root.

nano /etc/clickhouse-server/users.d/20-local-admin.xml

Replace the entire file with the XML below. Substitute the hash you copied for SHA256_HASH and save the file.

<clickhouse>
    <users>
        <default>
            <networks replace="replace">
                <ip>127.0.0.1</ip>
                <ip>::1</ip>
            </networks>
            <access_management>1</access_management>
            <password remove="1" />
            <password_sha256_hex>SHA256_HASH</password_sha256_hex>
        </default>
    </users>
</clickhouse>

Restart ClickHouse. Use your new password when creating the database and at later prompts for the default account.

systemctl restart clickhouse-server

Create the database and Zabbix account

In Bash on the ClickHouse host, create the otel database and run the script that creates its metrics, traces, and logs tables. The -t 604800 option sets seven-day retention.

clickhouse-client --host 127.0.0.1 --ask-password \
  --query 'CREATE DATABASE IF NOT EXISTS otel'

read -rsp 'Password for the ClickHouse default account: ' CH_PASSWORD
echo
/usr/share/zabbix/sql-scripts/clickhouse/otel_all.sh \
  -s http://127.0.0.1:8123 -d otel -u default -p "$CH_PASSWORD" -t 604800
unset CH_PASSWORD

Open the ClickHouse console. Create otel_zabbix for proxy writes and server and frontend reads. Set a new password. The last command lists the tables, including otel_traces and otel_logs.

clickhouse-client --host 127.0.0.1 --ask-password
-- Enter your own password for Zabbix.
CREATE USER otel_zabbix IDENTIFIED WITH sha256_password BY 'REPLACE_OTEL_PASSWORD';
GRANT SELECT, INSERT ON otel.* TO otel_zabbix;
SHOW TABLES FROM otel;

Set up ClickHouse in the Zabbix frontend

Open Administration → Data source. Under APM, select Enable global data source. Saving this form sets the shared source for proxy writes, queries on Zabbix server, and APM views in the frontend. Zabbix stores it in its main database, and the server passes the setting to the proxy during synchronization. Set TelemetryProvider in their files only if that component needs its own connection.

Frontend fieldValue
Database typeClickHouse
URLhttp://clickhouse.example.com:8123 with your hostname or IP
AuthenticationUsername and password
Usernameotel_zabbix
PasswordThe password you set in ClickHouse
Databaseotel

Click Test, then click Update after the test succeeds. The test checks the frontend’s connection to ClickHouse and requires PHP cURL.

1 Enable the source · 2 Enter the connection details · 3 Click Test. Then save with Update.
1 Enable the source · 2 Enter the connection details · 3 Click Test. Then save with Update.

Install Zabbix proxy with APM

Move to the Zabbix proxy host. If it runs a different OS from ClickHouse, switch the OS selection above and return to the proxy installation. Add the Zabbix 8.0 repository.

. /etc/os-release
rpm -Uvh "https://repo.zabbix.com/zabbix/8.0/release/${ID%linux}/${VERSION_ID%%.*}/noarch/zabbix-release-latest.el${VERSION_ID%%.*}.noarch.rpm"

On EL 10, the APM-capable proxy comes in a separate package with the -apm suffix. For SQLite, install zabbix-proxy-sqlite3-apm. The commands also enable CRB and EPEL for dependencies.

dnf install dnf-plugins-core

. /etc/os-release
if [ "$ID" = rhel ]; then
  subscription-manager repos --enable "codeready-builder-for-rhel-10-$(arch)-rpms"
  dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm
else
  dnf config-manager --set-enabled crb
  dnf install epel-release
fi

dnf --setopt='epel.excludepkgs=zabbix*' install zabbix-proxy-sqlite3-apm nano

This package supports standard agent and SNMP monitoring. It also works as a regular proxy with APM off. Set StartAPMCollector=1 to add OpenTelemetry ingestion.

Prepare ClickHouse for telemetry

Install ClickHouse

On the dedicated ClickHouse host, add the official LTS repository and list the available version.

sudo apt update
sudo apt install ca-certificates curl gnupg nano
sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://packages.clickhouse.com/rpm/lts/repodata/repomd.xml.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/clickhouse.gpg
ARCH=$(dpkg --print-architecture)
printf '%s\n' "deb [arch=$ARCH signed-by=/etc/apt/keyrings/clickhouse.gpg] https://packages.clickhouse.com/deb lts main" \
  | sudo tee /etc/apt/sources.list.d/clickhouse.list
sudo apt update
apt-cache policy clickhouse-server clickhouse-client

Run the next command if the offered version meets the current Zabbix APM requirements. If it is outside that range, choose matching supported server and client versions from the official ClickHouse packages. When installing interactively via apt, choose your own password at the Set up the password for the default user prompt.

sudo apt install clickhouse-server clickhouse-client

Install the Zabbix schema package

On the same ClickHouse host, add the Zabbix 8.0 repository. The zabbix-sql-scripts package provides the script that creates the APM tables.

sudo apt update
sudo apt install curl ca-certificates
curl -fLO https://repo.zabbix.com/zabbix/8.0/release/debian/pool/main/z/zabbix-release/zabbix-release_latest_8.0+debian13_all.deb
sudo dpkg -i zabbix-release_latest_8.0+debian13_all.deb
sudo apt update
sudo apt install zabbix-sql-scripts

Configure access to ClickHouse

The firewall commands in this guide assume that firewalld is running on the host named in each step. To set up a firewall on Debian 13 with firewalld, install and start it as shown below. If you already manage the firewall with another tool, allow the same ports there.

sudo apt install firewalld
sudo systemctl enable --now firewalld

On the ClickHouse host, allow 8123/TCP for the Zabbix server, frontend, and proxy. Run the firewalld commands as root. The example uses the public zone on the private network interface.

firewall-cmd --permanent --zone=public --add-port=8123/tcp
firewall-cmd --reload

After configuring the firewall, create the following file on the ClickHouse host.

nano /etc/clickhouse-server/config.d/20-otel-listen.xml
<clickhouse>
    <listen_host>0.0.0.0</listen_host>
</clickhouse>

0.0.0.0 listens on every IPv4 interface. It replaces the default local-only listening so the other hosts can reach the database. To listen only on a specific private IP, use that IP instead of 0.0.0.0 and add a second listen_host with 127.0.0.1 for local administration.

clickhouse-client uses port 9000 for its native SQL connection. The built-in SQL console runs at http://127.0.0.1:8123/play and uses the same accounts and permissions. access_management lets the default account manage users and access through SQL commands such as CREATE USER and GRANT. We restrict that account to local connections. ClickHouse web console

nano /etc/clickhouse-server/users.d/20-local-admin.xml
<clickhouse>
    <users>
        <default>
            <networks replace="replace">
                <ip>127.0.0.1</ip>
                <ip>::1</ip>
            </networks>
            <access_management>1</access_management>
        </default>
    </users>
</clickhouse>

Start ClickHouse and check its version. When prompted, enter the password you set for the default account during installation.

sudo systemctl enable clickhouse-server
sudo systemctl restart clickhouse-server
clickhouse-client --host 127.0.0.1 --ask-password --query 'SELECT version()'
If the installer did not ask for a password

On a fresh installation with an empty password, check the version without --ask-password. Then run the following block in Bash and enter a new password.

read -rsp 'New password for the ClickHouse default account: ' CH_PASSWORD
printf '\n'
printf '%s' "$CH_PASSWORD" | sha256sum | cut -d ' ' -f 1
unset CH_PASSWORD

Copy the printed hash into a temporary note. Then open the configuration file as root.

nano /etc/clickhouse-server/users.d/20-local-admin.xml

Replace the entire file with the XML below. Substitute the hash you copied for SHA256_HASH and save the file.

<clickhouse>
    <users>
        <default>
            <networks replace="replace">
                <ip>127.0.0.1</ip>
                <ip>::1</ip>
            </networks>
            <access_management>1</access_management>
            <password remove="1" />
            <password_sha256_hex>SHA256_HASH</password_sha256_hex>
        </default>
    </users>
</clickhouse>

Restart ClickHouse. Use your new password when creating the database and at later prompts for the default account.

sudo systemctl restart clickhouse-server

Create the database and Zabbix account

In Bash on the ClickHouse host, create the otel database and run the script that creates its metrics, traces, and logs tables. The -t 604800 option sets seven-day retention.

clickhouse-client --host 127.0.0.1 --ask-password \
  --query 'CREATE DATABASE IF NOT EXISTS otel'

read -rsp 'Password for the ClickHouse default account: ' CH_PASSWORD
echo
/usr/share/zabbix/sql-scripts/clickhouse/otel_all.sh \
  -s http://127.0.0.1:8123 -d otel -u default -p "$CH_PASSWORD" -t 604800
unset CH_PASSWORD

Open the ClickHouse console. Create otel_zabbix for proxy writes and server and frontend reads. Set a new password. The last command lists the tables, including otel_traces and otel_logs.

clickhouse-client --host 127.0.0.1 --ask-password
-- Enter your own password for Zabbix.
CREATE USER otel_zabbix IDENTIFIED WITH sha256_password BY 'REPLACE_OTEL_PASSWORD';
GRANT SELECT, INSERT ON otel.* TO otel_zabbix;
SHOW TABLES FROM otel;

Set up ClickHouse in the Zabbix frontend

Open Administration → Data source. Under APM, select Enable global data source. Saving this form sets the shared source for proxy writes, queries on Zabbix server, and APM views in the frontend. Zabbix stores it in its main database, and the server passes the setting to the proxy during synchronization. Set TelemetryProvider in their files only if that component needs its own connection.

Frontend fieldValue
Database typeClickHouse
URLhttp://clickhouse.example.com:8123 with your hostname or IP
AuthenticationUsername and password
Usernameotel_zabbix
PasswordThe password you set in ClickHouse
Databaseotel

Click Test, then click Update after the test succeeds. The test checks the frontend’s connection to ClickHouse and requires PHP cURL.

1 Enable the source · 2 Enter the connection details · 3 Click Test. Then save with Update.
1 Enable the source · 2 Enter the connection details · 3 Click Test. Then save with Update.

Install Zabbix proxy with APM

Move to the Zabbix proxy host. If it runs a different OS from ClickHouse, switch the OS selection above and return to the proxy installation. Add the Zabbix 8.0 repository.

sudo apt update
sudo apt install curl ca-certificates
curl -fLO https://repo.zabbix.com/zabbix/8.0/release/debian/pool/main/z/zabbix-release/zabbix-release_latest_8.0+debian13_all.deb
sudo dpkg -i zabbix-release_latest_8.0+debian13_all.deb
sudo apt update

On Debian 13, APM support is built into zabbix-proxy-sqlite3.

sudo apt install zabbix-proxy-sqlite3 nano

This package supports standard agent and SNMP monitoring. It also works as a regular proxy with APM off. Set StartAPMCollector=1 to add OpenTelemetry ingestion.

Start the proxy and enable APM collection

Open Administration → Proxies → Create proxy. Enter apm-proxy as the name and select Active. In Proxy address, enter the source IP from which the proxy connects to the server. Under APM, enable Data collection enabled and click Add.

1 Open the APM tab · 2 Enable data collection · 3 Click Add.
1 Open the APM tab · 2 Enable data collection · 3 Click Add.

Run the system administration commands in the shared steps as root on the host specified in each step.

To configure SQLite, edit only DBName in /etc/zabbix/zabbix_proxy.conf. Enter the full path to the .sqlite file. The proxy automatically creates the database file on its first start and manages it thereafter.

For a running proxy, keep its current DBName value. For a new proxy, use a persistent directory writable by the zabbix user. If you do not already have one, create /var/lib/zabbix with the command below.

install -d -o zabbix -g zabbix -m 0750 /var/lib/zabbix

On the proxy host, open the configuration in Nano. Edit these settings in place and keep the rest of the package configuration. Replace Server with your Zabbix server address. Hostname must match the proxy name in the frontend.

nano /etc/zabbix/zabbix_proxy.conf
# …
Server=zabbix.example.com
Hostname=apm-proxy
DBName=/var/lib/zabbix/zabbix_proxy.sqlite

StartAPMCollector=1
APMListenIP=0.0.0.0
# …

The default 0.0.0.0 lets the proxy accept OTLP/gRPC on all IPv4 interfaces on port 4317. A Collector on the same host connects to 127.0.0.1:4317. If the local Collector is the only sender, keep inbound 4317/TCP from other hosts blocked at the firewall.

Before starting the proxy, allow the zabbix-server service (10051/TCP) on the Zabbix server. Skip this block if you already enabled the service during server installation.

firewall-cmd --permanent --zone=public --add-service=zabbix-server

For direct OTLP/gRPC from another host, allow inbound 4317/TCP on the proxy host. The command opens the port for the entire public zone. This example assumes a private network for the senders.

firewall-cmd --permanent --zone=public --add-port=4317/tcp

After adding the rules, reload the firewall once on each host where you changed the rules.

firewall-cmd --reload

Validate the configuration and start the service. In Administration → Proxies, check that Last seen (age) updates for the proxy.

zabbix_proxy -T
systemctl enable zabbix-proxy
systemctl restart zabbix-proxy

On its first configuration sync, Zabbix proxy receives the APM collection setting and global data source from Zabbix server. With StartAPMCollector=1, it then starts the OTLP/gRPC receiver automatically on port 4317.

Connect an application over HTTP or HTTPS

An application using OTLP/HTTP sends data to the OpenTelemetry Collector. The Collector forwards it to Zabbix proxy over OTLP/gRPC. If your applications send telemetry directly to the proxy over gRPC, you can skip to the gRPC test.

Collector configuration for HTTP only

On the proxy host, save the configuration to /etc/otelcol-contrib/config.yaml first. The package’s default configuration uses port 4317, which Zabbix proxy uses in this guide.

mkdir -p /etc/otelcol-contrib
nano /etc/otelcol-contrib/config.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
processors:
  batch: {}
exporters:
  otlp_grpc/zabbix:
    endpoint: 127.0.0.1:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc/zabbix]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc/zabbix]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp_grpc/zabbix]

Receivers defines where the Collector accepts data. 0.0.0.0:4318 makes the receiver available on the proxy host’s network interfaces. Exporters defines where the Collector sends data. 127.0.0.1:4317 is the local Zabbix proxy gRPC receiver. The pipelines connect reception, batching, and export separately for traces, metrics, and logs.

Install and start the Collector

When switching from the earlier container setup, stop the old Collector first so the service can use its port.

Install the official otelcol-contrib package. It includes a systemd unit. The commands below use x86-64 packages. On ARM64, use the package with arm64 in the filename in place of amd64.

Alma/CentOS/RHEL/Rocky 10

dnf install https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.165.0/otelcol-contrib_0.165.0_linux_amd64.rpm

Debian 13. The --force-confold option preserves the configuration you created above.

curl -fLO https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.165.0/otelcol-contrib_0.165.0_linux_amd64.deb
dpkg --force-confold -i otelcol-contrib_0.165.0_linux_amd64.deb

Allow 4318/TCP for OTLP over HTTP on the Collector host.

firewall-cmd --permanent --zone=public --add-port=4318/tcp
firewall-cmd --reload

Start otelcol-contrib and enable it at boot.

systemctl enable --now otelcol-contrib

Configure the application to send telemetry

On the host running the application that sends telemetry, set these OpenTelemetry SDK variables in the application’s process environment. In OTEL_EXPORTER_OTLP_ENDPOINT, replace collector.example.com with the Collector host’s DNS name or IP address. In this guide, the Collector runs on the proxy host.

OTEL_SERVICE_NAME=checkout-demo
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector.example.com:4318

Collector configuration for HTTP and HTTPS

On the Collector host, open /etc/otelcol-contrib/config.yaml. The complete example below keeps HTTP on port 4318 and adds HTTPS on port 8443. Both receivers run at the same time and forward traces, metrics, and logs to the same Zabbix proxy.

nano /etc/otelcol-contrib/config.yaml
receivers:
  otlp/http:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
  otlp/https:
    protocols:
      http:
        endpoint: 0.0.0.0:8443
        tls:
          cert_file: /etc/otelcol-contrib/tls/collector.crt
          key_file: /etc/otelcol-contrib/tls/collector.key
processors:
  batch: {}
exporters:
  otlp_grpc/zabbix:
    endpoint: 127.0.0.1:4317
    tls:
      insecure: true
service:
  pipelines:
    traces:
      receivers: [otlp/http, otlp/https]
      processors: [batch]
      exporters: [otlp_grpc/zabbix]
    metrics:
      receivers: [otlp/http, otlp/https]
      processors: [batch]
      exporters: [otlp_grpc/zabbix]
    logs:
      receivers: [otlp/http, otlp/https]
      processors: [batch]
      exporters: [otlp_grpc/zabbix]

The service runs as the otelcol-contrib user. Put the certificate and key at the paths shown above and make both files readable by this user. You can use root:otelcol-contrib ownership and mode 0640 for the key.

For HTTPS only, remove the entire otlp/http receiver from the configuration and set receivers: [otlp/https] in all three pipelines.

Allow 8443/TCP on the Collector host.

firewall-cmd --permanent --zone=public --add-port=8443/tcp

When switching to HTTPS only, remove the firewall rule for 4318/TCP if it was added solely for this Collector.

After saving the configuration, reload the firewall and restart the Collector.

firewall-cmd --reload
systemctl restart otelcol-contrib

On the host running the application that sends telemetry over HTTPS, set OTEL_EXPORTER_OTLP_ENDPOINT to https://collector.example.com:8443. The protocol is http/protobuf for HTTPS too. The application must trust the Collector certificate.

Connect an application directly over gRPC

If the application sends OTLP/gRPC directly from another host, point it to the proxy’s private IP or DNS name. On the proxy, allow 4317/TCP only from the sender addresses and configure mutual TLS (mTLS). When using both routes, configure a trusted CA, client certificate, and key in the Collector exporter and use an address that matches the proxy certificate. TLS parameters for Zabbix APM

Test telemetry ingestion

  • OTLP/gRPC goes directly from the sender to Zabbix proxy on port 4317.
  • OTLP over HTTP or HTTPS goes from the sender to the OpenTelemetry Collector on port 4318 or 8443. The Collector forwards the trace to Zabbix proxy over gRPC.

Choose the test that matches your setup. For telemetry sent directly to the proxy over gRPC, run the first test. If you send telemetry through the Collector, use the HTTP or HTTPS test.

Send a test trace directly over gRPC

On the host that will send the test, install the otel-cli v0.4.5 package as root for the sender’s OS using the selector above. The commands are for Linux on x86-64. On ARM64, replace amd64 with arm64 in the filename.

Alma/CentOS/RHEL/Rocky 10

dnf install https://github.com/equinix-labs/otel-cli/releases/download/v0.4.5/otel-cli_0.4.5_linux_amd64.rpm

Debian 13

curl -fLO https://github.com/equinix-labs/otel-cli/releases/download/v0.4.5/otel-cli_0.4.5_linux_amd64.deb &&
  apt install ./otel-cli_0.4.5_linux_amd64.deb

Run without a system installation. You can extract the prebuilt binary into the directory where you will save the script. On ARM64, use linux_arm64 in the archive filename.

archive=otel-cli_0.4.5_linux_amd64.tar.gz
curl -fLO "https://github.com/equinix-labs/otel-cli/releases/download/v0.4.5/$archive" &&
  tar -xzf "$archive" otel-cli

Open a new file. If you use the extracted binary, save the script next to it.

nano test-apm-grpc.sh

Paste the following contents and save the file.

#!/usr/bin/env bash
set -euo pipefail

endpoint=${1:?Usage: ./test-apm-grpc.sh PROXY_ENDPOINT [otel-cli options]}
shift
unset TRACEPARENT

PATH="$(dirname -- "$0"):$PATH" otel-cli span \
  --protocol grpc --endpoint "$endpoint" --timeout 5s \
  --service apm-test-grpc --name test-request \
  --fail --verbose --tp-print "$@"
printf '\nService: apm-test-grpc\n'

Make the file executable.

chmod +x test-apm-grpc.sh

Local test without TLS. Run it on the Zabbix proxy host if its OTLP/gRPC receiver runs without TLS.

./test-apm-grpc.sh 127.0.0.1:4317 --insecure

Test with mutual TLS. Run this command on the sender. Replace proxy.example.com with the name on the proxy certificate. The ca.crt file verifies the proxy certificate. client.crt and client.key are the sender’s certificate and key. Adjust the file paths as needed.

./test-apm-grpc.sh proxy.example.com:4317 \
  --tls-ca-cert ca.crt --tls-client-cert client.crt --tls-client-key client.key

After a successful send, the script prints the trace ID and the service name. Use trace id to identify the trace in Zabbix.

# trace id: 7ba3eae5ce58b4a7b54e077e4e324029
#  span id: 1e205f37114e72b6
TRACEPARENT=00-7ba3eae5ce58b4a7b54e077e4e324029-1e205f37114e72b6-01

Service: apm-test-grpc

Send a test trace through the Collector

On the Linux host that will send telemetry, open a new file.

nano test-apm-http.sh

Paste the following contents and save the file.

#!/usr/bin/env bash
set -euo pipefail

collector_url=${1:?Usage: ./test-apm-http.sh COLLECTOR_URL}
now_ns=$(date +%s%N)
read -r trace_id < /proc/sys/kernel/random/uuid
trace_id=${trace_id//-/}

curl -fsS --max-time 10 \
  -H 'Content-Type: application/json' \
  --data-binary @- "${collector_url%/}/v1/traces" <<JSON
{
  "resourceSpans": [{
    "resource": {
      "attributes": [{"key": "service.name", "value": {"stringValue": "apm-test-http"}}]
    },
    "scopeSpans": [{"spans": [{
      "traceId": "$trace_id",
      "spanId": "0000000000000001",
      "name": "test-request",
      "kind": 2,
      "startTimeUnixNano": "$((now_ns - 10000000))",
      "endTimeUnixNano": "$now_ns"
    }]}]
  }]
}
JSON
printf '\nService: apm-test-http\nTrace ID: %s\n' "$trace_id"

Make the script executable and choose the command that matches your Collector configuration. In either example, replace collector.example.com with your Collector’s address.

chmod +x test-apm-http.sh

HTTP on port 4318

./test-apm-http.sh http://collector.example.com:4318

HTTPS on port 8443. Run this test if the Collector uses TLS and the sender trusts its certificate.

./test-apm-http.sh https://collector.example.com:8443

Example output after the Collector accepts the request. The Trace ID changes on every run.

{"partialSuccess":{}}
Service: apm-test-http
Trace ID: be034f426cd944f9afbb09e198313f93

Verify the first data in Zabbix

Open APM → Traces, select the last 15 minutes, and find apm-test-http for the test through the Collector or apm-test-grpc for the direct proxy test. Open the resulting trace and compare its Trace ID with the script output. To verify your own application, trigger a request and search for its OTEL_SERVICE_NAME value. If the app also exports metrics and logs, find them under APM → Metrics and APM → Logs. You can look up related logs by Trace ID when the app includes it.

① Copy the Trace ID to look up related logs.
① Copy the Trace ID to look up related logs.

If data is missing, check the proxy log. If you use the Collector, check its log too. For ClickHouse connection errors, verify the URL, account, and permissions in the data source.

tail -n 50 /var/log/zabbix/zabbix_proxy.log
journalctl -u otelcol-contrib -n 50 --no-pager

uxMAX for APM

For working with APM, we recommend the free uxMAX module. In Traces, Metrics and Logs, use Save as to keep your own filters and selected time range for later use. Save changes to an existing filter with Update. When configuring Telemetry query items, the timeline illustrates the read window, Granularity buckets and possible gaps or overlaps. Open the guided installer in the initMAX Portal, select your operating system and run the commands on the server hosting your Zabbix frontend.

uxMAX in APM Metrics – named filter tabs and the Save as button
uxMAX adds named filter tabs to APM. Save as stores your conditions and selected time range.
uxMAX Telemetry query timeline with Update interval, Time shift, Lookback limit and Granularity
The uxMAX timeline for a Telemetry query item connects Update interval, Time shift, Lookback limit and Granularity. This is an illustration. The actual period depends on the last successful query.

Optionally override the global data source in a configuration file

Use this section when a component needs its own database or account. A file setting overrides Administration → Data source only for the component where you set it. Add these blocks to the existing configuration files and retain the other settings.

Proxy for telemetry writes

On the Zabbix proxy host, open its configuration.

nano /etc/zabbix/zabbix_proxy.conf
# Set the URL, database, account and password for this component.
TelemetryProvider=clickhouse;url=http://clickhouse.example.com:8123,db=otel,username=otel_zabbix,password="REPLACE_OTEL_PASSWORD"
zabbix_proxy -T
systemctl restart zabbix-proxy

Server for telemetry queries

On the Zabbix server host, open its configuration.

nano /etc/zabbix/zabbix_server.conf
# Set the URL, database, account and password for this component.
TelemetryProvider=clickhouse;url=http://clickhouse.example.com:8123,db=otel,username=otel_zabbix,password="REPLACE_OTEL_PASSWORD"
zabbix_server -T
systemctl restart zabbix-server

Frontend for APM views

On the frontend host, add the full block to the existing PHP file, which also contains the connection to the main Zabbix database.

nano /etc/zabbix/web/zabbix.conf.php
// Set the ClickHouse connection for the frontend.
$TELEMETRY_PROVIDERS[] = [
    'provider' => 'clickhouse',
    'url' => 'http://clickhouse.example.com:8123',
    'db' => 'otel',
    'username' => 'otel_zabbix',
    'password' => 'REPLACE_OTEL_PASSWORD'
];
php -l /etc/zabbix/web/zabbix.conf.php

Reload APM to verify the change. The global data source Test button checks the values entered in its form. Telemetry data source documentation

Zabbix APM documentation · Official ClickHouse packages · Collector configuration

×Shopping Cart

Your cart is empty.