Zabbix/partitioning

Zabbix Database Partitioning Guide (Python based)

This guide describes how to set up and manage database partitioning for Zabbix using the zabbix_partitioning.py script.

Overview

The script manages MySQL table partitions based on time (Range Partitioning on the clock column). It automatically:

1. Creates future partitions to ensure new data can be written.
2. Drops old partitions based on configured retention periods.

Benefits:

• Performance: Faster cleanup of old data (dropping a partition is instantaneous compared to Zabbix internal housekeeping).
• Recommended: For database bigger than 100GB.
• Must have!: For database bigger than 500G.

Warning

Support for MySQL/MariaDB only. Always BACKUP your database before initializing partitioning!

---

1. Prerequisites

• Python 3.6+
• Python Libraries: pymysql, pyyaml

  # Debian/Ubuntu
  sudo apt install python3-pymysql python3-yaml
  # RHEL/AlmaLinux/Rocky
  sudo dnf install python3-pymysql python3-pyyaml
  # Or via pip
  pip3 install pymysql pyyaml
  

• Database Permissions: The user configured in the script needs:
  • SELECT, INSERT, CREATE, DROP, ALTER on the Zabbix database.
  • SUPER or SESSION_VARIABLES_ADMIN privilege (required to disable binary logging via SET SESSION sql_log_bin=0 if replicate_sql: False).

---

2. Installation

1. Copy the script and config to a precise location (e.g., /usr/local/bin or specialized directory).

   mkdir -p /opt/zabbix_partitioning
   cp zabbix_partitioning.py /opt/zabbix_partitioning/
   cp zabbix_partitioning.conf /etc/zabbix/
   chmod +x /opt/zabbix_partitioning/zabbix_partitioning.py
   

---

3. Configuration

Edit /etc/zabbix/zabbix_partitioning.conf:

database:
    host: localhost
    user: zbx_part
    passwd: YOUR_PASSWORD
    db: zabbix
    # port: 3306  # Optional, default is 3306

partitions:
    daily:
        - history: 14d
        - history_uint: 14d
        - trends: 365d
        # ... add other options as needed. Please check the config file for more options.

Important Notes:

• replicate_sql:
  • False (Default): Partitioning maintenance commands are NOT replicated to slaves. Recommended if you manage partitions separately on each node or want to reduce replication lag.
  • True: Commands are replicated.
• auditlog:
  • In Zabbix 7.0+, the auditlog table does not have the clock column in its Primary Key by default. Do not add it to the config unless you have manually altered the table schema.

---

4. Zabbix Preparation (CRITICAL)

Before partitioning, you must disable Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts.

1. Log in to Zabbix Web Interface.
2. Go to Administration -> General -> Housekeeping.
3. Uncheck the following (depending on what you partition):
   • Enable internal housekeeping for History
   • Enable internal housekeeping for Trends
4. Click Update.

---

5. Initialization

This step converts existing standard tables into partitioned tables.

1. Dry Run (Verify what will happen):

   /opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run
   

   Check the output for any errors.

2. Execute Initialization:

   /opt/zabbix_partitioning/zabbix_partitioning.py --init
   

   This may take time depending on table size.

---

6. Automation (Cron Job)

Set up a daily cron job to create new partitions and remove old ones.

1. Open crontab:

   crontab -e
   

2. Add the line (run daily at 00:30):

   30 0 * * * /usr/bin/python3 /opt/zabbix_partitioning/zabbix_partitioning.py -c /etc/zabbix/zabbix_partitioning.conf >> /var/log/zabbix_partitioning.log 2>&1
   

---

7. Automation (Systemd Timer) — Recommended

Alternatively, use systemd timers for more robust scheduling and logging.

1. Create Service Unit (/etc/systemd/system/zabbix-partitioning.service):

   [Unit]
   Description=Zabbix Database Partitioning Service
   After=network.target mysql.service
   
   [Service]
   Type=oneshot
   User=root
   ExecStart=/usr/bin/python3 /opt/zabbix_partitioning/zabbix_partitioning.py -c /etc/zabbix/zabbix_partitioning.conf
   

2. Create Timer Unit (/etc/systemd/system/zabbix-partitioning.timer):

   [Unit]
   Description=Run Zabbix Partitioning Daily
   
   [Timer]
   OnCalendar=*-*-* 00:30:00
   Persistent=true
   
   [Install]
   WantedBy=timers.target
   

3. Enable and Start:

   systemctl daemon-reload
   systemctl enable --now zabbix-partitioning.timer
   

4. View Logs:

   journalctl -u zabbix-partitioning.service
   

---

8. Troubleshooting

• Connection Refused: Check host, port in config. Ensure MySQL is running.
• Access Denied (1227): The DB user needs SUPER privileges to disable binary logging (replicate_sql: False). Either grant the privilege or set replicate_sql: True (if replication load is acceptable).
• Primary Key Error: "Primary Key does not include 'clock'". The table cannot be partitioned by range on clock without schema changes. Remove it from config.