Zabbix/partitioning
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: move pg_cron job management instructions to generic maintenance section

Browse Source
This commit is contained in:
Maksym Buz
2026-03-30 20:46:46 +00:00
parent 505933e880
commit 32a587172e
4 changed files with 113 additions and 146 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
postgresql/procedures/00_schema_create.sql
View File

@@ -22,7 +22,6 @@ CREATE TABLE IF NOT EXISTS partitions.version (
description text
);
-- Set initial version
INSERT INTO partitions.version (version, description) VALUES ('7-1', 'Zabbix 7.4 and 7.0 compatible version')
ON CONFLICT (version) DO NOTHING;

99
postgresql/procedures/README.md
View File

@@ -1,17 +1,15 @@
# PostgreSQL Partitioning for Zabbix
This is the declarative (PostgreSQL procedures based) partitioning implementation for Zabbix `history`, `trends`, and `auditlog` tables on PostgreSQL. This solution is intended to replace standard Zabbix housekeeping for the configured tables. Partitioning is very useful for large environments because it completely eliminates the housekeeper from the process. Instead of huge DELETE queries on several million rows, fast DDL queries (ALTER TABLE) are executed, which drop an entire partition.
This is the declarative partitioning implementation for Zabbix `history*`, `trends*`, and `auditlog` tables on PostgreSQL. This solution is intended to replace standard Zabbix housekeeping for the configured tables. Partitioning is very useful for large environments because it completely eliminates the housekeeper from the process. Instead of huge DELETE queries on several million rows, fast DDL queries (ALTER TABLE) are executed, which drop an entire partition.
> [!WARNING]
> **High-Load Environments**:
> 1. **Data Visibility**: After enabling partitioning, old data remains in `*_old` tables and is **NOT visible** in Zabbix. You must migrate data manually if needed.
> 2. **Disable Housekeeping**: You **MUST** disable Zabbix Housekeeper for History and Trends in *Administration -> Housekeeping*. Failure to do so will cause massive `DELETE` loads.
> 2. **Disable Housekeeping**: You **MUST** disable Zabbix Housekeeper for History and Trends in *Administration -> Housekeeping*.
## Table of Contents
- [Architecture](#architecture)
- [Components](#components)
- [Prerequisites: Database & User Creation](#prerequisites-database--user-creation)
- [Installation](#installation)
- [Configuration](#configuration)
- [Modifying Retention](#modifying-retention)
@@ -24,7 +22,6 @@ This is the declarative (PostgreSQL procedures based) partitioning implementatio
- [`auditlog` Table](#auditlog-table)
- [Converting Existing Tables](#converting-existing-tables)
- [Upgrades](#upgrades)
- [Appendix: Zabbix Server & Frontend RDS Configuration](#appendix-zabbix-server--frontend-rds-configuration)
## Architecture
@@ -37,21 +34,6 @@ All procedures, information, statistics and configuration are stored in the `par
3. **Monitoring View**: `partitions.monitoring` provides system state visibility.
4. **Version Table**: `partitions.version` provides information about installed version of the partitioning solution.
## Prerequisites: Database & User Creation
If you are deploying Zabbix on a fresh database instance (like AWS RDS) rather than a local server, you must first create the `zabbix` user and database using your administrator account (e.g., `postgres`).
1. Connect to your DB instance as the administrator:
```bash
psql "host=YOUR_RDS_HOST port=5432 user=postgres dbname=postgres sslmode=require"
```
2. Create the user and database:
```sql
CREATE USER zabbix WITH PASSWORD 'your_secure_password';
-- On Cloud DBs like RDS, the master user must inherit the new role to grant ownership
GRANT zabbix TO postgres;
CREATE DATABASE zabbix OWNER zabbix;
```
## Installation
The installation is performed by executing the SQL procedures in the following order:
@@ -66,9 +48,9 @@ You can deploy these scripts manually against your Zabbix database using `psql`.
```bash
# Connect as the zabbix database user
export PGPASSWORD="your_zabbix_password"
DB_HOST="localhost" # Or your RDS endpoint
DB_HOST="localhost" # Or your DB endpoint
DB_NAME="zabbix"
DB_USER="zabbix"
DB_USER="zbxpart_admin"
for script in 00_schema_create.sql 01_maintenance.sql 02_enable_partitioning.sql 03_monitoring_view.sql; do
echo "Applying $script..."
@@ -113,25 +95,21 @@ CALL partitions.run_maintenance();
To ensure partitions are created in advance and old data is cleaned up, the maintenance procedure should be scheduled to run automatically.
It is recommended to run the maintenance **twice a day** (e.g., at 05:30 and 23:30).
It is recommended to run the maintenance **twice a day** and not in round hours because of the way housekeeper works (e.g., at 05:30 and 23:30).
* **Primary Run**: Creates new future partitions and drops old ones.
* **Secondary Run**: Acts as a safety check. Since the procedure is idempotent (safe to run multiple times), a second run ensures everything is consistent if the first run failed or was interrupted.
You can schedule this using one of the following methods:
#### Option 1: `pg_cron` (Recommended)
`pg_cron` is a cron-based job scheduler that runs directly inside the database as an extension.
> [!NOTE]
> **Cloud Managed Databases (AWS RDS, Aurora, Azure, GCP):**
> Managed databases generally have `pg_cron` pre-installed and handle the authentication/connections securely for you automatically. You do **not** need to install OS packages or configure a `.pgpass` file! Simply modify your RDS Parameter Group to include `shared_preload_libraries = 'pg_cron'` and `cron.database_name = 'zabbix'`, reboot the instance, and execute `CREATE EXTENSION pg_cron;`.
`pg_cron` is a cron-based job scheduler that runs directly inside the database as an extension. It is very useful for cloud based databases like AWS RDS, Aurora, Azure, GCP, because it handles the authentication/connections securely for you automatically and its available as a managed extension. You do **not** need to install OS packages or configure anything. Simply modify the RDS Parameter Group to include `shared_preload_libraries = 'pg_cron'` and `cron.database_name = 'zabbix'`, reboot the instance, and execute `CREATE EXTENSION pg_cron;`.
**Setup `pg_cron` (Self-Hosted):**
1. Install the package via your OS package manager (e.g., `postgresql-15-cron` on Debian/Ubuntu, or `pg_cron_15` on RHEL/CentOS).
2. Configure it modifying `postgresql.conf`:
```ini
shared_preload_libraries = 'pg_cron'
cron.database_name = 'zabbix' # Define the database where pg_cron will run
cron.database_name = 'zabbix'
```
3. Restart PostgreSQL:
```bash
@@ -145,10 +123,6 @@ You can schedule this using one of the following methods:
```sql
SELECT cron.schedule('zabbix_partition_maintenance', '30 5,23 * * *', 'CALL partitions.run_maintenance();');
```
6. **Manage your `pg_cron` jobs** (run as superuser):
- To **list all active schedules**: `SELECT * FROM cron.job;`
- To **view execution logs/history**: `SELECT * FROM cron.job_run_details;`
- To **remove/unschedule** the job: `SELECT cron.unschedule('zabbix_partition_maintenance');`
**⚠️ Troubleshooting `pg_cron` Connection Errors:**
If your cron jobs fail to execute and you see `FATAL: password authentication failed` in your PostgreSQL logs, it is because `pg_cron` attempts to connect via TCP (`localhost`) by default, which usually requires a password.
@@ -224,9 +198,17 @@ If running in Docker, you can execute it via the host's cron by targeting the co
```bash
30 5,23 * * * docker exec zabbix-db-test psql -U zabbix -d zabbix -c "CALL partitions.run_maintenance();"
```
### Managing `pg_cron` Jobs
If you are using `pg_cron` for scheduling, you can verify and manage your jobs (run as superuser):
- To **list all active schedules**: `SELECT * FROM cron.job;`
- To **view execution logs/history**: `SELECT * FROM cron.job_run_details;`
- To **remove/unschedule** the job: `SELECT cron.unschedule('zabbix_partition_maintenance');`
## Monitoring & Permissions
System state can be monitored via the `partitions.monitoring` view. It includes a `future_partitions` column which counts how many partitions exist *after* the current period. This is useful for alerting (e.g., trigger if `future_partitions < 2`).
System state can be monitored via the `partitions.monitoring` view. It includes the information about number of future partitions and the time since the last maintenance run. Plus it includes the total size of the partitioned table in bytes.
```sql
SELECT * FROM partitions.monitoring;
@@ -238,17 +220,17 @@ To check the installed version of the partitioning solution:
SELECT * FROM partitions.version ORDER BY installed_at DESC LIMIT 1;
```
### Least Privilege Access (`zbx_monitor`)
### Least Privilege Access (`zbxpart_admin`)
For monitoring purposes, it is recommended to create a dedicated user with read-only access to the monitoring view.
```sql
CREATE USER zbx_monitor WITH PASSWORD 'secure_password';
GRANT USAGE ON SCHEMA partitions TO zbx_monitor;
GRANT SELECT ON partitions.monitoring TO zbx_monitor;
CREATE USER zbxpart_admin WITH PASSWORD 'secure_password';
GRANT USAGE ON SCHEMA partitions TO zbxpart_admin;
GRANT SELECT ON partitions.monitoring TO zbxpart_admin;
```
> [!NOTE]
> If you ever apply updates to `03_monitoring_view.sql`, you should run the script as the `zabbix` database user (the original creator of the view). The script drops and recreates the view, so running it as `zabbix` ensures the view retains its ownership and preserves existing `GRANT` permissions for read-only users.
> If you ever apply updates to `03_monitoring_view.sql`, you should run the script as the `zbxpart_admin` database user (the original creator of the view). The script drops and recreates the view, so running it as `zbxpart_admin` ensures the view retains its ownership and preserves existing `GRANT` permissions for read-only users.
## Implementation Details
@@ -267,41 +249,4 @@ The enablement script guarantees practically zero downtime by automatically rena
When upgrading Zabbix:
1. **Backup**: Ensure a full database backup exists.
2. **Compatibility**: Zabbix upgrade scripts may attempt to `ALTER` tables. PostgreSQL supports `ALTER TABLE` on partitioned tables for adding columns, which propagates to partitions.
3. **Failure Scenarios**: If an upgrade script fails due to partitioning, the table may need to be temporarily reverted or the partition structure manually adjusted.
---
## Appendix: Zabbix Server & Frontend RDS Configuration
If you are running Zabbix against an external Cloud database (like AWS RDS) via SSL (`verify-full`), you must explicitly configure both the Zabbix Server daemon and the Web Frontend to enforce SSL and locate the downloaded Root CA Certificate.
**Prerequisite:** Download your cloud provider's root certificate (e.g., `global-bundle.pem`) and place it in a secure location on your Zabbix Server (e.g., `/etc/zabbix/global-bundle.pem`).
### 1. Zabbix Server (`/etc/zabbix/zabbix_server.conf`)
Ensure the following database lines are active:
```ini
DBHost=YOUR_RDS_ENDPOINT.amazonaws.com
DBPort=5432
DBName=zabbix
DBUser=zabbix
DBPassword=your_secure_password
DBTLSConnect=verify_full
DBTLSCAFile=/etc/zabbix/global-bundle.pem
```
### 2. Zabbix Frontend PHP (`/etc/zabbix/web/zabbix.conf.php`)
If you used the Web Setup Wizard, it might not configure the Root CA File correctly. Update your config array to enforce encryption and verify the host certificate:
```php
$DB['TYPE'] = 'POSTGRESQL';
$DB['SERVER'] = 'YOUR_RDS_ENDPOINT.amazonaws.com';
$DB['PORT'] = '5432';
$DB['DATABASE'] = 'zabbix';
$DB['USER'] = 'zabbix';
$DB['PASSWORD'] = 'your_secure_password';
$DB['SCHEMA'] = '';
$DB['ENCRYPTION'] = true;
$DB['VERIFY_HOST'] = true;
$DB['CA_FILE'] = '/etc/zabbix/global-bundle.pem';
```
3. **Failure Scenarios**: If an upgrade script fails due to partitioning, the table may need to be temporarily reverted or the partition structure manually adjusted.