feat: introduce configurable future partition buffer and add monitoring for future partitions.
This commit is contained in:
@@ -1,22 +1,23 @@
|
|||||||
# PostgreSQL Partitioning for Zabbix
|
# PostgreSQL Partitioning for Zabbix
|
||||||
|
|
||||||
This document describes the declarative partitioning implementation for Zabbix `history`, `trends`, and `auditlog` tables on PostgreSQL. This solution replaces standard Zabbix housekeeping for the configured tables.
|
This is the declarative (PostgreSQL procedures based) partitioning implementation for Zabbix `history`, `trends`, and `auditlog` tables on PostgreSQL. This solution is intended to replace standard Zabbix housekeeping for the configured tables. Partitioning is very useful for large environments because it completely eliminates the housekeeper from the process. Instead of huge DELETE queries on several million rows, fast DDL queries (ALTER TABLE) are executed, which drop an entire partition.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
The solution uses PostgreSQL native declarative partitioning (`PARTITION BY RANGE`).
|
The solution uses PostgreSQL native declarative partitioning (`PARTITION BY RANGE`).
|
||||||
All procedures and configuration are stored in the `partitions` schema to maintain separation from Zabbix schema.
|
All procedures, information, statistics and configuration are stored in the `partitions` schema to maintain full separation from Zabbix schema.
|
||||||
|
|
||||||
### Components
|
### Components
|
||||||
1. **Configuration Table**: `partitions.config` defines retention policies.
|
1. **Configuration Table**: `partitions.config` defines retention policies.
|
||||||
2. **Maintenance Procedure**: `partitions.run_maintenance()` manages partition lifecycle.
|
2. **Maintenance Procedure**: `partitions.run_maintenance()` manages partition lifecycle.
|
||||||
3. **Monitoring View**: `partitions.monitoring` provides system state visibility.
|
3. **Monitoring View**: `partitions.monitoring` provides system state visibility.
|
||||||
|
4. **Version Table**: `partitions.version` provides information about installed version of the partitioning solution.
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
The installation is performed by executing the SQL procedures in the following order:
|
The installation is performed by executing the SQL procedures in the following order:
|
||||||
1. Initialize schema (`00_partitions_init.sql`).
|
1. Initialize schema (`00_partitions_init.sql`).
|
||||||
2. Prepare tables (e.g., `auditlog` PK adjustment) (`01_auditlog_prep.sql`).
|
2. Auditlog PK adjustment (`01_auditlog_prep.sql`).
|
||||||
3. Install maintenance procedures (`02_maintenance.sql`).
|
3. Install maintenance procedures (`02_maintenance.sql`).
|
||||||
4. Enable partitioning on tables (`03_enable_partitioning.sql`).
|
4. Enable partitioning on tables (`03_enable_partitioning.sql`).
|
||||||
5. Install monitoring views (`04_monitoring_view.sql`).
|
5. Install monitoring views (`04_monitoring_view.sql`).
|
||||||
@@ -30,6 +31,7 @@ Partitioning policies are defined in the `partitions.config` table.
|
|||||||
| `table_name` | text | Name of the Zabbix table (e.g., `history`, `trends`). |
|
| `table_name` | text | Name of the Zabbix table (e.g., `history`, `trends`). |
|
||||||
| `period` | text | Partition interval: `day`, `week`, or `month`. |
|
| `period` | text | Partition interval: `day`, `week`, or `month`. |
|
||||||
| `keep_history` | interval | Data retention period (e.g., `30 days`, `12 months`). |
|
| `keep_history` | interval | Data retention period (e.g., `30 days`, `12 months`). |
|
||||||
|
| `future_partitions` | integer | Number of future partitions to pre-create (buffer). Default: `5`. |
|
||||||
| `last_updated` | timestamp | Timestamp of the last successful maintenance run. |
|
| `last_updated` | timestamp | Timestamp of the last successful maintenance run. |
|
||||||
|
|
||||||
### Modifying Retention
|
### Modifying Retention
|
||||||
@@ -44,7 +46,7 @@ WHERE table_name = 'history';
|
|||||||
## Maintenance
|
## Maintenance
|
||||||
|
|
||||||
The maintenance procedure `partitions.run_maintenance()` is responsible for:
|
The maintenance procedure `partitions.run_maintenance()` is responsible for:
|
||||||
1. Creating future partitions (current period + 3 future buffers).
|
1. Creating future partitions (current period + `future_partitions` buffer).
|
||||||
2. Creating past partitions (backward coverage based on `keep_history`).
|
2. Creating past partitions (backward coverage based on `keep_history`).
|
||||||
3. Dropping partitions older than `keep_history`.
|
3. Dropping partitions older than `keep_history`.
|
||||||
|
|
||||||
@@ -56,7 +58,7 @@ CALL partitions.run_maintenance();
|
|||||||
|
|
||||||
## Monitoring & Permissions
|
## Monitoring & Permissions
|
||||||
|
|
||||||
System state can be monitored via the `partitions.monitoring` view.
|
System state can be monitored via the `partitions.monitoring` view. It includes a `future_partitions` column which counts how many partitions exist *after* the current period. This is useful for alerting (e.g., trigger if `future_partitions < 2`).
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
SELECT * FROM partitions.monitoring;
|
SELECT * FROM partitions.monitoring;
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ CREATE TABLE IF NOT EXISTS partitions.config (
|
|||||||
table_name text NOT NULL,
|
table_name text NOT NULL,
|
||||||
period text NOT NULL CHECK (period IN ('day', 'week', 'month', 'year')),
|
period text NOT NULL CHECK (period IN ('day', 'week', 'month', 'year')),
|
||||||
keep_history interval NOT NULL,
|
keep_history interval NOT NULL,
|
||||||
|
future_partitions integer NOT NULL DEFAULT 5,
|
||||||
last_updated timestamp WITH TIME ZONE DEFAULT now(),
|
last_updated timestamp WITH TIME ZONE DEFAULT now(),
|
||||||
PRIMARY KEY (table_name)
|
PRIMARY KEY (table_name)
|
||||||
);
|
);
|
||||||
|
|||||||
@@ -105,7 +105,8 @@ $$;
|
|||||||
CREATE OR REPLACE PROCEDURE partitions.maintain_table(
|
CREATE OR REPLACE PROCEDURE partitions.maintain_table(
|
||||||
p_table_name text,
|
p_table_name text,
|
||||||
p_period text,
|
p_period text,
|
||||||
p_keep_history interval
|
p_keep_history interval,
|
||||||
|
p_future_partitions integer DEFAULT 5
|
||||||
) LANGUAGE plpgsql AS $$
|
) LANGUAGE plpgsql AS $$
|
||||||
DECLARE
|
DECLARE
|
||||||
v_start_time timestamp with time zone;
|
v_start_time timestamp with time zone;
|
||||||
@@ -139,8 +140,8 @@ BEGIN
|
|||||||
RETURN;
|
RETURN;
|
||||||
END IF;
|
END IF;
|
||||||
|
|
||||||
-- 1. Create Future Partitions (Current + 3 ahead)
|
-- 1. Create Future Partitions (Current + Buffer)
|
||||||
FOR i IN 0..3 LOOP
|
FOR i IN 0..p_future_partitions LOOP
|
||||||
CALL partitions.create_partition(
|
CALL partitions.create_partition(
|
||||||
p_table_name,
|
p_table_name,
|
||||||
v_start_time + (i * v_period_interval),
|
v_start_time + (i * v_period_interval),
|
||||||
@@ -176,7 +177,7 @@ DECLARE
|
|||||||
v_row record;
|
v_row record;
|
||||||
BEGIN
|
BEGIN
|
||||||
FOR v_row IN SELECT * FROM partitions.config LOOP
|
FOR v_row IN SELECT * FROM partitions.config LOOP
|
||||||
CALL partitions.maintain_table(v_row.table_name, v_row.period, v_row.keep_history);
|
CALL partitions.maintain_table(v_row.table_name, v_row.period, v_row.keep_history, v_row.future_partitions);
|
||||||
END LOOP;
|
END LOOP;
|
||||||
END;
|
END;
|
||||||
$$;
|
$$;
|
||||||
|
|||||||
@@ -27,9 +27,9 @@ BEGIN
|
|||||||
|
|
||||||
-- 3. Create initial partitions
|
-- 3. Create initial partitions
|
||||||
RAISE NOTICE 'Creating initial partitions for %...', v_table;
|
RAISE NOTICE 'Creating initial partitions for %...', v_table;
|
||||||
CALL partitions.maintain_table(v_table, v_row.period, v_row.keep_history);
|
CALL partitions.maintain_table(v_table, v_row.period, v_row.keep_history, v_row.future_partitions);
|
||||||
|
|
||||||
-- 4. (Optional) Copy data?
|
-- Optional: Migrate existing data
|
||||||
-- EXECUTE format('INSERT INTO public.%I SELECT * FROM public.%I', v_table, v_old_table);
|
-- EXECUTE format('INSERT INTO public.%I SELECT * FROM public.%I', v_table, v_old_table);
|
||||||
|
|
||||||
ELSIF EXISTS (SELECT 1 FROM pg_class WHERE relname = v_table AND relkind = 'p') THEN
|
ELSIF EXISTS (SELECT 1 FROM pg_class WHERE relname = v_table AND relkind = 'p') THEN
|
||||||
|
|||||||
@@ -10,6 +10,12 @@ SELECT
|
|||||||
c.period,
|
c.period,
|
||||||
c.keep_history,
|
c.keep_history,
|
||||||
count(child.relname) AS partition_count,
|
count(child.relname) AS partition_count,
|
||||||
|
count(child.relname) FILTER (
|
||||||
|
WHERE
|
||||||
|
(c.period = 'day' AND child.relname > (parent.relname || '_p' || to_char(now(), 'YYYYMMDD')))
|
||||||
|
OR
|
||||||
|
(c.period = 'month' AND child.relname > (parent.relname || '_p' || to_char(now(), 'YYYYMM')))
|
||||||
|
) AS future_partitions,
|
||||||
pg_size_pretty(sum(pg_total_relation_size(child.oid))) AS total_size,
|
pg_size_pretty(sum(pg_total_relation_size(child.oid))) AS total_size,
|
||||||
min(child.relname) AS oldest_partition,
|
min(child.relname) AS oldest_partition,
|
||||||
max(child.relname) AS newest_partition,
|
max(child.relname) AS newest_partition,
|
||||||
|
|||||||
Reference in New Issue
Block a user