Compare commits

...

4 Commits

4 changed files with 328 additions and 152 deletions

View File

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.6.0] - 2025-12-20
### Added
- **SSL**: Added support for simplified SSL configuration (`ssl: required`) for Managed/Cloud databases and updated Wizard/Config.
- **Safety**: Added automatic cleanup of `housekeeper` table during init (targeted) and conflict detection during maintenance.
- **CLI**: Added short flags for all arguments (`-vv`, `-f`, `-d`, `-s`, `-w`).
- **Validation**: Enforced strict unit validation for retention periods and strict table existence checks.
- **Error Handling**: Improved error reporting used to identify specific tables causing configuration errors.
### Changed
- **CLI**: Refactored `--init` and `--fast-init` into mutually exclusive standalone commands (Breaking Change).
- **CLI**: Reorganized argument parsing logic for better maintainability.
- **Safety**: Updated initialization warnings to explicitly mention duration ("SEVERAL HOURS") and disk space requirements.
- **Logic**: Refined housekeeper conflict check to only warn about tables actively configured for partitioning.
## [0.5.0] - 2025-12-16
### Added
- **Wizard**: Added interactive configuration wizard (`--wizard`).
- **CLI**: Added `--fast-init` flag to skip table scanning during initialization, instead of config option.
- **Monitoring**: Added `--stats` argument to output JSON statistics (Size, Count, Days Left).
- **Template**: Updated `zabbix_mysql_partitioning_template.yaml` to use Master/Dependent items for efficient stats collection.
### Changed
- **Config**: Removed `initial_partitioning_start` from configuration file (moved to `--fast-init`).
- **Template**: Replaced legacy check item with comprehensive stats master item.
- **CLI**: Removed legacy `--check-days` argument (replaced by `--stats`).
## [0.4.1] - 2025-12-16 ## [0.4.1] - 2025-12-16
### Added ### Added
- **CLI**: Added `--verbose` / `-v` flag to switch between INFO (default) and DEBUG logging levels. - **CLI**: Added `--verbose` / `-v` flag to switch between INFO (default) and DEBUG logging levels.

View File

@@ -47,8 +47,16 @@ The script manages MySQL table partitions based on time (Range Partitioning on t
--- ---
## 3. Configuration ## 3. Configuration
Edit `/etc/zabbix/zabbix_partitioning.conf`: The preferred way to create the configuration file is using the **Interactive Wizard**:
```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --wizard
```
Follow the on-screen prompts to configure your database connection, retention periods, and other settings. The wizard will generate the YAML configuration file for you.
### Automatic Config Structure (Reference)
If you prefer to edit manually `/etc/zabbix/zabbix_partitioning.conf`:
```yaml ```yaml
database: database:
host: localhost host: localhost
@@ -56,6 +64,7 @@ database:
passwd: YOUR_PASSWORD passwd: YOUR_PASSWORD
db: zabbix db: zabbix
# port: 3306 # Optional, default is 3306 # port: 3306 # Optional, default is 3306
# socket: /var/run/mysqld/mysqld.sock # Overrides host if present
partitions: partitions:
daily: daily:
@@ -74,9 +83,7 @@ partitions:
- **`yearly`**: Partitions are created for each year. - **`yearly`**: Partitions are created for each year.
- Retention Format: `14d` (days), `12w` (weeks), `12m` (months), `1y` (years). - Retention Format: `14d` (days), `12w` (weeks), `12m` (months), `1y` (years).
- **`initial_partitioning_start`**: Controls how the very FIRST partition is determined during initialization (`--init` mode). - **`replicate_sql`**: Controls MySQL Binary Logging for partitioning commands.
- `db_min`: (Default) Queries the table for the oldest record (`MIN(clock)`). Accurate but **slow** on large tables.
- `retention`: (Recommended for large DBs) Skips the query. Calculates the start date as `Now - Retention Period`. Creates a single `p_archive` partition for all data older than that date.
- **`premake`**: Number of future partitions to create in advance. - **`premake`**: Number of future partitions to create in advance.
- Default: `10`. Ensures you have a buffer if the script fails to run for a few days. - Default: `10`. Ensures you have a buffer if the script fails to run for a few days.
@@ -90,7 +97,25 @@ partitions:
--- ---
## 4. Zabbix Preparation (CRITICAL) ---
## 4. Runtime Parameters
| Argument | Description |
|---|---|
| `-c`, `--config FILE` | Path to configuration file (Default: `/etc/zabbix/zabbix_partitioning.conf`) |
| `-i`, `--init` | Initialize partitions (Standard Mode: Scan Table). |
| `-f`, `--fast-init` | Initialize partitions (Fast Mode: Skip Scan, use Retention). |
| `--wizard` | Launch interactive configuration wizard. |
| `-r`, `--dry-run` | Simulate queries without executing. Logs expected actions (Safe mode). |
| `-v`, `--verbose` | Enable debug logging (DEBUG level). |
| `--stats TABLE` | Output JSON statistics (Size, Count, Days Left) for a specific table. |
| `--discovery` | Output Zabbix Low-Level Discovery (LLD) JSON. |
| `-V`, `--version` | Show script version. |
---
## 5. Zabbix Preparation (CRITICAL)
Before partitioning, you **must disable** Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts. Before partitioning, you **must disable** Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts.
1. Log in to Zabbix Web Interface. 1. Log in to Zabbix Web Interface.
@@ -102,38 +127,52 @@ Before partitioning, you **must disable** Zabbix's internal housekeeping for the
--- ---
## 5. Initialization ## 6. Initialization
This step converts existing standard tables into partitioned tables. This step converts existing standard tables into partitioned tables.
1. **Dry Run** (Verify what will happen): 1. **Dry Run** (Verify what will happen):
```bash ```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run /opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run
``` ```
*Check the output for any errors.*
2. **Execute Initialization**: 2. **Execute Initialization**:
> [!WARNING]
> For large databases, standard initialization may take **SEVERAL HOURS** and require significant disk space (approx 2x table size).
There are two strategies for initialization:
**A. Standard Initialization (`--init`)**:
Scans the database to find the oldest record (`MIN(clock)`) and creates partitions from that point forward.
*Best for smaller databases or when you need to retain ALL existing data.*
```bash ```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --init /opt/zabbix_partitioning/zabbix_partitioning.py --init
``` ```
*This may take time depending on table size.*
**B. Fast Initialization (`--fast-init`)**:
Skips the slow table scan. It calculates the start date based on your configured **Retention Period** (e.g., `Now - 365d`).
It creates a single catch-all `p_archive` partition for all data older than the retention start date, then creates granular partitions forward.
*Recommended for large databases to avoid long table locks/scans.*
```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --fast-init
```
--- ---
## 6. Automation (Cron Job) ## 7. Automation (Cron Job)
Set up a daily cron job to create new partitions and remove old ones. Set up a cron job to create new partitions and remove old ones.
1. Open crontab: 1. Open crontab:
```bash ```bash
crontab -e crontab -e
``` ```
2. Add the line (run daily at 00:30): 2. Add the line (run twice a day at 00:10 and 04:10):
```cron ```cron
30 0 * * * /usr/bin/python3 /opt/zabbix_partitioning/zabbix_partitioning.py -c /etc/zabbix/zabbix_partitioning.conf >> /var/log/zabbix_partitioning.log 2>&1 10 0,4 * * * /usr/bin/python3 /opt/zabbix_partitioning/zabbix_partitioning.py -c /etc/zabbix/zabbix_partitioning.conf >> /var/log/zabbix_partitioning.log 2>&1
``` ```
--- ---
## 7. Automation (Systemd Timer) — Recommended ## 8. Automation (Systemd Timer) — Recommended
Alternatively, use systemd timers for more robust scheduling and logging. Alternatively, use systemd timers for more robust scheduling and logging.
1. **Create Service Unit** (`/etc/systemd/system/zabbix-partitioning.service`): 1. **Create Service Unit** (`/etc/systemd/system/zabbix-partitioning.service`):
@@ -151,10 +190,10 @@ Alternatively, use systemd timers for more robust scheduling and logging.
2. **Create Timer Unit** (`/etc/systemd/system/zabbix-partitioning.timer`): 2. **Create Timer Unit** (`/etc/systemd/system/zabbix-partitioning.timer`):
```ini ```ini
[Unit] [Unit]
Description=Run Zabbix Partitioning Daily Description=Run Zabbix Partitioning twice a day
[Timer] [Timer]
OnCalendar=*-*-* 00:30:00 OnCalendar=*-*-* 00:10:00 *-*-* 04:10:00
Persistent=true Persistent=true
[Install] [Install]
@@ -176,12 +215,12 @@ Alternatively, use systemd timers for more robust scheduling and logging.
--- ---
## 8. Troubleshooting ## 9. Troubleshooting
- **Connection Refused**: Check `host`, `port` in config. Ensure MySQL is running. - **Connection Refused**: Check `host`, `port` in config. Ensure MySQL is running.
- **Access Denied (1227)**: The DB user needs `SUPER` privileges to disable binary logging (`replicate_sql: False`). Either grant the privilege or set `replicate_sql: True` (if replication load is acceptable). - **Access Denied (1227)**: The DB user needs `SUPER` privileges to disable binary logging (`replicate_sql: False`). Either grant the privilege or set `replicate_sql: True` (if replication load is acceptable).
- **Primary Key Error**: "Primary Key does not include 'clock'". The table cannot be partitioned by range on `clock` without schema changes. Remove it from config. - **Primary Key Error**: "Primary Key does not include 'clock'". The table cannot be partitioned by range on `clock` without schema changes. Remove it from config.
## 9. Docker Usage ## 10. Docker Usage
You can run the partitioning script as a stateless Docker container. This is ideal for Kubernetes CronJobs or environments where you don't want to manage Python dependencies on the host. You can run the partitioning script as a stateless Docker container. This is ideal for Kubernetes CronJobs or environments where you don't want to manage Python dependencies on the host.
@@ -267,8 +306,8 @@ docker run --rm \
| `RETENTION_TRENDS` | 365d | Retention for `trends*` tables | | `RETENTION_TRENDS` | 365d | Retention for `trends*` tables |
| `RETENTION_AUDIT` | 365d | Retention for `auditlog` (if enabled) | | `RETENTION_AUDIT` | 365d | Retention for `auditlog` (if enabled) |
| `ENABLE_AUDITLOG_PARTITIONING` | false | Set to `true` to partition `auditlog` | | `ENABLE_AUDITLOG_PARTITIONING` | false | Set to `true` to partition `auditlog` |
| `RUN_MODE` | maintenance | `init`, `maintenance`, `dry-run`, `discovery`, `check` | | `RUN_MODE` | maintenance | `init`, `maintenance`, `dry-run`, `discovery`, `stats` |
| `CHECK_TARGET` | - | Required if `RUN_MODE=check`. Table name to check (e.g. `history`). | | `CHECK_TARGET` | - | Required if `RUN_MODE=stats`. Table name to check (e.g. `history`). |
| `PARTITION_DAILY_[TABLE]` | - | Custom daily retention (e.g., `PARTITION_DAILY_mytable=30d`) | | `PARTITION_DAILY_[TABLE]` | - | Custom daily retention (e.g., `PARTITION_DAILY_mytable=30d`) |
| `PARTITION_WEEKLY_[TABLE]` | - | Custom weekly retention | | `PARTITION_WEEKLY_[TABLE]` | - | Custom weekly retention |
| `PARTITION_MONTHLY_[TABLE]` | - | Custom monthly retention | | `PARTITION_MONTHLY_[TABLE]` | - | Custom monthly retention |
@@ -282,36 +321,37 @@ docker run --rm \
zabbix-partitioning zabbix-partitioning
``` ```
#### Scenario G: Monitoring (Health Check) #### Scenario G: Monitoring (Stats)
Check days remaining for a specific table (e.g., `history`). Returns integer days. Get detailed JSON statistics for a specific table.
```bash ```bash
docker run --rm \ docker run --rm \
-e DB_HOST=zabbix-db \ -e DB_HOST=zabbix-db \
-e RUN_MODE=check \ -e RUN_MODE=stats \
-e CHECK_TARGET=history \ -e CHECK_TARGET=history \
zabbix-partitioning zabbix-partitioning
``` ```
--- ---
## 10. Monitoring ## 11. Monitoring
The script includes built-in features for monitoring the health of your partitions via Zabbix. The script includes built-in features for monitoring the health of your partitions via Zabbix.
### 10.1 CLI Usage ### 10.1 CLI Usage
- **Discovery (LLD)**: - **Discovery (LLD)**:
Output the list of configured tables for Zabbix Discovery rules.
```bash ```bash
./zabbix_partitioning.py --discovery ./zabbix_partitioning.py --discovery
# Output: [{"{#TABLE}": "history", "{#PERIOD}": "daily"}, ...] # Output: [{"{#TABLE}": "history", "{#PERIOD}": "daily"}, ...]
``` ```
- **Check Days**: - **Statistics (JSON)**:
Get detailed stats (Size, Partition Count, Days Remaining).
```bash ```bash
./zabbix_partitioning.py --check-days history ./zabbix_partitioning.py --stats history
# Output: 30 (integer days remaining) # Output: {"table": "history", "size_bytes": 102400, "partition_count": 5, "days_left": 30}
``` ```
- **Version**: - **Version**:
```bash ```bash
./zabbix_partitioning.py --version ./zabbix_partitioning.py --version
# Output: zabbix_partitioning.py 0.3.1-test
``` ```
### 10.2 Zabbix Template ### 10.2 Zabbix Template

View File

@@ -11,6 +11,11 @@ database:
user: zbx_part user: zbx_part
passwd: <password> passwd: <password>
db: zabbix db: zabbix
# ssl: required # Use system CAs (Mandatory for Azure/AWS/GCP)
# ssl: # Or use custom certificates
# ca: /etc/ssl/certs/ca-cert.pem
# cert: /etc/ssl/certs/client-cert.pem
# key: /etc/ssl/certs/client-key.pem
# partitions: Define retention periods for tables. # partitions: Define retention periods for tables.
# Format: table_name: duration (e.g., 14d, 12w, 1m, 1y) # Format: table_name: duration (e.g., 14d, 12w, 1m, 1y)

View File

@@ -1,9 +1,8 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
# -*- coding: utf-8 -*- # -*- coding: utf-8 -*-
""" """
Zabbix Database Partitioning Management Script Zabbix MySQL Partitioning Script
Refactored for Zabbix 7.x compatibility, better maintainability, and standard logging.
""" """
import os import os
@@ -21,7 +20,7 @@ from typing import Optional, Dict, List, Any, Union, Tuple
from contextlib import contextmanager from contextlib import contextmanager
# Semantic Versioning # Semantic Versioning
VERSION = '0.4.1' VERSION = '0.6.0'
# Constants # Constants
PART_PERIOD_REGEX = r'([0-9]+)(h|d|m|y)' PART_PERIOD_REGEX = r'([0-9]+)(h|d|m|y)'
@@ -74,7 +73,12 @@ class ZabbixPartitioner:
connect_args['host'] = self.db_host connect_args['host'] = self.db_host
if self.db_ssl: if self.db_ssl:
connect_args['ssl'] = self.db_ssl if self.db_ssl is True or (isinstance(self.db_ssl, str) and self.db_ssl.lower() == 'required'):
# Enable SSL with default context (uses system CAs)
# Useful for Cloud DBs (Azure, AWS) that require TLS by default
connect_args['ssl'] = {}
else:
connect_args['ssl'] = self.db_ssl
# PyMySQL SSL options # PyMySQL SSL options
# Note: valid ssl keys for PyMySQL are 'ca', 'capath', 'cert', 'key', 'cipher', 'check_hostname' # Note: valid ssl keys for PyMySQL are 'ca', 'capath', 'cert', 'key', 'cipher', 'check_hostname'
@@ -177,9 +181,11 @@ class ZabbixPartitioner:
""" """
Calculate the retention date based on config string (e.g., "30d", "12m"). Calculate the retention date based on config string (e.g., "30d", "12m").
""" """
# Ensure string
period_str = str(period_str)
match = re.search(PART_PERIOD_REGEX, period_str) match = re.search(PART_PERIOD_REGEX, period_str)
if not match: if not match:
raise ConfigurationError(f"Invalid period format: {period_str}") raise ConfigurationError(f"Invalid period format: '{period_str}' (Expected format like 30d, 12w, 1y)")
amount = int(match.group(1)) amount = int(match.group(1))
unit = match.group(2) unit = match.group(2)
@@ -233,10 +239,9 @@ class ZabbixPartitioner:
raise DatabaseError("Could not determine MySQL version") raise DatabaseError("Could not determine MySQL version")
# MySQL 8.0+ supports partitioning natively # MySQL 8.0+ supports partitioning natively
# (Assuming MySQL 8+ or MariaDB 10+ for modern Zabbix)
self.logger.info(f"MySQL Version: {version_str}") self.logger.info(f"MySQL Version: {version_str}")
# 2. Check Zabbix DB Version (optional info) # 2. Check Zabbix DB Version
try: try:
mandatory = self.execute_query('SELECT `mandatory` FROM `dbversion`', fetch='one') mandatory = self.execute_query('SELECT `mandatory` FROM `dbversion`', fetch='one')
if mandatory: if mandatory:
@@ -244,6 +249,18 @@ class ZabbixPartitioner:
except Exception: except Exception:
self.logger.warning("Could not read 'dbversion' table. Is this a Zabbix DB?") self.logger.warning("Could not read 'dbversion' table. Is this a Zabbix DB?")
def validate_table_exists(self, table: str) -> bool:
"""Return True if table exists in the database."""
# Use simple select from information_schema
exists = self.execute_query(
"SELECT 1 FROM information_schema.tables WHERE table_schema = %s AND table_name = %s",
(self.db_name, table), fetch='one'
)
if not exists:
self.logger.error(f"Table '{table}' does NOT exist in database '{self.db_name}'. Skipped.")
return False
return True
def get_table_min_clock(self, table: str) -> Optional[datetime]: def get_table_min_clock(self, table: str) -> Optional[datetime]:
ts = self.execute_query(f"SELECT MIN(`clock`) FROM `{table}`", fetch='one') ts = self.execute_query(f"SELECT MIN(`clock`) FROM `{table}`", fetch='one')
return datetime.fromtimestamp(int(ts)) if ts else None return datetime.fromtimestamp(int(ts)) if ts else None
@@ -373,95 +390,131 @@ class ZabbixPartitioner:
for name in to_drop: for name in to_drop:
self.execute_query(f"ALTER TABLE `{table}` DROP PARTITION {name}") self.execute_query(f"ALTER TABLE `{table}` DROP PARTITION {name}")
def initialize_partitioning(self, table: str, period: str, premake: int, retention_str: str):
"""Initial partitioning for a table (convert regular table to partitioned)."""
self.logger.info(f"Initializing partitioning for {table}")
def _check_init_prerequisites(self, table: str) -> bool:
"""Return True if partitioning can proceed."""
if self.has_incompatible_primary_key(table): if self.has_incompatible_primary_key(table):
self.logger.error(f"Cannot partition {table}: Primary Key does not include 'clock' column.") self.logger.error(f"Cannot partition {table}: Primary Key does not include 'clock' column.")
return return False
# If already partitioned, skip
if self.get_existing_partitions(table): if self.get_existing_partitions(table):
self.logger.info(f"Table {table} is already partitioned.") self.logger.info(f"Table {table} is already partitioned.")
return return False
# init_strategy = self.config.get('initial_partitioning_start', 'db_min') # Removed in favor of flag # Disk Space & Lock Warning
# but flag needs to be passed to this method or accessed from somewhere. msg = (
# Since I can't easily change signature without affecting calls, I'll pass it in kwargs or check self.fast_init if I add it to class. f"WARNING: Partitioning table '{table}' requires creating a copy of the table.\n"
pass f" Ensure you have free disk space >= Data_Length + Index_Length (approx 2x table size).\n"
f" For large databases, this operation may take SEVERAL HOURS to complete."
)
self.logger.warning(msg)
def initialize_partitioning(self, table: str, period: str, premake: int, retention_str: str, fast_init: bool = False): # Interactive Check
"""Initial partitioning for a table (convert regular table to partitioned).""" if sys.stdin.isatty():
self.logger.info(f"Initializing partitioning for {table}") print(f"\n{msg}")
if input("Do you have enough free space and is Zabbix stopped? [y/N]: ").lower().strip() != 'y':
self.logger.error("Initialization aborted by user.")
return False
if self.has_incompatible_primary_key(table): return True
self.logger.error(f"Cannot partition {table}: Primary Key does not include 'clock' column.")
return
# If already partitioned, skip def _generate_init_sql(self, table: str, period: str, start_dt: datetime, premake: int, p_archive_ts: int = None):
if self.get_existing_partitions(table): """Generate and execute ALTER TABLE command."""
self.logger.info(f"Table {table} is already partitioned.")
return
start_dt = None
p_archive_ts = None
if fast_init:
self.logger.info(f"Strategy 'fast-init': Calculating start date from retention ({retention_str})")
retention_date = self.get_lookback_date(retention_str)
# Start granular partitions from the retention date
start_dt = self.truncate_date(retention_date, period)
# Create a catch-all for anything older
p_archive_ts = int(start_dt.timestamp())
else:
# Default 'db_min' strategy
self.logger.info("Strategy 'db_min': Querying table for minimum clock (may be slow)")
min_clock = self.get_table_min_clock(table)
if not min_clock:
# Empty table. Start from NOW
start_dt = self.truncate_date(datetime.now(), period)
else:
# Table has data.
start_dt = self.truncate_date(min_clock, period)
# Build list of partitions from start_dt up to NOW + premake
target_dt = self.get_next_date(self.truncate_date(datetime.now(), period), period, premake) target_dt = self.get_next_date(self.truncate_date(datetime.now(), period), period, premake)
curr = start_dt
partitions_def = {}
# If we have an archive partition, add it first
if p_archive_ts:
partitions_def['p_archive'] = str(p_archive_ts)
while curr < target_dt:
name = self.get_partition_name(curr, period)
desc = self.get_partition_description(curr, period)
partitions_def[name] = desc
curr = self.get_next_date(curr, period, 1)
# Re-doing the loop to be cleaner on types
parts_sql = [] parts_sql = []
# 1. Archive Partition
if p_archive_ts: if p_archive_ts:
parts_sql.append(f"PARTITION p_archive VALUES LESS THAN ({p_archive_ts}) ENGINE = InnoDB") parts_sql.append(f"PARTITION p_archive VALUES LESS THAN ({p_archive_ts}) ENGINE = InnoDB")
# 2. Granular Partitions
# We need to iterate again from start_dt
curr = start_dt curr = start_dt
while curr < target_dt: while curr < target_dt:
name = self.get_partition_name(curr, period) name = self.get_partition_name(curr, period)
desc_date_str = self.get_partition_description(curr, period) # Returns "YYYY-MM-DD HH:MM:SS" desc_date_str = self.get_partition_description(curr, period)
parts_sql.append(PARTITION_TEMPLATE % (name, desc_date_str)) parts_sql.append(PARTITION_TEMPLATE % (name, desc_date_str))
curr = self.get_next_date(curr, period, 1) curr = self.get_next_date(curr, period, 1)
if not parts_sql:
self.logger.warning(f"No partitions generated for {table}")
return
query = f"ALTER TABLE `{table}` PARTITION BY RANGE (`clock`) (\n" + ",\n".join(parts_sql) + "\n)" query = f"ALTER TABLE `{table}` PARTITION BY RANGE (`clock`) (\n" + ",\n".join(parts_sql) + "\n)"
self.logger.info(f"Applying initial partitioning to {table} ({len(parts_sql)} partitions)") self.logger.info(f"Applying initial partitioning to {table} ({len(parts_sql)} partitions)")
self.execute_query(query) self.execute_query(query)
def _get_configured_tables(self) -> List[str]:
"""Return a list of all tables configured for partitioning."""
tables_set = set()
partitions_conf = self.config.get('partitions', {})
for tables in partitions_conf.values():
if not tables: continue
for item in tables:
tables_set.add(list(item.keys())[0])
return list(tables_set)
def clean_housekeeper_table(self):
"""
Clean up Zabbix housekeeper table during initialization.
Removes pending tasks ONLY for configured tables to prevent conflicts.
"""
configured_tables = self._get_configured_tables()
if not configured_tables:
return
self.logger.info(f"Cleaning up 'housekeeper' table for: {', '.join(configured_tables)}")
# Construct IN clause
placeholders = ', '.join(['%s'] * len(configured_tables))
query = f"DELETE FROM `housekeeper` WHERE `tablename` IN ({placeholders})"
self.execute_query(query, configured_tables)
def check_housekeeper_execution(self):
"""
Check if Zabbix internal housekeeper is running for configured partitioned tables.
"""
configured_tables = self._get_configured_tables()
if not configured_tables:
return
query = "SELECT DISTINCT `tablename` FROM `housekeeper` WHERE `tablename` != 'events'"
rows = self.execute_query(query, fetch='all')
if rows:
found_tables = {r[0] for r in rows}
configured_set = set(configured_tables)
conflicts = found_tables.intersection(configured_set)
if conflicts:
self.logger.error(f"CRITICAL: Found pending housekeeper tasks for partitioned tables: {', '.join(conflicts)}")
self.logger.error("Please DISABLE Zabbix internal housekeeper for these tables in Administration -> General -> Housekeeping!")
def initialize_partitioning_fast(self, table: str, period: str, premake: int, retention_str: str):
"""Fast Init: Use retention period to determine start date."""
self.logger.info(f"Initializing partitioning for {table} (Fast Mode)")
if not self._check_init_prerequisites(table): return
self.logger.info(f"Strategy 'fast-init': Calculating start date from retention ({retention_str})")
retention_date = self.get_lookback_date(retention_str)
start_dt = self.truncate_date(retention_date, period)
p_archive_ts = int(start_dt.timestamp())
self._generate_init_sql(table, period, start_dt, premake, p_archive_ts)
def initialize_partitioning_scan(self, table: str, period: str, premake: int):
"""Scan Init: Scan table for minimum clock value."""
self.logger.info(f"Initializing partitioning for {table} (Scan Mode)")
if not self._check_init_prerequisites(table): return
self.logger.info("Strategy 'db_min': Querying table for minimum clock (may be slow)")
min_clock = self.get_table_min_clock(table)
if not min_clock:
start_dt = self.truncate_date(datetime.now(), period)
else:
start_dt = self.truncate_date(min_clock, period)
self._generate_init_sql(table, period, start_dt, premake)
def discovery(self): def discovery(self):
"""Output Zabbix Low-Level Discovery logic JSON.""" """Output Zabbix Low-Level Discovery logic JSON."""
partitions_conf = self.config.get('partitions', {}) partitions_conf = self.config.get('partitions', {})
@@ -554,32 +607,6 @@ class ZabbixPartitioner:
self.discovery() self.discovery()
return return
# --- Check Mode ---
if mode == 'check':
if not target_table:
# Check all and print simple status? Or error?
# Zabbix usually queries one by one.
# Implementing simple check which returns days for specific table
raise ConfigurationError("Target table required for check mode")
# Find period for table
found_period = None
for period, tables in partitions_conf.items():
for item in tables:
if list(item.keys())[0] == target_table:
found_period = period
break
if found_period: break
if not found_period:
# Table not in config?
print("-1") # Error code
return
days_left = self.check_partitions_coverage(target_table, found_period)
print(days_left)
return
# --- Stats Mode --- # --- Stats Mode ---
if mode == 'stats': if mode == 'stats':
if not target_table: if not target_table:
@@ -593,6 +620,11 @@ class ZabbixPartitioner:
self.check_compatibility() self.check_compatibility()
premake = self.config.get('premake', 10) premake = self.config.get('premake', 10)
if mode == 'init':
self.clean_housekeeper_table()
else:
self.check_housekeeper_execution()
for period, tables in partitions_conf.items(): for period, tables in partitions_conf.items():
if not tables: if not tables:
continue continue
@@ -601,12 +633,24 @@ class ZabbixPartitioner:
table = list(item.keys())[0] table = list(item.keys())[0]
retention = item[table] retention = item[table]
if not self.validate_table_exists(table):
continue
if mode == 'init': if mode == 'init':
self.initialize_partitioning(table, period, premake, retention, fast_init=self.fast_init) try:
if self.fast_init:
self.initialize_partitioning_fast(table, period, premake, retention)
else:
self.initialize_partitioning_scan(table, period, premake)
except ConfigurationError as e:
raise ConfigurationError(f"Table '{table}': {e}")
else: else:
# Maintenance mode (Add new, remove old) # Maintenance mode (Add new, remove old)
self.create_future_partitions(table, period, premake) try:
self.remove_old_partitions(table, retention) self.create_future_partitions(table, period, premake)
self.remove_old_partitions(table, retention)
except ConfigurationError as e:
raise ConfigurationError(f"Table '{table}': {e}")
# Housekeeping extras # Housekeeping extras
if mode != 'init' and not self.dry_run: if mode != 'init' and not self.dry_run:
@@ -615,6 +659,31 @@ class ZabbixPartitioner:
if mode != 'init' and not self.dry_run: if mode != 'init' and not self.dry_run:
pass pass
def get_retention_input(prompt: str, default: str = None, allow_empty: bool = False) -> str:
"""Helper to get and validate retention period input."""
while True:
val = input(prompt).strip()
# Handle Empty Input
if not val:
if default:
return default
if allow_empty:
return ""
# If no default and not allow_empty, continue loop
continue
# Handle Unit-less Input (Reject)
if val.isdigit():
print(f"Error: '{val}' is missing a unit. Please use 'd', 'w', 'm', or 'y' (e.g., {val}d).")
continue
# Validate Format
if re.search(PART_PERIOD_REGEX, val):
return val
print("Invalid format. Use format like 30d, 12w, 1y.")
def run_wizard(): def run_wizard():
print("Welcome to Zabbix Partitioning Wizard") print("Welcome to Zabbix Partitioning Wizard")
print("-------------------------------------") print("-------------------------------------")
@@ -645,11 +714,26 @@ def run_wizard():
config['database']['passwd'] = input("Database Password: ").strip() config['database']['passwd'] = input("Database Password: ").strip()
config['database']['db'] = input("Database Name [zabbix]: ").strip() or 'zabbix' config['database']['db'] = input("Database Name [zabbix]: ").strip() or 'zabbix'
# 1.1 SSL
if input("Use SSL/TLS for connection? [y/N]: ").lower().strip() == 'y':
print(" Mode 1: Managed/Cloud DB (Use system CAs, e.g. Azure/AWS)")
print(" Mode 2: Custom Certificates (Provide ca, cert, key)")
if input(" Use custom certificates? [y/N]: ").lower().strip() == 'y':
ssl_conf = {}
ssl_conf['ca'] = input(" CA Certificate Path: ").strip()
ssl_conf['cert'] = input(" Client Certificate Path: ").strip()
ssl_conf['key'] = input(" Client Key Path: ").strip()
# Filter empty
config['database']['ssl'] = {k: v for k, v in ssl_conf.items() if v}
else:
config['database']['ssl'] = 'required'
# 2. Auditlog # 2. Auditlog
print("\n[Auditlog]") print("\n[Auditlog]")
print("Note: To partition 'auditlog', ensure its Primary Key includes the 'clock' column.") print("Note: To partition 'auditlog', ensure its Primary Key includes the 'clock' column.")
if input("Partition 'auditlog' table? [y/N]: ").lower().strip() == 'y': if input("Partition 'auditlog' table? [y/N]: ").lower().strip() == 'y':
ret = input("Auditlog retention (e.g. 365d) [365d]: ").strip() or '365d' ret = get_retention_input("Auditlog retention (e.g. 365d) [365d]: ", "365d")
config['partitions']['weekly'].append({'auditlog': ret}) config['partitions']['weekly'].append({'auditlog': ret})
# 3. History Tables # 3. History Tables
@@ -659,12 +743,12 @@ def run_wizard():
print("\n[History Tables]") print("\n[History Tables]")
# Separate logic as requested # Separate logic as requested
if input("Set SAME retention for all history tables? [Y/n]: ").lower().strip() != 'n': if input("Set SAME retention for all history tables? [Y/n]: ").lower().strip() != 'n':
ret = input("Retention for all history tables (e.g. 30d) [30d]: ").strip() or '30d' ret = get_retention_input("Retention for all history tables (e.g. 30d) [30d]: ", "30d")
for t in history_tables: for t in history_tables:
config['partitions']['daily'].append({t: ret}) config['partitions']['daily'].append({t: ret})
else: else:
for t in history_tables: for t in history_tables:
ret = input(f"Retention for '{t}' (e.g. 30d, skip to ignore): ").strip() ret = get_retention_input(f"Retention for '{t}' (e.g. 30d, skip to ignore): ", allow_empty=True)
if ret: if ret:
config['partitions']['daily'].append({t: ret}) config['partitions']['daily'].append({t: ret})
@@ -672,12 +756,12 @@ def run_wizard():
trends_tables = ['trends', 'trends_uint'] trends_tables = ['trends', 'trends_uint']
print("\n[Trends Tables]") print("\n[Trends Tables]")
if input("Set SAME retention for all trends tables? [Y/n]: ").lower().strip() != 'n': if input("Set SAME retention for all trends tables? [Y/n]: ").lower().strip() != 'n':
ret = input("Retention for all trends tables (e.g. 365d) [365d]: ").strip() or '365d' ret = get_retention_input("Retention for all trends tables (e.g. 365d) [365d]: ", "365d")
for t in trends_tables: for t in trends_tables:
config['partitions']['monthly'].append({t: ret}) config['partitions']['monthly'].append({t: ret})
else: else:
for t in trends_tables: for t in trends_tables:
ret = input(f"Retention for '{t}' (e.g. 365d, skip to ignore): ").strip() ret = get_retention_input(f"Retention for '{t}' (e.g. 365d, skip to ignore): ", allow_empty=True)
if ret: if ret:
config['partitions']['monthly'].append({t: ret}) config['partitions']['monthly'].append({t: ret})
@@ -731,22 +815,43 @@ def setup_logging(config_log_type: str, verbose: bool = False):
logger.addHandler(handler) logger.addHandler(handler)
def parse_args(): def parse_args():
parser = argparse.ArgumentParser(description='Zabbix Partitioning Manager') parser = argparse.ArgumentParser(
parser.add_argument('-c', '--config', default='/etc/zabbix/zabbix_partitioning.conf', help='Config file path') description='Zabbix Database Partitioning Script',
parser.add_argument('-i', '--init', action='store_true', help='Initialize partitions') epilog='''
parser.add_argument('-r', '--dry-run', action='store_true', help='Simulate queries') Examples:
parser.add_argument('-v', '--verbose', action='store_true', help='Enable debug logging') # 1. Interactive Configuration (Initial config file creation)
%(prog)s --wizard
# Monitoring args # 2. Initialization (First Run)
parser.add_argument('--discovery', action='store_true', help='Output Zabbix LLD JSON') # A. Standard (Scans DB for oldest record - Recommended):
parser.add_argument('--check-days', type=str, help='Check days of future partitions left for table', metavar='TABLE') %(prog)s --init
parser.add_argument('--stats', type=str, help='Output detailed table statistics in JSON', metavar='TABLE')
# Wizard & Flags # B. Fast (Start from retention period - Best for large DBs):
parser.add_argument('--wizard', action='store_true', help='Launch interactive configuration wizard') %(prog)s --fast-init
parser.add_argument('--fast-init', action='store_true', help='Skip MIN(clock) check during init, start from retention')
parser.add_argument('-V', '--version', action='version', version=f'%(prog)s {VERSION}', help='Show version and exit') # 3. Regular Maintenance (Cron/Systemd)
# Creates future partitions and drops old ones.
%(prog)s
# 4. Monitoring (Zabbix Integration)
# Discovery (LLD): %(prog)s --discovery
# Statistics (JSON): %(prog)s --stats history
''',
formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument('--config','-c', default='/etc/zabbix/zabbix_partitioning.conf', help='Path to configuration file')
# Mutually Exclusive Actions
group = parser.add_mutually_exclusive_group()
group.add_argument('--init', '-i', action='store_true', help='Initialize partitions (Standard: Scans DB for oldest record)')
group.add_argument('--fast-init', '-f', action='store_true', help='Initialize partitions (Fast: Starts FROM retention period, skips scan)')
group.add_argument('--discovery', '-d', action='store_true', help='Output Zabbix Low-Level Discovery (LLD) JSON')
group.add_argument('--stats', '-s', type=str, help='Output table statistics (Size, Count, Usage) in JSON', metavar='TABLE')
group.add_argument('--wizard', '-w', action='store_true', help='Launch interactive configuration wizard')
parser.add_argument('--version', '-V', action='version', version=f'%(prog)s {VERSION}', help='Show version and exit')
parser.add_argument('--verbose', '-vv', action='store_true', help='Enable debug logging')
parser.add_argument('--dry-run', '-r', action='store_true', help='Simulate queries without executing them')
return parser.parse_args() return parser.parse_args()
@@ -781,16 +886,16 @@ def main():
config['logging'] = 'console' # Force console for discovery? Or suppress? config['logging'] = 'console' # Force console for discovery? Or suppress?
# actually we don't want logs mixing with JSON output # actually we don't want logs mixing with JSON output
# so checking mode before setup logging # so checking mode before setup logging
elif args.check_days:
mode = 'check'
target = args.check_days
elif args.stats: elif args.stats:
mode = 'stats' mode = 'stats'
target = args.stats target = args.stats
elif args.init: mode = 'init' elif args.init:
mode = 'init'
elif args.fast_init:
mode = 'init'
# Setup logging # Setup logging
if mode in ['discovery', 'check', 'stats']: if mode in ['discovery', 'stats']:
logging.basicConfig(level=logging.ERROR) # Only show critical errors logging.basicConfig(level=logging.ERROR) # Only show critical errors
else: else:
setup_logging(config.get('logging', 'console'), verbose=args.verbose) setup_logging(config.get('logging', 'console'), verbose=args.verbose)