Change: Adjusted README to the new changes. Added runtime parameters.

This commit is contained in:
2025-12-16 23:05:44 +01:00
parent 3b2bebf6b9
commit d2de7f8b02

View File

@@ -47,8 +47,16 @@ The script manages MySQL table partitions based on time (Range Partitioning on t
--- ---
## 3. Configuration ## 3. Configuration
Edit `/etc/zabbix/zabbix_partitioning.conf`: The preferred way to create the configuration file is using the **Interactive Wizard**:
```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --wizard
```
Follow the on-screen prompts to configure your database connection, retention periods, and other settings. The wizard will generate the YAML configuration file for you.
### Automatic Config Structure (Reference)
If you prefer to edit manually `/etc/zabbix/zabbix_partitioning.conf`:
```yaml ```yaml
database: database:
host: localhost host: localhost
@@ -56,6 +64,7 @@ database:
passwd: YOUR_PASSWORD passwd: YOUR_PASSWORD
db: zabbix db: zabbix
# port: 3306 # Optional, default is 3306 # port: 3306 # Optional, default is 3306
# socket: /var/run/mysqld/mysqld.sock # Overrides host if present
partitions: partitions:
daily: daily:
@@ -74,9 +83,7 @@ partitions:
- **`yearly`**: Partitions are created for each year. - **`yearly`**: Partitions are created for each year.
- Retention Format: `14d` (days), `12w` (weeks), `12m` (months), `1y` (years). - Retention Format: `14d` (days), `12w` (weeks), `12m` (months), `1y` (years).
- **`initial_partitioning_start`**: Controls how the very FIRST partition is determined during initialization (`--init` mode). - **`replicate_sql`**: Controls MySQL Binary Logging for partitioning commands.
- `db_min`: (Default) Queries the table for the oldest record (`MIN(clock)`). Accurate but **slow** on large tables.
- `retention`: (Recommended for large DBs) Skips the query. Calculates the start date as `Now - Retention Period`. Creates a single `p_archive` partition for all data older than that date.
- **`premake`**: Number of future partitions to create in advance. - **`premake`**: Number of future partitions to create in advance.
- Default: `10`. Ensures you have a buffer if the script fails to run for a few days. - Default: `10`. Ensures you have a buffer if the script fails to run for a few days.
@@ -90,7 +97,25 @@ partitions:
--- ---
## 4. Zabbix Preparation (CRITICAL) ---
## 4. Runtime Parameters
| Argument | Description |
|---|---|
| `-c`, `--config FILE` | Path to configuration file (Default: `/etc/zabbix/zabbix_partitioning.conf`) |
| `-i`, `--init` | Initialize partitions (converts tables). |
| `--fast-init` | Skip slow table scan during initialization. Starts from retention period. |
| `--wizard` | Launch interactive configuration wizard. |
| `-r`, `--dry-run` | Simulate queries without executing. Logs expected actions (Safe mode). |
| `-v`, `--verbose` | Enable debug logging (DEBUG level). |
| `--stats TABLE` | Output JSON statistics (Size, Count, Days Left) for a specific table. |
| `--discovery` | Output Zabbix Low-Level Discovery (LLD) JSON. |
| `-V`, `--version` | Show script version. |
---
## 5. Zabbix Preparation (CRITICAL)
Before partitioning, you **must disable** Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts. Before partitioning, you **must disable** Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts.
1. Log in to Zabbix Web Interface. 1. Log in to Zabbix Web Interface.
@@ -102,24 +127,35 @@ Before partitioning, you **must disable** Zabbix's internal housekeeping for the
--- ---
## 5. Initialization ## 6. Initialization
This step converts existing standard tables into partitioned tables. This step converts existing standard tables into partitioned tables.
1. **Dry Run** (Verify what will happen): 1. **Dry Run** (Verify what will happen):
```bash ```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run /opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run
``` ```
*Check the output for any errors.*
2. **Execute Initialization**: 2. **Execute Initialization**:
There are two strategies for initialization:
**A. Standard Initialization (Default)**:
Scans the database to find the oldest record (`MIN(clock)`) and creates partitions from that point forward.
*Best for smaller databases or when you need to retain ALL existing data.*
```bash ```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --init /opt/zabbix_partitioning/zabbix_partitioning.py --init
``` ```
*This may take time depending on table size.*
**B. Fast Initialization (`--fast-init`)**:
Skips the slow table scan. It calculates the start date based on your configured **Retention Period** (e.g., `Now - 365d`).
It creates a single catch-all `p_archive` partition for all data older than the retention start date, then creates granular partitions forward.
*Recommended for large databases to avoid long table locks/scans.*
```bash
/opt/zabbix_partitioning/zabbix_partitioning.py --init --fast-init
```
--- ---
## 6. Automation (Cron Job) ## 7. Automation (Cron Job)
Set up a daily cron job to create new partitions and remove old ones. Set up a daily cron job to create new partitions and remove old ones.
1. Open crontab: 1. Open crontab:
@@ -133,7 +169,7 @@ Set up a daily cron job to create new partitions and remove old ones.
--- ---
## 7. Automation (Systemd Timer) — Recommended ## 8. Automation (Systemd Timer) — Recommended
Alternatively, use systemd timers for more robust scheduling and logging. Alternatively, use systemd timers for more robust scheduling and logging.
1. **Create Service Unit** (`/etc/systemd/system/zabbix-partitioning.service`): 1. **Create Service Unit** (`/etc/systemd/system/zabbix-partitioning.service`):
@@ -176,12 +212,12 @@ Alternatively, use systemd timers for more robust scheduling and logging.
--- ---
## 8. Troubleshooting ## 9. Troubleshooting
- **Connection Refused**: Check `host`, `port` in config. Ensure MySQL is running. - **Connection Refused**: Check `host`, `port` in config. Ensure MySQL is running.
- **Access Denied (1227)**: The DB user needs `SUPER` privileges to disable binary logging (`replicate_sql: False`). Either grant the privilege or set `replicate_sql: True` (if replication load is acceptable). - **Access Denied (1227)**: The DB user needs `SUPER` privileges to disable binary logging (`replicate_sql: False`). Either grant the privilege or set `replicate_sql: True` (if replication load is acceptable).
- **Primary Key Error**: "Primary Key does not include 'clock'". The table cannot be partitioned by range on `clock` without schema changes. Remove it from config. - **Primary Key Error**: "Primary Key does not include 'clock'". The table cannot be partitioned by range on `clock` without schema changes. Remove it from config.
## 9. Docker Usage ## 10. Docker Usage
You can run the partitioning script as a stateless Docker container. This is ideal for Kubernetes CronJobs or environments where you don't want to manage Python dependencies on the host. You can run the partitioning script as a stateless Docker container. This is ideal for Kubernetes CronJobs or environments where you don't want to manage Python dependencies on the host.
@@ -267,8 +303,8 @@ docker run --rm \
| `RETENTION_TRENDS` | 365d | Retention for `trends*` tables | | `RETENTION_TRENDS` | 365d | Retention for `trends*` tables |
| `RETENTION_AUDIT` | 365d | Retention for `auditlog` (if enabled) | | `RETENTION_AUDIT` | 365d | Retention for `auditlog` (if enabled) |
| `ENABLE_AUDITLOG_PARTITIONING` | false | Set to `true` to partition `auditlog` | | `ENABLE_AUDITLOG_PARTITIONING` | false | Set to `true` to partition `auditlog` |
| `RUN_MODE` | maintenance | `init`, `maintenance`, `dry-run`, `discovery`, `check` | | `RUN_MODE` | maintenance | `init`, `maintenance`, `dry-run`, `discovery`, `stats` |
| `CHECK_TARGET` | - | Required if `RUN_MODE=check`. Table name to check (e.g. `history`). | | `CHECK_TARGET` | - | Required if `RUN_MODE=stats`. Table name to check (e.g. `history`). |
| `PARTITION_DAILY_[TABLE]` | - | Custom daily retention (e.g., `PARTITION_DAILY_mytable=30d`) | | `PARTITION_DAILY_[TABLE]` | - | Custom daily retention (e.g., `PARTITION_DAILY_mytable=30d`) |
| `PARTITION_WEEKLY_[TABLE]` | - | Custom weekly retention | | `PARTITION_WEEKLY_[TABLE]` | - | Custom weekly retention |
| `PARTITION_MONTHLY_[TABLE]` | - | Custom monthly retention | | `PARTITION_MONTHLY_[TABLE]` | - | Custom monthly retention |
@@ -282,36 +318,37 @@ docker run --rm \
zabbix-partitioning zabbix-partitioning
``` ```
#### Scenario G: Monitoring (Health Check) #### Scenario G: Monitoring (Stats)
Check days remaining for a specific table (e.g., `history`). Returns integer days. Get detailed JSON statistics for a specific table.
```bash ```bash
docker run --rm \ docker run --rm \
-e DB_HOST=zabbix-db \ -e DB_HOST=zabbix-db \
-e RUN_MODE=check \ -e RUN_MODE=stats \
-e CHECK_TARGET=history \ -e CHECK_TARGET=history \
zabbix-partitioning zabbix-partitioning
``` ```
--- ---
## 10. Monitoring ## 11. Monitoring
The script includes built-in features for monitoring the health of your partitions via Zabbix. The script includes built-in features for monitoring the health of your partitions via Zabbix.
### 10.1 CLI Usage ### 10.1 CLI Usage
- **Discovery (LLD)**: - **Discovery (LLD)**:
Output the list of configured tables for Zabbix Discovery rules.
```bash ```bash
./zabbix_partitioning.py --discovery ./zabbix_partitioning.py --discovery
# Output: [{"{#TABLE}": "history", "{#PERIOD}": "daily"}, ...] # Output: [{"{#TABLE}": "history", "{#PERIOD}": "daily"}, ...]
``` ```
- **Check Days**: - **Statistics (JSON)**:
Get detailed stats (Size, Partition Count, Days Remaining).
```bash ```bash
./zabbix_partitioning.py --check-days history ./zabbix_partitioning.py --stats history
# Output: 30 (integer days remaining) # Output: {"table": "history", "size_bytes": 102400, "partition_count": 5, "days_left": 30}
``` ```
- **Version**: - **Version**:
```bash ```bash
./zabbix_partitioning.py --version ./zabbix_partitioning.py --version
# Output: zabbix_partitioning.py 0.3.1-test
``` ```
### 10.2 Zabbix Template ### 10.2 Zabbix Template