Change: Adjusted README to the new changes. Added runtime parameters.
This commit is contained in:
@@ -47,8 +47,16 @@ The script manages MySQL table partitions based on time (Range Partitioning on t
|
|||||||
---
|
---
|
||||||
|
|
||||||
## 3. Configuration
|
## 3. Configuration
|
||||||
Edit `/etc/zabbix/zabbix_partitioning.conf`:
|
The preferred way to create the configuration file is using the **Interactive Wizard**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
/opt/zabbix_partitioning/zabbix_partitioning.py --wizard
|
||||||
|
```
|
||||||
|
|
||||||
|
Follow the on-screen prompts to configure your database connection, retention periods, and other settings. The wizard will generate the YAML configuration file for you.
|
||||||
|
|
||||||
|
### Automatic Config Structure (Reference)
|
||||||
|
If you prefer to edit manually `/etc/zabbix/zabbix_partitioning.conf`:
|
||||||
```yaml
|
```yaml
|
||||||
database:
|
database:
|
||||||
host: localhost
|
host: localhost
|
||||||
@@ -56,6 +64,7 @@ database:
|
|||||||
passwd: YOUR_PASSWORD
|
passwd: YOUR_PASSWORD
|
||||||
db: zabbix
|
db: zabbix
|
||||||
# port: 3306 # Optional, default is 3306
|
# port: 3306 # Optional, default is 3306
|
||||||
|
# socket: /var/run/mysqld/mysqld.sock # Overrides host if present
|
||||||
|
|
||||||
partitions:
|
partitions:
|
||||||
daily:
|
daily:
|
||||||
@@ -74,9 +83,7 @@ partitions:
|
|||||||
- **`yearly`**: Partitions are created for each year.
|
- **`yearly`**: Partitions are created for each year.
|
||||||
- Retention Format: `14d` (days), `12w` (weeks), `12m` (months), `1y` (years).
|
- Retention Format: `14d` (days), `12w` (weeks), `12m` (months), `1y` (years).
|
||||||
|
|
||||||
- **`initial_partitioning_start`**: Controls how the very FIRST partition is determined during initialization (`--init` mode).
|
- **`replicate_sql`**: Controls MySQL Binary Logging for partitioning commands.
|
||||||
- `db_min`: (Default) Queries the table for the oldest record (`MIN(clock)`). Accurate but **slow** on large tables.
|
|
||||||
- `retention`: (Recommended for large DBs) Skips the query. Calculates the start date as `Now - Retention Period`. Creates a single `p_archive` partition for all data older than that date.
|
|
||||||
|
|
||||||
- **`premake`**: Number of future partitions to create in advance.
|
- **`premake`**: Number of future partitions to create in advance.
|
||||||
- Default: `10`. Ensures you have a buffer if the script fails to run for a few days.
|
- Default: `10`. Ensures you have a buffer if the script fails to run for a few days.
|
||||||
@@ -90,7 +97,25 @@ partitions:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Zabbix Preparation (CRITICAL)
|
---
|
||||||
|
|
||||||
|
## 4. Runtime Parameters
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|---|---|
|
||||||
|
| `-c`, `--config FILE` | Path to configuration file (Default: `/etc/zabbix/zabbix_partitioning.conf`) |
|
||||||
|
| `-i`, `--init` | Initialize partitions (converts tables). |
|
||||||
|
| `--fast-init` | Skip slow table scan during initialization. Starts from retention period. |
|
||||||
|
| `--wizard` | Launch interactive configuration wizard. |
|
||||||
|
| `-r`, `--dry-run` | Simulate queries without executing. Logs expected actions (Safe mode). |
|
||||||
|
| `-v`, `--verbose` | Enable debug logging (DEBUG level). |
|
||||||
|
| `--stats TABLE` | Output JSON statistics (Size, Count, Days Left) for a specific table. |
|
||||||
|
| `--discovery` | Output Zabbix Low-Level Discovery (LLD) JSON. |
|
||||||
|
| `-V`, `--version` | Show script version. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Zabbix Preparation (CRITICAL)
|
||||||
Before partitioning, you **must disable** Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts.
|
Before partitioning, you **must disable** Zabbix's internal housekeeping for the tables you intend to partition. If you don't, Zabbix will try to delete individual rows while the script tries to drop partitions, causing conflicts.
|
||||||
|
|
||||||
1. Log in to Zabbix Web Interface.
|
1. Log in to Zabbix Web Interface.
|
||||||
@@ -102,24 +127,35 @@ Before partitioning, you **must disable** Zabbix's internal housekeeping for the
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Initialization
|
## 6. Initialization
|
||||||
This step converts existing standard tables into partitioned tables.
|
This step converts existing standard tables into partitioned tables.
|
||||||
|
|
||||||
1. **Dry Run** (Verify what will happen):
|
1. **Dry Run** (Verify what will happen):
|
||||||
```bash
|
```bash
|
||||||
/opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run
|
/opt/zabbix_partitioning/zabbix_partitioning.py --init --dry-run
|
||||||
```
|
```
|
||||||
*Check the output for any errors.*
|
|
||||||
|
|
||||||
2. **Execute Initialization**:
|
2. **Execute Initialization**:
|
||||||
|
There are two strategies for initialization:
|
||||||
|
|
||||||
|
**A. Standard Initialization (Default)**:
|
||||||
|
Scans the database to find the oldest record (`MIN(clock)`) and creates partitions from that point forward.
|
||||||
|
*Best for smaller databases or when you need to retain ALL existing data.*
|
||||||
```bash
|
```bash
|
||||||
/opt/zabbix_partitioning/zabbix_partitioning.py --init
|
/opt/zabbix_partitioning/zabbix_partitioning.py --init
|
||||||
```
|
```
|
||||||
*This may take time depending on table size.*
|
|
||||||
|
**B. Fast Initialization (`--fast-init`)**:
|
||||||
|
Skips the slow table scan. It calculates the start date based on your configured **Retention Period** (e.g., `Now - 365d`).
|
||||||
|
It creates a single catch-all `p_archive` partition for all data older than the retention start date, then creates granular partitions forward.
|
||||||
|
*Recommended for large databases to avoid long table locks/scans.*
|
||||||
|
```bash
|
||||||
|
/opt/zabbix_partitioning/zabbix_partitioning.py --init --fast-init
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6. Automation (Cron Job)
|
## 7. Automation (Cron Job)
|
||||||
Set up a daily cron job to create new partitions and remove old ones.
|
Set up a daily cron job to create new partitions and remove old ones.
|
||||||
|
|
||||||
1. Open crontab:
|
1. Open crontab:
|
||||||
@@ -133,7 +169,7 @@ Set up a daily cron job to create new partitions and remove old ones.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 7. Automation (Systemd Timer) — Recommended
|
## 8. Automation (Systemd Timer) — Recommended
|
||||||
Alternatively, use systemd timers for more robust scheduling and logging.
|
Alternatively, use systemd timers for more robust scheduling and logging.
|
||||||
|
|
||||||
1. **Create Service Unit** (`/etc/systemd/system/zabbix-partitioning.service`):
|
1. **Create Service Unit** (`/etc/systemd/system/zabbix-partitioning.service`):
|
||||||
@@ -176,12 +212,12 @@ Alternatively, use systemd timers for more robust scheduling and logging.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 8. Troubleshooting
|
## 9. Troubleshooting
|
||||||
- **Connection Refused**: Check `host`, `port` in config. Ensure MySQL is running.
|
- **Connection Refused**: Check `host`, `port` in config. Ensure MySQL is running.
|
||||||
- **Access Denied (1227)**: The DB user needs `SUPER` privileges to disable binary logging (`replicate_sql: False`). Either grant the privilege or set `replicate_sql: True` (if replication load is acceptable).
|
- **Access Denied (1227)**: The DB user needs `SUPER` privileges to disable binary logging (`replicate_sql: False`). Either grant the privilege or set `replicate_sql: True` (if replication load is acceptable).
|
||||||
- **Primary Key Error**: "Primary Key does not include 'clock'". The table cannot be partitioned by range on `clock` without schema changes. Remove it from config.
|
- **Primary Key Error**: "Primary Key does not include 'clock'". The table cannot be partitioned by range on `clock` without schema changes. Remove it from config.
|
||||||
|
|
||||||
## 9. Docker Usage
|
## 10. Docker Usage
|
||||||
|
|
||||||
You can run the partitioning script as a stateless Docker container. This is ideal for Kubernetes CronJobs or environments where you don't want to manage Python dependencies on the host.
|
You can run the partitioning script as a stateless Docker container. This is ideal for Kubernetes CronJobs or environments where you don't want to manage Python dependencies on the host.
|
||||||
|
|
||||||
@@ -267,8 +303,8 @@ docker run --rm \
|
|||||||
| `RETENTION_TRENDS` | 365d | Retention for `trends*` tables |
|
| `RETENTION_TRENDS` | 365d | Retention for `trends*` tables |
|
||||||
| `RETENTION_AUDIT` | 365d | Retention for `auditlog` (if enabled) |
|
| `RETENTION_AUDIT` | 365d | Retention for `auditlog` (if enabled) |
|
||||||
| `ENABLE_AUDITLOG_PARTITIONING` | false | Set to `true` to partition `auditlog` |
|
| `ENABLE_AUDITLOG_PARTITIONING` | false | Set to `true` to partition `auditlog` |
|
||||||
| `RUN_MODE` | maintenance | `init`, `maintenance`, `dry-run`, `discovery`, `check` |
|
| `RUN_MODE` | maintenance | `init`, `maintenance`, `dry-run`, `discovery`, `stats` |
|
||||||
| `CHECK_TARGET` | - | Required if `RUN_MODE=check`. Table name to check (e.g. `history`). |
|
| `CHECK_TARGET` | - | Required if `RUN_MODE=stats`. Table name to check (e.g. `history`). |
|
||||||
| `PARTITION_DAILY_[TABLE]` | - | Custom daily retention (e.g., `PARTITION_DAILY_mytable=30d`) |
|
| `PARTITION_DAILY_[TABLE]` | - | Custom daily retention (e.g., `PARTITION_DAILY_mytable=30d`) |
|
||||||
| `PARTITION_WEEKLY_[TABLE]` | - | Custom weekly retention |
|
| `PARTITION_WEEKLY_[TABLE]` | - | Custom weekly retention |
|
||||||
| `PARTITION_MONTHLY_[TABLE]` | - | Custom monthly retention |
|
| `PARTITION_MONTHLY_[TABLE]` | - | Custom monthly retention |
|
||||||
@@ -282,36 +318,37 @@ docker run --rm \
|
|||||||
zabbix-partitioning
|
zabbix-partitioning
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Scenario G: Monitoring (Health Check)
|
#### Scenario G: Monitoring (Stats)
|
||||||
Check days remaining for a specific table (e.g., `history`). Returns integer days.
|
Get detailed JSON statistics for a specific table.
|
||||||
```bash
|
```bash
|
||||||
docker run --rm \
|
docker run --rm \
|
||||||
-e DB_HOST=zabbix-db \
|
-e DB_HOST=zabbix-db \
|
||||||
-e RUN_MODE=check \
|
-e RUN_MODE=stats \
|
||||||
-e CHECK_TARGET=history \
|
-e CHECK_TARGET=history \
|
||||||
zabbix-partitioning
|
zabbix-partitioning
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10. Monitoring
|
## 11. Monitoring
|
||||||
The script includes built-in features for monitoring the health of your partitions via Zabbix.
|
The script includes built-in features for monitoring the health of your partitions via Zabbix.
|
||||||
|
|
||||||
### 10.1 CLI Usage
|
### 10.1 CLI Usage
|
||||||
- **Discovery (LLD)**:
|
- **Discovery (LLD)**:
|
||||||
|
Output the list of configured tables for Zabbix Discovery rules.
|
||||||
```bash
|
```bash
|
||||||
./zabbix_partitioning.py --discovery
|
./zabbix_partitioning.py --discovery
|
||||||
# Output: [{"{#TABLE}": "history", "{#PERIOD}": "daily"}, ...]
|
# Output: [{"{#TABLE}": "history", "{#PERIOD}": "daily"}, ...]
|
||||||
```
|
```
|
||||||
- **Check Days**:
|
- **Statistics (JSON)**:
|
||||||
|
Get detailed stats (Size, Partition Count, Days Remaining).
|
||||||
```bash
|
```bash
|
||||||
./zabbix_partitioning.py --check-days history
|
./zabbix_partitioning.py --stats history
|
||||||
# Output: 30 (integer days remaining)
|
# Output: {"table": "history", "size_bytes": 102400, "partition_count": 5, "days_left": 30}
|
||||||
```
|
```
|
||||||
- **Version**:
|
- **Version**:
|
||||||
```bash
|
```bash
|
||||||
./zabbix_partitioning.py --version
|
./zabbix_partitioning.py --version
|
||||||
# Output: zabbix_partitioning.py 0.3.1-test
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 10.2 Zabbix Template
|
### 10.2 Zabbix Template
|
||||||
|
|||||||
Reference in New Issue
Block a user