88 lines
2.0 KiB
Markdown
88 lines
2.0 KiB
Markdown
|
# M3DB Backfill Tools
|
||
|
|
|
||
|
|
Scripts to backfill historical metrics from Mimir to M3DB.
|
||
|
|
|
||
|
|
## Prerequisites
|
||
|
|
|
||
|
|
Copy `.env` and set credentials:
|
||
|
|
```bash
|
||
|
|
cp .env.example .env
|
||
|
|
# Edit .env with your credentials
|
||
|
|
```
|
||
|
|
|
||
|
|
Required environment variables:
|
||
|
|
- `MIMIR_USERNAME` - Mimir API username
|
||
|
|
- `MIMIR_PASSWORD` - Mimir API password
|
||
|
|
|
||
|
|
## Files
|
||
|
|
|
||
|
|
| File | Purpose |
|
||
|
|
|------|---------|
|
||
|
|
| `backfill.py` | Main backfill script — pulls from Mimir, writes to M3DB |
|
||
|
|
| `backfill-gap.py` | Lightweight script for filling specific time gaps |
|
||
|
|
| `backfill-pod.yaml` | Kubernetes pod manifest for running backfill |
|
||
|
|
| `BACKFILL_RUNBOOK.md` | Detailed runbook with lessons learned |
|
||
|
|
| `test-metrics.py` | Test script for verifying data flow |
|
||
|
|
|
||
|
|
## Quick Usage
|
||
|
|
|
||
|
|
### Full Backfill
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Edit START_TS and END_TS in backfill.py first
|
||
|
|
# Format: Unix timestamps (seconds since epoch)
|
||
|
|
|
||
|
|
# Create configmap and run
|
||
|
|
kubectl create configmap backfill-script --from-file=backfill.py=backfill.py -n m3db
|
||
|
|
kubectl apply -f backfill-pod.yaml
|
||
|
|
kubectl logs -f backfill -n m3db
|
||
|
|
```
|
||
|
|
|
||
|
|
### Fill a Specific Gap
|
||
|
|
|
||
|
|
Edit `backfill-gap.py` to set your time range:
|
||
|
|
|
||
|
|
```python
|
||
|
|
START_TS = 1774175400 # Unix timestamp
|
||
|
|
END_TS = 1774243800 # Unix timestamp
|
||
|
|
```
|
||
|
|
|
||
|
|
Then run:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
kubectl create configmap backfill-gap-script --from-file=backfill-gap.py=backfill-gap.py -n m3db
|
||
|
|
kubectl apply -f backfill-gap-pod.yaml
|
||
|
|
kubectl logs -f backfill-gap -n m3db
|
||
|
|
```
|
||
|
|
|
||
|
|
## Timestamp Helpers
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Convert date to Unix timestamp
|
||
|
|
date -u -d '2026-03-22 10:30:00' +%s
|
||
|
|
|
||
|
|
# Convert Unix timestamp to date
|
||
|
|
date -u -d @1774175400
|
||
|
|
```
|
||
|
|
|
||
|
|
## Requirements
|
||
|
|
|
||
|
|
- Mimir credentials (in script)
|
||
|
|
- M3DB coordinator endpoint: `http://m3coordinator.m3db.svc.cluster.local:7201`
|
||
|
|
- `bufferPast` must be >= the age of data you're backfilling (currently 21 days)
|
||
|
|
|
||
|
|
## Metrics Backfilled
|
||
|
|
|
||
|
|
- `vllm:prompt_tokens_total`
|
||
|
|
- `vllm:generation_tokens_total`
|
||
|
|
- `DCGM_FI_DEV_GPU_UTIL`
|
||
|
|
|
||
|
|
## Cleanup
|
||
|
|
|
||
|
|
After backfill completes:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
kubectl delete pod backfill -n m3db
|
||
|
|
kubectl delete configmap backfill-script -n m3db
|
||
|
|
```
|