M3DB Backfill Tools

Scripts to backfill historical metrics from Mimir to M3DB.

Prerequisites

Copy .env and set credentials:

cp .env.example .env
# Edit .env with your credentials

Required environment variables:

MIMIR_USERNAME - Mimir API username
MIMIR_PASSWORD - Mimir API password

Files

File	Purpose
`backfill.py`	Main backfill script — pulls from Mimir, writes to M3DB
`backfill-gap.py`	Lightweight script for filling specific time gaps
`backfill-pod.yaml`	Kubernetes pod manifest for running backfill
`BACKFILL_RUNBOOK.md`	Detailed runbook with lessons learned
`test-metrics.py`	Test script for verifying data flow

Quick Usage

Full Backfill

# Edit START_TS and END_TS in backfill.py first
# Format: Unix timestamps (seconds since epoch)

# Create configmap and run
kubectl create configmap backfill-script --from-file=backfill.py=backfill.py -n m3db
kubectl apply -f backfill-pod.yaml
kubectl logs -f backfill -n m3db

Fill a Specific Gap

Edit backfill-gap.py to set your time range:

START_TS = 1774175400  # Unix timestamp
END_TS = 1774243800    # Unix timestamp

Then run:

kubectl create configmap backfill-gap-script --from-file=backfill-gap.py=backfill-gap.py -n m3db
kubectl apply -f backfill-gap-pod.yaml
kubectl logs -f backfill-gap -n m3db

Timestamp Helpers

# Convert date to Unix timestamp
date -u -d '2026-03-22 10:30:00' +%s

# Convert Unix timestamp to date
date -u -d @1774175400

Requirements

Mimir credentials (in script)
M3DB coordinator endpoint: http://m3coordinator.m3db.svc.cluster.local:7201
bufferPast must be >= the age of data you're backfilling (currently 21 days)

Metrics Backfilled

vllm:prompt_tokens_total
vllm:generation_tokens_total
DCGM_FI_DEV_GPU_UTIL

Cleanup

After backfill completes:

kubectl delete pod backfill -n m3db
kubectl delete configmap backfill-script -n m3db

2.0 KiB Raw Blame History