# 2026-05-01

## Inference Image Generation API - Multi-arch Docker Build

- Project: `ewr.vultrcr.com/vultrserverlessinference/inference-image-generation-api`
- Mike needed multi-arch Docker image (amd64 + arm64) for FLUX.1-dev inference API
- amd64 builds natively on sandbox; arm64 was attempted via QEMU emulation through buildx
- Cleared ~115G disk space on sandbox to make room for builds
- First successful arm64 push via QEMU (bitsandbytes compiled from source for arm64 CUDA kernels)
- Multi-arch manifest `latest` created pointing to both `latest-amd64` and `latest-arm64`

### CUDA OOM Fix on T4 (2x 16GB)
- FLUX.1-dev (23GB model) on 2x T4 was only using GPU 0 → OOM
- Root cause: `from_pipe()` in generate endpoint was losing multi-GPU device map
- Fix: removed `from_pipe()`, swap scheduler directly on `shared_pipe`
- Changed `device_map="auto"` to `device_map="balanced"` with `max_memory` param per GPU
- `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` env var added
- Bugs fixed along the way: `total_mem` → `total_memory`, `"auto"` → `"balanced"` (diffusers only supports balanced/cuda/cpu)

### Arm64 Build Issue (unresolved)
- Buildx only has amd64 nodes — arm64 runs via QEMU emulation, extremely slow
- bitsandbytes `kernels.cu` compile keeps OOMing during QEMU-emulated arm64 build
- Previous successful build's cache was lost, recompile keeps failing
- Options discussed: (1) register GH200 as remote buildx node for native arm64, (2) skip bitsandbytes on arm64, (3) reduce CUDA capabilities to just `90`
- Mike hasn't provided GH200 SSH details yet