Files

biondizzle f63eed5cfd Purge INT4 references — expert weights are FP4 (E2M1), not INT4

All docs and scripts updated. Historical memory entries annotated.

2026-05-08 02:33:46 +00:00

2026-05-01

Inference Image Generation API - Multi-arch Docker Build

Project: ewr.vultrcr.com/vultrserverlessinference/inference-image-generation-api
Mike needed multi-arch Docker image (amd64 + arm64) for FLUX.1-dev inference API
amd64 builds natively on sandbox; arm64 was attempted via QEMU emulation through buildx
Cleared ~115G disk space on sandbox to make room for builds
First successful arm64 push via QEMU (bitsandbytes compiled from source for arm64 CUDA kernels)
Multi-arch manifest latest created pointing to both latest-amd64 and latest-arm64

FLUX.1-dev (23GB model) on 2x T4 was only using GPU 0 → OOM
Root cause: from_pipe() in generate endpoint was losing multi-GPU device map
Fix: removed from_pipe(), swap scheduler directly on shared_pipe
Changed device_map="auto" to device_map="balanced" with max_memory param per GPU
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True env var added
Bugs fixed along the way: total_mem → total_memory, "auto" → "balanced" (diffusers only supports balanced/cuda/cpu)

Buildx only has amd64 nodes — arm64 runs via QEMU emulation, extremely slow
bitsandbytes kernels.cu compile keeps OOMing during QEMU-emulated arm64 build
Previous successful build's cache was lost, recompile keeps failing
Options discussed: (1) register GH200 as remote buildx node for native arm64, (2) skip bitsandbytes on arm64, (3) reduce CUDA capabilities to just 90
Mike hasn't provided GH200 SSH details yet