biondizzle
07468031db
Previously the Dockerfile was copying the old version that had cudaMemPrefetchAsync completely removed, which caused cublasGemmEx crashes because model weights stayed in LPDDR and cuBLAS couldn't access them. This version prefetches allocations <2 GiB (model weights) to GPU but skips large allocations (KV cache).
VLLM images for GH200
Hosted here
docker login
# Alternative
# docker buildx build --platform linux/arm64 --memory=600g -t rajesh550/gh200-vllm:0.9.0.1 .
docker build --memory=450g --platform linux/arm64 -t rajesh550/gh200-vllm:0.11.1rc2 . 2>&1 | tee build.log
docker push rajesh550/gh200-vllm:0.11.1rc2