[ROCm] Enable bitsandbytes quantization support on ROCm (#34688)
Signed-off-by: badaoui <abdennacerbadaoui0@gmail.com>
This commit is contained in:
committed by
GitHub
parent
2aab2bb543
commit
8dc8a99b56
@@ -7,7 +7,7 @@ Compared to other quantization methods, BitsAndBytes eliminates the need for cal
|
||||
Below are the steps to utilize BitsAndBytes with vLLM.
|
||||
|
||||
```bash
|
||||
pip install bitsandbytes>=0.46.1
|
||||
pip install bitsandbytes>=0.49.2
|
||||
```
|
||||
|
||||
vLLM reads the model's config file and supports both in-flight quantization and pre-quantized checkpoint.
|
||||
|
||||
Reference in New Issue
Block a user