42 lines
1.2 KiB
Markdown
42 lines
1.2 KiB
Markdown
|
# --8<-- [start:installation]
|
||
|
|
|
||
|
|
vLLM has been adapted to work on ARM64 CPUs with NEON support, leveraging the CPU backend initially developed for the x86 platform.
|
||
|
|
|
||
|
|
ARM CPU backend currently supports Float32, FP16 and BFloat16 datatypes.
|
||
|
|
|
||
|
|
!!! warning
|
||
|
|
There are no pre-built wheels or images for this device, so you must build vLLM from source.
|
||
|
|
|
||
|
|
# --8<-- [end:installation]
|
||
|
|
# --8<-- [start:requirements]
|
||
|
|
|
||
|
|
- OS: Linux
|
||
|
|
- Compiler: `gcc/g++ >= 12.3.0` (optional, recommended)
|
||
|
|
- Instruction Set Architecture (ISA): NEON support is required
|
||
|
|
|
||
|
|
# --8<-- [end:requirements]
|
||
|
|
# --8<-- [start:set-up-using-python]
|
||
|
|
|
||
|
|
# --8<-- [end:set-up-using-python]
|
||
|
|
# --8<-- [start:pre-built-wheels]
|
||
|
|
|
||
|
|
# --8<-- [end:pre-built-wheels]
|
||
|
|
# --8<-- [start:build-wheel-from-source]
|
||
|
|
|
||
|
|
--8<-- "docs/getting_started/installation/cpu/cpu/build.inc.md"
|
||
|
|
|
||
|
|
Testing has been conducted on AWS Graviton3 instances for compatibility.
|
||
|
|
|
||
|
|
# --8<-- [end:build-wheel-from-source]
|
||
|
|
# --8<-- [start:set-up-using-docker]
|
||
|
|
|
||
|
|
# --8<-- [end:set-up-using-docker]
|
||
|
|
# --8<-- [start:pre-built-images]
|
||
|
|
|
||
|
|
# --8<-- [end:pre-built-images]
|
||
|
|
# --8<-- [start:build-image-from-source]
|
||
|
|
|
||
|
|
# --8<-- [end:build-image-from-source]
|
||
|
|
# --8<-- [start:extra-information]
|
||
|
|
# --8<-- [end:extra-information]
|