biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

[Doc] neuron documentation update (#8671)

Browse Source 
Signed-off-by: omrishiv <327609+omrishiv@users.noreply.github.com>
This commit is contained in:
omrishiv
2024-09-20 15:04:37 -07:00
committed by GitHub
parent b4e4eda92e
commit 7c8566aa4f
2 changed files with 3 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/source/getting_started/neuron-installation.rst
View File

@@ -3,8 +3,8 @@
Installation with Neuron
========================
vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK.
At the moment Paged Attention is not supported in Neuron SDK, but naive continuous batching is supported in transformers-neuronx.
vLLM 0.3.3 onwards supports model inferencing and serving on AWS Trainium/Inferentia with Neuron SDK with continuous batching.
Paged Attention and Chunked Prefill are currently in development and will be available soon.
Data types currently supported in Neuron SDK are FP16 and BF16.
Requirements