Migrate docs from Sphinx to MkDocs (#18145)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
16
docs/models/extensions/tensorizer.md
Normal file
16
docs/models/extensions/tensorizer.md
Normal file
@@ -0,0 +1,16 @@
|
||||
---
|
||||
title: Loading models with CoreWeave's Tensorizer
|
||||
---
|
||||
[](){ #tensorizer }
|
||||
|
||||
vLLM supports loading models with [CoreWeave's Tensorizer](https://docs.coreweave.com/coreweave-machine-learning-and-ai/inference/tensorizer).
|
||||
vLLM model tensors that have been serialized to disk, an HTTP/HTTPS endpoint, or S3 endpoint can be deserialized
|
||||
at runtime extremely quickly directly to the GPU, resulting in significantly
|
||||
shorter Pod startup times and CPU memory usage. Tensor encryption is also supported.
|
||||
|
||||
For more information on CoreWeave's Tensorizer, please refer to
|
||||
[CoreWeave's Tensorizer documentation](https://github.com/coreweave/tensorizer). For more information on serializing a vLLM model, as well a general usage guide to using Tensorizer with vLLM, see
|
||||
the [vLLM example script](https://docs.vllm.ai/en/latest/getting_started/examples/tensorize_vllm_model.html).
|
||||
|
||||
!!! note
|
||||
Note that to use this feature you will need to install `tensorizer` by running `pip install vllm[tensorizer]`.
|
||||
Reference in New Issue
Block a user