Migrate docs from Sphinx to MkDocs (#18145)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
39
docs/deployment/integrations/llamastack.md
Normal file
39
docs/deployment/integrations/llamastack.md
Normal file
@@ -0,0 +1,39 @@
|
||||
---
|
||||
title: Llama Stack
|
||||
---
|
||||
[](){ #deployment-llamastack }
|
||||
|
||||
vLLM is also available via [Llama Stack](https://github.com/meta-llama/llama-stack) .
|
||||
|
||||
To install Llama Stack, run
|
||||
|
||||
```console
|
||||
pip install llama-stack -q
|
||||
```
|
||||
|
||||
## Inference using OpenAI Compatible API
|
||||
|
||||
Then start Llama Stack server pointing to your vLLM server with the following configuration:
|
||||
|
||||
```yaml
|
||||
inference:
|
||||
- provider_id: vllm0
|
||||
provider_type: remote::vllm
|
||||
config:
|
||||
url: http://127.0.0.1:8000
|
||||
```
|
||||
|
||||
Please refer to [this guide](https://llama-stack.readthedocs.io/en/latest/distributions/self_hosted_distro/remote-vllm.html) for more details on this remote vLLM provider.
|
||||
|
||||
## Inference via Embedded vLLM
|
||||
|
||||
An [inline vLLM provider](https://github.com/meta-llama/llama-stack/tree/main/llama_stack/providers/inline/inference/vllm)
|
||||
is also available. This is a sample of configuration using that method:
|
||||
|
||||
```yaml
|
||||
inference
|
||||
- provider_type: vllm
|
||||
config:
|
||||
model: Llama3.1-8B-Instruct
|
||||
tensor_parallel_size: 4
|
||||
```
|
||||
Reference in New Issue
Block a user