requirements.txt

ninja  # For faster builds.
psutil
ray >= 2.9
sentencepiece  # Required for LLaMA tokenizer.
numpy
torch == 2.1.2
transformers >= 4.37.0 # Required for Qwen2
xformers == 0.0.23.post1  # Required for CUDA 12.1.
fastapi
uvicorn[standard]
pydantic >= 2.0  # Required for OpenAI server.
aioprometheus[starlette]
pynvml == 11.5.0
triton >= 2.1.0
cupy-cuda12x == 12.3.0  # Required for CUDA graphs. CUDA 11.8 users should install cupy-cuda11x instead.
Specify python package dependencies in requirements.txt (#78) 2023-05-07 16:30:43 -07:00			`ninja # For faster builds.`
			`psutil`
Update Ray version requirements (#2636) 2024-01-28 14:27:22 -08:00			`ray >= 2.9`
Specify python package dependencies in requirements.txt (#78) 2023-05-07 16:30:43 -07:00			`sentencepiece # Required for LLaMA tokenizer.`
			`numpy`
Pin PyTorch & xformers versions (#2155) 2023-12-17 01:46:54 -08:00			`torch == 2.1.2`
Add qwen2 (#2495) 2024-01-23 06:34:21 +08:00			`transformers >= 4.37.0 # Required for Qwen2`
[Minor] Fix xformers version (#2158) 2023-12-17 02:28:02 -08:00			`xformers == 0.0.23.post1 # Required for CUDA 12.1.`
Specify python package dependencies in requirements.txt (#78) 2023-05-07 16:30:43 -07:00			`fastapi`
Use standard extras for uvicorn (#1166) 2023-09-27 21:41:36 -03:00			`uvicorn[standard]`
migrate pydantic from v1 to v2 (#2531) 2024-01-22 01:05:56 +01:00			`pydantic >= 2.0 # Required for OpenAI server.`
Add Production Metrics in Prometheus format (#1890) 2023-12-02 16:37:44 -08:00			`aioprometheus[starlette]`
Implement custom all reduce kernels (#2192) 2024-01-28 04:46:35 +08:00			`pynvml == 11.5.0`
Require triton >= 2.1.0 (#2746) Co-authored-by: yangrui1 <yangrui@lanjingren.com> 2024-02-05 15:07:36 +08:00			`triton >= 2.1.0`
Use CuPy for CUDA graphs (#2811) 2024-02-13 11:32:06 -08:00			`cupy-cuda12x == 12.3.0 # Required for CUDA graphs. CUDA 11.8 users should install cupy-cuda11x instead.`