biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Add full API docs and improve the UX of navigating them (#17485)

Browse Source 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
This commit is contained in:
Harry Mellor
2025-05-04 03:42:43 +01:00
committed by GitHub
parent 46fae69cf0
commit d6484ef3c3
101 changed files with 872 additions and 980 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
vllm/transformers_utils/tokenizer.py
View File

@@ -39,9 +39,9 @@ def decode_tokens(
) -> str:
"""
Backend-agnostic equivalent of HF's
:code:`tokenizer.decode(token_ids, ...)`.
`tokenizer.decode(token_ids, ...)`.
:code:`skip_special_tokens=None` means to use the backend's default
`skip_special_tokens=None` means to use the backend's default
settings.
"""
if skip_special_tokens is not None:
@@ -61,9 +61,9 @@ def encode_tokens(
) -> list[int]:
"""
Backend-agnostic equivalent of HF's
:code:`tokenizer.encode(text, ...)`.
`tokenizer.encode(text, ...)`.
:code:`add_special_tokens=None` means to use the backend's default
`add_special_tokens=None` means to use the backend's default
settings.
"""