[core] cudagraph output with tensor weak reference (#9724)

Signed-off-by: youkaichao <youkaichao@gmail.com>
This commit is contained in:
youkaichao
2024-10-27 00:19:28 -07:00
committed by GitHub
parent 67a6882da4
commit 8549c82660
4 changed files with 50 additions and 28 deletions

View File

@@ -18,6 +18,9 @@
TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
// vLLM custom ops
ops.def("weak_ref_tensor(Tensor input) -> Tensor");
ops.impl("weak_ref_tensor", torch::kCUDA, &weak_ref_tensor);
// Attention ops
// Compute the attention between an input query and the cached
// keys/values using PagedAttention.