biondizzle/vllm
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

Move query quantization to attention layer for Flashinfer & Triton. (#26534)

Browse Source 
Signed-off-by: adabeyta <aabeyta@redhat.com>
Signed-off-by: Adrian Abeyta <aabeyta@redhat.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
This commit is contained in:
Adrian Abeyta
2025-10-15 18:01:38 -05:00
committed by GitHub
parent e5b438a247
commit 0a9ef0cfce
6 changed files with 43 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
tests/compile/test_fusion_attn.py
View File

@@ -421,7 +421,9 @@ def test_attention_quant_pattern(
]
if any(attn_fusion_supported):
# Check quantization ops in the graph before and after fusion
test_backend.check_before_ops([QUANT_OPS[quant_key]], fully_replaced=True)
# Note: fully_replaced=False because query quant ops remain in graph.
# Only output quant ops are fused into attention.
test_backend.check_before_ops([QUANT_OPS[quant_key]], fully_replaced=False)
# access the underlying `AttnFusionPass` on the `LazyInitPass`
assert attn_pass.pass_.matched_count == sum(attn_fusion_supported)