|
|
faf92b30ad
|
E1: Wire LayerCacheHandle gather methods + CUDA gather kernels
- gather_compressed_kv: CSA top-k gather via existing gather_kv.cu
- gather_all_compressed_kv: HCA dense gather via new gather_all_compressed_kernel
- gather_swa_kv: SWA ring buffer gather via new gather_swa_kernel
- Added gather_swa.cu with both SWA + all-compressed gather kernels
- Added gather.py Python wrapper (torch.utils.cpp_extension JIT)
- Updated handle.py: added schema field, num_query_heads/head_dim properties
- Updated manager.py: passes schema + num_query_heads to handle
All gather kernels: FP8→BF16 dequant + BF16 RoPE concat in single launch.
Output: dense BF16 tensors ready for FMHA consumption.
|
2026-05-30 21:09:21 +00:00 |
|
|
|
b4d58df620
|
KV Cache: schema, allocator, pools, manager, append_swa kernel
Complete KV cache substrate for DSV4 inference:
schema.py: Per-layer cache shape derived from LayerSpec.
- CSA: 32 entries/block, 32 indexer entries, tail=3
- HCA: 1 entry/block, no indexer, tail=127
- SWA: no classical pool, no tail
- BLOCK_SIZE_ORIGINAL_TOKENS=128 (lcm of compression ratios)
- compute_block_budget() for allocator sizing
allocator.py: Fixed-size block free-list.
- GPU stack with pinned host top pointer
- acquire/release between graph captures only
- OOM raises on exhaustion
paged_cache.py: Per-layer classical KV storage.
- FP8 (uint8) for non-RoPE dims, BF16 for RoPE dims (paper 2.3.4)
- Per-entry inverse scale for FP8 dequant
- FP4 indexer keys for CSA layers (NVFP4 scheme)
- memory_bytes() tracking
state_cache.py: Per-layer SWA window + tail buffer.
- Ring buffer with position tracking (swa_head, swa_pos)
- CSA: dual streams (ka/za/kb/zb) for overlapping compression
- HCA: single stream (ka/za only)
- SWA: no tail buffer
- reset_slot() for request completion
handle.py: LayerCacheHandle — typed per-call view.
- write_swa(), read_swa_view(), read_classical_view(), read_indexer_view()
- No GPU allocation in acquire() — 0 bytes delta (cudagraph safe)
- SWAView/ClassicalView/IndexerView dataclasses for kernel signatures
manager.py: KVCacheManager — owns everything.
- Per-layer schema, pool, and allocator construction
- admit_request()/release_request() lifecycle
- allocate_block() for compression flush
- acquire() returns LayerCacheHandle (zero-alloc)
append_swa.cu: Native kernel for SWA writes.
- One block per token, 128 threads per block
- Warp-level amax reduction, BF16->FP8 E4M3 quantization
- Atomic ring buffer head increment
- FP8/BF16 split write + inv_scale + position metadata
- FP8 round-trip: <3.6% relative error
- RoPE half: exact match (no quantization)
All tests pass on B200:
- Schema correctness for CSA/HCA/SWA
- Allocator acquire/release/OOM
- Pool shapes match architecture spec
- Manager lifecycle (admit/release/recycle/exhaustion)
- Zero-alloc acquire() (cudagraph safe)
- append_swa kernel: positions, RoPE exact, FP8 quality, wrap-around, multi-request isolation
|
2026-05-22 00:08:38 +00:00 |
|
