|
|
b4d58df620
|
KV Cache: schema, allocator, pools, manager, append_swa kernel
Complete KV cache substrate for DSV4 inference:
schema.py: Per-layer cache shape derived from LayerSpec.
- CSA: 32 entries/block, 32 indexer entries, tail=3
- HCA: 1 entry/block, no indexer, tail=127
- SWA: no classical pool, no tail
- BLOCK_SIZE_ORIGINAL_TOKENS=128 (lcm of compression ratios)
- compute_block_budget() for allocator sizing
allocator.py: Fixed-size block free-list.
- GPU stack with pinned host top pointer
- acquire/release between graph captures only
- OOM raises on exhaustion
paged_cache.py: Per-layer classical KV storage.
- FP8 (uint8) for non-RoPE dims, BF16 for RoPE dims (paper 2.3.4)
- Per-entry inverse scale for FP8 dequant
- FP4 indexer keys for CSA layers (NVFP4 scheme)
- memory_bytes() tracking
state_cache.py: Per-layer SWA window + tail buffer.
- Ring buffer with position tracking (swa_head, swa_pos)
- CSA: dual streams (ka/za/kb/zb) for overlapping compression
- HCA: single stream (ka/za only)
- SWA: no tail buffer
- reset_slot() for request completion
handle.py: LayerCacheHandle — typed per-call view.
- write_swa(), read_swa_view(), read_classical_view(), read_indexer_view()
- No GPU allocation in acquire() — 0 bytes delta (cudagraph safe)
- SWAView/ClassicalView/IndexerView dataclasses for kernel signatures
manager.py: KVCacheManager — owns everything.
- Per-layer schema, pool, and allocator construction
- admit_request()/release_request() lifecycle
- allocate_block() for compression flush
- acquire() returns LayerCacheHandle (zero-alloc)
append_swa.cu: Native kernel for SWA writes.
- One block per token, 128 threads per block
- Warp-level amax reduction, BF16->FP8 E4M3 quantization
- Atomic ring buffer head increment
- FP8/BF16 split write + inv_scale + position metadata
- FP8 round-trip: <3.6% relative error
- RoPE half: exact match (no quantization)
All tests pass on B200:
- Schema correctness for CSA/HCA/SWA
- Allocator acquire/release/OOM
- Pool shapes match architecture spec
- Manager lifecycle (admit/release/recycle/exhaustion)
- Zero-alloc acquire() (cudagraph safe)
- append_swa kernel: positions, RoPE exact, FP8 quality, wrap-around, multi-request isolation
|
2026-05-22 00:08:38 +00:00 |
|
