107d62dd76
docs: update PERFORMANCE_AUDIT.md — Part 1 (P0-P3) landed, Part 2 KV cache next
2026-06-02 09:30:06 +00:00
040b2eb6e7
perf: P0/P1/P2 — fused SwiGLU for MoE+SE, eliminate per-call gsa fill
...
P0: Enable fused SwiGLU for all MoE instances (moe._fused_swiglu = True).
Eliminates ~8 BF16 kernel launches per MoE per token (gate/up split,
SiLU, clamp, elementwise multiply → single fused kernel launch).
P1: Enable fused SwiGLU for shared expert (SE):
- Added set_fused_swiglu() method to Nvfp4SharedExpert
- Added _run_l1_fused() using run_fused_swiglu_grouped_gemm (1-group)
- Interleave L1 weights at finalize time for fused kernel compatibility
- Fused kernel handles SwiGLU + clamp in registers, outputs BF16
P2: Eliminate per-call _gsa_buf.fill_() in Nvfp4Linear:
- _activation_global_scale is set once at warmup, never changes after
- Skip redundant fill_() via _gsa_buf_initialized flag
- Saves 244 CPU→GPU scalar fills per token (4 linears × 61 layers)
P3: Deferred (in-kernel RoPE fusion — kernel-side change, not single_shot)
2026-06-02 06:59:25 +00:00
106f42c93c
auto: pre-test commit
2026-06-01 23:01:34 +00:00
828ba73dff
Update PERFORMANCE_AUDIT.md: P0 complete, P2/P3/P5 done
2026-06-01 22:21:31 +00:00
c9b92cd840
Remove P1 from audit — multi-GPU layout is correct for the reference script
...
The single_shot is a reference for vLLM/SGLang integration. The layer-pipeline
sharding (gpu = li % NUM_GPUS) is the right pattern for this reference.
EP/TP sharding belongs in the actual vLLM integration, not here.
2026-06-01 21:07:59 +00:00
e0607c9e2f
P0: Add fused_amax_quantize.cu kernel + CUDA module loader with compile-once caching
...
- fused_amax_quantize.cu: Single kernel launch computes amax → gsa → NVFP4 quantize
Zero CPU-GPU syncs. gsa written to GPU buffer for downstream GEMM global_scale_a.
- dsv4/kernels/cuda/__init__.py: Module loader that compiles .cu once and caches.
Eliminates JIT recompilation overhead (was ~100ms per call, ~500x per token).
- P1 audit corrected: layer-pipe at batch=1 is wrong, but single-GPU doesn't fit
(800GB weights vs 192GB HBM). Correct fix is EP=8 for MoE + TP/replicate for dense.
2026-06-01 21:02:03 +00:00
d279965db4
Update PERFORMANCE_AUDIT.md: remove invalidated items, add WIP status
...
- Removed: RoPE 8x duplication (INVALIDATED), mHC BF16 bmm (INVALIDATED),
Router .float() cast (INVALIDATED)
- Added: WIP section documenting current session's work and status
- Added: Cardinal rule violation warning (must use test harness)
- Added: Compilation issues found (c10::, x.options())
- P0 marked PARTIAL: amax_gsa kernel written, GEMM path sync-free,
quantize kernel still needs .item()
- P4 marked DONE
- All other items NOT STARTED or DEFERRED
2026-06-01 20:55:44 +00:00
360f76b970
Performance audit fixes: eliminate CPU-GPU syncs
...
PERFORMANCE_AUDIT.md validation results:
1. Nvfp4Linear .item() sync (610/step) → FIXED: compute_amax_gsa_gpu kernel
2. MoE .item() sync (183/step) → FIXED: same kernel
3. SharedExpert .item() sync (122/step) → FIXED: same kernel
4. FMHA V clone → FIXED: V=K, transpose creates copy implicitly
5. torch.cuda.synchronize in moe_forward → FIXED: conditional on VERBOSE
6. RoPE 8x duplication → INVALIDATED: necessary for per-GPU HBM access
7. mHC BF16 bmm → INVALIDATED: 28K FLOPs, not a bottleneck
8. Router .float() cast → INVALIDATED: needed for FP32 topk, ~1μs
New files:
- dsv4/kernels/cuda/amax_gsa.cu: GPU-only amax→gsa kernel
- dsv4/ops/quantize.py: compute_amax_gsa_gpu() wrapper
Net effect: ~915 fewer CPU-GPU syncs per decode step
Remaining syncs: ~10 per layer (quantize kernel parameter) + diagnostics
2026-06-01 20:40:19 +00:00
