|
|
4f698baa5d
|
Production fused CUDA sampler + decode loop optimizations
- Add dsv4/kernels/cuda/sampler.cu: fused temperature + repetition penalty
+ top-k + top-p (nucleus) sampling, single kernel launch, zero CPU syncs
- Add dsv4/model/sampler.py: CUDASampler wrapper + PyTorch reference
- Update single_shot_inference.py:
- Use CUDASampler for non-greedy decoding (temperature=0.6, top_k=50, top_p=0.95)
- Pre-allocate decode buffers (no per-step torch.tensor allocation)
- Track thinking tokens (128821/128822) — not garbage for reasoning model
- Reduce diagnostic CPU syncs (top-5 every 5 steps, NaN check every 20)
- Add --top-k and --top-p CLI args
- Default: temperature=0.6 (was 0.0 greedy), rep_penalty=1.1 (was 1.2)
|
2026-06-01 20:29:57 +00:00 |
|
|
|
3fb3c925af
|
Restructure: cutedsl/ -> dsv4/ with proper layering
- Split bridge.py -> ops/quantize.py, ops/layouts.py, ops/gemm_runner.py
- Renamed classes: CuTeDSLNvfp4Linear -> Nvfp4Linear, etc.
- Moved kernel code to dsv4/kernels/ (gemm, attention, compressor, decode, cuda)
- Moved PyTorch bridges to dsv4/ops/
- Moved nn.Module layers to dsv4layers/
- Moved reference implementations to dsv4/reference/
- Moved vendored CUTLASS code to vendored/
- Archived ~190 debug tests to tests/archive/
- Kept ~15 canonical tests in tests/unit/
- Updated all import paths
- Added stubs for future components (model/, cache/, loader/)
- Updated pyproject.toml: dsv4-inference package name
|
2026-05-21 17:30:44 +00:00 |
|
