**Status:** GEMM verified correct, SF remap verified correct, B layout verified correct, L2 slot_token cleaned up. vLLM still produces garbage. The checkpoint `input_scale` red herring is documented below. The bug remains unidentified.
Mike's code review suggested using the checkpoint's `input_scale` as the activation global scale instead of the dynamic `amax/(6*448)`. This was **wrong** and has been reverted (commit `79b9bec`).
The checkpoint `input_scale` is NOT the `input_global_scale = amax/(6*448)` normalization constant. They are different quantities:
-`input_global_scale` normalizes data to [0,1] before FP4 quantization
-`input_scale` is a calibration constant from the Quark quantization tool
The calibration amax for `input_scale = 2.86e-4` would be `2.86e-4 * 6 * 448 = 0.77`. Runtime hidden states have amax ~5-10. The `input_scale` was computed on a different data distribution (probably calibration data, not actual inference data).
**The correct use of `input_scale` is still unknown.** The Quark path computes `alpha = input_scale * weight_scale_2`, but this may assume BF16 activations (not FP4-quantized). Our CUTLASS kernel requires FP4 input, so we must quantize with the dynamic scale.
**Preserved for future use:** `_w13_input_scale` and `_w2_input_scale` are now saved in `finalize_weights` (not dropped) in case we need them for alpha computation later.
**Checkpoint input_scale values (layer 0, all experts identical):**
The 0.2 cosine against the Python BF16 dequantization reference was a RED HERRING. The reference is wrong, not the GEMM. 8+ iterations of SF remap changes all produced the same 0.2 cosine because it was never about the remap. **A wrong reference is worse than no reference.**
1.**E2M1 packing:** Confirmed correct — element 2j in low nibble, 2j+1 in high nibble. Suggested hardware oracle with `__nv_cvt_bfloat16raw2_to_fp4x2`.
2.**A RowMajor:** Confirmed correct — no micro-tiling for A.
3.**B ColumnMajor:** Byte transpose confirmed correct by test. Mike flagged theoretical concern about nibble-level transpose but our test passed.
4.**Alpha/global scale:** Mike suggested `alpha = input_scale * weight_scale_2` (from checkpoint). We tried it — wrong for activation normalization. The correct use of `input_scale` in our pipeline is still TBD.
5.**Gate/up correction:** Mathematically valid. `up_half *= up_weight_gs / gate_weight_gs` is equivalent to per-column alpha.
1.**Compare against BF16 model** — run the same prompt on a known-good implementation to see if the attention layers are working and only MoE is broken
2.**Check the vLLM model integration** — how does the MoE output get mixed with the residual? Is `hc_post` correct?
3.**Understand the Quark input_scale contract** — maybe we need to NOT quantize activations to FP4 and instead use BF16 input
4.**Add per-layer token output logging** — see which layer the tokens go off the rails
5.**Check o_a_proj BF16 handling** — it's kept in BF16 in the checkpoint, is it being processed correctly?
