deepseek-v4-quant/memory/2026-05-06-lmcache-speculative-decoding.md

Session: 2026-05-06 16:22:53 UTC

• Session Key: agent:main:main
• Session ID: 6e5fde2d-305e-45b8-9f0c-65b531130bd4
• Source: webchat

Conversation Summary

user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Wed 2026-05-06 16:17 UTC] Take a gander at /home/openclaw/dev/vllm-with-lmcache

Notice in lmcache-config-glm-51.yaml i have save_decode_cache: false ?

Would the prevent the storing of tokens that were generated during speculative decode? assistant: Good question. Short answer: no, save_decode_cache: false does NOT prevent storing spec-decode tokens specifically.

save_decode_cache controls whether KV cache from the decode phase (i.e., autoregressive token generation after prefill) gets saved to the cache. When it's false, LMCache skips persisting KV blocks that are computed during normal decode steps.

Here's the nuance with speculative decoding:

• Spec-decode generates candidate tokens, then verifies them against the target model. The verified (accepted) tokens become part of the actual decode sequence.
• The KV cache for those accepted spec tokens is computed during what vLLM considers the decode phase — they're just decode steps that happen to be verified in bulk rather than one at a time.
• So save_decode_cache: false would skip caching the KV for all decode-phase tokens, whether they came from speculative decoding or regular autoregressive decoding. It's not selective about the source.

What it actually does:

save_decode_cache	What gets cached
true	Prefill KV + decode KV (including spec-decode accepted tokens)
false	Only prefill KV — no decode-phase KV stored at all

If your goal is to prevent caching of only the spec-decode candidate tokens (e.g. to avoid caching potentially-unverified KV), that's not what this flag does. LMCache doesn't currently have a knob for that — by the time KV hits the cache, spec-decode has already accepted/rejected candidates, and the accepted ones are indistinguishable from regular decode tokens.

If you set it to false to save memory/bandwidth and you're only interested in prefix caching (shared system prompts, etc.), that's the right call. But if you were hoping it'd let prefill cache through while filtering spec-decode artifacts — that's not the distinction it makes. user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Wed 2026-05-06 16:19 UTC] No. I get this error sometimes:

 (Worker_TP2 pid=767) [2026-05-06 15:56:03,745] LMCache ERROR: Request chatcmpl-3fb08ae0-a0cc-4f52-86e0-de85aeee8f3c-83f6e6b1The number of retrieved tokens is less than the expected number of tokens! This should not happen! (vllm_v1_adapter.py:878:lmcache.integration.vllm.vllm_v1_adapter)
(Worker_TP2 pid=767) [2026-05-06 15:56:03,745] LMCache ERROR: Num retrieved tokens: 49664, num expected tokens: 55808 (vllm_v1_adapter.py:884:lmcache.integration.vllm.vllm_v1_adapter)

And thinking maybe its because its not storing the draft tokens since theyre part of the decode phase and that's set to false assistant: That's a solid hypothesis. The math checks out too — 55808 - 49664 = 6144 missing tokens, which at your chunk_size of 256 is exactly 24 chunks. Feels too clean to be coincidence.

Here's the likely sequence of events:

1. Request A runs, hits a shared prefix during prefill (cached ✅), then enters decode + spec-decode to generate tokens. Those decode KV blocks are not saved because save_decode_cache: false.
2. Request B comes in with the same prefix plus some of the tokens that Request A generated during decode. LMCache sees a matching prefix, expects to find KV for the full shared portion (including the decode tokens), but only retrieves the prefill portion.
3. 💥 Token count mismatch.

The adapter is basically saying: "I know we processed these 55808 tokens before, but I only have 49664 tokens worth of KV cache — where's the rest?"

So yes — save_decode_cache: false creates a gap where cached prefix tracking thinks tokens exist in the cache but they were never actually stored. With spec-decode, the gap is even larger per step (multiple tokens per step), making the mismatch more noticeable.

Fix options:

1. Set save_decode_cache: true — stores all decode KV, eliminates the gap. Costs more memory/storage but consistent.
2. If you want to keep it false, this might be a bug in LMCache's v1 adapter — it should account for uncached decode tokens when computing expected retrieval counts, rather than assuming the full sequence was cached. Worth filing upstream.

Want me to dig into the adapter code at that line number to see exactly how it computes the expected token count? user: Sender (untrusted metadata):

{
  "label": "openclaw-control-ui",
  "id": "openclaw-control-ui"
}

[Wed 2026-05-06 16:20 UTC] No. I'm going to roll this out. It's just a time consuming process, so I wanted to verify before doing it assistant: Makes sense — verify before you burn hours. Go get 'em. 🔧