Qwen3.8-27B on RTX 3090
Baseline (no speculation)
Best — official MTP head + ngram 16–32
Hill climbing median tok/s per step
Experiments tok/s
| Configuration | Median | Mean | vs baseline | Notes |
|---|---|---|---|---|
| Baseline (sampling, no spec) | 39.9 | — | 1.00× | reference point |
--spec-type draft-mtp -n 2 | 54.5 | — | +37% | sweet spot; code +64% |
--spec-default (ngram) + mtp | 54–60 | 71 | +36…51% | high variance, code 105–135 |
| MTP n=3 | 52.5 | 52.7 | +32% | worse than n=2 — too many verifications |
Round 2 — fine-grained tuning tok/s
| Configuration | Median | Mean | vs MTP n2 | Notes |
|---|---|---|---|---|
| MTP n2 (anchor) | 52.4 | 53.8 | 1.00× | repeat — consistent with round 1 (54.5) |
--spec-draft-n-min 2 | 52.2 | 54.3 | −0.4% | no benefit — 1-token drafts are cheap anyway |
--spec-draft-p-min 0.1 | 52.5 | 53.0 | +0.2% | noise — nothing gained |
| Hybrid ngram 16–32 | 57.9 | 74.2 | +10% / +38% | tuned ngram window instead of default 48–64; code up to 132 tok/s |
| Hybrid ngram 8–16 | 54.2 | 64.3 | +3% | worse than 16–32 — too-short matches add noise |
| MTP n2 + draft KV q8 | 51.6 | 52.6 | −1.5% | −240 MiB VRAM — nearly free memory win |
Round 2 takeaway: the ngram window is the real lever we missed. Default --spec-ngram-mod-n-min 48 / --spec-ngram-mod-n-max 64 is tuned for large drams; cutting the window to 16–32 lets short repetitive patterns (code, JSON) be accepted earlier — median +10%, and +38% on mean because code prompts spike to 132 tok/s. Draft KV q8_0: −240 MiB VRAM at −1.5% speed — worth it when VRAM is tight.
Round 3 — official ggml-org MTP head tok/s, code probe
All Round 1–3 numbers below use the code-only probe (upper-bound workload). See Round 4 for the mixed-workload re-check.
The MTP weights shipped in the unsloth GGUF are only the training-time heads. ggml-org publishes a standalone MTP head (mtp-Qwen3.8-27B-Q4_0.gguf, 1.6 GB) that plugs in via --spec-draft-model. Compared on the same server binary (b10446), same probe:
| Configuration | Median | Mean | vs built-in MTP | Notes |
|---|---|---|---|---|
| Built-in MTP n2 (unsloth heads) | 52.4 | 53.8 | 1.00× | round-2 anchor |
| Official head Q4_0, MTP-only | 61.8 | 62.5 | +18% | better drafts than shipped heads |
| Official head + ngram 16–32 | 68.2 | 69.8 | +30% | new best median; VRAM 23.3 GiB fits 24 GB |
| Built-in MTP + ngram 16–32 (re-run b10446) | 61.3 | 74.6 | +17% | mean 74.6 but median 61.3 — variance dominated by code spikes |
Round 3 takeaway: the official MTP head beats the built-in heads at every point — MTP-only 61.8 vs 52.4 (+18%), and with ngram it reaches 68.2 median. The Q8_0 head (2.7 GB) does not fit at 128K context (OOM: 25.2 GiB > 24 GiB) — Q4_0 is the right pick and still gains. Note on stability: built-in+ngram has a higher mean (74.6) but lower median — the official head is the more consistent win, +71% over baseline.
Fresh-master bonus: the rebuild (b10446, pulled 2026-08-15) also exposes --reasoning-preserve, --agent, --no-agent, --spec-draft-ngl — new levers not in the May build.
Production config updated: hybrid MTP n2 + ngram 16–32, draft KV q8_0 (see below).
Round 4 — robust paired benchmark same build, diverse probe
Rounds 1–3 used a code-only probe (3 prompts: merge-sorted-lists, mmap-vs-read, bash-watcher) — code is the best case for ngram speculation, so 68.2 was an upper bound, not a mixed-workload number. Round 4 re-runs the champion configs on the same server binary (b10446) with a diverse probe: 8 prompts (code, log parsing, JSON schema, prose, math, Polish, poem) × 4 runs × 800 tokens, randomized order across rounds (kills GPU clock/thermal drift bias) = 32 measurements per config × 2 independent rounds.
| Configuration | Round 1 med | Round 2 med | Pooled median | Pooled mean | vs baseline | Notes |
|---|---|---|---|---|---|---|
| Baseline (no spec) | 36.4 | 36.2 | 36.3 | 36.5 | 1.00× | stable, low variance |
| Built-in MTP + ngram 16–32 | 53.3 | 51.6 | 52.5 | 55.3 | +44% | code spikes to 87–90 |
| Official head + ngram 16–32 | 57.9 | 55.8 | 56.1 | 60.0 | +54% | wins both rounds; code up to 138 |
Round 4 takeaway: the real mixed-workload number is ~56 median / 60 mean — still +54% over baseline and the official head still beats built-in in both independent rounds (57.9 vs 53.3, 55.8 vs 51.6). Per-prompt medians (official+ngram): code/logs/JSON/math 66–76 t/s, prose/Polish/poem 40–48 t/s — speculation gains are concentrated in structured text, exactly as theory predicts. 68.2 remains valid as the code-workload number.
Results
1. Official MTP head > built-in heads
The standalone ggml-org head (--spec-draft-model mtp-*.gguf) drafts better than the heads baked into the unsloth GGUF: 61.8 vs 52.4 tok/s median, MTP-only (+18%, code probe). Combined with tuned ngram it hits 68.2 median on code workloads — +71% over baseline with zero quality loss; on a diverse mixed-workload probe (Round 4) it is 56.1 median / 60.0 mean (+54%), still winning both independent rounds vs built-in (57.9/55.8 vs 53.3/51.6). Q8_0 head OOMs at 128K ctx; Q4_0 fits (23.3 GiB).
2. ngram-mod — free boost, window 16–32 is the sweet spot
ngram speculation (match against history) stacks with MTP: 68.2 median / 69.8 mean tok/s (code probe), 56.1 / 60.0 on mixed workload with a 16–32 window. Default 48–64 window is tuned for big batch — on interactive coding, short repetitive patterns (code, JSON) get accepted earlier with 16–32. Code prompts spike to 105–138 tok/s.
3. KV q8_0 — free VRAM savings, zero degradation
Perplexity on a 297 KB corpus (BDH paper, ctx 8192): f16 = 6.3356, q8_0 = 6.3300 (−0.09%, within noise), q4_0 = 6.3463 (+0.17%, also within error bar ±0.086). KV cache quantization is practically lossless — q8_0 used in production.
Quality — perplexity (KV q8 vs q4 vs f16)
| KV config | PPL | vs f16 | Verdict |
|---|---|---|---|
| f16 (reference) | 6.3356 | — | full precision |
| q8_0 | 6.3300 | −0.09% | measurement noise — zero degradation |
| q4_0 | 6.3463 | +0.17% | within error bar ±0.086 |
Corpus: 297 KB (scientific paper, en), ctx 8192, flash-attn on, t=16. Error bar ±0.086 makes all differences statistically insignificant. Takeaway: KV cache can be safely quantized — q8_0 chosen for production as the sweet spot.
Seed-match: MTP does not change the distribution
Same --seed 42 + same prompt: 406 shared characters (entire private-think identical), divergence only in content sampling — MTP consumes RNG differently, but from the same distribution. Speculative decoding is distributionally transparent: MTP changes speed only, not quality.
Production configuration v3 — official head
llama-server -m /data/models/qwen3.8/Qwen3.8-27B-Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 -ngl 99 -t 16 \
-c 131072 --no-context-shift --flash-attn on \
--cache-type-k q8_0 --cache-type-v q8_0 \
--jinja --reasoning-format deepseek \
--temp 0.7 --top-p 0.80 --top-k 20 --min-p 0.0 --presence-penalty 1.5 \
--spec-type draft-mtp,ngram-mod \
--spec-draft-model /data/models/qwen3.8/ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf \
--spec-draft-n-max 2 \
--spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 32 \
--spec-draft-type-k q8_0 --spec-draft-type-v q8_0 -np 1VRAM — budget (v3)
- Model + ctx 128K + KV q8: ~22.0 GiB
- Official MTP head Q4_0: +1.9 GiB
- MTP context (n=2) + draft KV q8_0: ~+760 MiB
- Total (measured): 23.3 GiB — fits, ~700 MiB headroom
What is MTP?
Multi-Token Prediction: "heads" in the model predict 2 next tokens. Speculative decoding verifies drafts in a single pass — that's the win. Use the official ggml-org head (mtp-Qwen3.8-27B-Q4_0.gguf) for better drafts than the training-time heads baked into the unsloth GGUF — --spec-draft-model loads it without touching the main GGUF.
Next experiments
- ✔
--spec-draft-n-min/--spec-draft-p-min— no effect, ruled out - ✔ ngram window tuning — 16–32 is the sweet spot
- ✔ Draft-side KV q8_0 — −240 MiB, adopted
- ✔ Official ggml-org MTP head vs built-in — official Q4_0 wins (+18%), adopted
- ✔ Round 4 paired benchmark — diverse probe confirms official head wins both rounds (56.1 vs 52.5 pooled median, +54% vs baseline)
- Q8_0 official head — OOMs at 128K ctx (25.2 GiB); viable only with
-c 81920or--spec-draft-nglpartial offload - 128K context fill + generation under v3 config — VRAM check at full context (23.3 GiB measured at startup; fill may grow)
- New levers from b10446:
--reasoning-preserve,--agent,--reasoning-effort,--spec-draft-ngl