POCKET vs Bonsai measured generation speed
Same machine, stock llama.cpp. POCKET wins generation on CPU 2.69x and on GPU 2.22x; Bonsai wins GPU prompt processing.
Generation speed (tok/s) — same machine, stock llama.cpp
POCKET-35B IQ1_M (7.66 GB) vs Bonsai-27B Q1_0 (3.53 GB) · higher is better
CPU · Xeon 16 threads
POCKET
27.0
Bonsai
10.1
POCKET 2.69x faster
GPU · H100 NVL
POCKET
197
Bonsai
89
POCKET 2.22x faster
Honest caveat
GPU prompt-processing: Bonsai wins (1816 vs 753 tok/s). Quality (HellaSwag 400): tie, 61.0% vs 60.0%.