Your local LLM forgets its cache after every restart. I measured how much time that wastes.
When you run a local LLM, the model has to "read" your conversation before it can answer. That reading step is called prefill — the model processes all the tokens in your prompt and conversation history, and the longer the context, the longer it takes.
Here is the problem. If you stop llama-server, switch models, reboot your machine, or the server crashes, the conversation's KV cache is gone. On the next message the model does not continue from where it left off — it starts over and reads the whole conversation again from token one. At small context sizes this is annoying. At large context sizes it becomes painful: on my M3 Max, with a 64K-token context, a cold restart meant waiting more than seven minutes before the first word came back.
The surprising part is that llama.cpp already has a way to avoid this. It has --slot-save-path and two REST endpoints:
/slots/0?action=save
/slots/0?action=restore
These can save a conversation's KV cache to disk and restore it later, even after restarting the server. The feature works — but almost nobody automates it, and I could not find a proper measurement of how much it helps, when it breaks, or whether a restored cache gives the same result as recomputing from scratch.
So I measured it. I ran 480 benchmark rows plus a supervised session, and published the full benchmark as stillwarm-bench, including the dataset and a hash-chained evidence log. I also built the missing automation as a small zero-dependency tool:
pip install stillwarm
The setup
I wanted the results to be easy to reproduce, so I froze the setup as much as possible: an Apple M3 Max with 36 GB unified memory, llama.cpp pinned to release b9871 by commit, and models pinned by SHA-256 — Llama-3.1-8B-Instruct Q4_K_M as the main model, plus Qwen2.5-7B-Instruct and Gemma-3-4B-it. The test document was Frankenstein, which is public domain, cut into exact token sizes from 2K to 64K. Every run used greedy decoding, flash attention was explicitly turned on for every measured run, and each cell had one warmup plus five measured reps.
For every warm or restored run, I checked two things. First, a 64-token byte-diff probe compared the restored result against a cold recompute. Second, I checked whether the server actually reused the cache — the server-reported prompt_n had to look like a small question, not the full document. That second check turned out to be very important: a restore can report success and still not save you time. More on that below.
Main result: 71× to 201× faster
| context | cold prefill median | resume from disk restore + TTFT | speedup |
|---|---|---|---|
| 8K | 18.995 s | 0.266 s | 71× |
| 32K | 157.42 s | 0.989 s | 159× |
| 64K | 441.58 s | 2.197 s | 201× |
This was the same session, with cold and restore runs interleaved, so the machine's temperature would not unfairly help one side. After restore, decode speed stayed the same — the benefit is in avoiding the full prefill. All 15 supervised restores were byte-identical to recompute.
I also want to be honest about how the headline number changed. My first internal number was 1191×. That was wrong, because I only measured the time after restore and did not include the restore time itself. After fixing that, the number became 225×. Then I controlled thermal state more carefully — the cold baseline had been running on a hotter machine, which made it look worse than it really was. After fixing that too, the final number became 201×. So the headline got smaller twice. That is a good thing: the number became more honest each time.
When does restore beat recompute?
The answer is: almost always. With a warm page cache, restore starts winning after roughly 30 tokens of context.
Then I tested a harsher case. Before each rep, I deliberately purged the page cache with sudo purge, so the restore had to read from disk under worse conditions. Even then, restore won once the context was around 132 to 295 tokens, depending on the test size. In the purged-cache case, restores read at 5.27 GB/s against a raw-disk dd reference of 6.40 GB/s — around 84% to 95% of what the SSD could physically deliver.
The practical rule is simple: if your session is longer than about 300 tokens, restoring the KV cache is faster than making the model re-read everything. That was true on this machine even under the worst conditions I tested.
What makes a saved cache invalid?
This was one of the most useful parts of the work. I tested a 46-cell portability matrix to see what actually breaks a saved KV cache.
Flash attention mismatch breaks it. If a cache was saved with flash attention on and you try to restore it with flash attention off, or the other way around, llama.cpp refuses cleanly. This was the main true invalidator.
The saved state must fit the context window. If the saved state is larger than the new context window, restore also refuses cleanly. That makes sense — you cannot load a larger state into a smaller context window.
Different llama.cpp builds worked in my test. Files saved on a release from about five weeks earlier restored fine, in both directions. That does not mean the format is guaranteed to stay stable forever — it only means this specific build drift worked in practice. So I treat build changes as a warning, not an automatic failure.
Changing -ngl survived. Changing the GPU layer split did not break the cache. That surprised me, but the measurements were consistent.
Based on these results, stillwarm uses this cache key: model SHA, flash-attention setting, KV cache types, SWA mode, and prefix hash. The build tag is recorded, but it is only a warning.
Three findings that surprised me
1. A restore can succeed but still be useless
This happened with Gemma, which uses sliding-window attention. Without --swa-full, the restore endpoint reported success, and the answer was also correct — but the server quietly re-prefilled almost the entire document. From the outside, everything looked fine. Performance-wise, the restore did not really help.
That is why correctness is not enough: you also need to check whether the cache was actually reused. This is why stillwarm includes a reuse warning — if a "successful" restore is followed by near-full re-prefill, the tool warns you.
2. Where you save the cache matters
I expected save_point to be simple bookkeeping. It is not — it changes behavior. If you save the state after generation, the cache includes sampled tokens from the answer, and a new question does not always extend cleanly from that. With sliding-window attention, the cache may not be able to roll back far enough, and the server can fall back to a full re-prefill. But if you save the state after document prefill only, the saved state is a strict prefix of any future question, and partial reuse works correctly. So stillwarm records which type of save you have in the sidecar file.
3. Quantized KV caches restore deterministically, but not always like a fresh recompute
This was the strangest result. With a q4_0 KV cache, the restored continuation diverged from a same-config cold recompute in 5 out of 5 reps — the first different token appeared around position 47, and both outputs were fluent. With q8_0, this happened in 1 out of 5 reps. With f16, 0 out of 5.
The odd detail was that the restored q4 output matched the f16 baseline generation — the cold q4 path was the one that looked different. My best explanation is batch-shape numerics in quantized attention.
The practical consequence is important: stillwarm's --verify does not compare a restore against a brand-new recompute. Instead, it compares against a probe hash captured at save time. That checks deterministic equality against the saved state, which is the thing we actually care about.
A thermal lesson from benchmarking on a laptop
I also learned a very normal but easy-to-ignore lesson: laptop benchmarks are affected by heat. The same config gave around 473 tok/s on a cool machine and around 310 tok/s after about an hour of sustained load.
The weirdest row in the dataset was this: purged-cache restores on a cool machine took around 1.50 to 1.69 seconds at 64K, while page-cache-warm restores on a hot machine took around 1.78 to 1.86 seconds. That is backwards from what page-cache behavior alone would predict. A dd contrast showed 21.2 GB/s warm versus 6.4 GB/s cold, so the page cache was not the explanation. My labeled hypothesis is SoC thermal state — I did not isolate the exact mechanism.
The lesson is simple: if you benchmark on a laptop, interleave your test conditions in one session. Otherwise, your baseline may depend more on heat than on the thing you are trying to measure.
Energy savings
Since I was already measuring performance, I also measured energy with powermetrics. A 32K cold prefill ran 116.5 seconds at about 33.8 W — about 1.102 Wh. The same resume from disk was about 0.021 Wh, and that number is an upper bound, because the restore was shorter than the 4-sample measurement window.
A simple way to say it: resuming from disk used around 50× less energy than recomputing the same context.
The tool: stillwarm
stillwarm wraps llama-server and does the cache work automatically: restore before the request, save after the response. It includes cache keys based on the measured invalidators, refusal handling for incompatible restores, warnings for ineffective restores, --verify support, and LRU pruning under a disk budget. Zero runtime dependencies, Python 3.9+.
Quickstart:
from stillwarm import LlamaServer, ServerProfile, Session
# llama-server ... -fa on --slot-save-path ~/.stillwarm/ (must be running)
server = LlamaServer("http://127.0.0.1:8080")
profile = ServerProfile(model_sha256="<sha256 of your .gguf>",
flash_attn="on", cache_type_k="f16", cache_type_v="f16",
swa_full=False, ctx_size=16384)
sw = Session(server, profile, cache_dir="~/.stillwarm",
server_build_tag="b9871", disk_budget_gb=50)
r = sw.ask("paper-review", document + "\n\nQ: Summarize.\nA:", n_predict=256)
# ... llama-server restarts ...
r = sw.ask("paper-review", document + "\n\nQ: And the weaknesses?\nA:", n_predict=256)
print(r.restored, r.prompt_n) # True, ~a few dozen tokens instead of thousands
cache_dir must be the directory passed to --slot-save-path. CLI:
stillwarm list ~/.stillwarm
stillwarm inspect ~/.stillwarm/paper-review.bin
stillwarm prune ~/.stillwarm --budget-gb 50 --dry-run
Scope is intentionally limited: this is for llama-server. Ollama and LM Studio are built on llama.cpp, but they do not expose the slot save and restore endpoints. I tested this on macOS with llama.cpp b9871; the demo Space runs the same mechanism on Linux CPU. Windows is not tested yet.
Can you just download a KV cache?
I also published one cache artifact: an 8K-token KV cache of Frankenstein for Qwen2.5-7B — 471 MB with a full sidecar. You can find it here: stillwarm-kv-cache-artifact.
But the honest answer is that downloading caches is usually not worth it. On this machine, downloading the cache beats recomputing only if your internet speed is above about 215 Mbps. Below that, your GPU is faster. Above that — especially on gigabit fiber — downloading can win. The bigger point is that KV caches are heavy, and they are tied to the exact model and settings, so shipping caches is usually not practical. But now we have a measured threshold instead of a guess.
Limitations
This was tested on one main machine: an Apple M3 Max with 36 GB unified memory, on the Metal backend. Cross-OS file portability is still untested — the demo Space shows that the mechanism works on Linux, but moving a Mac-saved file to Linux is still open. Windows is also untested.
There is no guarantee that llama.cpp's slot save format will stay stable across versions. In my test, a build difference of about five weeks worked; that should not be treated as a long-term promise.
This project also focuses on one slot. It is not trying to be a full serving framework. For large vLLM fleets, LMCache already exists and is excellent. For Apple-side multi-agent scheduling on MLX, there is related agent-memory work. This project focuses on the llama.cpp consumer path: measurements, portability data, and a small verified tool.
Everything is public
- Benchmark, harness, and figures: https://github.com/vimalnakrani08/stillwarm-bench
- Full dataset, evidence chain, and raw powermetrics: https://huggingface.co/datasets/vimalnakrani/stillwarm-bench-results
- Tool: https://github.com/vimalnakrani08/stillwarm · https://pypi.org/project/stillwarm/
- Live demo: https://huggingface.co/spaces/vimalnakrani/stillwarm-demo
- Cache artifact: https://huggingface.co/datasets/vimalnakrani/stillwarm-kv-cache-artifact