---
base_model:
- Qwen/Qwen3-8B
datasets:
- open-r1/OpenR1-Math-220k
license: apache-2.0
pipeline_tag: text-generation
tags:
- math
- trimkv
- KV
- Cache
- Compression
---

# TrimKV-Qwen3-8B-Math

> **TRIM-KV** is an efficient and learnable key–value eviction strategy designed to improve the efficiency of large language models (LLMs) in long-horizon inference.

This model is a version of `TrimKV-Qwen3-8B-Math` enhanced with TRIM-KV, as presented in the paper [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://huggingface.co/papers/2512.03324).

The core idea behind TRIM-KV is to learn the intrinsic importance of each key–value pair at creation time, which we call *token retention*, and then decay this importance exponentially over time to mimic standard inference running with eviction. 

The retention score is query-agnostic and captures the long-term utility of tokens. This is different from attention scores, which are query-dependent: they capture the short-term utility for predicting the next token and are recomputed at every step.

- **Paper:** [Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs](https://huggingface.co/papers/2605.09649)
- **Code:** [GitHub - ngocbh/trimkv](https://github.com/ngocbh/trimkv)
- **Arxiv:** [2512.03324](https://arxiv.org/abs/2512.03324)


## Quick Start

### Installation

To use this model, you need to install the `trimkv` library:

```sh
git clone https://github.com/ngocbh/trimkv.git
cd trimkv
pip install -e .
````

### Inference

```python
import torch
from trimkv.models.qwen3 import TrimKVQwen3ForCausalLM
from trimkv.cache_utils import TrimKVCache
from transformers import AutoTokenizer

model_path = "ngocbh/TrimKV-Qwen3-8B-Math"

model = TrimKVQwen3ForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,
    load_trimkv_weights=True,
    download_from="huggingface",
    use_cache=True,
    device_map="cuda",
)

# Configure TRIM-KV settings
model.config._attn_implementation = "flash_attention_2"
model.config.compress_memory = True
model.config.memory_size = 512
model.config.buffer_size = 128

tokenizer = AutoTokenizer.from_pretrained(
    model.config.base_model,
    use_fast=True,
    padding_side="left",
)

# Use model.generate as normal.
# Note: TRIM-KV uses TrimKVCache under the hood. So please pass TrimKVCache to model.generate
```

## Citation

```bibtex
@article{bui2025cache,
  title={Cache what lasts: Token retention for memory-bounded kv cache in llms},
  author={Bui, Ngoc and Sharma, Shubham and Lamba, Simran and Mishra, Saumitra and Ying, Rex},
  journal={arXiv preprint arXiv:2512.03324},
  year={2025}
}
@article{bui2025make,
  title={Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction},
  author={Bui, Ngoc and Nguyen, Hieu Trung and Cohan, Arman and Ying, Rex},
  journal={arXiv preprint arXiv:2512.03324},
  year={2025}
}
```