File size: 2,345 Bytes
068bc97 ef9503c 068bc97 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
license: apache-2.0
base_model: answerdotai/ModernBERT-base
tags:
- token-classification
- compression
- context-compression
- headroom
language:
- en
---
# kompress-v3
Token compression classifier fine-tuned from [chopratejas/kompress-v2-base](https://huggingface.co/chopratejas/kompress-v2-base) (ModernBERT-base, 149M params). Trained as part of the [ultrawhale](https://github.com/peterlodri-sec/ultrawhale) fine-tuning loop.
Kompress classifies each token in a message as keep (1) or drop (0). Used by the [headroom proxy](https://github.com/headroomlabs-ai/headroom) to compress LLM context before it reaches the model.
## Eval results (heretic adversarial benchmark)
[Heretic-style prompts](https://github.com/p-e-w/heretic) generate responses maximally dense with must-keep tokens (chemical formulas, CVE identifiers, memory addresses, line numbers). The benchmark measures what fraction of those tokens survive compression.
| Metric | Value |
|---|---|
| heretic exact_pct | 0.942 |
| keep_rate | 0.728 |
| override_delta | +0.027 |
| base model | kompress-v2-base |
[Full progression across all versions](https://pocoo.vaked.dev/posts/2026-06-25-kompress-heretic-eval)
## Training
Self-labeled references via kompress-v2-base, 1802 training pairs. mk_in_ref improved from ~0.5 (Q&A labels) to 0.720. First iteration of the ultrawhale self-labeling loop.
## Usage
```python
# Via headroom proxy (recommended)
# ANTHROPIC_BASE_URL=http://localhost:8787 claude
# Direct library use
from headroom import compress, CompressConfig
result = compress(messages, config=CompressConfig(kompress_model="PeetPedro/kompress-v3"))
```
## CONCLUSION
Self-labeling works but noisy labels limit precision. Override delta +0.027 — model still needs external regex.
## USECASE
First self-labeling experiment. Use if studying the self-labeling loop pattern.
## Series
| Version | heretic | keep_rate | Notes |
|---|---|---|---|
| v3 | 0.942 | 0.728 | first self-label |
| v3.1 | 0.925 | — | domain data |
| v3.2 | 0.929 | — | domain refined |
| v3.3 | 0.942 | — | domain-only, overfit |
| **v4** | **0.967** | **0.823** | override internalized |
| v5 | 0.961 | — | loop converged |
| **v6** | **0.962** | **0.854** | agent-distribution |
Training code: [ultrawhale](https://github.com/peterlodri-sec/ultrawhale)
|