---
license: apache-2.0
base_model: answerdotai/ModernBERT-base
tags:
  - token-classification
  - compression
  - context-compression
  - headroom
language:
  - en
---

# kompress-v3

Token compression classifier fine-tuned from [chopratejas/kompress-v2-base](https://huggingface.co/chopratejas/kompress-v2-base) (ModernBERT-base, 149M params). Trained as part of the [ultrawhale](https://github.com/peterlodri-sec/ultrawhale) fine-tuning loop.

Kompress classifies each token in a message as keep (1) or drop (0). Used by the [headroom proxy](https://github.com/headroomlabs-ai/headroom) to compress LLM context before it reaches the model.

## Eval results (heretic adversarial benchmark)

[Heretic-style prompts](https://github.com/p-e-w/heretic) generate responses maximally dense with must-keep tokens (chemical formulas, CVE identifiers, memory addresses, line numbers). The benchmark measures what fraction of those tokens survive compression.

| Metric | Value |
|---|---|
| heretic exact_pct | 0.942 |
| keep_rate | 0.728 |
| override_delta | +0.027 |
| base model | kompress-v2-base |

[Full progression across all versions](https://pocoo.vaked.dev/posts/2026-06-25-kompress-heretic-eval)

## Training

Self-labeled references via kompress-v2-base, 1802 training pairs. mk_in_ref improved from ~0.5 (Q&A labels) to 0.720. First iteration of the ultrawhale self-labeling loop.

## Usage

```python
# Via headroom proxy (recommended)
# ANTHROPIC_BASE_URL=http://localhost:8787 claude

# Direct library use
from headroom import compress, CompressConfig
result = compress(messages, config=CompressConfig(kompress_model="PeetPedro/kompress-v3"))
```


## CONCLUSION

Self-labeling works but noisy labels limit precision. Override delta +0.027 — model still needs external regex.

## USECASE

First self-labeling experiment. Use if studying the self-labeling loop pattern.

## Series

| Version | heretic | keep_rate | Notes |
|---|---|---|---|
| v3 | 0.942 | 0.728 | first self-label |
| v3.1 | 0.925 | — | domain data |
| v3.2 | 0.929 | — | domain refined |
| v3.3 | 0.942 | — | domain-only, overfit |
| **v4** | **0.967** | **0.823** | override internalized |
| v5 | 0.961 | — | loop converged |
| **v6** | **0.962** | **0.854** | agent-distribution |

Training code: [ultrawhale](https://github.com/peterlodri-sec/ultrawhale)