File size: 1,777 Bytes
3c6d98a c283b04 3c6d98a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
license: mit
---
# gated-deltanet-nsa-0.4B-10B
Gated DeltaNet + native sparse attention (0.4B params, 10B tokens)
## Overview
* **Training**: gated-deltanet-nsa-0.4B-10B was trained on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), which is realeased under [ODC-By v1.0](https://opendatacommons.org/licenses/by/1-0/)
* **Parameters**: 0.4B
* **Task**: Language modeling
* **Framework**: HuggingFace, [flash-linear-attention](https://github.com/fla-org/flash-linear-attention)
* **Output structure**: [batch_size, sequence_length, num_logits]
## Performance
Various; available in paper
## Running Code
* Minimal code to instantiate the model and perform inference:
```python
# Requires flash-linear-attention (https://github.com/fla-org/flash-linear-attention)
import fla
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(path_to_model).cuda()
tokenizer = AutoTokenizer.from_pretrained(path_to_model).cuda()
input_ids = tokenizer("All human beings are", return_tensors="pt").input_ids
outputs = model.generate(input_ids, max_length=15)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
## License
Gated DeltaNet is released under [MIT License](LICENSE.txt)
## Citation
If you find our work useful, please cite the following publication:
```bibtex
@misc{he_alleviating_2025,
title = {Alleviating {Forgetfulness} of {Linear} {Attention} by {Hybrid} {Sparse} {Attention} and {Contextualized} {Learnable} {Token} {Eviction}},
url = {http://arxiv.org/abs/2510.20787},
doi = {10.48550/arXiv.2510.20787},
publisher = {arXiv},
author = {He, Mutian and Garner, Philip N.},
month = oct,
year = {2025},
note = {arXiv:2510.20787 [cs]},
}
```
|