---
license: apache-2.0
language:
- en
base_model:
- openai-community/gpt2-xl
datasets:
- wikitext-103
---
# GPT2-large-Finetuned-WikiText103

## Model Description

A fine-tuned version of GPT2-large on the WikiText-103 dataset.

- **Paper:** [MLP Memory: A Retriever-Pretrained Memory for Large Language Models](https://arxiv.org/pdf/2508.01832)
- **GitHub:** [https://github.com/Rubin-Wei/MLPMemory](https://github.com/Rubin-Wei/MLPMemory)
- **Base Model:** GPT2-large
- **Training Dataset:** WikiText-103
- **Model Size:** 744M parameters

## Performance on WikiText-103

| Model | Perplexity | Improvement |
|:------|:----------:|:-----------:|
| GPT2-large (baseline) | 15.80 | - |
| **GPT2-large-Finetuned** | **10.42** | **-5.38** |

## Training Details

- **Training Data:** WikiText-103 (103M tokens)
- **Optimizer:** AdamW
- **Learning Rate:** 2e-5 with cosine schedule

## Citation

This model was released as part of the paper ["MLP Memory: A Retriever-Pretrained Memory for Large Language Models"](https://arxiv.org/pdf/2508.01832).

For more information, see: [https://github.com/Binn0/MLPMemory](https://github.com/Binn0/MLPMemory).

If you use this model, please cite:

```bibtex
@inproceedings{Wei2025MLPMA,
  title={MLP Memory: A Retriever-Pretrained Memory for Large Language Models},
  author={Rubin Wei and Jiaqi Cao and Jiarui Wang and Jushi Kai and Qipeng Guo and Bowen Zhou and Zhouhan Lin},
  year={2025},
  url={https://api.semanticscholar.org/CorpusID:281658735}
}
```