---
license: mit
language:
- en
tags:
- gpt-2
- wikitext
- magic
- bergson
- fine-tuned
base_model: gpt2
datasets:
- Salesforce/wikitext
---

# bergson MAGIC checkpoint — GPT-2 fine-tuned on wikitext-2

GPT-2 (124M) fine-tuned on
`Salesforce/wikitext` `wikitext-2-raw-v1` `train` (chunked at 512
tokens) via the [`bergson`](https://github.com/EleutherAI/bergson)
MAGIC pipeline. This is the exact checkpoint used to generate the
attribution scores published at
[`EleutherAI/bergson-magic-scores-gpt-2`](https://huggingface.co/datasets/EleutherAI/bergson-magic-scores-gpt-2).

## Loading

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("EleutherAI/bergson-magic-gpt-2")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/bergson-magic-gpt-2")
```

## YAML used to produce this checkpoint

```yaml
run_path: runs/gpt2_wikitext
model: gpt2
overwrite: true

data:
  dataset: Salesforce/wikitext
  subset: wikitext-2-raw-v1
  split: "train"
  chunk_length: 512

query:
  dataset: Salesforce/wikitext
  subset: wikitext-2-raw-v1
  split: "test[3:4]"
  chunk_length: 0

distributed:
  nproc_per_node: 4
  nnode: 4

batch_size: 256
num_epochs: 2
lr_schedule:
  lr_scheduler_type: polynomial
  lr: 0.0008
  lr_start: 1e-6
  lr_end: 0.00008
  warmup_steps: 0.25

subset_strategy: random
wandb_project: magic
```

Saved as `examples/magic/gpt2_wikitext.yaml` in the bergson repo.

Run with:

```bash
bergson magic examples/magic/gpt2_wikitext.yaml
```

The `bergson magic` step trains the model on the train split via its
own training loop (it must, because MAGIC's attribution scores are
the gradients of query loss with respect to per-example training
weights, computed by back-propagating through training). The final
trained weights end up at the `hf_model/` subdirectory of the run
path; that's what was uploaded here.