---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
- mlfoundations/dclm-baseline-1.0-parquet
language:
- en
tags:
- boris
- nmai
- 125M
---

![Boris](Boris-125M.png)

# Boris-125M
Boris-125M is a Large Language Model created by New Millennium Artificial Intelligence, trained on 2.5B tokens and contains 125 million parameters.

Trained for 33:38:48 on one RTX 3060.

# Final Stats:
### Loss - 3.2998
### Grad Norm - 0.281
### Learning Rate - 6.00e-05


![Benchmarks](benchmark.png)


## Details

| | |
|---|---|
| Architecture | GPT-2 (pre-LN, learned positional embeddings, tied embeddings) |
| Layers / heads / d_model | 12 / 12 / 768 |
| Context length | 1024 |
| Vocab | 50304 (GPT-NeoX-20B BPE, padded) |
| Tokenizer | `EleutherAI/gpt-neox-20b` |
| Precision | trained in bf16 autocast with fp32 master weights |

Base model; not instruction-tuned or aligned. Outputs may be inaccurate or
offensive.