LLMPretrain — a 101M decoder-only LM trained from scratch

Every weight in this model was produced by this project. There is no from_pretrained, no base checkpoint, and no adapter: the tokenizer, the architecture code, the data pipeline and the training loop are all bespoke, and training started from a random initialisation.

Trained on a single RTX 5070 Ti (16 GB).

Architecture

Written by hand — no transformers model classes were used to define it.

Type	decoder-only, pre-norm transformer
Parameters	100,684,032
Layers	12
d_model	768
Attention heads	12 query / 4 KV (GQA)
Head dim	64
FFN	SwiGLU, hidden 2048 (≈ 8/3 × d_model)
Normalisation	RMSNorm (pre-norm) + QK-norm
Positions	RoPE (θ=10000) — no learned position embeddings
Context	1024
Vocab	32,768 (own byte-level BPE, padded to a multiple of 128)
Biases	none, in any linear layer
Embeddings	tied between input and LM head

Parameter split: 25,165,824 embedding (25.0%) / 75,517,440 transformer blocks.

Training

Tokens seen	9,999,745,024
Steps	19,073
Schedule	warmup-stable-decay (WSD)
Optimizer	fused AdamW, betas (0.9, 0.95), wd 0.1 on 2D params only
Precision	bf16 autocast, fp32 master weights
Hardware	1 × RTX 5070 Ti 16 GB

Evaluation

Bits per byte is the headline metric. This model's vocabulary (32,768) differs from GPT-2's (50,257), and cross-entropy per token is not comparable across tokenizers. BPB divides that out and is directly comparable.

Metric	This model	GPT-2 124M
Parameters	100,684,032	124,439,808
Validation loss	2.9725	—
Bits per byte	0.9312	—
HellaSwag (0-shot, len-norm)	pending	29.55%

Validation loss is measured on a held-out slice of the training corpus and is not comparable to GPT-2's OpenWebText number. Bits per byte is, because it is tokenizer-independent.

Data provenance and licensing

Source	HF id	License
fineweb-edu	HuggingFaceFW/fineweb-edu	ODC-By 1.0

Attribution: FineWeb-Edu (HuggingFaceFW/fineweb-edu), ODC-By 1.0

ODC-By requires attribution, which is given above. Whether pretraining-data licenses flow through to model weights is legally unsettled; permissively licensed sources were chosen deliberately as the most defensible posture. Model weights are released under cc-by-nc-4.0.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Dikshan1234/LLMPretrain")
model = AutoModelForCausalLM.from_pretrained("Dikshan1234/LLMPretrain",
                                             trust_remote_code=True)

inputs = tok("Once upon a time", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=100, do_sample=True,
                     temperature=0.8, top_p=0.95)
print(tok.decode(out[0]))

trust_remote_code=True is required because the architecture (RoPE + SwiGLU + GQA + QK-norm) is not a stock transformers model. The two files it loads, configuration_llmpretrain.py and modeling_llmpretrain.py, are in this repo and are short enough to read.

Sample generations

===== step 19073 =====

--- [greedy] 'Once upon a time'
Once upon a time, the world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power.
The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great wealth and power.
The world was a place of great wealth and power. The world was a place of great wealth and power. The world was a place of great

--- [t=0.8] 'Once upon a time'
Once upon a time, a man had a big dream. What would Jesus have done to make that dream come true?
Jesus answered that: “It is not possible to make any other man come to my aid than you.” (John 8:45)
I believe it is the ultimate goal of every successful leader to be his or her own helper. What we must do is to not only make sure that we will be the ideal leader, but that we will be his or her own helper.
Jesus went on to say: “I have chosen the time to do all it takes to bring the

--- [greedy] 'The capital of France is'
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is Paris.
The capital of France is

--- [t=0.8] 'The capital of France is'
The capital of France is Paris. The capital of France is Paris.
French is the official language. The French language is known as Creole. French spoken in France is French, but not Creole. France is a French-speaking country.
French is a Germanic language. The French language is a Germanic language.
French is a Germanic language. The French language is a Germanic language. France is a Germanic-speaking country.
French is a Germanic language. The French language is a Germanic language.
French is a Germanic language. The French language is a Germanic-speaking country.
French is a Germanic

--- [greedy] 'In 1969, humans first'
In 1969, humans first began to use the word “machine” to describe the machines that they used to make the world. The word “machine” was first used in the context of the computer industry, and it was used to describe the machines that were used to make the world.
The word “machine” is also used to describe the machines that are used to make the world. The word “machine” is used to describe the machines that are used to make the world. The word “machine” is also used to describe the machines that are used to make the world. The word “

--- [t=0.8] 'In 1969, humans first'
In 1969, humans first began to show signs of humanlike behavior, and as many as 10,000 people have been diagnosed with autism.
In 2009, a study published in the journal Nature Neuroscience shed light on the link between autism and the gut microbiota. By studying the gut microbiota in autistic children, researchers were able to trace the genes that are responsible for the onset of the condition.
There was some evidence that the human gut bacteria play an important role in the development of autism. Specifically, researchers found that the gut bacteria in autistic children have genetic links to autism.
Researchers also found that the gut bacteria in autistic

--- [greedy] 'The three states of matter are'
The three states of matter are:
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance that is composed of atoms and molecules.
- Matter is a substance th

Limitations

This is a base model trained on a modest token budget. It has had no instruction tuning, no RLHF and no safety alignment. It will produce confidently wrong text, and it reflects whatever biases exist in its training corpus. It is a research and educational artifact, not a product.