File size: 3,068 Bytes
7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 7bfe5c1 690e121 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
language:
- en
license: apache-2.0
tags:
- text-generation
- pytorch
- causal-lm
- reasoning
- coding
- grok-distillation
- mature
base_model: distilgpt2
---
# distilgpt2-grok-coder-reasoning
## Model Details
### Model Description
This model is a full fine-tuned version of **DistilGPT2**, exposed to an aggressive, completely uncapped curriculum of Grok-4 level distillation traces, hyper-creative and logic datasets, comprehensive coding logic, and mature internet discourse. It is designed to act as a highly responsive, analytical engine capable of deep structural reasoning and complex logic emulation.
Trained natively at an accelerated maximum learning rate with a cosine decay schedule, the model synthesizes diverse programmatic and theoretical domains from a massive multi-repository corpus, processed at the model's absolute maximum context window of 1024 tokens.
- **Developed by:** GODsStrongestSoldier
- **Model type:** Causal Language Model (Transformer Decoder)
- **Language:** English
- **License:** Apache 2.0
- **Finetuned from model:** `distilgpt2`
---
## Datasets Used for Fine-Tuning
This model was trained comprehensively on the full, uncapped contents of the following datasets:
- [WithinUsAI/Grok4.4_heavy_max_distill_god_seed_25k](https://huggingface.co/datasets/WithinUsAI/Grok4.4_heavy_max_distill_god_seed_25k)
- [WithinUsAI/GOD_Coder_Complete_DataSet](https://huggingface.co/datasets/WithinUsAI/GOD_Coder_Complete_DataSet)
- [acheong08/nsfw_reddit](https://huggingface.co/datasets/acheong08/nsfw_reddit)
- [TeichAI/grok-code-fast-1-1000x](https://huggingface.co/datasets/TeichAI/grok-code-fast-1-1000x)
- [TeichAI/brainstorm-v3.1-grok-4-fast-200x](https://huggingface.co/datasets/TeichAI/brainstorm-v3.1-grok-4-fast-200x)
- [Crownelius/Hyper-Creative-Grok-V1](https://huggingface.co/datasets/Crownelius/Hyper-Creative-Grok-V1)
- [Crownelius/Hyper-UltraData-Grok-V1](https://huggingface.co/datasets/Crownelius/Hyper-UltraData-Grok-V1)
---
## Training Details
### Training Procedure
The model underwent **full fine-tuning** without the use of adapters or LoRA layers. All native parameters of the base model were globally updated. The training harness dynamically parsed heavily nested dataset repositories, enforcing a strict shape constraint to generate mathematically perfect 1024-token continuous sequences for the GPU, maxing out the DistilGPT2 context window.
To maximize adaptation to the Grok-level reasoning data, an absolute peak learning rate (`3e-4`) was utilized alongside a 5% warmup phase and a cosine scheduler.
#### Hardware
- **Environment:** Kaggle
- **Accelerators:** Dual NVIDIA T4 GPUs (15GB VRAM each)
#### Hyperparameters
- **Epochs:** 1
- **Context Window / Block Size:** 1024
- **Per-Device Batch Size:** 4
- **Gradient Accumulation Steps:** 16
- **Effective Global Batch Size:** 128
- **Peak Learning Rate:** 3e-04
- **Learning Rate Scheduler:** Cosine
- **Warmup Ratio:** 0.05
- **Optimizer:** Fused AdamW (`adamw_torch_fused`)
- **Mixed Precision:** fp16
- **Gradient Checkpointing:** Enabled
|