PY-AI-Dev commited on
Commit
dfe4362
·
verified ·
1 Parent(s): 59e2777

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +0 -44
README.md CHANGED
@@ -115,50 +115,6 @@ print(f"Best answer: {best.strip()}")
115
  # → Best answer: biomass
116
  ```
117
 
118
- ### Text generation
119
-
120
- ```python
121
- from transformers import AutoTokenizer, AutoModelForCausalLM
122
- import torch
123
-
124
- model = AutoModelForCausalLM.from_pretrained(
125
- "liodon-ai/slm-10m",
126
- trust_remote_code=True,
127
- dtype=torch.bfloat16,
128
- ).to("cuda")
129
- tokenizer = AutoTokenizer.from_pretrained("liodon-ai/slm-10m", trust_remote_code=True)
130
-
131
- inputs = tokenizer("The quick brown fox", return_tensors="pt").to("cuda")
132
- output = model.generate(**inputs, max_new_tokens=50, do_sample=True, temperature=0.8, top_k=50)
133
- print(tokenizer.decode(output[0], skip_special_tokens=True))
134
- ```
135
-
136
- > **Note:** Free-text generation quality is limited at this scale. The model's strength is in relative likelihood scoring, as used by the benchmark evaluations above.
137
-
138
- ## Reproduce
139
-
140
- ```bash
141
- git clone https://github.com/liodon-ai/slm-pretrain
142
- pip install -r requirements.txt
143
-
144
- # Prepare data
145
- python prepare_data.py
146
-
147
- # Train (25B tokens)
148
- python train.py
149
-
150
- # Export to HF format
151
- python export.py --checkpoint checkpoints/step_0044000.pt --out hf_model
152
-
153
- # Evaluate (4 lm-eval benchmarks)
154
- PYTHONPATH=. lm_eval --model hf \
155
- --model_args pretrained=hf_model,trust_remote_code=True \
156
- --tasks hellaswag,arc_easy,arc_challenge,piqa \
157
- --device cuda
158
-
159
- # ArithMark-2.0
160
- python eval_arithmark.py
161
- ```
162
 
163
  ## Citation
164
 
 
115
  # → Best answer: biomass
116
  ```
117
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
118
 
119
  ## Citation
120