language:
- en
license: apache-2.0
pipeline_tag: text-generation
tags:
- mathematics
- reasoning
- education
- adaption-labs
- autoscientist
- llama4
- lora
- instruction-tuning
- sft
datasets:
- Charley890/adaption-adaptive-math-2
Overview
Adaptive Math 2
A mathematics-specialized instruction dataset designed to improve reasoning, structured problem solving, and educational AI assistants through supervised fine-tuning with Adaptation Labs AutoScientist.
Research Snapshot
| Property | Value |
|---|---|
| Domain | Mathematics |
| Dataset Type | Instruction Tuning |
| Framework | Adaptation Labs AutoScientist |
| Base Model | Llama-4 Scout 17B |
| Fine-tuning | LoRA (SFT) |
| Grade | A |
| Quality Score | 9.5 / 10 |
Dataset : https://huggingface.co/datasets/Charley890/adaption-adaptive-math-2
Adaptive Math 2 focuses on educational mathematical reasoning rather than simple answer prediction.
Covers
- Algebra
- Geometry
- Arithmetic
- Number Theory
- Statistics
- Word Problems
- Mathematical Reasoning
- Multi-step Solutions
Characteristics
Structured instruction format
Educational explanations
Curriculum-oriented questions
Reasoning-aware responses
Clean supervised fine-tuning format
Example Dataset Samples
Example
Instruction
Solve:
4x - 9 = 19
Expected Response
4x = 28
x = 7
Educational Impact
Adaptive Math 2 is intended for:
- AI tutors
- Educational assistants
- Mathematical reasoning
- Homework support
- Classroom demonstrations
- STEM education
- Benchmark evaluation
The dataset emphasizes transparent reasoning instead of answer memorization.
Mathematical Training Specification
model: base_model: "meta-llama/Llama-4-Scout-17B-16E-Instruct" approximate_model_size: "109B parameters" training_method: "Supervised Fine-Tuning (SFT)" adaptation_method: "LoRA" data_format: "Chat"
mathematical_formulation:
objective: description: "The adapted model minimizes the supervised language-modeling loss over the Adaptive Math 2 dataset." equation: | θ* = argmin_θ L(θ)
language_model_loss: equation: | L(θ) = -Σᵢ log Pθ(yᵢ | xᵢ)
lora: description: "Instead of updating the full model weights, LoRA learns a low-rank update." equation: | W' = W + ΔW ΔW = (α/r)BA
parameters:
rank_r: 16
alpha: 32
dropout: 0
scaling_factor: |
α/r = 32/16 = 2
effective_update:
equation: |
ΔW = 2BA
optimization: learning_rate: 0.00005 weight_decay: 0 max_gradient_norm: 2 optimizer_constraint: | ||g||₂ ≤ 2
training_schedule: epochs: 5 evaluations: 5 evaluation_frequency: | 5 evaluations / 5 epochs = 1 evaluation per epoch
scheduler:
type: "Linear"
num_cycles: 0.5
warmup_ratio: 0.03
warmup:
equation: |
T_warmup = 0.03T
batch: batch_size: "max"
target_modules: count: 10 modules: - "k_proj" - "o_proj" - "q_proj" - "v_proj" - "shared_expert.gate" - "shared_expert.up_proj" - "shared_expert.down_proj" - "feed_forward.gate_proj" - "feed_forward.up_proj" - "feed_forward.down_proj"
training_objective: equation: | θ_LoRA* = argmin_{A,B} L(W + (α/r)BA)
interpretation: rank: "r = 16 controls the low-rank adaptation capacity." scaling: "α/r = 2 controls the magnitude of the LoRA update." regularization: "LoRA dropout = 0 and weight decay = 0." stability: "Gradient norm is clipped at 2." schedule: "Learning rate follows a linear schedule after a 3% warmup."
Training Interpretation
benchmark: adaptation_strategy: "Parameter-efficient fine-tuning" objective: "Improve mathematical reasoning while preserving the pretrained model." full_parameter_update: false low_rank_update: true
key_result: statement: | Adaptive Math 2 applies a low-rank parameter update rather than retraining the complete 109B-parameter model.
mathematical_summary: | W_adapted = W_base + 2BA
meaning: - "W_base represents the pretrained model." - "A and B are learned low-rank matrices." - "r = 16 defines the adaptation rank." - "α = 32 gives a scaling factor of 2." - "Only the selected target modules receive LoRA updates."
Reproducibility
configuration: training_method: "SFT" training_type: "LoRA" epochs: 5 learning_rate: 0.00005 warmup_ratio: 0.03 weight_decay: 0 max_grad_norm: 2 lora_rank: 16 lora_alpha: 32 lora_dropout: 0 scheduler: "linear" scheduler_cycles: 0.5 evaluations: 5 batch_size: "max"
credit: adaptive_data: "Adaptive Data by Adaption Labs" training_evaluation: "AutoScientist"
Adaptive Math 2 was developed using the Adaption Lab AutoScientist pipe
📊 Model Performance
{
"job_id": "8db3bddd-326c-44ba-8440-2456d10d33f2",
"training_experiment_id": "78a0fd31-7d13-40cf-bc55-fb2d2bf9e92c",
"original_model_name": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
"trained_model_name": "adaption_adaptive_math_2",
"training_method": "sft",
"training_type": "lora",
"data_format": "chat",
"hyperparams": {
"lora": "true",
"lora_r": 16,
"n_evals": 5,
"n_epochs": 5,
"batch_size": "max",
"lora_alpha": 32,
"lora_dropout": 0,
"min_lr_ratio": 0.1,
"warmup_ratio": 0.03,
"weight_decay": 0,
"learning_rate": 0.00005,
"max_grad_norm": 2,
"base_model_size": "109B",
"train_on_inputs": "false",
"training_method": "sft",
"lr_scheduler_type": "linear",
"scheduler_num_cycles": 0.5,
"lora_trainable_modules": "k_proj,o_proj,q_proj,v_proj,shared_expert.gate_proj,shared_expert.up_proj,shared_expert.down_proj,feed_forward.gate_proj,feed_forward.up_proj,feed_forward.down_proj"
}
}
Training Data
The model was trained on 1,306 rows of adapted data with the following domain distribution: math (77%), code (8%), science (8%), academic-education (8%).
Model Evaluation
The model was evaluated on an in-distribution held-out test set as well as a broader domain-specific test set to measure generalization.
| Domain | Win rate vs. base model |
|---|---|
| math | 66% |
How to use
pip install torch transformers peft
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
ADAPTER = "<this-repo-id>"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float32 if device == "cpu" else torch.bfloat16
base = AutoModelForCausalLM.from_pretrained(BASE, dtype=dtype).to(device)
model = PeftModel.from_pretrained(base, ADAPTER)
# Optional: merge the LoRA weights into the base for faster inference
model = model.merge_and_unload()
model.eval()
tokenizer = AutoTokenizer.from_pretrained(BASE)
messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(device)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))