---
library_name: peft
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
tags:
- axolotl
- base_model:adapter:Qwen/Qwen3-4B-Instruct-2507
- lora
- transformers
datasets:
- jalasoft/typst-instruct
pipeline_tag: text-generation
model-index:
- name: qwen-3-4B-it-ft-typ
results: []
---
[
](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config
axolotl version: `0.13.0.dev0`
```yaml
base_model: Qwen/Qwen3-4B-Instruct-2507
# optionally might have model_type or tokenizer_type
# Works for Qwen
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
hub_model_id: jalasoft/qwen-3-4B-it-ft-typ
load_in_8bit: false
load_in_4bit: true
chat_template: qwen3
eot_tokens:
- <|im_end|>
datasets:
- path: jalasoft/typst-instruct
type:
system_prompt: 'You are an expert in Typst markup language. Generate clean, well-formatted Typst code based on user instructions:'
field_instruction: prompt
field_output: completion
val_set_size: 0.1
output_dir: /workspace-data/output
adapter: qlora
# Average 1k - 1.5 k dataset
lora_r: 32
lora_alpha: 64
lora_dropout: 0.05
# Explicit targeting for Qwen architecture
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
# Support long academic papers (up to ~15-20k tokens = 16384)
# Support medium articles (up to ~5-10k tokens = 8192)
sequence_len: 4096
# Ensures consistent memory usage
pad_to_sequence_len: true
# Disable packing during evaluation for accuracy
eval_sample_packing: false
# Pack multiple samples for efficiency
sample_packing: true
# Better batching with packing enabled
multipack_real_batches: true
wandb_project: qwen-3-4B-it-ft-typ
wandb_entity:
wandb_watch:
wandb_name:
wandb_log_model:
# Keep at 1 for log sequences (16k tokens)
micro_batch_size: 6
# Accumulate gradients for stable training 32 (Memory issues with 16)
gradient_accumulation_steps: 4
# Evaluation batch size
eval_batch_size: 4
# More epochs for better convergence
num_epochs: 5
# Fastest optimizer for CUDA
optimizer: adamw_torch_fused
# Cosine annealing for smooth learning rate decay
lr_scheduler: cosine
learning_rate: 8e-5
bf16: auto
tf32: true
# Note, we should define the best values for the next params in order to avoid memory leaks
gradient_checkpointing: true
gradient_checkpointing_kwargs:
use_reentrant: false
resume_from_checkpoint:
logging_steps: 2
flash_attention: true
warmup_ratio: 0.1
weight_decay: 0.03
max_grad_norm: 1.0
evals_per_epoch: 4
saves_per_epoch: 2
special_tokens:
```
# qwen-3-4B-it-ft-typ
This model is a fine-tuned version of [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) on the jalasoft/typst-instruct dataset.
It achieves the following results on the evaluation set:
- Loss: 0.8641
- Memory/max Active (gib): 26.06
- Memory/max Allocated (gib): 26.06
- Memory/device Reserved (gib): 56.71
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 8e-05
- train_batch_size: 6
- eval_batch_size: 4
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 24
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 5
- training_steps: 50
### Training results
| Training Loss | Epoch | Step | Validation Loss | Active (gib) | Allocated (gib) | Reserved (gib) |
|:-------------:|:------:|:----:|:---------------:|:------------:|:---------------:|:--------------:|
| No log | 0 | 0 | 1.1808 | 25.87 | 25.87 | 26.11 |
| 2.5428 | 0.2791 | 3 | 1.1722 | 49.04 | 49.04 | 56.71 |
| 2.2131 | 0.5581 | 6 | 1.0901 | 26.06 | 26.06 | 56.71 |
| 1.9746 | 0.8372 | 9 | 1.0184 | 49.04 | 49.04 | 56.71 |
| 1.5513 | 1.0930 | 12 | 0.9782 | 26.06 | 26.06 | 56.71 |
| 1.3567 | 1.3721 | 15 | 0.9514 | 49.04 | 49.04 | 56.71 |
| 1.2133 | 1.6512 | 18 | 0.9300 | 26.06 | 26.06 | 56.71 |
| 1.1362 | 1.9302 | 21 | 0.9145 | 49.04 | 49.04 | 56.71 |
| 1.0783 | 2.1860 | 24 | 0.9022 | 26.06 | 26.06 | 56.71 |
| 1.016 | 2.4651 | 27 | 0.8923 | 49.04 | 49.04 | 56.71 |
| 0.9807 | 2.7442 | 30 | 0.8841 | 26.06 | 26.06 | 56.71 |
| 0.9721 | 3.0 | 33 | 0.8776 | 49.04 | 49.04 | 56.71 |
| 0.9597 | 3.2791 | 36 | 0.8723 | 26.06 | 26.06 | 56.71 |
| 0.9817 | 3.5581 | 39 | 0.8683 | 49.04 | 49.04 | 56.71 |
| 0.9456 | 3.8372 | 42 | 0.8659 | 26.06 | 26.06 | 56.71 |
| 0.9041 | 4.0930 | 45 | 0.8646 | 49.04 | 49.04 | 56.71 |
| 0.95 | 4.3721 | 48 | 0.8641 | 26.06 | 26.06 | 56.71 |
### Framework versions
- PEFT 0.17.1
- Transformers 4.57.1
- Pytorch 2.8.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.1