Sanskrit-qwen-7B-Translate

This model is a fine-tuned version of Qwen/Qwen2.5-7B-Instruct-1M optimized for Sanskrit language tasks.

Model Description

This is a merged version of a fine-tuned Qwen 2.5 7B model, specifically trained for Sanskrit language understanding and translation tasks. The model has been trained on a custom Sanskrit dataset to enhance its capabilities in handling Sanskrit text.

Intended Uses & Limitations

Intended Uses

Sanskrit text understanding and generation
Sanskrit-English translation tasks
Sanskrit language processing

Limitations

Performance may vary based on the complexity of Sanskrit text
Model should be used within ethical and legal guidelines

Training Data

The model was trained on the diabolic6045/Sanskrit-llama dataset.

Training Procedure

Training Details

Base Model: Qwen/Qwen2.5-7B-Instruct-1M
Training Type: Fine-tuning
Hardware: Multi-GPU setup
Training Parameters:
- Learning Rate: 2e-05
- Epochs: 1
- Batch Size: 2 (total)
- Optimizer: AdamW
- LR Scheduler: Cosine with warmup

Framework Versions

Transformers 4.49.0
Pytorch 2.5.1+cu121
Datasets 3.2.0
Tokenizers 0.21.0

See axolotl config

axolotl version: 0.8.0.dev0


base_model: Qwen/Qwen2.5-7B-Instruct-1M
load_in_8bit: false
load_in_4bit: true
strict: false

datasets:
  - path: diabolic6045/Sanskrit-llama
    type: alpaca
dataset_prepared_path:
val_set_size: 0
output_dir: ./outputs/qlora-out

adapter: qlora
lora_model_dir:

sequence_len: 1024
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:

hub_model_id: Sanskrit-qwen-8B

wandb_project: संस्कृतम्-llama
wandb_entity: 
wandb_watch: all
wandb_name: संस्कृतम्-llama
wandb_log_model: 

gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 1
optimizer: paged_adamw_8bit
lr_scheduler: cosine
cosine_min_lr_ratio: 0.2
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: false
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: false

#gpu_memory_limit: 20GiB
#lora_on_cpu: true         

warmup_steps: 10
evals_per_epoch: 4
saves_per_epoch: 1
debug:
deepspeed: deepspeed_configs/zero1.json
weight_decay: 0.0
special_tokens:
   pad_token: <|end_of_text|>