2026-04-05 05:47:31 - ReXMoE - INFO - ================================================================================ 2026-04-05 05:47:31 - ReXMoE - INFO - ReXMoE Training Log - 0504_054731 2026-04-05 05:47:31 - ReXMoE - INFO - Log file: ./rexmoe_recovered_50_pruned_50_0304_033137_10_rexmoe_natural_phi_mini_moe_R3/logs/rexmoe_training_0504_054731.log 2026-04-05 05:47:31 - ReXMoE - INFO - ================================================================================ 2026-04-05 05:47:31 - ReXMoE - INFO - ================================================================================ 2026-04-05 05:47:31 - ReXMoE - INFO - ReXMoE Recovery Training (Fine-tuning Pruned Model) 2026-04-05 05:47:31 - ReXMoE - INFO - ================================================================================ 2026-04-05 05:47:31 - ReXMoE - INFO - Checkpoint Path: ./pruned_50_0304_033137_10_rexmoe_natural_phi_mini_moe_R3 2026-04-05 05:47:31 - ReXMoE - INFO - Dataset Path: ../dataset/alpaca_data_cleaned.json 2026-04-05 05:47:31 - ReXMoE - INFO - Dataset Mode: IF 2026-04-05 05:47:31 - ReXMoE - INFO - Num Samples: 5000 2026-04-05 05:47:31 - ReXMoE - INFO - Num Epochs: 1 2026-04-05 05:47:31 - ReXMoE - INFO - Batch Size: 1 2026-04-05 05:47:31 - ReXMoE - INFO - Max Seq Length: 512 2026-04-05 05:47:31 - ReXMoE - INFO - Learning Rate: 0.0001 2026-04-05 05:47:31 - ReXMoE - INFO - Save Path: ./rexmoe_recovered_50_pruned_50_0304_033137_10_rexmoe_natural_phi_mini_moe_R3 2026-04-05 05:47:31 - ReXMoE - INFO - Gradient Checkpointing: False 2026-04-05 05:47:31 - ReXMoE - INFO - Full LoRA (q,k,v,o): True 2026-04-05 05:47:31 - ReXMoE - INFO - Use Scheduler: True 2026-04-05 05:47:31 - ReXMoE - INFO - Evaluation Steps: 5000 2026-04-05 05:47:31 - ReXMoE - INFO - Gradient Checkpointing: False 2026-04-05 05:47:31 - ReXMoE - INFO - Using device: cuda 2026-04-05 05:47:31 - ReXMoE - INFO - [1/5] Loading pruned model from: ./pruned_50_0304_033137_10_rexmoe_natural_phi_mini_moe_R3 2026-04-05 05:54:05 - ReXMoE - INFO - Loading dense model for KD from microsoft/Phi-mini-MoE-instruct 2026-04-05 05:54:10 - ReXMoE - INFO - [2/5] Setting up LoRA for fine-tuning... 2026-04-05 05:54:12 - ReXMoE - INFO - Attempting to load and remap pretrained LoRA from: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3 2026-04-05 05:54:12 - ReXMoE - INFO - ✓ Loaded remapped LoRA weights. 2026-04-05 05:54:12 - ReXMoE - INFO - - Remapped expert weights: 0 2026-04-05 05:54:12 - ReXMoE - INFO - - Skipped pruned weights: 0 2026-04-05 05:54:12 - ReXMoE - INFO - Total parameters: 4,719,726,912 2026-04-05 05:54:12 - ReXMoE - INFO - Trainable parameters: 76,002,304 (1.6103%) 2026-04-05 05:54:17 - ReXMoE - INFO - Prior Evaluation: 2026-04-05 05:54:17 - ReXMoE - INFO - Evaluating model with 3 sample prompts... 2026-04-05 05:54:41 - ReXMoE - INFO - --- Prompt 1/3 --- 2026-04-05 05:54:41 - ReXMoE - INFO - Instruction: What is the capital of France? 2026-04-05 05:54:41 - ReXMoE - INFO - Input: None 2026-04-05 05:54:41 - ReXMoE - INFO - Generated completion (len 100): capital? capital?? capital?? capital??? capital?? capital?? capital?? capital?? capital?? capital?? capital?? capital??? capital??? capital????? capital???? capital?????????????? a capital?????????????? capital??? capital???? capital?? 2026-04-05 05:55:05 - ReXMoE - INFO - --- Prompt 2/3 --- 2026-04-05 05:55:05 - ReXMoE - INFO - Instruction: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense. What will most likely result if a high-pressure system remains in an area for a long period of time? A. fog B. rain C. drought D. tornado Answer: 2026-04-05 05:55:05 - ReXMoE - INFO - Input: None 2026-04-05 05:55:05 - ReXMoE - INFO - Generated completion (len 100): The air with high pressure will stay in an area for a long period of time. A. fog B. rain C. drought D. tornado Answer: the air with high pressure can stay in an area for a long period of time A. fog B. rain C. dr 2026-04-05 05:55:28 - ReXMoE - INFO - --- Prompt 3/3 --- 2026-04-05 05:55:28 - ReXMoE - INFO - Instruction: Given the fact: predators eat prey Question: Predators eat A. lions B. humans C. bunnies D. grass Answer: 2026-04-05 05:55:28 - ReXMoE - INFO - Input: None 2026-04-05 05:55:28 - ReXMoE - INFO - Generated completion (len 100): lion is a predator. : as are predators : the as are predators : the as are predators : who are predators : predators are : the e is a predator : the e is a predator : e is a predator . animals are predators . animals are predators animals are predators 2026-04-05 05:55:28 - ReXMoE - INFO - Evaluation of all 3 prompts complete. 2026-04-05 05:55:28 - ReXMoE - INFO - Epoch 1/1 2026-04-05 06:19:10 - ReXMoE - INFO - Step 500/5000 | Loss: 1.0398 | KD Loss: 0.4316 2026-04-05 06:43:01 - ReXMoE - INFO - Step 1000/5000 | Loss: 1.9496 | KD Loss: 0.8789 2026-04-05 07:07:09 - ReXMoE - INFO - Step 1500/5000 | Loss: 1.3975 | KD Loss: 0.8164 2026-04-05 07:31:38 - ReXMoE - INFO - Step 2000/5000 | Loss: 1.6832 | KD Loss: 1.0781 2026-04-05 07:56:05 - ReXMoE - INFO - Step 2500/5000 | Loss: 1.4381 | KD Loss: 0.8203 2026-04-05 08:20:25 - ReXMoE - INFO - Step 3000/5000 | Loss: 0.9392 | KD Loss: 0.8086 2026-04-05 08:44:48 - ReXMoE - INFO - Step 3500/5000 | Loss: 0.8403 | KD Loss: 0.8125 2026-04-05 09:08:46 - ReXMoE - INFO - Step 4000/5000 | Loss: 1.1178 | KD Loss: 0.5469 2026-04-05 09:33:09 - ReXMoE - INFO - Step 4500/5000 | Loss: 1.3528 | KD Loss: 0.6992 2026-04-05 09:57:21 - ReXMoE - INFO - Step 5000/5000 | Loss: 1.9670 | KD Loss: 1.3125 2026-04-05 09:57:21 - ReXMoE - INFO - Evaluation at step 5000: 2026-04-05 09:57:21 - ReXMoE - INFO - Evaluating model with 3 sample prompts... 2026-04-05 09:57:24 - ReXMoE - INFO - --- Prompt 1/3 --- 2026-04-05 09:57:24 - ReXMoE - INFO - Instruction: What is the capital of France? 2026-04-05 09:57:24 - ReXMoE - INFO - Input: None 2026-04-05 09:57:24 - ReXMoE - INFO - Generated completion (len 9): The capital of France is Paris. 2026-04-05 09:57:26 - ReXMoE - INFO - --- Prompt 2/3 --- 2026-04-05 09:57:26 - ReXMoE - INFO - Instruction: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense. What will most likely result if a high-pressure system remains in an area for a long period of time? A. fog B. rain C. drought D. tornado Answer: 2026-04-05 09:57:26 - ReXMoE - INFO - Input: None 2026-04-05 09:57:26 - ReXMoE - INFO - Generated completion (len 7): C. drought. 2026-04-05 09:57:29 - ReXMoE - INFO - --- Prompt 3/3 --- 2026-04-05 09:57:29 - ReXMoE - INFO - Instruction: Given the fact: predators eat prey Question: Predators eat A. lions B. humans C. bunnies D. grass Answer: 2026-04-05 09:57:29 - ReXMoE - INFO - Input: None 2026-04-05 09:57:29 - ReXMoE - INFO - Generated completion (len 12): The correct answer is C. bunnies. 2026-04-05 09:57:29 - ReXMoE - INFO - Evaluation of all 3 prompts complete. 2026-04-05 09:57:29 - ReXMoE - INFO - Epoch 1 completed. Average Loss: nan 2026-04-05 09:57:36 - ReXMoE - INFO - Training complete.