metadata
base_model: HuggingFaceTB/SmolLM-135M-Instruct
datasets: HumanLLMs/Human-Like-DPO-Dataset
library_name: transformers
model_name: llm-course-hw2-reward-model
tags:
- generated_from_trainer
- trl
- reward-trainer
🏆 Model Card for llm-course-hw2-reward-model
This model is a fine-tuned reward model based on HuggingFaceTB/SmolLM-135M-Instruct, trained on the HumanLLMs/Human-Like-DPO-Dataset dataset.
It has been trained using TRL to evaluate and rank responses based on human preferences, playing a crucial role in RLHF (Reinforcement Learning from Human Feedback) for models like SmolLM-135M-PPO.
📝 Overview
- Base Model: SmolLM-135M-Instruct
- Fine-Tuned Dataset: HumanLLMs/Human-Like-DPO-Dataset
- Objective: Learn to assign higher scores to more engaging, structured, and emotional responses.
- Use Case: Used in PPO-based RLHF training to reinforce human-like response quality.
Training Method
- The model was fine-tuned using Direct Preference Comparisons:
- Each sample contains a chosen response (preferred) and a rejected response.
- The model learns to assign higher rewards to the chosen response and lower rewards to the rejected one.
- This reward function was used in PPO fine-tuning to optimize response generation.