ajiayi/llama-3.1-8b-merged-unsloth-justification

Model Card: Llama 3.1 8B Fine-tuned with Justification Distillation

Model Overview

This model is a specialized fine-tuned variant of Llama 3.1 8B Instruct designed for stock price prediction with explicit justification generation. The model has been trained through knowledge distillation using GPT-4o Mini as the teacher model, focusing on generating well-reasoned predictions with detailed financial explanations.

Model ID: ajiayi/llama-3.1-8b-merged-unsloth-justification

Model Size: 8 Billion Parameters

Base Model: Meta-Llama-3.1-8B-Instruct

Training Details

Fine-tuning Approach

• Method: Knowledge Distillation via Justification-Based Supervision
• Teacher Model: GPT-4o Mini (via OpenAI API)
• Framework: Unsloth + Hugging Face Transformers
• Quantization: 4-bit QLoRA (Quantized Low-Rank Adaptation)
• Training Environment: Google Colab with GPU acceleration

LoRA Configuration

- r (Rank): Optimized via rank finder (tested ranks: 8, 16, 32, 64)
- lora_alpha: 16
- lora_dropout: 0
- Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Bias: None
- Use Gradient Checkpointing: Unsloth (optimized)

Training Hyperparameters

• Batch Size: 4 (per device)
• Learning Rate: 2e-4
• Epochs: 3
• Max Sequence Length: 2048 tokens
• Optimizer: AdamW with bf16 precision (where supported)

Training Data

• Train Set: 8,698 samples
• Validation Set: 1,243 samples
• Total Training Data: 9,941 samples
• Data Format: JSONL with instruction-following format
• Dataset: train_with_justifications_instruction_format.jsonl

Input/Output Specification

Training Data Format

This model was trained using the Alpaca instruction-following format with the following structure:

{
  "instruction": "[System prompt describing task]",
  "input": "[Market data input]",
  "output": "[JSON-formatted prediction output]"
}

Expected Input Format

The model expects inputs formatted as financial analysis prompts containing:

TICKER: [Stock Symbol]
DATE: [YYYY-MM-DD]

RECENT CLOSING PRICES (most recent last): [comma-separated prices]

TECHNICAL INDICATORS:
SMA_20=[value], SMA_50=[value],
EMA_12=[value], EMA_26=[value],
RSI_14=[value], MACD=[value], MACD_signal=[value], MACD_hist=[value],
BB_width_20_2=[value]

SENTIMENT AGGREGATES:
headline_count=[count], sent_compound_mean=[score]

HEADLINES (concise):
[News headlines if available]

System Instruction

You are a financial analyst with expertise in stock market forecasting.
Your task is to analyze market data and predict the next trading day stock price.
Use historical price trends, technical indicators, and sentiment analysis to provide an informed forecast.
Ensure that your predictions are well-justified, considering multiple financial factors.

• Predicted Stock Price: The forecasted close price for the next trading day.
• Price Movement Likelihood: The likelihood of the predicted stock price.
• Justification: Provide an explanation for the predicted stock price and the corresponding likelihood, considering the following:
  - Historical market data (e.g., recent closing prices).
  - Technical indicators (e.g., SMA, EMA, RSI, MACD, Bollinger Bands).
  - Sentiment analysis (e.g., news sentiment, market sentiment).

Please weigh these signals and justify the predicted stock price.

Return STRICT JSON with keys:
- predicted_close (float, next-day close price),
- likelihood (float in [0,1]),
- justification (string, 1–2 sentences).

Expected Output Format

The model should produce output in the following JSON format:

{
  "predicted_close": 27.18,
  "likelihood": 0.5,
  "justification": "The predicted price reflects recent trading activity, with closing prices showing a sideways trend around the 27.2 range. The low RSI indicates that the stock is oversold, potentially suggesting a minor corrective bounce, while the negative sentiment from recent headlines contributes to uncertainty, leading to a moderate likelihood of this forecast."
}

Data Characteristics

Stocks Covered

The training data includes diverse stocks across sectors:

• Technology: AAPL (Apple)
• Consumer Goods: PEP (PepsiCo)
• Banking: HSBC
• International Markets: 0700.HK (Hong Kong), 7203.T (Japan)

Data Splits

• Training Period: 2015-01-01 to 2021-12-31
• Validation Period: 2022-01-01 to 2022-12-31
• Temporal Coverage: ~7 years of historical market data

Technical Features Included

• Price Indicators: SMA (20, 50), EMA (12, 26), Bollinger Bands
• Momentum: RSI (14), MACD with signal line and histogram
• Sentiment: VADER compound sentiment scores from news headlines
• Temporal Information: Historical closing prices (5-day window)

Model Performance

Key Strengths

1. Justification Quality: Generates detailed, coherent explanations for predictions
2. Reasoning Consistency: Superior reasoning over baseline models through knowledge distillation
3. Financial Context Awareness: Understands market technical indicators and sentiment
4. Instruction Following: Strong at following the strict JSON output format

Limitations

1. Data Scarcity: Limited to sectors/stocks in training data
2. Historical Bias: Reflects market patterns from 2015-2021 training period
3. Sentiment Dependency: Relies on headline sentiment which may be incomplete
4. Financial Expertise: While improved through distillation, remains a language model without real financial planning capabilities

Usage

Installation

pip install transformers torch huggingface_hub

Basic Inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ajiayi/llama-3.1-8b-merged-unsloth-justification"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype="auto")

# Prepare your input following the format above
prompt = """[Your financial analysis input]"""

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Hugging Face Inference Endpoints

This model supports Hugging Face Inference Endpoints for production deployment:

1. Go to ajiayi/llama-3.1-8b-merged-unsloth-justification
2. Click "Deploy → Inference Endpoints"
3. Configure hardware and enable endpoint
4. Use the provided API URL for inference requests

With Pipeline

from transformers import pipeline

pipe = pipeline("text-generation", model="ajiayi/llama-3.1-8b-merged-unsloth-justification")
results = pipe("[Your prompt]", max_new_tokens=256, temperature=0.7)
print(results[0]['generated_text'])

Ethical Considerations

Intended Use

• Financial analysis and prediction research
• Educational purposes to demonstrate knowledge distillation
• Baseline model for stock price forecasting tasks

Not Recommended For

• Making real financial investment decisions without human oversight
• Financial advice to retail investors
• Mission-critical financial applications without additional validation

Limitations & Disclaimers

• This model generates predictions based on limited historical data
• Market conditions change; historical patterns may not persist
• Sentiment analysis may be incomplete or subject to manipulation
• Model should not be used as sole basis for financial decisions
• Past performance does not guarantee future results

Dataset Attribution

Training data derived from:

• Stock Prices: Historical market data from public sources
• News Sentiment: Google News scraped with VADER sentiment analysis
• Technical Indicators: Computed using standard financial libraries

The distillation targets (teacher model outputs) were generated using OpenAI's GPT-4o Mini API.

Citation

If you use this model in research, please cite:

@model{llama_justification_distillation_2025,
  title={Llama 3.1 8B Fine-tuned with Justification Distillation for Stock Price Prediction},
  author={Jiayi Ang},
  year={2025},
  howpublished={Hugging Face Model Hub},
  note={Knowledge distillation from GPT-4o Mini}
}

Model Card Contact

For questions or issues related to this model, please open an issue on the model repository or contact the model creator.

Version History

Version	Date	Changes
1.0	2025	Initial release with merged LoRA weights

Related Models

• Socratic CoT Variant: ajiayi/llama-3.1-8b-merged-unsloth-cot
• Base Model: meta-llama/Llama-3.1-8B-Instruct
• Teacher Model: GPT-4o Mini (OpenAI)

License

This model follows the license of the base Llama 3.1 8B Instruct model. Please refer to the base model's license for details.