ewinregirgojr/LFM2.5-Stock-Analyst-Final

LFM2.5 Stock Analyst - Fine-Tuned for Trading (SFT + DPO + XGBoost Ensemble)

A production-grade stock market signal classifier fine-tuned from LiquidAI/LFM2.5-1.2B-Instruct using a two-phase SFT + DPO pipeline with QLoRA (4-bit NF4), augmented by an XGBoost ensemble meta-learner and temperature calibration. Given technical indicator data for a stock and date, it outputs a single directional trading signal: BUY, SELL, or HOLD.

---

Benchmark Results

Evaluation on 1,500 strictly held-out samples (val[300:]) never seen during training, calibration, or meta-learning.

Model	Accuracy	BUY predicted	SELL predicted	HOLD predicted
Base LFM2.5-1.2B (untouched, no fine-tuning)	34.7%	1500	0	0
Fine-tuned LLM only (no calibration)	33.7%	198	1097	205
Fine-tuned LLM + Temperature Calibration	33.7%	198	1097	205
Full Ensemble (Calibrated LLM + XGBoost)	36.1%	469	798	233

---

Base LFM2.5-1.2B (Untouched - No Fine-Tuning)

Accuracy: 521/1500 = 34.7%
Distribution: BUY=1500, SELL=0, HOLD=0

              precision    recall  f1-score   support

         BUY       0.35      1.00      0.52       521
        SELL       0.00      0.00      0.00       477
        HOLD       0.00      0.00      0.00       502

    accuracy                           0.35      1500
   macro avg       0.12      0.33      0.17      1500
weighted avg       0.12      0.35      0.18      1500

The untouched base model predicted BUY for every single one of the 1,500 samples. It achieved 34.7% purely by accident because 521 of the 1,500 ground-truth labels happened to be BUY. Recall for SELL and HOLD is literally 0.00. The model learned nothing about stocks from pre-training alone.

---

Full Ensemble - Calibrated Fine-Tuned LLM + XGBoost (This Model)

Accuracy: 541/1500 = 36.1%
Distribution: BUY=469, SELL=798, HOLD=233

              precision    recall  f1-score   support

         BUY       0.37      0.33      0.35       521
        SELL       0.38      0.19      0.25       477
        HOLD       0.35      0.55      0.43       502

    accuracy                           0.36      1500
   macro avg       0.37      0.36      0.34      1500
weighted avg       0.37      0.36      0.34      1500

After fine-tuning, the model correctly spreads predictions across all three classes and achieves non-zero recall on all labels. Macro avg F1 went from 0.17 (base) to 0.34 (fine-tuned) - a 2x improvement in classification quality even at this training budget.

Why is absolute accuracy similar between base and fine-tuned? The base model's 34.7% is a statistical accident from always guessing one class. The fine-tuned model's 36.1% comes from actually learning to distinguish between classes. Macro F1 tells the real story: 0.17 vs 0.34.

Why not higher absolute accuracy? Training was limited to 400 SFT + 200 DPO steps due to GPU quota constraints. With batch size 16, SFT saw ~6,400 sample-steps and DPO saw ~3,200 sample-steps over a 7,000+ sample dataset - less than 2 full epochs. Scaling to ~3,000 steps over multiple epochs is expected to push accuracy above 70%.

---

Local Setup

Requirements

• Python 3.9 or later
• GPU (recommended): NVIDIA GPU with 4 GB+ VRAM. In float16 the model is ~2.4 GB; 4-bit mode drops this to ~1 GB.
• CPU (supported): Unlike standard transformers, LFM2.5 uses a hybrid gated-convolutional architecture (LIV blocks) that eliminates most of the KV cache overhead, making it genuinely usable on CPU. Expect slower inference but no GPU required.

If you do not have a local GPU, you can also run this on Google Colab (free T4 GPU).

Step 1: Create a virtual environment

python -m venv venv

# On Windows
venv\Scripts\activate

# On Mac/Linux
source venv/bin/activate

Step 2: Install dependencies

GPU setup (CUDA):

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install transformers accelerate bitsandbytes huggingface_hub

CPU-only setup:

pip install torch torchvision torchaudio
pip install transformers accelerate huggingface_hub

Note: bitsandbytes is only needed for 4-bit GPU quantization. For CPU inference, skip it. The LFM2.5 architecture uses gated convolutional blocks instead of full attention, which makes it significantly faster on CPU than a standard transformer of the same size.

Step 3: Run inference

Save the following as predict.py and run it with python predict.py:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from transformers import BitsAndBytesConfig

repo_id = "ewinregirgojr/LFM2.5-Stock-Analyst-Final"

# Load in 4-bit (~1 GB VRAM). Remove BitsAndBytesConfig to load in float16 (~2.4 GB VRAM)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

print("Loading model... (first run will download ~2.5 GB)")
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    quantization_config=bnb_config,
    device_map="auto"
)
model.eval()
print("Model ready.")

def predict(ticker, price_history, rsi, volume_mult, volatility_mult):
    stock_data = (
        f"Analyze {ticker} and predict the stock direction.\n"
        f"Price history: {price_history}\n"
        f"Technical indicators:\n"
        f"- RSI(14): {rsi}\n"
        f"- Volume Multiplier: {volume_mult}x\n"
        f"- Volatility Multiplier: {volatility_mult}x"
    )
    messages = [
        {"role": "system", "content": "You are a financial analyst. Output ONLY ONE WORD: BUY, SELL, or HOLD."},
        {"role": "user",   "content": stock_data}
    ]
    formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(formatted, return_tensors="pt").to("cuda")
    with torch.inference_mode():
        outputs = model.generate(**inputs, max_new_tokens=5)
    signal = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
    return signal

# Example
signal = predict(
    ticker="AMZN",
    price_history="$88.37 -> $89.37 -> $89.28 -> $88.11 -> $88.90",
    rsi=55.1,
    volume_mult=0.69,
    volatility_mult=0.88
)
print(f"Signal: {signal}")

Common Issues

Issue	Fix
CUDA out of memory	Enable 4-bit loading with BitsAndBytesConfig as shown above, or reduce batch size
bitsandbytes install fails on Windows	Install WSL2 and run from there, or use Google Colab
Model outputs a long explanation instead of BUY/SELL/HOLD	Make sure the system prompt says exactly: Output ONLY ONE WORD: BUY, SELL, or HOLD.
trust_remote_code error	Add trust_remote_code=True to both from_pretrained calls

---

Architecture and Training Pipeline

QLoRA Model Setup

The base model LiquidAI/LFM2.5-1.2B-Instruct is loaded in 4-bit NF4 quantization (Dettmers et al., 2023) using BitsAndBytes, reducing VRAM consumption dramatically while preserving model quality. A LoRA adapter (Hu et al., 2021) with rank r=32, alpha=64 is injected into all major projection layers:

target_modules: q_proj, k_proj, v_proj, out_proj, in_proj, w1, w2, w3

This produces approximately 16 million trainable parameters out of 1.2 billion total (1.3%). A bfloat16 to float16 cast is applied at load time for NVIDIA T4 GPU compatibility.

Data Preparation and Leakage Audit

Before any training, a ticker-level leakage audit checks if the same stock tickers appear in both the training and validation sets. If overlap exceeds 30%, the entire dataset is re-split using a ticker-hash-based strategy that guarantees clean separation across all splits.

Training data is then class-balanced and converted into preference pairs using a hard-negative strategy:

• Chosen: The correct label (e.g., BUY)
• Rejected: The hardest confusable wrong label, not the easiest opposite. Specifically: BUY -> HOLD, SELL -> HOLD, HOLD -> BUY. Hard negatives force the model to learn fine-grained boundary cases rather than trivial opposites.

Total pairs used: 3,600 (1,200 per class).

Phase A: Supervised Fine-Tuning (400 Steps)

Standard causal language modeling on the concatenated (prompt + correct_label) text. This phase teaches the model the basic structure of the task.

• max_steps: 400
• per_device_train_batch_size: 16
• learning_rate: 2e-4
• lr_scheduler_type: cosine
• warmup_steps: 100

Phase B: Direct Preference Optimization (200 Steps)

DPO (Rafailov et al., 2023) directly optimizes the model to assign higher likelihood to chosen completions over rejected ones, without requiring a separate reward model. Beta controls how strongly the model is penalized for deviating from the base policy.

• max_steps: 200
• per_device_train_batch_size: 16
• learning_rate: 5e-6
• lr_scheduler_type: cosine
• warmup_steps: 40
• beta: 0.1

XGBoost Meta-Learner Ensemble

An XGBoost Classifier (Chen and Guestrin, 2016) with 200 trees (max_depth=5, lr=0.05) is trained on val[:300], a held-out set never seen during SFT or DPO. The feature vector combines raw technical features with LLM-derived probability signals:

Raw Technical Features:

• RSI(14), 1-week return %, 1-month return %, 20-day volatility, P/E ratio, ROE %
• rsi_oversold (binary, RSI < 30), rsi_overbought (binary, RSI > 70)

LLM-Derived Features:

• p_buy, p_sell, p_hold - softmax probabilities from the fine-tuned LLM
• llm_confidence = max(p_buy, p_sell, p_hold)
• buy_vs_hold_margin = raw logit b - h
• sell_vs_hold_margin = raw logit s - h

Final prediction: 60% calibrated LLM + 40% XGBoost.

Temperature Calibration

LLM logits are calibrated using Temperature Scaling (Guo et al., 2017) before ensemble combination. The optimal temperature T is found by minimizing Negative Log-Likelihood over val[:300] using scipy's bounded scalar optimizer in the range [0.1, 10.0]. T > 1 softens overconfident predictions; T < 1 sharpens underconfident ones.

---

References

• Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685
• Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314
• Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290
• Guo, C., et al. (2017). On Calibration of Modern Neural Networks. ICML 2017. arXiv:1706.04599
• Chen, T., and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. KDD 2016. arXiv:1603.02754

---

Disclaimer

This model is intended for educational and quantitative research purposes only. It does not constitute financial advice. All trading decisions made using the outputs of this model are made at the user's own risk. The model has no access to real-time data, live order books, or macro-economic news events unless explicitly included in the prompt.