morningstarxcdcode/adaption-finance-qa-sft-20b-model

Finance QA SFT 20B

LoRA adapter weights for fine-tuned GPT-OSS 20B on financial question-answering tasks. Trained on 20,000 Investopedia-derived QA pairs using Adaption's Adaptive Data platform. Part 1 Finance submission for the AutoScientist Challenge.

Model Details

• Base model: togethercomputer/gpt-oss-20b-bf16
• Fine-tuning method: LoRA (Low-Rank Adaptation)
• LoRA rank: 4
• LoRA alpha: 8
• LoRA dropout: 0
• Target modules: q_proj, v_proj
• Training epochs: 1
• Training steps: 21
• Final eval loss: 0.98
• License: Apache 2.0

Training Data

20,000 rows of financial question-answer pairs covering:

• Insurance and investing topics
• Banking and corporate finance
• Market analysis and financial reporting
• Regulatory compliance questions
• Personal finance guidance

Data sourced from Investopedia articles with self-verification to minimize hallucinations. Generated through Adaption's Adaptive Data platform.

Results

Metric	Base	Adapted	Change
Win Rate	41%	59%	+43.9%
Quality Score	6.0	8.6	+43.3%
Grade	C	B	Improved
Percentile	12.2	19.2	+57.4%

How to Use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    "togethercomputer/gpt-oss-20b-bf16",
    device_map="auto",
    torch_dtype="bfloat16"
)

model = PeftModel.from_pretrained(
    base_model,
    "morningstarxcdcode/adaption-finance-qa-sft-20b-model"
)

tokenizer = AutoTokenizer.from_pretrained(
    "morningstarxcdcode/adaption-finance-qa-sft-20b-model"
)

messages = [
    {"role": "system", "content": "You are a financial advisor assistant."},
    {"role": "user", "content": "What are the key differences between term life and whole life insurance?"}
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Bias, Risks, and Limitations

This model was trained on synthetic financial data derived from Investopedia articles. It should not be used as the sole basis for financial decisions. The model may produce plausible-sounding but incorrect financial information. Always consult qualified financial professionals for financial advice.

Technical Specifications

• Architecture: GptOssForCausalLM (24 layers, 64 attention heads, 32 experts, 4 active per token)
• Hidden size: 2880
• Vocab size: 201,088
• Max position embeddings: 131,072
• Precision: bfloat16
• PEFT version: 0.15.1

Links

• Dataset (HF): morningstarxcdcode/adaption-investopedia-finance-qa
• Demo: morningstarxcdcode/adaption-finance-qa-demo
• GitHub: LusterSourav/adaption-autoscientist-challenge

Team

Sourav Rajak, Priyanshu Tomar, Roshan G, Vivek Rajput

Acknowledgments

Built using Adaption Labs' AutoScientist and Adaptive Data platforms for the AutoScientist Challenge.