tejadhith/asc-mixtral-8x7b-finance-lora

Mixtral-8x7B Finance Reasoning — LoRA Adapter (E24, 20K)

AutoScientist Challenge — Finance category (Part 1, Jul 2026) Adapter weights: tejadhith/asc-mixtral-8x7b-finance-lora Released dataset: tejadhith/asc-finance-reasoning-20k (also on Kaggle) — the exact set this adapter trained on.

---

What this is

A PEFT LoRA adapter over mistralai/Mixtral-8x7B-Instruct-v0.1 trained to improve finance reasoning across all major sub-domains: quantitative/numerical, financial-statement analysis, markets and instruments, regulatory/compliance, fixed income, derivatives, portfolio theory, macro, corporate valuation, risk, personal finance, ESG, crypto, and advisory decision-making.

The adapter is the lead entry (E24) for the AutoScientist Challenge Finance category. It was trained on AutoScientist's free compute (Together AI, 2× H100 80 GB) using our 4,018-row curated seed expanded to ~20,018 rows at training time. Headline win-rates (Gemini 3.1 Pro judge, vs untuned Mixtral base): held-out 39 → 61 (+22), domain/General 27 → 73 (+46). Scoring is LLM-as-judge win-rate vs the same untuned Mixtral base on private held-out finance tasks, with scores size-normalized across entries.

---

Training

Training-data lineage

4,018-row curated seed  ──►  AutoScientist expansion  ──►  ~20,018-row training set  ──►  E24 adapter
   (quality 9.3 / A)          (+8K domain, +8K general)      (the released dataset)

The seed was the whole curation effort: ~2,430 geo-neutral authored rows (numerically verified) across all 14 finance sub-domains + ~1,588 canonical rows (ConvFinQA, BizBench, MMLU finance subsets), LLM-judge filtered (114 dropped), scoring 9.3/10 (grade A) on Adaptive Data. The 20K is that seed expanded with platform chain-of-thought augmentation for volume + anti-overfit. Both ship in the released dataset tejadhith/asc-finance-reasoning-20k — filter to the authored + real layers (2,430 + 1,588 = 4,018 rows) to recover the clean seed (the ~16K augmentation rows have an empty layer). Detail below.

Base model and method

• Base: mistralai/Mixtral-8x7B-Instruct-v0.1
• Method: SFT, LoRA on the attention projections (q_proj, k_proj, v_proj, o_proj) — verified from the released adapter_config.json (target_modules). (Note: the AutoScientist recipe UI reported all-linear, but the actual trained adapter targets attention only — a platform recipe-vs-artifact discrepancy; the released weights are authoritative and are attention-only.)
• LoRA rank / alpha / dropout / bias: r64 / alpha 128 / 0 / none; task_type: CAUSAL_LM
• Scheduler: cosine; peak LR: ~1.5e-4; steps: ~400 (val loss flat, no overfit)
• Platform: AutoScientist (Adaption Labs) on Together AI

Training data

Curated seed (core of the released 20K): 4,018 rows. Composition:

• ~2,430 authored rows: numerically verified, geography-neutral reasoning, Claude Sonnet 4.6, all 14 finance sub-domains
• ~1,588 real canonical rows: ConvFinQA, BizBench, MMLU finance subsets
• Difficulty: ~26 / 48 / 25 easy / medium / hard; 5 prompt archetypes; 114 bad rows dropped (2.8%)
• Reasoning traces: enhanced_completion generated by Adaption's open-weight trace engine (not OpenAI/Anthropic); provenance gate cleared for redistribution
• Adaptive Data quality score: 9.3 / 10, grade A

Augmentation (also part of the released 20K): AutoScientist expanded the seed to ~20,018 rows by adding ~8,000 domain-synthetic and ~8,000 general-synthetic rows (a ~40% general slice for anti-overfit regularization). These rows are broader and noisier than the curated seed — the general slice is cross-domain by design, and augmentation rows carry sparse metadata. The released dataset is this full ~20,018-row training set (published to match the model exactly); the curated 4,018-row seed is its high-quality core, and downstream users wanting only clean finance reasoning can filter to the seed layers.

---

Results

Win-rates are base → adapted (delta), judged by Gemini 3.1 Pro as an LLM-as-judge comparing the adapted model vs the same untuned Mixtral base head-to-head on private held-out finance tasks. The evaluation splits 50/50 across a finance-specific held-out set and a pooled cross-submission finance test ("General/Domain") drawn from all participants' submissions. Both halves are scored on finance reasoning competence; the final score is size-normalized across model sizes.

Run	Training rows	Held-out delta	Domain/General delta	Notes
E18 — Mixtral LoRA, filtered 4K, enhanced prompt	4,018	0	+10	Best 4K enhanced-prompt result
E20 — Mixtral LoRA, 4K, original prompt	4,018	−5	+22	Outlier draw; not reproduced
E21 — repeat of E20	4,018	+4	+7	Under-trained auto-recipe (22 steps)
E22 — repeat of E20	4,018	−5	+3	Well-trained (66 steps); confirms +22 was variance
E23 — Mixtral LoRA, 20K	20,018	+24	+25	Alternate 1; base collapsed to 38%; all 3 candidate models positive
E24 — Mixtral LoRA, 20K (this model)	20,018	+22	+46	Lead entry; held 39→61, domain 27→73; both halves strongly positive

4K original-prompt distribution (n=3): {+22, +7, +3}, median ~+7, range 19 pts. The held-out half carries ±8–10 pts run-to-run noise because AutoScientist re-rolls the training recipe each run (#14) and selects the surfaced candidate on the noisy held-out half; the domain/General half is the low-variance, decisive metric.

20K robustness: unlike the 4K runs (where 2 of 3 candidates were negative on held-out), all 3 E23 candidates were strongly positive on held-out (+14 / +17 / +24). E24 (the confirmation run) also landed strongly positive on both halves (+22 held / +46 domain), with all 3 candidate models positive — the step-change over 4K (+7…+10 domain) is now confirmed across two independent runs. E24 is the lead entry; E23 is Alternate 1.

---

Usage

Extract the adapter

The adapter archive is Zstandard-compressed despite the .tgz extension. Use tar xf (auto-detects) or tar --zstd -xf — not tar -z (gzip flag).

# Extract (auto-detect compression)
tar xf adaption_mixtral_8x7b_instruc_finance_exam_qa_pairs_b7516c5c.tgz

# Contents: adapter_config.json, adapter_model.safetensors,
#           tokenizer files, trainer_state.json

Load and run

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = "mistralai/Mixtral-8x7B-Instruct-v0.1"
adapter_path = "./adaption_mixtral_8x7b_instruc_finance_exam_qa_pairs_b7516c5c"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(
    base,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

prompt = "A bond has a face value of $1,000, an annual coupon rate of 6%, " \
         "and 5 years to maturity. If the required yield is 8%, what is the bond's price?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, temperature=0.1)
print(tokenizer.decode(out[0], skip_special_tokens=True))

The adapter's enhanced_completion training target includes chain-of-thought reasoning traces generated by Adaption's open-weight trace engine. The model is prompted with natural instruction-style queries (not the platform-rephrased enhanced_prompt) to match natural evaluation distributions.

---

Intended use

Finance reasoning tasks: quantitative problem-solving, financial-statement interpretation, instrument pricing, regulatory/compliance questions, portfolio and risk analysis, and scenario-based advisory questions. The model is trained to produce step-by-step reasoning before committing to an answer.

Suitable for: research, academic benchmarking, evaluation pipelines, and downstream fine-tuning.

---

Limitations and caveats

• Run-to-run variance. AutoScientist re-rolls the training recipe each run; domain win-rate swings ~±10 pts at 4K. At 20K, E23 (+24/+25) and E24 (+22/+46) are two independent confirmed runs — both halves positive in both. Report the range, not just the peak.
• Geography. Canonical sources (ConvFinQA, BizBench, MMLU) skew US-SEC filings and US financial context. The authored reasoning is analytically geography-neutral but supplements rather than replaces the US-skewed canonical distribution. Disclose this in downstream use.
• Training-time expansion quality. The 20K includes 16K AutoScientist-generated rows that are partially off-domain, contain null-instruction rows (21%), and carry no provenance metadata. Volume and diversity appear to have outweighed these quality issues at E23 scale, but the expansion rows should not be treated as clean training data for other purposes.
• Numeric drift. The Adaption reasoning-trace engine drifts ~5% of numeric answers (concentrated on hard multi-step rows); the curated 4,018-row seed filters the worst 2.8%. The expansion rows were not filtered.
• Authoring provenance. Authored seed rows were generated and verified by Claude Sonnet 4.6 fan-out agents. Redistribution of the instruction/response columns is permissible under CC-BY-4.0 and the challenge rules; users should consult Anthropic's usage policies for downstream applications.
• Not a financial advisor. Outputs are for research and evaluation; they do not constitute financial advice and should not be used for investment decisions.

---

Released dataset

tejadhith/asc-finance-reasoning-20k (also on Kaggle).

The released dataset is the full ~20,018-row training set — this adapter's exact training data. Its high-quality core is the 4,018-row curated seed (quality 9.3/A); the ~16K augmentation is broader/noisier.

Schema (columns): instruction, response, enhanced_prompt, enhanced_completion, reasoning_trace, user_enhanced_prompt, user_enhanced_completion, archetype, bucket, difficulty, final_answer, layer, license, source, subdomain.

The enhanced_* columns are Adaption Adaptive Data reasoning-trace CoT augmentations of instruction / response, generated by Adaption's open-weight engine.

License: CC-BY-4.0.

---

License

CC-BY-4.0 — adapter weights and released dataset. Attribution required; commercial use permitted.

Canonical source attributions: ConvFinQA (MIT/CC-BY-4.0), BizBench (Apache-2.0), MMLU (MIT). Reasoning traces generated by Adaption's open-weight trace engine; redistribution confirmed permissible by Adaption Labs.

---

Citation

If you use this adapter or dataset, please cite:

@misc{asc-finance-mixtral-2026,
  title   = {Mixtral-8x7B Finance Reasoning LoRA Adapter (AutoScientist Challenge, Finance)},
  author  = {Tejadhith Sankar},
  year    = {2026},
  url     = {https://huggingface.co/tejadhith/asc-mixtral-8x7b-finance-lora},
  note    = {AutoScientist Challenge, Finance category, Part 1. Dataset:
             https://huggingface.co/datasets/tejadhith/asc-finance-reasoning-20k}
}